now days we use curl/wget for web dev it seems? probably used it for the same thing back then but I also image that lots of people just used their own ad-hoc scripts or whatever; i wish i had firsthand knowledge of that.
i used wget to download copies of websites when i found one that was mostly documents/images rather than webapps.