Ask HN: Web scraping in production?

4 pointsarkmm1y ago5 comments

Are any of you maintaining any web scrapers in production?

I've done some for side projects, automated testing, and personal scripts (checking personal bank balances, getting a Global Entry interview slot, etc.), but it always feels very brittle.

Curious what applications people have in industry and what sorts of techniques people use for reliability.

4 pointsarkmm1y ago5 comments

Are any of you maintaining any web scrapers in production?

I've done some for side projects, automated testing, and personal scripts (checking personal bank balances, getting a Global Entry interview slot, etc.), but it always feels very brittle.

Curious what applications people have in industry and what sorts of techniques people use for reliability.

5 comments

5 comments · 2 top-level

sargstuff1y ago· 2 in thread

excel web scraping[0] (vs. using python[1] and/or odbc/delimited files)

A few 2025 use cases [2],[3]:

   Use publically available database information (construction, taxes, sales, traffic report, proposed building/zone changes, etc) to find out what's going on withing an area aka. zip code, housing area, 'vacation spot', etc

----

   creative take on topic:

      modern looming / static 'threaded' approach : https://news.ycombinator.com/item?id=43977384

      Structurally reprogrammable magnetic maetamaterials hold promise for biomedicine, soft robotics. ("web" support formed via scraping material in relevant patterns) : https://techxplore.com/news/2025-05-reprogrammable-magnetic-metamaterials-biomedicine-soft.html

      3d printed smart-fabrics : https://techxplore.com/news/2025-05-d-smart-fabrics-flexibility-ability.html

----

[0] : excel scraping : https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk

[1] : python scraping : https://www.youtube.com/watch?v=Oo8-nEuDBkk

[2] : https://dataforest.ai/blog/top-web-scraping-use-cases

[3] : https://www.parsehub.com/blog/web-scraping-examples/

arkmmOP1y ago

Neat - didn't realize there were affordances for scraping in Excel (but in hindsight I shouldn't be surprised).

I didn't follow the connection between modern looming and scraping though?

sargstuff1y ago

hint: silk spider webs & fabric threads.

Guess 3d printing should have been clarified as linear, fused deposition. Melted plastic line gets scraped along plate/material.

The 3d printed web reference, in this instance, being the in-fill pattern. [0]

Robotic metal pinching / incremental sheet forming might be bit more clearer example. [1]

-----

[0] : in-fill pattern : https://jlc3dp.com/blog/choosing-the-right-infill-structure-...

[1] : https://www.youtube.com/watch?v=Jc16Ob-yoDs

9d1y ago· 1 in thread

Scraping is inherently brittle, but it can be very useful for short-term scraping in very specific circumstances. I haven't had any in maybe 10 years.

sargstuff1y ago

IMHO, "untyped" format/delimited file yes. Directly placing/'compiling' in appropriate topological construct/environment works wonders. aka environment of database, spreadsheet, "reports" with information beyond raw data, etc

j / k navigate · click thread line to collapse

5 comments

5 comments · 2 top-level

sargstuff1y ago· 2 in thread

excel web scraping[0] (vs. using python[1] and/or odbc/delimited files)

A few 2025 use cases [2],[3]:

   Use publically available database information (construction, taxes, sales, traffic report, proposed building/zone changes, etc) to find out what's going on withing an area aka. zip code, housing area, 'vacation spot', etc

----

   creative take on topic:

      modern looming / static 'threaded' approach : https://news.ycombinator.com/item?id=43977384

      Structurally reprogrammable magnetic maetamaterials hold promise for biomedicine, soft robotics. ("web" support formed via scraping material in relevant patterns) : https://techxplore.com/news/2025-05-reprogrammable-magnetic-metamaterials-biomedicine-soft.html

      3d printed smart-fabrics : https://techxplore.com/news/2025-05-d-smart-fabrics-flexibility-ability.html

----

[0] : excel scraping : https://www.youtube.com/watch?app=desktop&v=6coVzIt93vk

[1] : python scraping : https://www.youtube.com/watch?v=Oo8-nEuDBkk

[2] : https://dataforest.ai/blog/top-web-scraping-use-cases

[3] : https://www.parsehub.com/blog/web-scraping-examples/

arkmmOP1y ago

Neat - didn't realize there were affordances for scraping in Excel (but in hindsight I shouldn't be surprised).

I didn't follow the connection between modern looming and scraping though?

sargstuff1y ago

hint: silk spider webs & fabric threads.

Guess 3d printing should have been clarified as linear, fused deposition. Melted plastic line gets scraped along plate/material.

The 3d printed web reference, in this instance, being the in-fill pattern. [0]

Robotic metal pinching / incremental sheet forming might be bit more clearer example. [1]

-----

[0] : in-fill pattern : https://jlc3dp.com/blog/choosing-the-right-infill-structure-...

[1] : https://www.youtube.com/watch?v=Jc16Ob-yoDs

9d1y ago· 1 in thread

Scraping is inherently brittle, but it can be very useful for short-term scraping in very specific circumstances. I haven't had any in maybe 10 years.

sargstuff1y ago

j / k navigate · click thread line to collapse