The second review I read was a customer complaining about profanity in a movie and then writing out all the examples. Who has time for that?
- how would you actually go about loading reviews if you really wanted to
- what kind of system would you need to work around the captcha and stuff
if you have a single worker trying to scrape a shit ton of products back to back to back you're going to get rate limited or their bot detection will catch you.
i also love that people will complain about the vulgar language in a book or movie by writing a review that contains a quote with the vulgar language
I'm going to publish an Airbnb example tomorrow where I scraped 1,406,718 photo URLs from public listing pages. For that I used https://docs.burla.dev/ which is a high-performance parallel processing python library I've been working on for a few years now.
Loved this until I remembered that these reviews are what AI is trained on and influenced by.
But at least he's employing hobos.
