undefined | Better HN

0 pointsbot4031mo ago0 comments

It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.

0 comments

That's still the same crawler system though. And it's lazy engineering to not build in something to track when you last requested a url.

And it's quite a trivial feature at that.

I sincerely doubt that search engines run their crawlers on a single machine and they got it figured out.

j / k navigate · click thread line to collapse

0 pointsbot4031mo ago0 comments

It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.

That's still the same crawler system though. And it's lazy engineering to not build in something to track when you last requested a url.

And it's quite a trivial feature at that.

I sincerely doubt that search engines run their crawlers on a single machine and they got it figured out.

j / k navigate · click thread line to collapse