undefined | Better HN

0 pointsdavidsojevic2mo ago0 comments

I don't mean that users are following the links to `acme.com` and `demo.com` type domains in documentation; I mean that bots are likely finding and following many links to them because of their widespread use in documentation.

If you search for `site:github.com "acme.com"` in Google, you'll find numerous instances of the domain being used in contrived links in documentation as an example of how URLs might be structured on an arbitrary domain and also in issues to demonstrate a fully qualified URL without giving away the actual domain people were using.

This means that numerous links are pointing to non-existent paths on `acme.com` because of the nature of how people are using them in documentation and examples.

0 comments

3 comments · 1 top-level

danaris2mo ago· 2 in thread

That is very possible.

But it is not necessary to see the results that are being described.

If sites like my tiny little browser game, with roughly 120 weekly unique users, are getting absolutely hammered by the scraper-bots (it was, last year, until I put the Wiki behind a login wall; now I still get a significant amount of bot traffic, it's just no longer enough to actually crash the game), then sites that people actually know and consider important like acme.com are very likely to be getting massive deluges of traffic purely from first-order hits.

grumbelbart22mo ago

The article describes that a lot of the requests are for non-existent URLs. Do you observe the same?

danaris2mo ago

Yes; I get a lot of requests for a mostly a small set of paths on my site that look like they're attempts at finding exploitable surfaces. Things like /auth/bind-session, /auth/check?jwt=, etc. (And those are just the ones that are coming up in the obvious error reports; when I go looking at the logs there are more.)

j / k navigate · click thread line to collapse