undefined | Better HN

0 pointscaydenm2y ago0 comments

OP here, I was trying to say that these pages were behind an authwall and loading with userids from a specific user but without any of their cookies to support that auth.

This led us to believe this page was MitM rather than crawled directly (as they would not be able to impersonate the user)

0 comments

FreakLegion2y ago

That's how I read it also. If the ids you're referring to were in the URL, it's almost certainly URL Filtering. The URLs are fed to the crawler via MITM, so you were basically right.

j / k navigate · click thread line to collapse

0 comments

FreakLegion2y ago

That's how I read it also. If the ids you're referring to were in the URL, it's almost certainly URL Filtering. The URLs are fed to the crawler via MITM, so you were basically right.

j / k navigate · click thread line to collapse