QVC Sues Shopping App for Web Scraping That Allegedly Triggered Site Outage (opens in new tab)

Yes. Did your read/understand what I posted? Especially the last part where I predicted that somebody will crawl out and post what you just did?

Rate limiting is not blocking.

Multiple tabs do not translate to concurrent requests. A shop like QVC should have a high limit and frankly an automatic system that flags stuff like this and notifies proper parties.

I assure you that if you had a proper system in place it would limit the exposure. We are talking about 600 r/s (QVC should be able to eat that anyways, but whatever).

At this level you should know this will happen and have placeholders in place.

If every single IP is unique, additional layer would be pattern matching traffic by different network classes and flagging it actively.

Another way to rate limit is by number of pages per session / time.

Then you can have another fail safe method where if X exceeds Y start throttling top X (dynamically scale up until X is returned to normal) while notifying DevOps of the issue.

Etc.

Very true, however the incidents I reference this was definitely not the case.

In fact for a while we would get Bing (MSN bot back then) crawl us everyday at the same time, almost on the dot.

Let me plug project honeypot (which I am in no way affiliated with). This is truly an awesome, and surprisingly accurate, free, service that does an amazing job at collecting heuristics on suspicious IP activity and exposing it in a easy to interpret way..

http://www.projecthoneypot.org/index.php

This is very true. Logistically (not technically) this wasn't an option - read: business decision. Before the days where everything had an API, and things were done with spreadsheets, and billing codes, and product codes, and typos... managing DNS for a large number of domains was not fun and rather expensive. Want to move a bunch of domains to a new IP address? $$$+Time + more time and money. We weren't able to trust our domain broker and had to double check EVERYTHING they did.

To you point, if you are a large provider, especially one that passively and actively sends a lot of money towards the search engines, there are some additional options at your disposal. We (the business units) had contacts with adsense etc, which would come in handy.

My reading of that is they were splitting their crawling amongst a large block of IPs they had. If they are using proxies, that is much easier because you can just block them all without any worry of accidentally blocking real consumers (in addition to the fact that you can also file abuse complaints). I think at Thumbtack we blocked all AWS IPs, a good deal of foreign IPs, in addition to the specific IPs of people who were abusively crawling.

At the same time, the onus shouldn't be on QVC to have to block this, result.ly should either be a good citizen or have to face a lawsuit. Granted, QVC's tech team should be able to deal with this because next time the person who is DOSing them might not be a US entity which can be sued, but that isn't entirely relevant in this situation.