undefined | Better HN

0 pointsmelvinram15y ago0 comments

I don't know if it's an easy fixer but it's certainly not difficult to eliminate 90% of scrapers. My sites get scraped all the time and if you look at these scrapper sites, they usually are not scrapping just one site. To simplify the issue, let's look a small data set:

  Site 1:
    * Content ABC
    * Content DEF
    * Content GHI

  Site 2:
    * Content JKL
    * Content MNO
    * Content PQR
    
  Site 3:
    * Content STU
    * Content VWX
    * Content YZ0

  Site 4:
    * Content ABC
    * Content DEF
    * Content MNO
    * Content PQR
    * Content STU

Which of these is a scrapper?

0 comments

4 comments · 3 top-level

Devilboy15y ago· 1 in thread

The scraper is the one where the content appeared last.

lurker1915y ago

How do you perform that measurement using practically bounded computing and networking resources?

jcampbell115y ago

When you add site 5 in the mix that has:

* Content ABC

* Content DEF

It makes it much harder to identify.

If you are a blackhat SEO, then you keep track of the last time google indexes you (including the anonymous crawlers, which is tricky), and backdate scraped content to just ahead of the time you were last indexed. Then you can send a complaint through googles tools about the site that wrote the original content.

Blackhats make content duplication a really challenging problem, and having a complaint form isn't going to solve much. The blackhats can take advantage of that as well.

lacker15y ago

Google does eliminate more than 90% of scrapers. It's just really easy to create scrapers; they can outnumber the original content by more than 1000-1. So you need many, many systems to remove scrapers.

j / k navigate · click thread line to collapse