undefined | Better HN

0 pointsAnalemma_9y ago0 comments

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.

0 comments

13 comments · 3 top-level

nmstoker9y ago· 6 in thread

With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.

bendbro9y ago

There isn't: the function for this standard includes expected benefit as an input. Every standard has inputs, so that certainly isn't the quality for making something a double standard. The only remaining quality is how unfair it feels, so it would probably be better to just address that, since it is obviously the only thing you disagree about.

lugg9y ago

With that logic decreased wages for women are not a double standard due to the potential for maternity leave affecting their output at work.

This is a double standard plain and simple, and a very dangerous one at that.

2 more replies

lerpa9y ago

It's their site they can block whatever they want. The problem is the stupid far reaching conclusion that this is trespassing.

Even normal trespassing laws are way too overreaching (see how it is handled in the UK for a saner example) but now you have the amazing possibility of remote trespassing.

The fun part is that it's just a matter of someone hiding something that says you cannot access the site in a place that you have to access the site in order to read -- the ToS. Suing people over this is idiotic.

The real problem is the involvement of Govt, and this kind of absurdity regarding ToS, EULAs and so on, is something that has been going on for decades. If you have the money you can make Govt your personal watch dogs.

mgalka9y ago

Whether or not it technically qualifies as a "double standard," in practice I don't see anything inherently unfair about it.

If a stranger enters my house without my permission, that's trespassing. But there's nothing unfair about letting in someone who I invite over.

nercht129y ago

That's a terrible analogy. Your home is private, websites are not. The fact is that websites are posted online for all to see, so it's more like saying certain people at a park may take pictures while others are not allowed. That's unfair. If everyone could take pictures, it would be fair. Yes, someone with an old bright bulb camera might be annoying people, but nobody said "fair" meant all players would be nice or that having a "fair" policy would somehow be more beneficial to the website owner. It's not, that's why site owners are selective. So they have a double standard, but it's for their benefit, not that of the site visitors (be they human or bot).

2 more replies

gakada9y ago

Using the law to block crawlers is more like saying:

1. Google can come in

2. Other Americans can't come in

3. Chinese people can come in (or anywhere else where US laws don't apply)

It might not be unfair, but it is certainly pointless and arbitrary.

1 more reply

tangue9y ago· 2 in thread

Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.

collinmanderson9y ago

Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)

lerpa9y ago

Exactly, market scrappers as search engines.

And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively.

The far reaching of government into this, and also the pirating stuff (which I do not condone but think that arresting people for that is waay too much) is what makes me want for the system to collapse under it's own weight. Like some website suing members of congress for visiting it while violating the ToS in this case.

I also secretly wanted Oracle to win vs Google so that cloning an API was piracy and that would extend to being a crime to purchase pirated goods which would make all clean room reverse engineering a criminal activity. That would lead to anyone that uses a PC without an authentic IBM BIOS (look up Phoenix BIOS) to be arrested, in theory, so even the US president would have to fall into that. It would have been a glorious shitstorm if Oracle won and IBM took that precedent to it's logical implications, the computer world would have failed, and the law would either be made even more arbitrary or be fixed, but at least it would be shown how idiotic the state of affairs was.

1 more reply

profeta9y ago· 2 in thread

because i may also come to the point where i am a direct competitor to google, but i will never get there because i can't scrap any site like they can.

your next argument may very well be a very racist one with the very same excuse you used above.

sjwright9y ago

And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.

greglindahl9y ago

A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.

1 more reply

j / k navigate · click thread line to collapse

0 comments

13 comments · 3 top-level

nmstoker9y ago· 6 in thread

bendbro9y ago

lugg9y ago

With that logic decreased wages for women are not a double standard due to the potential for maternity leave affecting their output at work.

This is a double standard plain and simple, and a very dangerous one at that.

2 more replies

lerpa9y ago

It's their site they can block whatever they want. The problem is the stupid far reaching conclusion that this is trespassing.

Even normal trespassing laws are way too overreaching (see how it is handled in the UK for a saner example) but now you have the amazing possibility of remote trespassing.

mgalka9y ago

Whether or not it technically qualifies as a "double standard," in practice I don't see anything inherently unfair about it.

If a stranger enters my house without my permission, that's trespassing. But there's nothing unfair about letting in someone who I invite over.

nercht129y ago

2 more replies

gakada9y ago

Using the law to block crawlers is more like saying:

1. Google can come in

2. Other Americans can't come in

3. Chinese people can come in (or anywhere else where US laws don't apply)

It might not be unfair, but it is certainly pointless and arbitrary.

1 more reply

tangue9y ago· 2 in thread

Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.

collinmanderson9y ago

Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)

lerpa9y ago

Exactly, market scrappers as search engines.

And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively.

1 more reply

profeta9y ago· 2 in thread

because i may also come to the point where i am a direct competitor to google, but i will never get there because i can't scrap any site like they can.

your next argument may very well be a very racist one with the very same excuse you used above.

sjwright9y ago

And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.

greglindahl9y ago

A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.

1 more reply

j / k navigate · click thread line to collapse