undefined | Better HN

0 pointsjiggawatts2y ago0 comments

Source addresses: the Google bot traffic comes from a small set of Google-owned IP address blocks.

Third parties bake this into things like Web Application Firewall (WAF) rules. For example, Azure App Gateway WAF has a policy category for “known bots” which includes Google but excludes your tiny AI startup.

It’s a moat built by giant corporations to keep tiny players in their place.

Google "helpfully" publishes their bot source IP addresses: https://developers.google.com/static/search/apis/ipranges/go...

AWS also provides named rules such as "bot:name:googlebot": https://docs.aws.amazon.com/waf/latest/developerguide/aws-ma...

0 comments

creato2y ago

I agree it’s a moat, but why would azure restrain competition from google? I think it’s just yet another example of an anti-abuse collateral damage, like email anti-spam blocking small unknown servers or Cloudflare blocking unknown IPs.

sokoloff2y ago

Because Azure customers want Google to be able to index their sites.

Would you host your e-commerce or social media site on a cloud provider that blocked Googlebot?

It's not Microsoft giving Google a handout out of the goodness of their hearts; it's Azure customers demanding that functionality. (Those Azure customers also don't care about a random little search startup and probably don't want to pay any egress fees to serve traffic to it.)

AWS offers the same: https://docs.aws.amazon.com/waf/latest/developerguide/waf-bo...

Oracle as well: https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_...

Akamai as well: https://www.akamai.com/products/bot-manager

whatshisface2y ago

Moats are anticompetitive, that's the entire idea behind why founders want them. Who else would a moat keep away, customers? ;)

kazinator2y ago

Customers are kept in by moats.

1 more reply

nova220332y ago

For example, Azure App Gateway WAF has a policy category for “known bots”

So why should we take Satya's testimony seriously? Azure is a Microsoft product and if what you're saying is true, Microsoft is part of the problem.

sokoloff2y ago

Almost surely that Azure customers want that.

https://news.ycombinator.com/item?id=37746480

j / k navigate · click thread line to collapse

0 pointsjiggawatts2y ago0 comments

Source addresses: the Google bot traffic comes from a small set of Google-owned IP address blocks.

It’s a moat built by giant corporations to keep tiny players in their place.

Google "helpfully" publishes their bot source IP addresses: https://developers.google.com/static/search/apis/ipranges/go...

AWS also provides named rules such as "bot:name:googlebot": https://docs.aws.amazon.com/waf/latest/developerguide/aws-ma...

0 comments

creato2y ago

sokoloff2y ago

Because Azure customers want Google to be able to index their sites.

Would you host your e-commerce or social media site on a cloud provider that blocked Googlebot?

AWS offers the same: https://docs.aws.amazon.com/waf/latest/developerguide/waf-bo...

Oracle as well: https://docs.oracle.com/en-us/iaas/Content/WAF/Bot/good_bot_...

Akamai as well: https://www.akamai.com/products/bot-manager

whatshisface2y ago

Moats are anticompetitive, that's the entire idea behind why founders want them. Who else would a moat keep away, customers? ;)

kazinator2y ago

Customers are kept in by moats.

1 more reply

nova220332y ago

For example, Azure App Gateway WAF has a policy category for “known bots”

So why should we take Satya's testimony seriously? Azure is a Microsoft product and if what you're saying is true, Microsoft is part of the problem.

sokoloff2y ago

Almost surely that Azure customers want that.

https://news.ycombinator.com/item?id=37746480

j / k navigate · click thread line to collapse