A couple of years back I worked with a company which maintained specific data which was the main traffic driver on that page. Google approached them and wanted to pay for the rights to get the data and display it on top of the search results, a feature which was fairly new back then.

This was an interesting dilemma because it was very clear that the money was way less than the loss in ad revenue due to traffic drop, but it was also clear that if we wouldn’t take the deal, a more desperate competitor would, which would result in the same traffic loss but without the extra google money. So the company took the deal.

History repeats itself here, with the difference that instead of paying for the data, the ai crawlers simply take it for free.

It's a catch-22. Without google crawling your site, you don't get any new traffic. But with google crawling your site, you also might not get any traffic.

AI summarization has already causes issues for sites like rtings where people are no longer visiting the site but still making use of the data presented there. Leading to rtings not getting enough traffic to continue to post their data.

It is an existential crisis for websites and when they go away it'll be an existential crisis for AI.

Internet is more and more becoming a commercialization platform. If you are selling something on your website, you still want Google (or ChatGPT for that matters) to expose customers to your product. The gate is the actual delivery of the product is behind a purchase/signup. Google and others want to control the entire customer journey, to the point the your website is simply a way to pass metadata to them. They are actually achieving this!

this kills the entire internet vibe of the 90s, early 2k

You're allowed to exist on the web. The alternative is you are pushed out, your site is not indexed and google / chrome labels it as a security risk when people are trying to reach it directly. The mandate is clear: give up the data or give up the spot.

If your site is all about disseminating information (like Wikipedia), then Google would provide a free mirror of sorts.

If your site is about your product, Google won't be able to serve the sign-up page from AI; the traffic would come your way. Same for a site that sell something: the traffic you're interested in would arrive at your checkout page.

Paid-content sites and ad-supported sites are screwed though, on top of their being screwed by archive.is and ad blockers.

That's Google making way for its disruptor. We'll see who that is. Imagine a search engine that just presents search results. Groundbreaking.

Sites pay good money to appear on top search results. Looks like the future is going to be sponsored AI sources. It's going to be even more difficult to figure out if google is presenting you with actual information instead of just an ad

I write things on the internet because I want to share ideas. If someone reads my post and tells a friend, that's great. If an AI crawls my posts and passes along the ideas that's great too.

(It doesn't work for ad-funded writing, but while I have substantial sympathy there this has historically been an unpopular argument on HN)

What you gain? Nothing, but they and other AI companies have decided not to respect your robots.txt

Who can lobby the EU for the right to abstain a website from being used for training data?

Depends on what you're trying to do. Sell ads over your content? Probably not great. Sell goods? Still good for you. Become influential and spread ideas? Good for you.

It's the news media that will suffer the most.

well its already happening and people are fighting over traffic crumbs already, they call it GEO

I asked the exact question back in 2011 when they were showing images directly leaving me pay for bandwidth out of my pocket

Free speculation: I could see a future where Google populates a footer on results with the website logos of the sources. Presumably, the new web will require users to memorize websites/brands and go directly to those sites if they see a lot of their results are being provided by one source.

Websites may go back to being simply labors of love.

Nothing, but idea is that you wont be able to prevent it from happening.

In that case, the consequence will be that people will stop having webs. It is already happening with personal and niche sites.

> allowing Googlebot to crawl my sites

As far as I know, you don't have a choice. They have no obligation to respect your wishes, and LLMs are legally allowed to scrape & republish your content.

Exactly nothing, nada, zero. And given it does not only crawl, it trains on it, so that never ever should people come back to search for it again.

The expected purpose of websites is to spread information, so whether users get it by making a request to your website or to Google is irrelevant. In fact, if they get it from Google it's better because it reduces website load.

If instead the purpose of your website is to manipulate users for financial gain (for instance by showing media attempting to manipulate their purchasing decisions, after receiving a bribe from a vendor), and the information is just a way to lure users, then maybe this malicious business model will finally be no longer possible.

Can you actually prevent Google from crawling your site?

Vastly less but still more traffic than if you didn’t participate. I’m sure they will calibrate it just so.

> then what exactly do I gain by allowing Googlebot to crawl my sites?

The counter argument is that sites are becoming more AI slop or may intentionally provide poison they don’t want to train on. There may be a cut off date after which training must be carefully curated; and the main body of data has already been collected.

Sites may still get traffic from agents searching for current information. Maybe even the resurgence of RSS? One can dream.

Websites tend to be updated and considered to be the source as well.

Allow? Deep down, do you think you have a choice?

Mechanisms might exist to make you think you have one, the same way copywrite should prevent millions of books being gobbled up by TheZuck but ultimately do you really have a choice?

Rules and laws don't exists for you.

> what exactly do I gain by allowing Googlebot to crawl my sites?

Site traffic

It's the Google AMP controversy all over again

The opportunity to feed it false information?

Maybe you want your ideas to spread? If your sites purpose is getting ad impressions then yea no point. But if your purpose is to spread ideas then it is still useful.

honest question here, why there are lot of comments about SEO is dead?

> then what exactly do I gain by allowing Googlebot to crawl my sites?

Mention

(You misspelled someone as some)

Google has always crawled your site and been an arse! Now you get to decide whether they are hallucinating!

You can drop pointers on Masto and other socials to your sites - that has not changed.

Do we need something else? ie you drop a link to somewhere else.

> then what exactly do I gain by allowing Googlebot to crawl my sites?

Making the information available that you put up your site for?