Google begins requiring JavaScript for Google Search (opens in new tab)

Given that curl-impersonate[1] exists and that a major player in this space is also looking for experience with this library, I'm pretty sure forcing the execution of JS using DOM stuff would be a much more effective deterrent to prevent scraping.

[1] https://github.com/lwthiker/curl-impersonate

I have been web searching using Google from the command line, with no Javascript, for decades. Until last week I never sent a User Agent HTTP header either. After this change I'm still searching from the command line, no Javascript. Thus "requiring Javascript" is not a correct phrase to describe this change. The requirement is a User Agent HTTP header with an approved value. The only difference in searching for me as of the last few days is that I now send a User Agent HTTP header, with an "approved" string.

Javascript does not stop bots. At least it does not stop Googlebot.

IMO, the change is to funnel more people (cf. bots) into seeing AI-generated search results. The "AI" garbage requires Javascript. That is why the spokesperson suggests "degraded" search results for people who are not using Javascript. For me, the results are improved by avoiding the AI garbage, not degraded.

"Why didn't they do it earlier?" is a fallacious argument.

If we accepted it, there would basically only be a single point in time where a change like this could be legitimately made. If the change is made before there is a large enough problem, you'll argue the change was unnecessary. If it's made after, you'll argue the change should have been made sooner.

"They've already done something else" isn't quite as logically fallacious, but shows that you don't experience dealing with adversarial application domains.

Adversarial problems, which scraping is, are dynamic and iterative games. The attacker and defender are stuck in an endless loop of game and counterplay, unless one side gives up. There's no point in defending against attacks that aren't happening -- it's not just useless, but probably harmful, because every defense has some cost in friction to legitimate users.

> My guess is it had more to do with squeezing out more profit from that supposed 0.1% of users.

Yes, that kind of thing is very easy to just assert. But just think about it for like two seconds. How much more revenue are you going to make per user? None. Users without JS are still shown ads. JS is not necessary for ad targeting either.

It seems just as plausible that this is losing them some revenue, because some proprortion of the people using the site without JS will stop using it rather than enable JS.

The problem is accountability. Imagine starting a trade show business in the physical world as an example.

One day you start getting a bunch of people come in to mess with the place. You can identify them and their organization, then promptly remove them. If they continue, there are legal ramifications.

On the web, these people can be robots that look just like real people until you spend a while studying their behavior. Worse if they’re real people being paid for sabotage.

In the real world, you arrest them and find the source. Online they can remain anonymous and protected. What recourse do we have beyond splitting the web into a “verified ID” web, and a pseudonymous analog? We can’t keep treating potential computer engagement the same as human forever. As AI agents inevitably get cheaper and harder to detect, what choice will we have?

> I'm appalled by some of the discussions here. Was I "enshittifying" my website out of unbridled "greed"? I don't think so. But every time I come here, I find these accusations, which makes me think that as a website with technical users, we can definitely do better.

It's if nothing else very evident most people fundamentally don't understand what an adversarial shit show running a public web service is.

There's a certain relatively tiny audience that has congregated on HN for whom hating ads is a kind of religion and google is the great satan.

Threads like this are where they come to affirm their beliefs with fellow adherents.

Comments like yours, those that imply there might be some valid reason for a move like this (even with degrees of separation) are simply heretical. I think these people cling to an internet circa 2002ish and the solution to all problems with the modern internet is to make the internet go back to 2002.

20000 RPS is very little — a web app / database running on an ordinary desktop computer can process up to 10000 RPS on a bare-metal configuration after some basic optimization. If that is half of your total average load, a single co-located server should be enough to eat entire "attack" without flinching. If you have "competitors" and I assume, that this is some kind of commercial product (including running profitable advertising-based business), you should probably have multiple geographically distributed servers and some kind of BGP-based DDoS protection.

Regarding Tor nodes — there is nothing wrong with locking them out, especially if your website isn't geo-blocked by any governments and there are no privacy concerns related to accessing it.

If, like Google, you lock out EVERYONE, even your logged in users, whose identities and payment details you have already confirmed, then... yes you are "enshittifying" or have ulterior motives.

> they were using real Gmail accounts to sign up

Using Gmail should be a red flag on its own. Google accounts can be purchased by millions, and immediately get resold after being blocked by target website. Same for phones. Only your own accounts / captchas / site rep can be treated as basis of trust. Confirmation e-mail is a mere formality to have some way of contacting your human users. By the time Reddit was created it was already useless as security measure.

Most black hat spammers use botnets, especially against bigger targets which have enough traffic to build statistics to fingerprint clients and map out bad ASNs and so on, and most botnets are low powered. You're not running chrome on a smart fridge or an enterprise router.

A major player in this space is apparently looking for people experienced in scraping without using browser automation. My guess is that not running a browser results in using far fewer resources, thus reducing their costs heavily.

Running a headless browser also means that any differences in the headless environment vs. a "headed" one can be discovered, as well as any of your Javascript executing within the page, which significantly makes it difficult to scale your operation.

so much more expensive and slow vs just scraping the html. It is not hard to scrape raw html if the target is well-defined (like google).

The only issue with a hash cash is there’s no way to know whether the user’s browser is the one who computed said proof of work, or has delegated it to a different system and is simply relaying its results. At scale, you’d end up with a large botnet that receives proof of work tokens to solve for the scraping network to use.

Have just updated my text: "ignoring non-free" :-)

As a serious computer user getting on for 25 years using text based search tools I've long made various "single-site" tools. A big inspiration way back was Surfraw [1], originally created by Julian Assange. Reality is, most of us use a small number of websites regularly. nearly all the info I want to touch is three keystrokes away on the command-line or from within emacs.

When search died, a few years ago practically now, I was still teaching a level-7 Research Methods course. The universities literally did not notice that all of the advice we gave students was totally obsolete and that it was not really possible to conduct academic research that way.

Research today is very much more like it was in the pre-interent era. You need to curate and keep in mind a set of reliable sources and personal, private collections.

Had the misfortune of needing to spend a week using a standard browser and sites like Google. It was beyond shocking. What I found I can only describe as a wastescape, a war zone, a bombed-out favela with burned out cars, overflowing sewers, piles of rubble and dead dogs lying in gutters.

My first thought was kinda, "Oh sweet Jesus Christ, what happened to my Internet?", and the very next one was "How does anyone get anything done now?" How does the economy still function? And of the course the answers are "They don't" and "It doesn't".

I think this is a really serious situation. There's simply no way that as "knowledge workers", scientists, or whatever people call us now, we can be as competitive as we were 10 or 20 years ago given the colossal degradation of our tools. We have to stop this foolish self-deception that things are "getting better". Google were a company that created free search. Well done. But that was then. We remain stuck in this strange mythology that advertising companies like Google and other enshitified BigTech are a net asset to the economy. Surely they're a vast parasitical drain and need digging into the ground so the rest of us can get on with something resembling progress?

[1] http://surfraw.org/

While I am glad that you seem to have found a new workflow that you like, your description strikes me as a personal experience.

I am aware that a lot of people use searches as a form of navigation, but it’s also very common that people use bookmarks, speed dial, history, pinned tabs, and other browser features instead of searching. My Firefox is configured to not do online searches when I type into the address bar, instead I get only history suggestions. This setup allows for quick navigation, and does not require any steps to set up new pages that I need to visit.

What I want to say that while you seem to imply that you found a different pattern of use that many people will soon migrate to, I think these patterns have always been popular. People discover and make use of them as needed.

It’s also strange that you put such a negative sentiment on interconnected documents. Do you not realize how important these connections were for you to be able to reach the point you are at now? How else would you have found the things that are useful to you? By watching ads?

Search engines are also … really not really a good example of the strengths of the interconnected web, as they are mostly a one way thing. Consider instead a Hacker News discussion about a blog, and some other blog linking to that discussion, creating these interconnected but still separate communities and documents.

> I've been experimenting with creating single-site browsers[1] for all websites I routinely visit, effectively removing navigational queries from search engines

Surely it would make more sense to use bookmarks?

The example repo is a little confusing to me, since it seems to use client-side JS to demonstrate that it doesn't need client-side JS: "It bootstraps a service worker and [...] No client-side JavaScript!"

But I guess the point is that the code in the service worker could have been on the server instead?

The trick seems to be using a template element with a slot and then slotting in the streamed content at the end. But you could probably also do it using just CSS to reposition the content from the bottom to the top, similarly to how many websites handle navigation menus, assuming that the client supports CSS.

You could use some simple CSS tricks (like flex-direction: column-reverse) to make the end of the HTML render above the search results and just keep appending the AI generated text to the HTML without closing the connection.

now that i think about it you could do it quite nicely with svg's and foreignObject

this is the only time in 15 years i've wished hn had a lol react. :D

How is that possible?

There's absolutely no need for JavaScript on a page that has a text input and two buttons and that has worked without JS for three decades. Given Google's reputation for privacy and the constant attempts at selling their users out, it's fair to assume that the reason they're requiring JavaScript is not noble.

Javascript is like Flash-lite. Is it evil? No. It's great, even.

What every last commercial site uses it for IS evil, without a doubt.

It's literally almost an anagram of Satan's Armpits.

It uses a lot of data, it is a security risk, it is a privacy risk, and it forces you to throw away your old devices.

It is in Google’s hands

No, limiting user freedom as to whether or not to use it is.

Javascript has always been evil.

Yes.

Yeah, roughly since 1996.

Always has been.

It saves me more than that most months. This month, a single superior query result saved me four hours of driving and $100.

For you?

For me, it's worth every cent.

You can find the endless reviews:

https://hn.algolia.com/?query=kagi&sort=byPopularity

Possibly one of the most reviewed SaaS companies here.

And yes, it's pretty great. But it's just Search (with some AI tossed in).

Incidentally, Kagi works with JavaScript disabled.

I tried it, it’s fine but I prefer the mix of uBlacklist and Bing with a duck branding and ChatGPT.

Yes

This gives me an idea: thanks to anti-spam mechanisms residential proxies + headless browsers provide a better experiences than regular browsing on real devices.

Instead of PoW, maybe just make the clients prove they are capable of proxying browser sessions?