Hacker News bans Google and all other search engines (opens in new tab)

(news.ycombinator.com)

318 pointsjsrfded16y ago120 comments

Looks like news.ycombinator.com is rejecting all search engine robots with "User-Agent: * Disallow: /". This is unfortuante, I often do site: searches to find old threads here. esp since news.ycombinator.com doesn't have its own site search, this is the only way to find old threads.

pg, what's up?

Hacker News bans Google and all other search engines

(news.ycombinator.com)

318 pointsjsrfded16y ago120 comments

pg, what's up?

120 comments

94 comments · 36 top-level

g0atbutt16y ago· 15 in thread

That's pretty disappointing. I'd love to hear an official explanation of why this choice was made.

cookiecaper16y ago

I quite agree. I use Google to search HN. I just used it the other day, because I didn't remember this one guy's startup or username, but I knew it involved family websites. I was able to find it after a bit of search-fu. I am sad to see that HN won't be indexed by Google anymore. :(

joe_the_user16y ago

Yes, the way that references vanish from hn, I really need Google to find things a couple of days old. I have tried and found other search methods not worthwhile.

Grumble...

dantheman16y ago

One way to search news.ycombinator is to visit searchyc.com

2 more replies

Qz16y ago

Hmm, does search-fu qualify as something you can put on a resume?

zachbeane16y ago

I wonder if part of the reason is that the Arc web server doesn't send HTTP status codes properly. Every URL path, valid or not, returns a 200, even if the content is only "Unknown."

For example, check the headers sent back for:

http://news.ycombinator.com/no/way/this/is/possibly/valid

    HTTP/1.1 200 OK
    Content-Type: text/html; charset=utf-8
    Connection: close

If a search engine sees that, it will periodically revisit it forever.

pierrefar16y ago

The correct solution is to fix the Arc server to return correct HTTP headers, not block everything using robots.txt.

2 more replies

sjs16y ago

Until someone creates an HTML link to such a document, how does this problem manifest itself?

2 more replies

Apreche16y ago

This sucks for one major reason. Sometimes I want to dig up a link that was on HN, and I don't know where else to find it. So I search google with site:ycombinator.com. If they block Google, then that won't work.

At the very least, add a search engine to HN itself so I can search for old links.

eswat16y ago

http://searchyc.com/ (assuming it's not blocked also)

Chirael16y ago

Agreed 100%

Why produce a great site and then not let your users access its valuable content?

Things scroll by the front page so fast it's not really fair not to provide a search feature.

I don't care if they let external search engines (Google et al) index it, but they should at least provide /some/ kind of internal search feature themselves.

IMO providing a search feature is probably at least as effective, if not more so, than the "noprocrast" flag.

After all, if I know I can search past items at any point in the future, I don't feel a pressure to load up the front page several times a day.

Whereas if the fast-scrolling front page is the only way to access items, it encourages that sort of rat-hitting-the-bar behavior I learned about in college psych classes ;)

pvg16y ago

HN is very, very slow. Removing crawlers likely helps with that.

tjic16y ago

Removing a leg or two could also help one towards a weight loss goal.

1 more reply

lsc16y ago

I would bet good money that it's nothing that a good dedicated server with, say, 32GiB ram and 2 or 4 sas spindles couldn't fix.

pjhyett16y ago

You can set gbot's crawler rate with google webmaster tools if that's the issue.

LouCypher16y ago

The robots.txt file's content-type is text/html, not text/plain. It won't work. I'm still able to access the contents of this site from Yahoo Pipes. Don't worry about it.

pg16y ago· 8 in thread

Don't worry, it doesn't mean anything. The software for ranking applications runs on the same server, and it is horribly inefficient (something 4 people use every 6 months doesn't tend to get optimized much). This weekend all of us were reading applications at the same time, and the system was getting so slow that I banned crawlers for a bit to buy us some margin. (Traffic from crawlers is much more expensive for us than traffic from human users, because it interacts badly with lazy item loading.) We only finished reading applications an hour before I had to leave for SXSW, so I forgot to set robots.txt back to the normal one, but I just did now.

g0atbutt16y ago

That makes sense. Thanks for the explanation pg!

lisper16y ago

Dude, why not just get yourself another server?

sbhat716y ago

"something 4 people use every 6 months doesn't tend to get optimized much"

1 more reply

snprbob8616y ago

I'd be really worried about some algorithm somewhere deciding to crawl less frequently even after the temporary ban has been lifted.

jsrfdedOP16y ago

Robot bans can render your archive.org records inaccessable, zap all your former code 200 urls out of the major search indices, and yes, kill your inclusion in the list of topical sites.

Much better to set a temporary Crawl-Delay directive. Otherwise you're not just telling the engines to pause crawling you, you're telling them "take all of my pages out of your index."

1 more reply

duck16y ago

Why is this still on the front page of HN? This seems like such a non-story.

ks16y ago

Have you considered using last-modified/etag headers to control the crawlers? If you detect a crawler, and it requests a page that it has visited before, you could just return a 304 without doing anything else.

robryan16y ago

On a related note the threads page seems to load very slowly now. I guess because it would place heavy load on the DB grabbing all your comments from different threads.

mikecane16y ago· 7 in thread

What is the reason for this?

idoh16y ago

My bet is that it is for technical reasons. Whenever you see a reply link (among other things) the server has to persist a continuation for it. Getting all of those pages crawled must generate a massive number of continuations to persist.

I know that this is one of the reasons that vote-links look the way they do.

If that's not the reason, then my #2 guess is that getting the whole site crawled is moving too many comments into the cache. If I remember correctly, whenever an uncached comment gets accessed, it gets cached (basically adding it to a hash), and thus added to the in-memory ram. That gradually raises the memory requirements for running the app, until pg does a restart, which resets the cache as well.

blasdel16y ago

And the regular restarts invalidate every single continuation-based URL

zitterbewegung16y ago

Probably to stop linkspam?

jsrfdedOP16y ago

It's unfortunate because I often use site: to find old threads on ycombinator. especially since news.ycombinator.com doesn't have its own search facility, external search engines are the old way to comb through old posts here.

1 more reply

Frazzydee16y ago

That doesn't make sense. Why not just add rel=nofollow to all links?

mikecane16y ago

OK, thanks.

drhodes16y ago

maybe to see what firms ignore robot.txt files?

tptacek16y ago· 4 in thread

I'm interested in the justification for this, but I'm happy about it. I'm actually uncomfortable with how high Hacker News comments score on Google.

jamesbressi16y ago

Please enlighten me as to why you are "uncomfortable with how high Hacker News comments score on Google." I mean it sincerely, I'd like to know because I have no clue why that is an issue?

tptacek16y ago

Talking to people in a relatively small forum like Hacker News: totally comfortable.

Talking with the entire world by having my comments associated in ways I don't expect with keywords: less comfortable.

1 more reply

chaosmachine16y ago

start posting your comments in rot13? ;)

earle16y ago

Some Brainfuck to implement this suggestion:

,[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>++++++++++++++<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>>>+++++[<<----->>-]<<<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>++++++++++++++<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>++++++++++++++<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>>>+++++[<<----->>-]<<<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>+<<-[>>++++++++++++++<<-[>>+<<-]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]>>.[-]<<,]

eel16y ago· 3 in thread

What does this mean for http://www.searchyc.com? Will it stop too? (Afterall, it must be a bot too, and a robots.txt like this implies that no bots /should/ scrape the site.) Or will it be allowed?

qeorge16y ago

That was my immediate question too. In addition to searchyc, we lose every other HN mash-up present and future.

Maybe this will finally bring about an HN API.

weaksauce16y ago

robots.txt is really only like a note on the front door of your unlocked house that says "please only look at these items in these rooms." or "please do not look in my house" If someone wants to write a program to index the site they can still do so. Google, MSN, Bing, yahoo, et. al. will respect the robots.txt file though.

mkull16y ago

Or searchyc could simply ignore robots.txt

pierrefar16y ago· 3 in thread

Does that also kill Search YC?

What about the "official" HNSearch (http://www.webmynd.com/html/hackernews.html )?

blasdel16y ago

The 'official' HNSearch is completely useless for anyone who would ever want to search HN. It's a Firefox Extension (!) that adds a 400px sidebar to your google search results pages showing site: searches along with an extra adsense ad. It's less useless than the 5 IE toolbars your annoying relatives live with, but just as obnoxious.

I just installed it again for a laugh, and it actually redirects to searchyc.com results when you click on its "Hacker News" header — even though pg staunchly refuses to ever link to searchyc. I made fun of him for it at his party before Startup School and he tried to laugh it off with "they're a YC company!" as if that weren't obvious on the face of it.

pierrefar16y ago

I don't use either. I'm interested for two reasons:

1. I use Google site: searches a LOT. I can remember reading about something I need now a long time ago on HN and I go find it.

2. One item on my "if I have time" projects list is to do "something" "fun" with HN submissions. Now I'm officially barred, regardless of what I would have thought up.

tptacek16y ago

What's pg's problem with SearchYC? It's incredibly useful.

2 more replies

tkiley16y ago· 3 in thread

Is this permanent?

HN is a good source of google juice for interesting new startups, and it would be a shame to see that go away...

qeorge16y ago

That's probably the main issue. HN's authority and DoFollow'd links make it a prime target for linkspam.

There's a lot of reasons I'm bummed about this, but I have to say: startups missing out on free PageRank is not one of them.

pierrefar16y ago

The way it's done now is that all new submissions are nofollowed until they hit a certain karma threshold. A while back when I looked at this, the threhold was about 3-5 up votes.

This is actually a very neat and simple way to exercise editorial control to remove the nofollow: if enough people in the community like it, there is something useful about it and it is vetted.

ericb16y ago

The links are nofollow'd now.

edit: Oh wait, seems that only comment links are nofollow'd.

1 more reply

tsally16y ago· 3 in thread

So is SearchYC done, or do they use a scraper not a crawler?

grinich16y ago

What's the difference?

tsally16y ago

In my mind legitimate crawlers are large engineered programs that respect robots.txt, whereas a scraper might just be a Perl script some dude wrote to check a website for updates periodically. Technically I suppose they all should probably respect robots.txt, but in my experience something you think of as a scraper rarely does.

Raphael16y ago

Well, what if I just saved every HTML page I visited. I'm clearly not a bot.

andrewcooke16y ago· 3 in thread

surprised no-one has suggested that this is to improve performance. this place can be very slow sometimes, and blocking robots can reduce load.

keltex16y ago

That's not a legitimate reason to ban google. You can always (using google's webmaster tools) throttle their indexing. Or you can use search-engine specific lines in robots.txt (Bing anyone?) to ban misbehaving bots.

The only reason to do this is if you don't want your site indexed by google. Which I really can't think of a legitimate reason to do so.

jonknee16y ago

... And you can speed the site up. There's no reason HN has to be so slow.

andrewcooke16y ago

true, but it's a busy time for yc. maybe this was just a stop-gap measure? when did it start?

1 more reply

proee16y ago· 2 in thread

This makes me sad. I often do site:news.ycombinator.com google searches for key topics.

Mc_Big_G16y ago

I do exactly the same thing at least once a day. I can always remember just a snippet of the post or subject I want to find and that type of query always gets me what I want. I just used it today to find the YC company who is doing the journalism stuff so I could send the link to my nephew.

Concours16y ago

That's my standard method as well to search HN.

brianr16y ago· 2 in thread

Whoa there, everything important is still crawlable and indexable. Here's what robots.txt says right now:

  User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /threads?

This just disallows those pages... not the home page, and not the /item? action (note the url of this page).

mcav16y ago

It used to just disallow all. PG must be changing it as we speak.

sahaj16y ago

looks like it's been updated.

sahaj16y ago· 1 in thread

i just googled this: http://www.google.com/search?q=site:news.ycombinator.com+Hac...

and found this: http://news.ycombinator.com/item?id=165279

one of the top replies:

My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google. As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.

borism16y ago

ugh

perhaps smart hackers shouldn't block everything out like this without any discretion whatsoever?

Matt_Cutts16y ago· 1 in thread

Does anyone know why HN decided to disallow search engines? It's certainly up to the website owner to decide, but there are good ways to (say) reduce the load on the web server without blocking search engines entirely.

seiji16y ago

The HN code keeps everything in memory (lazily loaded from disk). If every search engine and me-too crawler out there hits every page, you constantly have the entire HN corpus in memory. I assume the memory overload is what leads to the site being slow and unreliable. I don't know if the arc code has any LRU scheme or if everything sticks around until the server falls over.

HN just needs a super simple caching proxy in front of it to reduce load on the app server. Alternatively, just generate static pages for all topics older than 5 days.

Will sysadmin for YC dinner invites.

chaosmachine16y ago· 1 in thread

It appears to have been changed, just seconds ago. Now it reads:

  User-Agent: * 
  Disallow: /x?
  Disallow: /vote?
  Disallow: /reply?
  Disallow: /submitted?
  Disallow: /threads?

Matt_Cutts16y ago

That seems like a nice middle ground--it should save server load, but search engines can still get to useful content.

ddsmooth16y ago· 1 in thread

That notwithstanding, if you go to Google.com, type hacker news into the search box and click "I'm Feeling Lucky" you will find yourself at a familiar web page. ;-)

jsrfdedOP16y ago

Google indexes robots banned sites based on links. But notice that the snippet is gone. Google also won't be able to index HN posts and comments anymore.

jgavris16y ago· 1 in thread

very interesting

jgavris16y ago

the way i see it, they're going to have to put up a loginwall to stop crawlers...and then we'll just crawl back in anyway...

petercooper16y ago

Will this break Google Reader and other RSS readers from legitimately using the RSS feed here? After all, Google Reader uses a bot to read the feed and allows us to search it from within their app.. much like their search engine does with regular pages :-) (That is, "web robots" aren't just spiders.)

Also things like http://hacker-newspaper.gilesb.com/ and http://hnsort.com/ become less legitimate due to this. If the reason is to reduce the SEO benefits of getting a link on HN, just "nofollow" everything instead..

(Update: Googling on this topic brought up a page of my own where a Google Reader engineer explained how Google Reader deals with robots.txt - http://www.petercooper.co.uk/google-reader-ignores-robottxt-... - though their definition of Web robot is far from universal)

chaosmachine16y ago

If this is an anti-spam measure, I expect it will be about as effective as "no follow" was. There's still plenty of good reasons for spammers to submit crap. RSS and Twitter syndication of links, for example.

On the other hand, if the goal is to push HN back into semi-obscurity by making it harder to find, it might work.

simonw16y ago

I noticed that the other day while playing around with the YQL console - YQL obeys robots.txt, so the Hacker News data table doesn't work any more.

redsymbol16y ago

Everyone RELAX. There's probably a good explanation for this, maybe it was even an (easily correctable) accident. Give Paul a chance to respond before raising the pitchforks, okay?

WalkingDead16y ago

I guess that might have to do with load these search engines generate. Previosly in a thread we checked that comments posted in HN appear in google search after a minute of posting. That should create a good amount of load on HN. Multiply it for all the search engines in the wild, and PG probably have decided blocking those woun't do any harm. My guesss. Could be wrong, or it might be temporary or a mistake.

CoreDumpling16y ago

Well, at least my profile on HN will no longer be the first result for a Google search for my username (it can now revert to some blog that's not mine). But I really wish HN were searchable; I try to find insight on the comment threads here all the time.

ars16y ago

The robots page is returning the mime-type text/html instead of text/plain

nirmal16y ago

Looks like the Readable Feeds stuff is still working. Not sure how this will effect it in the future.

http://andrewtrusty.appspot.com/readability/

prs16y ago

Truth be told: On various occasions the site: operator on Google came in handy for me to dig out some nugget of information from the archives of HN.

I can't await to hear the reason behind that decision.

sev16y ago

I'd like to understand why this decision has been made, as well as why the explanation has been delayed (or will not be given at all) directly from the source.

euroclydon16y ago

Didn't I read here that the MSN bot refuses to obey the robots.txt file? Maybe I'll have to search HN with it.

AdamN16y ago

I think this is still on the front page because people want it to stay this way.

rmason16y ago

I think that it's brilliant. Short term it gets publicity and drives people to the group who are curious.

Long term PG signs a deal with Bing to be the exclusive search engine for Hacker News that pays for the servers and bandwidth.

Down mod me if you will but it's simply brilliant.

jchrisa16y ago

not strictly true (read the code), but of all the sites I know, HN is best poised to implement their own search and drop Google. go for it!

karlzt16y ago

I just searched in google and is not blocked

_pius16y ago

Very lame.

python12316y ago

This is horrible. Now what's a talentless, middle-aged career software engineer with hopeless dreams of entrepreneurship going to do with his time?

hackermom16y ago

i really enjoyed that the news item linked to the robots.txt file :) it's nerdy, it's short, it's clear and concise, all in one :)

quinto4216y ago

This is fantastic! No more spam links.

fnazeeri16y ago

Bye bye HN...it was good to know you.

j / k navigate · click thread line to collapse

120 comments

94 comments · 36 top-level

g0atbutt16y ago· 15 in thread

That's pretty disappointing. I'd love to hear an official explanation of why this choice was made.

cookiecaper16y ago

joe_the_user16y ago

Yes, the way that references vanish from hn, I really need Google to find things a couple of days old. I have tried and found other search methods not worthwhile.

Grumble...

dantheman16y ago

One way to search news.ycombinator is to visit searchyc.com

2 more replies

Qz16y ago

Hmm, does search-fu qualify as something you can put on a resume?

zachbeane16y ago

I wonder if part of the reason is that the Arc web server doesn't send HTTP status codes properly. Every URL path, valid or not, returns a 200, even if the content is only "Unknown."

For example, check the headers sent back for:

http://news.ycombinator.com/no/way/this/is/possibly/valid

    HTTP/1.1 200 OK
    Content-Type: text/html; charset=utf-8
    Connection: close

If a search engine sees that, it will periodically revisit it forever.

pierrefar16y ago

The correct solution is to fix the Arc server to return correct HTTP headers, not block everything using robots.txt.

2 more replies

sjs16y ago

Until someone creates an HTML link to such a document, how does this problem manifest itself?

2 more replies

Apreche16y ago

At the very least, add a search engine to HN itself so I can search for old links.

eswat16y ago

http://searchyc.com/ (assuming it's not blocked also)

Chirael16y ago

Agreed 100%

Why produce a great site and then not let your users access its valuable content?

Things scroll by the front page so fast it's not really fair not to provide a search feature.

I don't care if they let external search engines (Google et al) index it, but they should at least provide /some/ kind of internal search feature themselves.

IMO providing a search feature is probably at least as effective, if not more so, than the "noprocrast" flag.

After all, if I know I can search past items at any point in the future, I don't feel a pressure to load up the front page several times a day.

Whereas if the fast-scrolling front page is the only way to access items, it encourages that sort of rat-hitting-the-bar behavior I learned about in college psych classes ;)

pvg16y ago

HN is very, very slow. Removing crawlers likely helps with that.

tjic16y ago

Removing a leg or two could also help one towards a weight loss goal.

1 more reply

lsc16y ago

I would bet good money that it's nothing that a good dedicated server with, say, 32GiB ram and 2 or 4 sas spindles couldn't fix.

pjhyett16y ago

You can set gbot's crawler rate with google webmaster tools if that's the issue.

LouCypher16y ago

The robots.txt file's content-type is text/html, not text/plain. It won't work. I'm still able to access the contents of this site from Yahoo Pipes. Don't worry about it.

pg16y ago· 8 in thread

g0atbutt16y ago

That makes sense. Thanks for the explanation pg!

lisper16y ago

Dude, why not just get yourself another server?

sbhat716y ago

"something 4 people use every 6 months doesn't tend to get optimized much"

1 more reply

snprbob8616y ago

I'd be really worried about some algorithm somewhere deciding to crawl less frequently even after the temporary ban has been lifted.

jsrfdedOP16y ago

Robot bans can render your archive.org records inaccessable, zap all your former code 200 urls out of the major search indices, and yes, kill your inclusion in the list of topical sites.

Much better to set a temporary Crawl-Delay directive. Otherwise you're not just telling the engines to pause crawling you, you're telling them "take all of my pages out of your index."

1 more reply

duck16y ago

Why is this still on the front page of HN? This seems like such a non-story.

ks16y ago

robryan16y ago

On a related note the threads page seems to load very slowly now. I guess because it would place heavy load on the DB grabbing all your comments from different threads.

mikecane16y ago· 7 in thread

What is the reason for this?

idoh16y ago

I know that this is one of the reasons that vote-links look the way they do.

blasdel16y ago

And the regular restarts invalidate every single continuation-based URL

zitterbewegung16y ago

Probably to stop linkspam?

jsrfdedOP16y ago

1 more reply

Frazzydee16y ago

That doesn't make sense. Why not just add rel=nofollow to all links?

mikecane16y ago

OK, thanks.

drhodes16y ago

maybe to see what firms ignore robot.txt files?

tptacek16y ago· 4 in thread

I'm interested in the justification for this, but I'm happy about it. I'm actually uncomfortable with how high Hacker News comments score on Google.

jamesbressi16y ago

Please enlighten me as to why you are "uncomfortable with how high Hacker News comments score on Google." I mean it sincerely, I'd like to know because I have no clue why that is an issue?

tptacek16y ago

Talking to people in a relatively small forum like Hacker News: totally comfortable.

Talking with the entire world by having my comments associated in ways I don't expect with keywords: less comfortable.

1 more reply

chaosmachine16y ago

start posting your comments in rot13? ;)

earle16y ago

Some Brainfuck to implement this suggestion:

eel16y ago· 3 in thread

What does this mean for http://www.searchyc.com? Will it stop too? (Afterall, it must be a bot too, and a robots.txt like this implies that no bots /should/ scrape the site.) Or will it be allowed?

qeorge16y ago

That was my immediate question too. In addition to searchyc, we lose every other HN mash-up present and future.

Maybe this will finally bring about an HN API.

weaksauce16y ago

mkull16y ago

Or searchyc could simply ignore robots.txt

pierrefar16y ago· 3 in thread

Does that also kill Search YC?

What about the "official" HNSearch (http://www.webmynd.com/html/hackernews.html )?

blasdel16y ago

pierrefar16y ago

I don't use either. I'm interested for two reasons:

1. I use Google site: searches a LOT. I can remember reading about something I need now a long time ago on HN and I go find it.

2. One item on my "if I have time" projects list is to do "something" "fun" with HN submissions. Now I'm officially barred, regardless of what I would have thought up.

tptacek16y ago

What's pg's problem with SearchYC? It's incredibly useful.

2 more replies

tkiley16y ago· 3 in thread

Is this permanent?

HN is a good source of google juice for interesting new startups, and it would be a shame to see that go away...

qeorge16y ago

That's probably the main issue. HN's authority and DoFollow'd links make it a prime target for linkspam.

There's a lot of reasons I'm bummed about this, but I have to say: startups missing out on free PageRank is not one of them.

pierrefar16y ago

The way it's done now is that all new submissions are nofollowed until they hit a certain karma threshold. A while back when I looked at this, the threhold was about 3-5 up votes.

This is actually a very neat and simple way to exercise editorial control to remove the nofollow: if enough people in the community like it, there is something useful about it and it is vetted.

ericb16y ago

The links are nofollow'd now.

edit: Oh wait, seems that only comment links are nofollow'd.

1 more reply

tsally16y ago· 3 in thread

So is SearchYC done, or do they use a scraper not a crawler?

grinich16y ago

What's the difference?

tsally16y ago

Raphael16y ago

Well, what if I just saved every HTML page I visited. I'm clearly not a bot.

andrewcooke16y ago· 3 in thread

surprised no-one has suggested that this is to improve performance. this place can be very slow sometimes, and blocking robots can reduce load.

keltex16y ago

The only reason to do this is if you don't want your site indexed by google. Which I really can't think of a legitimate reason to do so.

jonknee16y ago

... And you can speed the site up. There's no reason HN has to be so slow.

andrewcooke16y ago

true, but it's a busy time for yc. maybe this was just a stop-gap measure? when did it start?

1 more reply

proee16y ago· 2 in thread

This makes me sad. I often do site:news.ycombinator.com google searches for key topics.

Mc_Big_G16y ago

Concours16y ago

That's my standard method as well to search HN.

brianr16y ago· 2 in thread

Whoa there, everything important is still crawlable and indexable. Here's what robots.txt says right now:

  User-Agent: * Disallow: /x? Disallow: /vote? Disallow: /reply? Disallow: /submitted? Disallow: /threads?

This just disallows those pages... not the home page, and not the /item? action (note the url of this page).

mcav16y ago

It used to just disallow all. PG must be changing it as we speak.

sahaj16y ago

looks like it's been updated.

sahaj16y ago· 1 in thread

i just googled this: http://www.google.com/search?q=site:news.ycombinator.com+Hac...

and found this: http://news.ycombinator.com/item?id=165279

one of the top replies:

borism16y ago

ugh

perhaps smart hackers shouldn't block everything out like this without any discretion whatsoever?

Matt_Cutts16y ago· 1 in thread

seiji16y ago

HN just needs a super simple caching proxy in front of it to reduce load on the app server. Alternatively, just generate static pages for all topics older than 5 days.

Will sysadmin for YC dinner invites.

chaosmachine16y ago· 1 in thread

It appears to have been changed, just seconds ago. Now it reads:

  User-Agent: * 
  Disallow: /x?
  Disallow: /vote?
  Disallow: /reply?
  Disallow: /submitted?
  Disallow: /threads?

Matt_Cutts16y ago

That seems like a nice middle ground--it should save server load, but search engines can still get to useful content.

ddsmooth16y ago· 1 in thread

That notwithstanding, if you go to Google.com, type hacker news into the search box and click "I'm Feeling Lucky" you will find yourself at a familiar web page. ;-)

jsrfdedOP16y ago

Google indexes robots banned sites based on links. But notice that the snippet is gone. Google also won't be able to index HN posts and comments anymore.

jgavris16y ago· 1 in thread

very interesting

jgavris16y ago

the way i see it, they're going to have to put up a loginwall to stop crawlers...and then we'll just crawl back in anyway...

petercooper16y ago

chaosmachine16y ago

On the other hand, if the goal is to push HN back into semi-obscurity by making it harder to find, it might work.

simonw16y ago

I noticed that the other day while playing around with the YQL console - YQL obeys robots.txt, so the Hacker News data table doesn't work any more.

redsymbol16y ago

Everyone RELAX. There's probably a good explanation for this, maybe it was even an (easily correctable) accident. Give Paul a chance to respond before raising the pitchforks, okay?

WalkingDead16y ago

CoreDumpling16y ago

ars16y ago

The robots page is returning the mime-type text/html instead of text/plain

nirmal16y ago

Looks like the Readable Feeds stuff is still working. Not sure how this will effect it in the future.

http://andrewtrusty.appspot.com/readability/

prs16y ago

Truth be told: On various occasions the site: operator on Google came in handy for me to dig out some nugget of information from the archives of HN.

I can't await to hear the reason behind that decision.

sev16y ago

I'd like to understand why this decision has been made, as well as why the explanation has been delayed (or will not be given at all) directly from the source.

euroclydon16y ago

Didn't I read here that the MSN bot refuses to obey the robots.txt file? Maybe I'll have to search HN with it.

AdamN16y ago

I think this is still on the front page because people want it to stay this way.

rmason16y ago

I think that it's brilliant. Short term it gets publicity and drives people to the group who are curious.

Long term PG signs a deal with Bing to be the exclusive search engine for Hacker News that pays for the servers and bandwidth.

Down mod me if you will but it's simply brilliant.

jchrisa16y ago

not strictly true (read the code), but of all the sites I know, HN is best poised to implement their own search and drop Google. go for it!

karlzt16y ago

I just searched in google and is not blocked

_pius16y ago

Very lame.

python12316y ago

This is horrible. Now what's a talentless, middle-aged career software engineer with hopeless dreams of entrepreneurship going to do with his time?

hackermom16y ago

i really enjoyed that the news item linked to the robots.txt file :) it's nerdy, it's short, it's clear and concise, all in one :)

quinto4216y ago

This is fantastic! No more spam links.

fnazeeri16y ago

Bye bye HN...it was good to know you.

j / k navigate · click thread line to collapse