Using PhantomJS at scale (opens in new tab)

(sorcery.smugmug.com)

17 pointschestone12y ago3 comments

3 comments

3 comments · 2 top-level

iLoch12y ago· 1 in thread

I face a similar issue going forward. I think http://prerender.io has been posted to HN before - seems to be a pretty reasonable plug and play solution.

rdoherty12y ago

Thanks for the link! We started working on this project in early 2013, prerender.io didn't exist then (I think).

prerender.io does do most of what we do. A few things we do differently are keeping a pool of PhantomJS workers always available, killing HTTP requests that never complete, putting it behind HAproxy and logging.

onethumb12y ago

This is a particularly big issue for us because we spend more than 50% of our resources on serving robots (not humans!).

I wrote about this back in July of 2010, and the problem has only gotten worse: http://don.blogs.smugmug.com/2010/07/15/great-idea-google-sh...

It's a crazy problem to have, but we haven't found a solution. We can't just disappear from all the various indexes and collections that these robots represent.

j / k navigate · click thread line to collapse