Show HN: Scaling PhantomJS With Ghost Town (opens in new tab)

(blog.buzzvil.com)

32 pointstkazec11y ago11 comments

11 comments

We developed the same thing: https://github.com/dailymuse/phantom-cluster

It was used for prerender [1] for some time, before being changed so that clustering happened at the HTTP worker level.

Scaling PhantomJS is no walk in the park. There's a lot of duct tape to get it to communicate with node.js, and it's easy for a memory leak to occur.

[1] https://github.com/prerender/prerender

gcb011y ago

you are all doing it wrong... we use phantomjs on its own. and run the test code in it. so its analogous to loading the target page, and then running your code/test in a browser dev console. you take care of concurrency and retries in your preferred shell. this also allows things like integration with Makefiles, etc.

using it inside node is just masochism.

codereflection11y ago

Can you please qualify the "neglected projects" statement? Looking at the commits on both projects show that they are not neglected.

phantomjs: https://github.com/ariya/phantomjs/commits/master

phantomjs-node: https://github.com/sgentle/phantomjs-node/commits/master

tkazecOP11y ago

PhantomJS: Extremely popular (10k+ stars!) with a comparative trickle of commits. The major flaws might be excusable, but the number of small issues and unmerged pull requests is not.

phantomjs-node: Popular but badly structured and documented. Many small ignored issues.

FootballMuse11y ago

At least for PhantomJS, there are 236 outstanding prs/issues since their last release Jan-26 with only 19 commits mostly dealing with Travis.

Also the development update Twitter stream @PhantomJS is not very active.

mootothemax11y ago

Has anyone had any luck with PhantomJS and web fonts?

I'd love to use PhantomJS for taking screenshots of designs supplied by third-parties to us, and changing the CSS involved by hand is not a realistic option.

The issue itself appears to be nearly 2 years old:

https://github.com/ariya/phantomjs/issues/10592

And PhantomJS's core developers appear to be putting off the fix until version 2 is released, whenever that may be:

https://github.com/ariya/phantomjs/pull/12108

Any ideas? I'm currently investigating using SlimerJS, which uses the Gecko engine, but that involves its own different set of challenges.

damian200011y ago

Interested to know if Selenium web driver integration was looked into? You can use Selenium Grid for example to run distributed jobs with a Phantom JS backend... https://github.com/detro/ghostdriver

dsl11y ago

> a system to reliably render images at scale

I'm curious what exactly "render" means in your context? Phantom.js is headless, so it would be futile to be simply rendering GIFs and JPGs within the browser I assume?

martin-adams11y ago

Imagine it like a web page to PNG converter. This doesn't need a desktop/display attached to do that. That's what "render" means in a headless context.

dsl11y ago

Ah. I get it. Brain did not parse "render" as the output side, but rather the input, so I was trying to figure out why they used phantom.js as an elaborate image format converter.

qhoc11y ago

I wrote my own doing something similar with leveraging the casperjs layer instead. I was just too lazy to deal with phantomjs directly.

j / k navigate · click thread line to collapse

11 comments

m0th8711y ago

We developed the same thing: https://github.com/dailymuse/phantom-cluster

It was used for prerender [1] for some time, before being changed so that clustering happened at the HTTP worker level.

Scaling PhantomJS is no walk in the park. There's a lot of duct tape to get it to communicate with node.js, and it's easy for a memory leak to occur.

[1] https://github.com/prerender/prerender

gcb011y ago

using it inside node is just masochism.

codereflection11y ago

Can you please qualify the "neglected projects" statement? Looking at the commits on both projects show that they are not neglected.

phantomjs: https://github.com/ariya/phantomjs/commits/master

phantomjs-node: https://github.com/sgentle/phantomjs-node/commits/master

tkazecOP11y ago

PhantomJS: Extremely popular (10k+ stars!) with a comparative trickle of commits. The major flaws might be excusable, but the number of small issues and unmerged pull requests is not.

phantomjs-node: Popular but badly structured and documented. Many small ignored issues.

FootballMuse11y ago

At least for PhantomJS, there are 236 outstanding prs/issues since their last release Jan-26 with only 19 commits mostly dealing with Travis.

Also the development update Twitter stream @PhantomJS is not very active.

mootothemax11y ago

Has anyone had any luck with PhantomJS and web fonts?

I'd love to use PhantomJS for taking screenshots of designs supplied by third-parties to us, and changing the CSS involved by hand is not a realistic option.

The issue itself appears to be nearly 2 years old:

https://github.com/ariya/phantomjs/issues/10592

And PhantomJS's core developers appear to be putting off the fix until version 2 is released, whenever that may be:

https://github.com/ariya/phantomjs/pull/12108

Any ideas? I'm currently investigating using SlimerJS, which uses the Gecko engine, but that involves its own different set of challenges.

damian200011y ago

Interested to know if Selenium web driver integration was looked into? You can use Selenium Grid for example to run distributed jobs with a Phantom JS backend... https://github.com/detro/ghostdriver

dsl11y ago

> a system to reliably render images at scale

I'm curious what exactly "render" means in your context? Phantom.js is headless, so it would be futile to be simply rendering GIFs and JPGs within the browser I assume?

martin-adams11y ago

Imagine it like a web page to PNG converter. This doesn't need a desktop/display attached to do that. That's what "render" means in a headless context.

dsl11y ago

Ah. I get it. Brain did not parse "render" as the output side, but rather the input, so I was trying to figure out why they used phantom.js as an elaborate image format converter.

qhoc11y ago

I wrote my own doing something similar with leveraging the casperjs layer instead. I was just too lazy to deal with phantomjs directly.

j / k navigate · click thread line to collapse