Ask HN: How to wget pages with javascript?

4 pointsbaconface14y ago8 comments

I need to wget a url, allow the javascript to execute, and capture the result in a file. I've tried wget, curl, httrack,plugins like lget, and links(2) and none of them execute JS.

I want something like $: firefox <url> -s <file>

Does anyone know how to accomplish this behavior without using JS?

8 comments

dstein14y ago

You can write a NodeJS script using the JSDom module to do this.

baconfaceOP14y ago

Thanks for the suggestion. Any idea how to accomplish this task without using JS?

I'm surprised chromium, konqueror and firefox don't offer this functionality for offline browsing.

dstein14y ago

Selenium (http://seleniumhq.org/) might work but it's not a simple command line script.

1 more reply

a3camero14y ago

iMacro? Rhino?

profitbaron14y ago

Personally, I'd use http://www.phantomjs.org/ and create a script which loads the page like a browser. Once you've done that you can get it to take screenshot or use JS to pull data off the page.

baconfaceOP14y ago

Thanks. I really want to get this done with the command line and it looks like Phantom JS will do the trick.

ddemchuk14y ago

perfect use case for celerity http://celerity.rubyforge.org/

j / k navigate · click thread line to collapse

Ask HN: How to wget pages with javascript?

4 pointsbaconface14y ago8 comments

I need to wget a url, allow the javascript to execute, and capture the result in a file. I've tried wget, curl, httrack,plugins like lget, and links(2) and none of them execute JS.

I want something like $: firefox <url> -s <file>

Does anyone know how to accomplish this behavior without using JS?

8 comments

dstein14y ago

You can write a NodeJS script using the JSDom module to do this.

baconfaceOP14y ago

Thanks for the suggestion. Any idea how to accomplish this task without using JS?

I'm surprised chromium, konqueror and firefox don't offer this functionality for offline browsing.

dstein14y ago

Selenium (http://seleniumhq.org/) might work but it's not a simple command line script.

1 more reply

a3camero14y ago

iMacro? Rhino?

profitbaron14y ago

Personally, I'd use http://www.phantomjs.org/ and create a script which loads the page like a browser. Once you've done that you can get it to take screenshot or use JS to pull data off the page.

baconfaceOP14y ago

Thanks. I really want to get this done with the command line and it looks like Phantom JS will do the trick.

ddemchuk14y ago

perfect use case for celerity http://celerity.rubyforge.org/

j / k navigate · click thread line to collapse