undefined | Better HN

0 pointsaustincheney8y ago0 comments

No, manually go there and copy/paste the code. Then when building your scraper bot use that code.

0 comments

3 comments · 1 top-level

AznHisoka8y ago· 2 in thread

but how do you use that code? its javascript, right? how would you use it if your crawler is written in Ruby or Python?

You could write a crawler in any language. Crawling is easy as you are listening for HTTP traffic and analyzing the HTML in the response.

To accurately get the content in dynamically executed pages you need to interact with the DOM. This is the reason Google updated its crawler to execute JavaScript.

AznHisoka8y ago

Yep, I know, but that means if I am writing the crawler in Ruby/Python, this is not something I can do, right?

1 more reply

j / k navigate · click thread line to collapse