Show HN: Web Scraping Language (WSL) (opens in new tab)

(scrape.it)

11 pointsscrape_it7y ago10 comments

10 comments

9 comments · 4 top-level

shubhamjain7y ago· 2 in thread

This is something that I wanted—a rich language for scraping that can help me avoid writing a lot of boilerplate. However, I am not too sure about the P2P/decentralized nature. For my projects, that would be a little too complicated. Would I be able to use just the WSL part?

scrape_itOP7y ago

yeah I've actually had a sudden change of heart - implementing a p2p/decentralized theme is quite a bit more work than I anticipated, mainly syncing problems. originally planned on releasing 3 weeks ago....this p2p is taking up almost all of my time and I'm not sure if ppl even give a shit about it....so much for my ideological mania

I'm going to put that p2p stuff on hold and focus on getting a REPL terminal app that you can start running WSL on this week.

russdpale7y ago

decentralized is very important, don't give up on it! Its fine if people don't care about it. People don't care about relational algebra but SQL is used everywhere in their life and they have no idea or care.

The idea of distributed cris-crossed data sets all over the internet, I think, is a powerful one in a day where data is wielded as power and is so asymmetrical in nature.

1 more reply

slig7y ago· 1 in thread

Hi, I see that you used our project's [1] layout as a template for your project, but forgot to change the image and the link on it. Just pointing it out. Good luck on your project.

[1]: https://github.com/javascript-obfuscator/javascript-obfuscat...

scrape_itOP7y ago

oh yeah forgot about that....its a great template, everything all one one html...many thanks. I will swap out the drawing.

Fudgel7y ago· 1 in thread

What does it use to scrape? Is it just an xml/html parser, or does it run a headless browser?

scrape_itOP7y ago

It uses a sort of a DSL that I came up with, and removes a lot of boilerplate.

Headless yes.

You write WSL (Web Scraping Language) to scrape 2 pages of HN:

    GOTO news.ycombinator.com | EXTRACT {.morelink, 2} {news: .storylink}

(you may certainly remove that ' , 2 ' business based on the moral system you subscribe to)

catchmeifyoucan7y ago· 1 in thread

Download links are broken

scrape_itOP7y ago

I'm working on it as I speak.

j / k navigate · click thread line to collapse

10 comments

9 comments · 4 top-level

shubhamjain7y ago· 2 in thread

scrape_itOP7y ago

I'm going to put that p2p stuff on hold and focus on getting a REPL terminal app that you can start running WSL on this week.

russdpale7y ago

The idea of distributed cris-crossed data sets all over the internet, I think, is a powerful one in a day where data is wielded as power and is so asymmetrical in nature.

1 more reply

slig7y ago· 1 in thread

Hi, I see that you used our project's [1] layout as a template for your project, but forgot to change the image and the link on it. Just pointing it out. Good luck on your project.

[1]: https://github.com/javascript-obfuscator/javascript-obfuscat...

scrape_itOP7y ago

oh yeah forgot about that....its a great template, everything all one one html...many thanks. I will swap out the drawing.

Fudgel7y ago· 1 in thread

What does it use to scrape? Is it just an xml/html parser, or does it run a headless browser?

scrape_itOP7y ago

It uses a sort of a DSL that I came up with, and removes a lot of boilerplate.

Headless yes.

You write WSL (Web Scraping Language) to scrape 2 pages of HN:

    GOTO news.ycombinator.com | EXTRACT {.morelink, 2} {news: .storylink}

(you may certainly remove that ' , 2 ' business based on the moral system you subscribe to)

catchmeifyoucan7y ago· 1 in thread

Download links are broken

scrape_itOP7y ago

I'm working on it as I speak.

j / k navigate · click thread line to collapse