Skip to content

Top Best Ask Show New Jobs

Show HN: News Searching API (opens in new tab)

(datanews.io)

40 pointscaballeto5y ago42 comments

42 comments

32 comments · 10 top-level

caballetoOP5y ago· 7 in thread

This is the side-project, that I have been building recently. The API allows searching over the data from many news sites and news aggregators, it also allows to easily extract whole news sites. I built it for two reasons. First, I wanted to have a source of recent news data to play with, essentially to run data analysis and train ml models on. Secondly, because alternative services had wildly high pricing, like 400-500$ per month, and I thought I could build and deploy such a system for a much cheaper price and improve my programming skills along the way. I would very much appreciate any comments/suggestions about the serive or how to make it better :)

bvm5y ago

How are you licensing the content?

He's not.

todsacerdoti5y ago

I would love to integrate your product into Pipedream.com so developers can quickly integrate Datanews with other apps such as Slack, Discord, Reddit, Google Sheets, etc. Please reach out -- tod at pipedream.

caballetoOP5y ago

Great idea, I like it.

sbr4645y ago

Nice work. Will share feedback soon after testing.

caballetoOP5y ago

Great, thanks, would appreciate it very much. I am in process of thinking about new features, so if you have some special use cases, I could think about that.

The prices are high because you’re paying for licensing costs.

moneywoes5y ago· 3 in thread

Can you talk about how you avoid getting blocked by captcha? Do you use RSS feeds? In that case do you not return images?

caballetoOP5y ago

The scraper only scraps the recent content, so it avoids scraping same links. As the daily number of articles published by the website is not very big, I make requests with a break in between, total number of requests is not that big, so the ip is not blocked. Also, adding proxy servers is a possible solution, i.e. having a proxy pool to proxy requests. I guess in this scenario it is easy to by pass capcha.

TechBro86155y ago

Nitpick: it’s “scrapes,” not “scraps.” To scrap something is to throw it away. To scrape something is to ingest it. The two words are, in fact, nearly opposite.

Sorry — I don’t mean to pick on you — it’s just that I see this mistake with an inexplicably high frequency, and it’s like nails on a chalkboard for me.

Also, very cool service!

caballetoOP5y ago

No, I don't use RSS feeds. I extract news links from the home page and sitemaps mainly. Most of the news articles have a news schema embedded within them. You can check info about it at schema.org. This is a piece of JSON which contains the description of the article, usually title, publish date, description & image links.

asdkhadsj5y ago· 2 in thread

Do you host the content as well? What's the legality of that?

I ask because I want to embed some archiving and "reader mode" logic into an app of mine that would be FOSS and self hosted. However that means each individual would be effectively scraping and archiving, and possibly p2p spreading, news content(as data sources).

So I'm curious if there is some underlying "fair use"-like mechanism that allows Archive, Outline.com, and you to consume news content without it being considered piracy.

caballetoOP5y ago

This is a great question. Thus far I thought, that the content that is publicly available without any limitation (e.g. membership access), can be scraped by anyone. You can take a look at hiQ Labs vs. LinkedIn [1]. LinkedIn's public data was scraped by data analytics company, the ruling was against LinkedIn.

I also think that the use case matters, I don't republish their content on the site, but merely provide it via API. Technically, it could be argued, that they could get this data themselves, but it is easier for them to use a service similar to this one to simplify things.

[1] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

artembugara5y ago

hiQ Labs vs. LinkedIn is a bad example. It is super specific.

LinkedIn did not owe the data (it was users').

In your case, you are reselling copyrighted product.

Of course you can scrape. It does not mean you can distribute this.

artembugara5y ago· 2 in thread

Disclaimer: I am a co-founder of a similar News API service https://newscatcherapi.com/

No news API solution returns the full body text of the article (including us). The reason is - copyright infringement in US and EU.

You can return only chunk of it

At least, that's what all lawyers I spoke to told me

caballetoOP5y ago

That's interesting. I've browsed a few of alternative apis, and if I get it correctly then newsapi.org returns full text articles for paid plans. I would research this question further, and maybe will change the api to return only part of the content.

bvm5y ago

Some do, you just have to pay for it.

xiekomb5y ago· 2 in thread

Hi,

Simple and effective API. The documentation lacks details about the query parameter. I want to search for A AND B but q=?A%20B does not yield the expected result (seems a OR query) or for exact phrase "A invests in B" do not seem to work. Can you please post details for advanced syntax like this?

caballetoOP5y ago

Hi, thanks for the feedback! So far, I haven't implemented this feature yet. The query parameter gets tokenized as is, and then tokens are used to search in the index. Could you provide more details about your use case? As I understand you need to have AND, OR, NOT, grouping () operators, anything else? I would try to implement this today, and write back to you.

yes this is exactly what I (most users?) would need. Search for exact phrase would be a plus also. Eg: "donald trump" instead of Donald AND trump. What kind of backend are you using, a simple database or search index like elasticsearch (which would be more appropriate for such project)?

ibdf5y ago· 2 in thread

Tried it.. pretty straight forward. Free account rate is good for personal use. I think it comes down to 4 calls per day which is reasonable if you are using this as maybe your daily news fetcher.

caballetoOP5y ago

Thanks for the feedback! Concerning rate limiting, I think I may have placed rules a bit too strict, as the system is hosted on AWS Free Tier. There is definitely a room for increase in RPS per user, but I wanted to play safe.

fluffernutter5y ago

This is great!

jslakro5y ago· 2 in thread

How do you get news from its sources?

caballetoOP5y ago

First, I extract the links to the news articles by searching them on the home page or sitemaps. Then, for each article I scrap raw html and extract key things like title, description, publish date, authors, etc. Usually each news article has a schema embedded within it, so it greatly simplifies things of extracting article details.

Do you scrape using web requests? Or do you use headless browsing to load the page and extract the contents out of the DOM?

graupel5y ago· 1 in thread

I used to run a large US publisher with 50+ sites writing 50,000 articles a month; if we found someone scraping full articles and offering them for use via an API it would be a problem - even moreso if that person was charging for it. I'm all for hacking and innovation around this space, just be careful here and don't break TOS.

caballetoOP5y ago

Hi, thanks for feedback. I am going to push changes to display only a snippets from the articles soon.

sagunsh5y ago· 1 in thread

What tech stack are you using? And how many news source does it collect data from?

caballetoOP5y ago

The tech stack is Python, Java, Spring, Elasticsearch, Redis/MySQL. This is a beta version with explicit support for 50 sources + Google News aggregator. I plan to release new version in about a week, which will support another 400-500 sources.

todsacerdoti5y ago

I added your API to Pipedream. Check out an example -- https://pipedream.com/@tod/datanews-io-example-p_ljC13p/edit

If you or your users need to build automated workflows with Datanews, please let us know any feedback.

j / k navigate · click thread line to collapse