New HN data dump available with over 14.5m entries (opens in new tab)

(archive.org)

79 pointscdman8y ago13 comments

13 comments

If you're interested in playing with Hacker News data and don't want to download the entire dataset (or don't have the CPU/memory to perform large JOINs on stories/comments), you can use the Google BigQuery HN dataset, which is now up-to-date: https://cloud.google.com/bigquery/public-data/hacker-news (specifically, the .full table, which combines both stories and comments; the dedicated tables are not up-to-date)

venning8y ago

I see this link mentioned all the time, but every time I try it I can't get it to work.

Specifically, the "GO TO THE HACKER NEWS DATASET" big blue button on that page. It kicks me over to a Google Cloud console link, which spins for a few seconds, and then brings up a "Welcome to BigQuery!" modal. The only thing I can do then is click "Create a Project", which then kicks me over to the generic console with a listing of all APIs.

Am I missing something?

minimaxir8y ago

You'll need to create a GCE project before you can use BigQuery (you don't need to provide a credit card if you remain in the free tier)

ers358y ago

See also: A dump of the stories, comments, and users from the Firebase API as a SQLite database with a full text search index: https://archive.org/details/hackernews-2017-05-18.db

B1FF_PSUVM8y ago

Can you tell about the time period and (estimate of) % of comments covered, for your DB and the dump posted?

Thanks.

ers358y ago

My DB covers from https://hacker-news.firebaseio.com/v0/item/1.json?print=pret... (10/09/2006 at 6:21pm UTC) to https://hacker-news.firebaseio.com/v0/item/14372035.json?pri... (05/18/2017 at 11:58pm UTC)

The dump posted claims to cover from 1 to 14566367 (06/16/2017 at 3:03am UTC)

B1FF_PSUVM8y ago

Ah, OK, forgot there's the item id for comments, ensuring 100% comment coverage.

(I read "the story vote count is inaccurate for certain stories because it is only scraped once and not updated" and thought some comments might be left out too.)

So, 145M at min. 10 sec. per comment, that's at least 40k hours worth, probably one order of magnitude more. Just writing time, reading maybe 3 orders of magnitude more.

Modern pyramids, they're impalpable ...

natch8y ago

Kudos to archive.org for hosting torrents. It would be helpful to know the size of the download up front. Nice clean web page design; would love to see that one bit of information added.

ers358y ago

You can see the size of the download by clicking show all: https://archive.org/download/14566367HackerNewsCommentsAndSt...

natch8y ago

Thanks. My point to the web designer stands though, the information should be on the first page. Before seeing your message, I looked and didn't find it; had to determine it by loading up the torrent in a client. Also as sillysaurus3 pointed out the expanded size is useful too.

fit2rule8y ago

The answer: 1.6 gigs.

sillysaurus38y ago

1.6gigs is no doubt the compressed answer. Anyone know the uncompressed size?

1 more reply

j / k navigate · click thread line to collapse

13 comments

minimaxir8y ago

venning8y ago

I see this link mentioned all the time, but every time I try it I can't get it to work.

Am I missing something?

minimaxir8y ago

You'll need to create a GCE project before you can use BigQuery (you don't need to provide a credit card if you remain in the free tier)

ers358y ago

See also: A dump of the stories, comments, and users from the Firebase API as a SQLite database with a full text search index: https://archive.org/details/hackernews-2017-05-18.db

B1FF_PSUVM8y ago

Can you tell about the time period and (estimate of) % of comments covered, for your DB and the dump posted?

Thanks.

ers358y ago

The dump posted claims to cover from 1 to 14566367 (06/16/2017 at 3:03am UTC)

B1FF_PSUVM8y ago

Ah, OK, forgot there's the item id for comments, ensuring 100% comment coverage.

(I read "the story vote count is inaccurate for certain stories because it is only scraped once and not updated" and thought some comments might be left out too.)

So, 145M at min. 10 sec. per comment, that's at least 40k hours worth, probably one order of magnitude more. Just writing time, reading maybe 3 orders of magnitude more.

Modern pyramids, they're impalpable ...

natch8y ago

Kudos to archive.org for hosting torrents. It would be helpful to know the size of the download up front. Nice clean web page design; would love to see that one bit of information added.

ers358y ago

You can see the size of the download by clicking show all: https://archive.org/download/14566367HackerNewsCommentsAndSt...

natch8y ago

fit2rule8y ago

The answer: 1.6 gigs.

sillysaurus38y ago

1.6gigs is no doubt the compressed answer. Anyone know the uncompressed size?

1 more reply

j / k navigate · click thread line to collapse