Tell HN: Hacker News BigQuery dataset is updated daily again

51 pointswietsevenema1y ago11 comments

As you might know, there's a Hacker News BigQuery dataset [1].

I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily.

SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC

[1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

11 comments

zX41ZdbW1y ago

Daily is too infrequently.

I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...

I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...

tarasglek1y ago

I love that this dataset is accessible via cors from web frontends. Got me into experiments with clickhouse

Same with github, much better than than throttled apis

dang1y ago

Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.

izyda1y ago

Any interest in getting this dataset on Snowflake too?

(I run a Snowflake funded data provider on Snowflake marketplace)

dang1y ago

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API), so anyone is free to do this.

2 more replies

throwaway2111y ago

How big is it?

wietsevenemaOP1y ago

Closing in on 41 million items

j / k navigate · click thread line to collapse

Tell HN: Hacker News BigQuery dataset is updated daily again

51 pointswietsevenema1y ago11 comments

As you might know, there's a Hacker News BigQuery dataset [1].

I recently noticed that the export was broken (since Nov 2022) and I worked with the team to help fix it. It'll now again run daily.

SELECT timestamp FROM `bigquery-public-data.hacker_news.full` ORDER BY id DESC LIMIT 1 # 2024-06-09 09:12:55 UTC

[1]: https://console.cloud.google.com/marketplace/product/y-combinator/hacker-news

11 comments

zX41ZdbW1y ago

Daily is too infrequently.

I have this dataset being updated in ClickHouse in real time: https://play.clickhouse.com/play?user=play#U0VMRUNUIG1heCh0a...

I also provide a way to export it or attach it to clickhouse-local and analyze it locally: https://github.com/ClickHouse/ClickHouse/issues/29693#issuec...

tarasglek1y ago

I love that this dataset is accessible via cors from web frontends. Got me into experiments with clickhouse

Same with github, much better than than throttled apis

dang1y ago

Thanks! I tried emailing my contacts on that team several times and never heard back, so I figured we no longer had contacts on the that team.

izyda1y ago

Any interest in getting this dataset on Snowflake too?

(I run a Snowflake funded data provider on Snowflake marketplace)

dang1y ago

The BigQuery dataset is entirely 3rd party, based on the public HN data (presumably via the HN Firebase API - https://github.com/HackerNews/API), so anyone is free to do this.

2 more replies

throwaway2111y ago

How big is it?

wietsevenemaOP1y ago

Closing in on 41 million items

j / k navigate · click thread line to collapse