undefined | Better HN

0 pointsmanigandham6y ago0 comments

GCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam).

There's a new feature called "Dataflow SQL" that lets you submit a declarative SQL query through the BQ UI as a job to Dataflow which can join both BigQuery tables and PubSub topics in the same job.

https://cloud.google.com/dataflow/docs/guides/sql/dataflow-s...

0 comments

chrisjc6y ago

Thank you for the correction. Sounds like a very slick solution bringing batch and streaming (the B and EAM obviously) together.

j / k navigate · click thread line to collapse

0 pointsmanigandham6y ago0 comments

GCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam).

There's a new feature called "Dataflow SQL" that lets you submit a declarative SQL query through the BQ UI as a job to Dataflow which can join both BigQuery tables and PubSub topics in the same job.

https://cloud.google.com/dataflow/docs/guides/sql/dataflow-s...

0 comments

chrisjc6y ago

Thank you for the correction. Sounds like a very slick solution bringing batch and streaming (the B and EAM obviously) together.

j / k navigate · click thread line to collapse