New Data Engineer, good learning resources?

1 pointswillnewby9y ago2 comments

Hello!

I just got a new job as a "platform" engineer, which looks like it's going to be a whole bunch of data engineering (i.e. getting data from point A --> redshift/etc). I come from primarily a webops/devops/sysadmin background, which means I've been feeling a little out of my depth on "what's all this data stuff and how does it fit together?". I'm curious for any good resources you may have.

Thanks!

2 comments

2 comments · 1 top-level

mattbillenstein9y ago· 1 in thread

That is a hard question to answer, I think you have to start looking at the big component (the data warehouse) and see what fits best for you.

At least what has worked best for me is using BigQuery and building custom tooling around that. Using Airflow, some simple python tools, .json.gz archives, and a mix of streaming data and hourly/daily exports has worked well and not been a huge task to keep operational. Of course, every situation is different; ymmv.

willnewbyOP9y ago

Gotcha. Good to know. Thanks!

j / k navigate · click thread line to collapse