undefined | Better HN

0 pointshappymellon2y ago0 comments

Completely agree.

I would also add that in my 25 years in the industry I have never actually come across an ETL workflow that was complex enough that it required people working it in code. Those opinions seem to occur before proper analysis happens.

0 comments

3 comments · 1 top-level

stvltvs2y ago· 2 in thread

So you have never needed to write SQL in your ETLs? I guess it depends on whether you consider that code, but at least 50% of the time I have to use SQL for one reason or another rather than just pointing the tool at a source and target.

what-the-grump2y ago

Honestly? If you are making the transformation in SQL you've lost the T to the database server. Does it mean we don't do it? Is water wet?

The reality is that we shouldn't be making the transformation in written data, now you have staging tables, procs, views, resulting tables procs, views, a staging database (often called staging or etl), and an unhappy DBA who yells at you every time you cross-apply incorrectly.

ETL should be done before the data lands in SQL.

stvltvs2y ago

I think you're describing ELT, an approach that I agree with and try to implement as often as possible.

I think that's orthogonal to the occasional need to use SQL to extract source data. And it's hard to beat the performance of RDBMS for your transformations. It's almost their whole raison d'être.

j / k navigate · click thread line to collapse