Agreed. I've been "the backend engineer who works with the data engineers" for several years now and I've seen their general trend of re-inventing the wheel the hard way a number of times.

I've spent the majority of my career building better tools for data-related tasks, then winning over my users by showing off performance and productivity gains.

I stepped into a Data Engineering Lead role in 2019. Stepped out of it in 2021. My team was the first in the org to really approach data engineering and we were all software engineers. I'm told that the systems we built have largely been replaced by Rube Goldberg machines pieced together by the folks who came after us.

Those replacement systems aren't even working, they're failing to deliver on the same simple data pipelines that we had working by the start of 2020. They're cobbled together using a million little AWS pieces and Docker and k8s... I'm glad that I left that role when I did, we were being pushed by a new-hire with a fancy Data Engineering VP title to do all sorts of asinine things. I went and looked just now and I see that he's Senior VP at a different company, he started there this summer. Onward and upward!