ifeedmartians

I'm planning to write a tutorial mini-series covering production-grade data engineering. This will include building and maintain an end-to-end data ecosystem. I would love for this to solve an actual problem and the tutorial progressively adds more complexity in solving that problem. Since I work in infrastructure, it would be naive for me to assume that I know of all use cases that would be helpful for the community so I'm hoping this community can help me understand a real-world data problem that a good portion of the community would like me to solve in these tutorials.

Ideally this would be something that covers data ingestion, processing, modeling and using that processed data for decision making.

An example would be Calculating the Intrinsic Value/Price for Stocks. I would cover ingesting all fundamental and price data, modeling and storing it in a usable format, processing it to calculate the intrinsic value and then using it to help with your current stock portfolio allocation. This could eventually also get into streaming the input data and making automated decisions on it.

Would love to hear your thoughts on real-world data problem which I can solve through this tutorial mini-series.

Many thanks!

Authentication has always been a daunting part of my SaaS because getting it right is necessary and there's very limited guidance around it.

Currently my SaaS has a web based authentication (which i'm very happy with). Sign Up: - gmail/github flow, create an account and store user details but no password - username/pass flow, hash the password using bcrypt and store it in my DB

Sign In: - Sign In using gmail/github/user-pass flow - Generate a session cookie for the user, store it on the client, which is sent with every request - When session cookie is invalid, user needs to sign up again

A very important component of my SaaS is a CLI.

> How do I integrate authentication using the CLI application?

Auth0/Okta are excluded as options because I used Auth0 in the past when building the web flow and I found it crazy hard to integrate, along with the crappy sign in page which comes up on the free tier, doing it myself was so much easier.

Thank You!