tchiotludo

We have launched Kestra (https://github.com/kestra-io/kestra), an open source data scheduling and orchestration platform that features high scalability and availability. To achieve this, we rely on Kafka as our core database.

We have received feedback that Kafka can be difficult to manage and is a no go on the project. The project did not pass the architecture team for this reason. While you can use Kafka without creating a full cluster (using a single node setup), it seems that Kafka is synonymous with complexity and problems.

We're asking for your thoughts on this. We are thinking of making it easier to onboard the project. So my question is this:

- Is Kafka really a scary technology, and do you feel like it can be a pain to manage and no go technologies?

- We also depend on ElasticSearch, do you have the same point of view?

- We have plans to add another option that will replace both dependencies with jdbc (https://github.com/kestra-io/kestra/pull/368), is theses dependencies more comfortable for you?

- If yes, what type of database is required? (mysql, postgres, others)

- We also see that Redis could have all the query interface needed for Kestra, what do you think about this one?

We've done some research and haven't found a cloud database at the moment that could meet all the needs (especially full text search and transaction support), if you have an idea it's welcome.

Thanks for your time, there's a lot of work to build this part, so your feedback is really important to avoid going in the wrong direction.