undefined | Better HN

0 points0x0085y ago0 comments

For example, we do a lot of of low-level RDD Operations through databricks. From skimming the Website I feel something like this is not in the scope of this project.

In the end, I feel, it is about wording. Databricks is a serverless spark environment with Azure integration and notebooks. Unless the product copies all the aspects (i.e. the hosting) it may not be wise to call it a databricks alternative.

If I reed the title as it is here on HN, I would think is about the infrastructure and not about a custom low-code JSON-based template language on top of spark sql.

0 comments

1 comments · 1 top-level

seddonm15y ago

Fair enough. This really replaces the Notebook experience of Databricks and gives you the option to self host and as it is dockerised it is easy to deploy on Kubernetes (with JupyterHub) or other container hosts. If you want to do low-level RDD operations you can easily build a plugin.

I think previously Spark was difficult to run, now with Docker and supported Kubernetes it is fairly easy.

Edit: I forgot to link to the Deploy repo which does have some scripts to easily deploy to cloud environments (but needs some work): https://github.com/tripl-ai/deploy

j / k navigate · click thread line to collapse