Torus: A Toolkit for Docker-First Data Science (opens in new tab)

(medium.com)

74 pointsaugustflanagan7y ago13 comments

13 comments

This is interesting! It sounds like this v1 gets your local environment up and running in a Docker container. I maintain something similar for analysts on my team, and we've seen success in terms of decreasing time spent on environment setup.

As another interesting use of Docker in the data space, I'm excited about Pachyderm [0] (though I haven't had the chance to use it in production). In particular, the data provenance story seems compelling.

0: https://github.com/pachyderm/pachyderm

jdoliner7y ago

Thanks for the plug saamm, I'm one of the creators of Pachyderm. I think Torus and Pachyderm would work very nicely together. You could go straight from developing code in the image Torus provides to deploying it on Pachyderm as a production pipeline that runs on new data as it comes in with just a few commands. Similarly, their Dockerized data science cookie-cutter could work nicely as a Pachyderm service, this would work similar to using the service on your laptop, except that you could easily deploy it on a cloud provider and schedule it with GPUs and it will get updated with new data as it comes in.

Very exciting to see more people applying containers to data science.

sdeymanifold7y ago

Yes to containers! We are trying to make it as seamless as possible to be Docker first in all things. And not reinvent the devops wheel. It just needs to be adapted for the needs ot data scientists. Pachyderm is really cool. I will have to check it out. We've recently moved to Airflow for all our pipeline management... how does Pachyderm fit in that ecosystem?

1 more reply

zimbatm7y ago

No to confuse with this other company at https://manifold.io/ (io, not ai) which deprecated their https://www.torus.sh/ project :)

voxadam7y ago

Or for that matter, the short lived distributed storage system from CoreOS.

https://coreos.com/blog/torus-distributed-storage-by-coreos....

plara7y ago

https://manifold.co ;-)

sdeymanifold7y ago

Yes, we found that super confusing thing later. I guess if you name your company Manifold you will name your projects after specific types of manifolds. We have Torus, next is Mobius, then ... Klein Bottle?

robohamburger7y ago

I think a more interesting direction would be for jupyter lab to ship an electron app and have it able to understand how to spin up and talk to containerized kernels.

I made a hacky version for work that proxies to a k8s pod but first class support would be cool.

po847y ago

https://github.com/jupyter-incubator/enterprise_gateway to launch kernels on a cluster and https://github.com/jupyter-incubator/nb2kg to make a notebook server aware of them might be of interest (sans electron app shell).

robohamburger7y ago

Thanks for sharing this! I didn't realize this project existed.

I have lots of questions now, like why this isn't using the zeromq based protocol, so I guess I will need to spend some time with it.

It does look like it closely overlaps with what I was describing. I didn't realize overriding/extending the http api was even a thing that could be done so I just used zeromq for my own purposes :)

mr_toad7y ago

Is cookie cutter running inside the docker container or on the host? The instructions imply setting up python, virtualenv, pip and cookie cutter all on the local machine...

1 more reply

stmw7y ago

Isn't this was Pachyderm was supposed to do?

j / k navigate · click thread line to collapse

13 comments

saamm7y ago

0: https://github.com/pachyderm/pachyderm

jdoliner7y ago

Very exciting to see more people applying containers to data science.

sdeymanifold7y ago

1 more reply

zimbatm7y ago

No to confuse with this other company at https://manifold.io/ (io, not ai) which deprecated their https://www.torus.sh/ project :)

voxadam7y ago

Or for that matter, the short lived distributed storage system from CoreOS.

https://coreos.com/blog/torus-distributed-storage-by-coreos....

plara7y ago

https://manifold.co ;-)

sdeymanifold7y ago

robohamburger7y ago

I think a more interesting direction would be for jupyter lab to ship an electron app and have it able to understand how to spin up and talk to containerized kernels.

I made a hacky version for work that proxies to a k8s pod but first class support would be cool.

po847y ago

robohamburger7y ago

Thanks for sharing this! I didn't realize this project existed.

I have lots of questions now, like why this isn't using the zeromq based protocol, so I guess I will need to spend some time with it.

It does look like it closely overlaps with what I was describing. I didn't realize overriding/extending the http api was even a thing that could be done so I just used zeromq for my own purposes :)

mr_toad7y ago

Is cookie cutter running inside the docker container or on the host? The instructions imply setting up python, virtualenv, pip and cookie cutter all on the local machine...

1 more reply

stmw7y ago

Isn't this was Pachyderm was supposed to do?

j / k navigate · click thread line to collapse