I love jupyter because its a great interactive programming environment that speeds up building scripts. But jupyter notebooks always end up being…notebooks. They are records of past work and thoughts that I had but I wouldn't present my paper notes to a coworker as finished documentation. I always end up extracting those jupyter snippets into runnable scripts and then I end up doing a lot of the work all over again as I parameterize things.

I have taken to using pycharm scientific mode and adding “cells” into my scripts: https://blog.jetbrains.com/pycharm/2018/04/pycharm-scientifi...

You get the interactivity of jupyter with the cells, the scientific view for plots and data, but you’re also writing a real script so you can still maintain some sanity in your code and good git history (and hopefully add some tests).

Then I combine these scripts together into a makefile and voila quick, easy and maintainable scripts. If you just stick to make, python, pandas and bash any programmer can modify your pipeline without needing specialist skills. I have written scrapers + ETL in <1.5 hour (journalist needed some data quick) with this method.

God I wish I still worked in data science I’m currently a front end dev and don’t get to do any of this at work. It’s such fun and offers so much room for creativity and problem solving.