💡 Get my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.

VSCode has this mode called “Python Interactive Window” where you have Jupyter-esque code blocks. The blocks are separated by a special comment (“# %%”) so the end result is still a script you can version control, unit test, debug, etc. It’s available through the Python extension.

Currently trying to refactor a jypiter notebook I got from our data science team - complete nightmare.

I've been using Jupyter for EDA and building pipelines. However, transitioning that pipeline into a standalone script has always been a bit of a journey for me. I would absolutely love to see a video on how to effectively make that transition from a Jupyter notebook to a full-fledged Python script, especially when it comes to keeping checks (maybe asserts to ensure data looks as expected?) for the exploratory nature of Jupyter while ensuring the robustness and maintainability of a script. Thanks for all the content you produce, and keep up the great work!

Im a computational chemist, so I use jupyter notebooks daily for explorative data analysis. Especially when analyising convergences, some paramaters have to be adjusted on the fly everytime and here notebooks are awesome to see the immediate effect of your choices without painfully loading in the huge amount of data again

For the Jupyter issue with regard to imports being present or missing due to editing "errors". I have a rule that I reload and run all features after finishing changes in a cell. This helps to ensure that I did not create a side effect, add, or remove something needed elsewhere. I can still run into issues, but the restart typically shows me the errors of my ways.

VScode actually supports doing annotated jupyter-style blocks in .py files. The advantage of this is that instead of a blob of json (jupyter is json with a lot of the output saved in there as literals) you're working in plain text and can therefore version control your file.

I've used Jupyter notebook a lot working in earth science modeling. I also manage the frontend (Java) and backend (Python) code for a website that process data based on a user's request. My thought process when using a notebook is so different than when I'm working on code that is part of the site backend. For instance, "testing" when working on the backend becomes more like "validation" when using notebooks for earth science modeling. With the backend code I may be using unit tests, while when I'm using notebooks I may be generating a plot or map to ensure that the data are being modified how I expect. To address the issues that can arise from running cells out if order, I am a stickler about using the "Restart and run all" command. Working in earth science modeling, Jupyter has been a big boon in regards to repeatably, reusability, and transparency. Thanks for another great video!

Great video! Coming from data science, I definitely see the value of exploratory data analysis with Jupyter notebooks. For your question, one annoying difficulty with Jupyter notebook files is version control. If you write a .py file and a coworker runs the file to see what it does, then there is no change to the .py file. Hence the version control software will not note the file as changed. But if the same scenario happens with a Jupyter notebook file, then the file changes! This is pretty annoying, especially if your coworkers are used to simply write git add .

I use notebooks for reports. Of course I run into the same issues you mentioned. This is why I try to define functions in a separate module. However, in many cases I use notebooks in an IPython shell which is convenient to explore code snippets.

I find jupyter useful for the reasons you have outlined. Plus they are a good way to prepare presentations where you want to show graphs and the like. Not to actually run the code, just to be able to see it with markdown providing reasonable headings, comments etc. Being able to access the presentation through a browser is also useful - you can demo through eg an iPad. The alternative of exporting the graphs/tables etc and then importing to PowerPoint is a pain. In the past I might of used XL to do something similar. I also find them useful for developing new code (where it’s not obvious what data manipulations are required up front) then once happy with the results, re-write the algorithm as a script. On the downside, version control is a pain in the arse. Merging always seems to go wrong with git. Also, they do seem to glitch in strange ways sometimes losing code or requiring a re-write.

I used to use Jupyter notebooks for data exploration and especially if there were intermediate results that took a long time to calculate. Eventually I gave up on them because it was too easy to save data off to csv, xslx, or even into a sqlite database (usually via diskcache) and then read them back in each time I re-ran.

I would have expected that you’ve touch on two other aspects: - version control challenges with Jupiter notebooks; and related - breaking the code -data separation paradigm in these notebooks that can be also a security/privacy risk.

Combining scripts with notebook is very useful for me in some situations!

I usually build pipelines on notebooks and transition that to regular python scripts. You just have to be aware of any changes to predefined variables. I do find notebooks to be much slower in magnitudes of 10s of minutes

Just a note - a square latlon will vary from an area perspective, with the largest at the equator and smallest at the poles. There are (lossy) projections to translate latlon to local distance in meters using the Azimuthal equidistant projection

I love Jupyter notebooks and use them almost every day, biggest issues I've found are version control and debugging. Although VScode has some limited debugging features for Jupyter it's definitely not as smooth as .py files.

These problems would also occur if the Python was being written as a script though. Don’t change the definition of functions and assignment of variables randomly throughout your code. Or if you must work that way, define each block with its own variables and definitions.

This was a nice video. Too often I see the more formal programmers, who don't have any experience with exploratory data analysis, dismiss notebooks upfront, without any nuance. Yeah, sure, I'll just run a script again and again, redoing the calculations and plots I again and again, super efficient. Notebooks are a great way of combining text, images, code and output, and have their downsides, of course, as everything in this planet. I've faced all the problems you mentioned in the video, and I'm now aware of the code smells. One golden rule I found was that, before "checking in" any notebook, or giving it to someone else, always restart the Kernel and run all cells. If it doesn't run to the end, except in some very specific cases, you have a problem that needs fixing. Ideally, restart and run everything every once in a while, like 1-2 hours. In the end, I consolidate some useful behavior into functions or classes and move them to a module that I can import in future notebooks, and which is properly unit tested and documented.

This happens a lot when dealing with researchers. Supposedly novel research with sota results in a notebook, but the state is broken because they've rerun cells in different orders with cell changes in between. So it becomes completely not reproducible.

I have run into problems with Jupyter notebooks. To avoid them I do gather function definitions and imports within .py modules. Also, it is always a good idea to restart the kernel and run up to the current cell to ensure the context is as intended within the sequence of the actions taken in cells. It is particularly risky to scroll up to a cell and re-run it.

One pitfall I hit when working with Jupyter notebooks and normal scripts together is that if you add a new function to your script it will not be imported into the notebook, even if you run the code block with the import command again. I believe the first time it runs the kernel caches the imports in some way to list what it's expecting, so a reimport doesn't load the new functions. I found restarting the kernel was necessary to allow me to import the new functions

Notebooks also crash in a very weird way at times. I don't really like them too because it is also hard to use them for debugging sometimes. Moving code into production, you have to redesign the program you have written. You can have the best of both worlds by using Python Interactive #%%.

Fantastic!! Thank you.

Just want to say that I'm loving using jupyter notebooks within vscode. You can easily connect them to a kernel without running a jupyter server and you can run copilot and other vscode plugins within jupyter, but still get the benefits of data exploration and running the code piece by piece. It appears you are using it already within vscode, but I bet many people are unaware and running it within an anaconda juptyer host.

its possible use lib "papermill" to create a python script to run a specific jupyter notebook with parameters seted in that python script, with a kernel of your choice. Very interesting lib to use notebooks as parametrized funcions with auto-saved state after each run.

I encountered the exact same problems you talk about in your video with Jupyter Notebook. Note I am not a data scientist. Same as you I can use it for deep data exploration that need visualisation. For simple data exploration I just import data in an sqlite DB which is more convenient to me when you already know SQL. I still sometimes use jupyter notebook to try some pieces of code since it's more practical and convenient than a simple python console. But still, I sometimes encounter a problem, sometimes I just try some pieces of code but I got hooked by the game and end up with a quite extensive script that is quite messy and that I need to take time to clean if I want to create a reusable script from it (outside of Jupyter Notebook). So I would say Jupyter Notebook can be absolutely great for data science, because of deep data exploration and because some code involving huge dataset or complex processing can take really long to execute in data science. Outside of this it can sometimes be an interesting tool to use to test some short code. But as I am not a data scientist I keep it more as a secondary tool, VS Code is far more convenient for 90% of my work.

Jupyter with right extensions it can show a image or play an audio(its very good for who people dealing with audio processing, you dont have to create audio file because of this) etc. Its very good for explonations and analysis

What browser do YOU use with jupyter notebook?

Missing the obvious usecase: Showcasing your code in a tutorial kind of style

It's only available for Julia, but Pluto notebooks solve a lot of the problems mentioned in this video - code blocks get reloaded when one of their dependencies change, making this sort of mistake considerably harder

I see the run button above the 'Square' Brackets... How is that done?

A load of the actual, real scientists (not just data scientists) I've talked to have been very keen on Jupyter.

We build data pipelines using Jupyter Notebooks and we do try to put much of the complex code into Python files that we import. One “gotcha” that still gets me sometimes is making changes to those Python files do not impact the “global state” even after running the imports again. The kernel needs to be restarted as well.

The problem you described for Jupyter can also be an issue for REPL-centered programming e.g. in Lisp/Emacs. :D

Does anyone know how to add the runtimer at the lower left corner when running a cell in the notebook? Thanks in advance!

Good video, I wanted to write about Jupyter notebooks myself, it's a very good tool

I found some specific behaviour when you using imports into Jupyter from your script files. If you change function in script file after you start working in the notebook your change wouldn't have effect in notebook. Repeating import doesn't help you, only restart Jupyter kernel.

Totally, I made a set of notebooks for someone to train their own language model and it was a nightmare managing the state and getting everything to return the same value all the time. Chasing ghosts in the machine.

Experimenting with data pipeline components in notebooks, then migrating to scripts or modules is super common on my teams. A couple major pain points: copying an instance method from a class in some module to a notebook, or back, is terrible due to the `self` or `cls` argument, or lack of it, breaking the expected functionality. And ensuring the ipykernel handles your custom module imports in the notebook the same way as Python does when running a script is often awkward. Modifying a custom module you imported in the notebook earlier means you must restart your kernel to ensure it uses the new version. And depending on where you store your notebooks relative to those modules, and where you run these from results in varying import behavior. I'm shocked at how often non-trivial data projects are built on popsickle sticks like this...and yet it (mostly) works.

This was extremely helpful, thank you!

My biggest issue with them is that they are not reuseable and the code in them becomes very cluttered and hard to work with ver, quickly

I use it in vscode when needed. Usually when I'm doing something with a 20min datalake query.

End-to-End PyMC project, please, yes. Thank you, bye! :D

Jupiter is good for reports or design but for pipelines scripts and vscode outline view are my preferred solution. But most important i noticed are your custom modules you generate over time, which increases the productivity. With Jupiter using your custom modules can get annoying because you have to reload the kernel each time after changing/importing your modules.

ipython (what jupyter notebook is built on top of) is also really nice. you can use it in place of the standard python repl. you get autocomplete and even some basic syntax highlighting

One of the biggest problems I have with jupyter is its interaction with git. Since it records not only the code but also the metadata and outputs, simply re-running a cell will lead to git detecting a change as the metadata has changed, and even worse, changing the code which generates an output will make git detect the change in the output, which can be hundreds of lines long. Worst case is if two people commit those type of “changes” in the same notebook in two different branches, it will lead to merge conflicts which are horrible to resolve, especially taking into account that standard conflicts resolution aid tools will not work (try finding the damn separator “=“s among hundreds of gibberish lines which represent the output). If you could make a video on ways to work around this, I’d be inmensely grateful

Refactoring and debugging is a nightmare

the function at 5:14 looks simple but why it took 4.5 minutes? how big was the UFO data?

Notebooks seem super odd to me. Not really software in the traditional sense. I cannot imagine a scenario where I'd ever use one. Seems only useful for munging data interactively. Which, obviously, (as evidenced by several of the comments below), is something many people do. That's just not something which would ever come up for me. But at least I now understand at a vague, conceptual level what they are and can happily ignore them.

What's with the "I love god slash design guide" at 0:32? Not sure if that's what you're actually saying (could it be "I love code"?) but the subtitles say "I love god..." 😂

You have to keep putting common functions in modules as you go. This leads to old notebooks being refactored or function signatures changing without being re-run. There are tools to re-run them to check compatibility but some things like training DL cannot be re-run easily. All global parameters should be defined once at the top. You have to get comfortable with the notebook violating single object responsibility. The notebooks are good for documentation but due to the fact they tend to get copied and pasted, you get out-of-date comments. The notebooks do not play nice with Git and review solutions but there are some solutions for this. Despite these drawbacks, they are good for running experiments and docummenting them.

Notebooks empower the mess! I try to avoid them as maximum as I can. I end up discovering that for me, using small scripts to do EDA is more productive, because you will always start by investigating some hypothesis on your data, and with a plus that you'll keep best practices for coding. To do data viz, I'm experiment to use streamlit apps, because are very easy to build and also will be useful if you'll need to show your analysis for someone else.

Great video....make more videos....we are new and confused on many points.

I have very mixed feelings about Jupyter notebooks. Working with it at least 5 years I definitely enjoy its way of being more interactive and easier to share than scripts. However, it’s an issue when you are transitioning from EDA to development and this happens not always in a well defined moment :) and eventually you have quite bad piece of code. So my rule is to run only high level functions with very well defined interfaces and put everything else into user defined functions and classes files.

Notebooks: never. Python files: always

People who run jupyter cells by clicking on the play button with the mouse are a big red flag xd

I suspect that the majority of comments will touch on the issues of data scientists only working in notebooks and not having any knowledge of software engineering principles. The unfortunate part of this is all of the junior level data scientists who are really a just jack of all trades and master of none. Many of the data science teams are poorly managed and offer no up skilling or mentorship, which will likely lead to more of these scenarios where competent engineers have to compensate for lack of ability within overgrown data science teams. If you are in a decision making role, please ensure that your data science team either trains people to code or produces cutting-edge work. The latter is extremely rare outside of world class research groups.

USS Cerritos

I'm going down the Arjan Rabbit Hole now. 🐇🕳 I watch one video, then you mention something I don't know, like partial functions, which leads me to watching another video about sometihng I don't know... 🤣🤣

6:25 is that a semicolon there, after calling sns.histplot?! Sacrilegious!

"... we can clearly see that the US is the most popular country" - lol, that's gotta be a giant red flag of some kinds 😆

:)

Simple. Never use Jupyter notebooks. Unless you really hate your customer. And yourself.