👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis

One thing that I learned long before starting my Python learning. Bugs seem to be inevitable and when one happens I ask myself, "How did this get past my unit tests?" So I go back and modify the test suite to catch the bug. Not quite 'test driven development', but really helpful with any sort of iterative-development or refactoring of a project.

If you use notebooks, I _highly_ recommend enabling Autoreload. I find myself using notebooks / VS Code interactive sessions frequently. One of my biggest frustrations with notebooks was that if I changed a function, I would have to rerun that cell every time to update the function definition. It was also less conducive to separating my code out into submodules (which are quite convenient). It was a total game changer for me to add "%load_ext autoreload %autoreload 2" to my Jupyter startup commands. In a way, this workflow promotes the use putting functions in submodules because any time you call a function, it will reload that file with any changes you have made.

I rewrote cookiecutter and turned it into a programmable configuration language. It's called `tackle` and I am almost done with the last rewrite before I start promoting it. Does everything cookiecutter does plus a ton of other features like making modular code generators (ex. a license module you can integrate to your code generator), flow control within the prompts (ex. if you select one option then expose a subset of other options), and schema validation (not really related to cookiecutter but is the main use case of the tool). It's more relatable to jsonnette, cue, and dhall but you can easily make code generators like cookiecutter as well. Almost done with it (a 4 year long personal project that I've rewrote 5 times) and hopefully it gets traction.

Biggest tip is to combine python scripts with Notebooks. Notebooks allow for fast and visual exploration of the data / problem. Then move pieces of code to a ./lib folder and use it from them. And start adding test. Most of the time you are performing the same operations of the data so you can end up with a shared lib across multiple projects. And that is very handy when starting a new project.

Love these videos! I found your channel from your Hydra config tutorial and all of your videos have been full of invaluable knowledge that I've already been using in my projects at work! Thank you and keep it up!

Official request for a full Taipy video 🙌

The fastest data storage I have found with python is arrow, but I unusually use csv or json although I have used quite a few databases. Also, I have been slowly learning tip number 5 over the past 10 years or so. Once I force myself to make code that others can use, I find myself being much more proficient and can see why the top coding people generally seem to make tools that others can use in one way or another. Thanks!

I have had a semi-emoitional experience listening to this :D I have been asbolutely solitary on a project this last 8 months. Deciding at the start, that instead of dealing with it with my old knowledge, with Excel and SQL I would tackle it while learning python along the way, with GPT. To hear these tips, to realize I just through trial and error or logic, got to these tips! I have not done so bad, is the feeling but also lots to learn because this is just the base. And I for sure got a foot in the door even if just slightly. Tip 1 - Common structure - totally tried to apply a common structure, but then some external stresses and pressures, I crumbled and it was visible in my code thereafter :/ Next time - this is a must. Tip 2 - Existing libraries - learning with GPT, can be limiting - specifically asking GPT for alternative libraries for the same solution, asking pros and cons etc, helps you immediately address as many issues with fewer packages or the perfect package directly. GPT didn't point that out as much as it could, so its on you. But learning in more depth, now, after idenitfying which packages I use all over, would aid me to understand under the hood. i.e. Pipeline tool - when I started, I wanted something like this. Head dev guy in the office said nothing like it exists. So made a non gui module of functions for handling data load and export for csv, xlsx, sql - to see Taipy... uff, can't wait. Tip 3 - Log Results - as in excel days, copy after copy, different folders, filenames, - to track every transformation to exactly as you said, back track for verification - plus makes it easier to answer external questions as to how and why. Tip 4 - that was just forced as along the way, given different data sources, data formats - I just refused to hard code sources, so dedicating just to data load phase and how outputs anywhere in the whole project will be presented, stored, displayed - whether in memory or sql or csv an logging where and when everything is stored for easier recalling downstream... was like a matrix Tip 5 - exactly, jupyter notebook once you have done with code - pop them into spyder, and call them whether as functions import or as I learned last week, i can run .py files and have results loaded into jupyter notebook - but I don't know what else there can be done - its sandbox place, but py files is the aim. Tip 6 - exactly again, it just had to happen - anything which you start seeing something repeat, think about the context, the occurrence, variations - pop them in a centralized place for easier management - like in excel, lookup tables :) Just trial and error - but to think of it in advance, is game changer for such bigger projects... Tip 7 - No idea - biggest wow moment for me this one so far - because exactly now I am asked to hand over the code I wrote and I am certain I put a little of tests break points in the code with comments for what change I did as to why to have the break point condition where it can be reverted to a different approach - didn't want to delete any of my test points but had a feeling it would look terrible handing that over... and I am sure the way you talk about it, is much neater than what I have formulated so if there is 'official approach' I would gladly revamp my approach because i literally just used user changeable boolean variables such as sample_mode =True to trigger df.head(1000) throughout the script for example... Next week I find out which team they move me to, as I just came in to clean data and present results and instead I now have code to take any addresses, validate them and proceed to secondary data point evaluations to consolidate data across foreign sources, to spot what is missing and shouldn't be but also validation on more granular data in this case ownership and tax rate. Its the most intense bit of work I have done alone - every single step needed something created and addressed - no normalization anywhere consistent in a single field of data point, crucial or otherwise. The reality even in the end, whatever you do, its only as good as the data you get, but the stories and scenarios of data which can be told after, if more people were interested in that and understanding of the influence of data diligence, its like a civilization booster in awareness :D

# 6 is such a great tip. Here's how I do it: - YAML file with the source code for defaults - (maybe user /machine dependant YAML file for some test system) - a YAML file with the data to override any values in evaluation YAML is great because it's human readable and has comments. So you can tell the user "put a config.yaml with the keys X, Y and Z to the data and off you go".

I’m a big fan of validating data I’m bringing in via Pandera. It is like defining a contract and if the data coming in breaks that contract (data types, column names, nullability, validation checks, etc.) I want to know about it BEFORE I start processing it. I also use Pandera typing heavily to define my function arguments and return types to make it clear that the data going in and out of my functions validate to a schema which is way better than the generic “pd.DataFrame” e.g. def myfunc(df: DataFrame[my_input_schema] -> DataFrame[my_output_schema]:

This was a really great video. I am a data scientist for over 2 years and was great to see that i already use developed a habit to use some points (probably because of other videos of you 😂❤) but also learned something new too! Could you do the same thing for data engineering?? That would be awesome!

Great video! As a data scientist myself, I would love to see you work through an example that uses something like MLFlow. It's a very common tool at every DS-job I've worked since it's open source but also part of many managed solutions. Specifically, I'd love to see how you build an MLFlow client, how you structure experiments/runs and when/where you feel it is best in the flow of an ML pipeline to log to/load from MLFlow. Most MLFlow tutorials I've seen are notebook based, which is great for learning the basics but there isn't much guidance out there on how to structure a project that leverages it.

As always, thank you for the great video! I'd love to see more DS content, as well as some content on mlflow and dvc.

Some good points in the video to get people thinking about different aspects. Scalability leads to a temptation for some to use multiple APIs and thereby an API management tool which in turn costs time and increases the probability of a ML library being used like a sledgehammer to crack a nut _(especially if time lack inclines a planner to be avoidant of dependency tree challenges)._ No matter the scalability of a software system _(whilst not seeing it as something to be seen as "regardless" of scale)_ databasing to keep track can become a bottleneck, and so retries and dead letter queues are worth it. Your mention of workflows is wise and jumping from one database to another _(e.g. MySQL to Postgres)_ is very likely to incur thoughts spent on workflows for that very task. You can optimise all you like _(which is noble)_ but these days people are more incentivised to "build-things" and so somebody might pip that "optimiser person" to the post by throwing computer horsepower at the challenge, thereby forcing something to be big rather than scalable in ways other than unidirectionally. Logs tend to mean hash tables. There are advantages to storing in a database like the choices available for DHT versus round robin. Environmental variables can be for ENV metadata to set up a virtual machine. If you own that server, like you suggest, it's an extra thing to secure _(for example against IGMP snooping)._ Containers and sandboxes are an extra layer of security rather than a replacement for security. Multiple BSD jails for example can be managed with ansible for instance. My comment has no hate in it and I do no harm. I am not appalled or afraid, boasting or envying or complaining... Just saying. Psalms23: Giving thanks and praise to the Lord and peace and love. Also, I'd say Matthew6.

Great video. I feel data-science projects are rarely examined in terms of design/structure quality. I hope to see more videos about it in the future. Perhaps on writing tests, I sometimes lack ideas about how to test data-science code.

Amazing video! I am very happy to notice that these are the bits of advice I have been pushing around to my work environments. I hope these things become the norm soon.

Great vid! Would love to see your approach to doing a data project eg download data, use airflow to process, train model and host via an API endpoint

Love these Data Science videos

Thank you for the good videos. Will there be any examples of design patterns using a GUI as an example?

As an intermediate data storage format, I typically use duckdb because I'm quite comfortable with SQL, and duckdb allows me to query large sets of data very quickly

Great video, and very great advice! I'll be giving a course about software development next semester and I think some of the points you talked about are worth mentioning !

Arjan can now fill the dutch city of Tilburg with his 200k subs! Impressive since he only passed the city of Breda (150k) a couple of months ago! Congratulations Arjan!

A .py file can be a notebook too! Just add "# %%" to create a cell - vscode will detect it automatically! Downside: the output isn't saved like you can do with a regular notebook Upside: code is easier to test, no visual fluff, etc.

In defense of notebooks, the exploratory aspect that it allows makes it really nice and quick to find problems within the data. You can look deeper at the objects or dataframes where it halts. If there is a way to combine this ability with a well structured set of scripts, it would be fantastic

It's a shame the pipeline suggestion came from a sponsor, as it would be very interesting to see you compare different tools like snakemake and nextflow. It's a very niche field and choosing a tools is dificult since you have to comit a lot. Great video though, would love to see that pipeline video eventually.

One thing I’ve began doing is using a flatter directory hierarchy and using SQLlite to catalog file paths along with useful, project specific metadata. This way I write a SQL query to pre-filter data by only fetching relevant Parquet file paths to pass into Dask for reading and analyzing.

Regarding Jupyter notebooks, a lot of things still work when opening the notebook in VS code, such as code formatters. You just may need to trigger it specifically in each cell (Alt+Shift+f for the black formatter, I believe).

I started doing Test Driven Development and it has saved me from more than one huge headache! Good advice!

That fight to the death line caught me mid sip and now I need to clean my monitor 😂

Good overview. Would really appreciate some physical examples.

Thanks sir, great video! Can you recommend a book on software design?

Great content as always. I’ve used csv, json, pickle, parquet, and sql files. I would argue there is no “standard” data science project. Once you get past boilerplate stats, you’ll inevitably have to write ad hoc functions to match the idiosyncrasies of your data either for deeper analysis or visualizations.

I use CSV many times 'cause it's simple and compact! I've also used parquet when I wanted to be sure about the data types when I've loaded back the data.

Thanks for the content. Quick feedback on the changing camera angles, it may look nice but it disturbs the flow of thought. Thank you very much.

I constantly use the Parquet data format. It makes loading data WAY faster. In Python, it works just as CSV (e.g., using Pandas, instead of using read_csv() you use read_parquet()). It is bundled with an intelligent way of compressing repeated values, so it has a way smaller HD memory footprint when compared to CSV and JSON. It stores data in a columnar fashion, so that if you only need some columns for a project, and other columns for another, you can avoid retrieving unwanted columns into memory. It also works well with Big Data environments (such as Apache Spark). Having a smaller HD memory footprint means you can transfer to other people easily as well. And store it in cloud solutions with a lower cost. And honestly, as a Data Scientist, you kind of never would open the CSV or JSON file and check it yourself. 99% of the time we use a library like Pandas or a software like Tableau to visualize and work with the data. So being human-readable is not really an advantage for data scientists, as it is for backend and frontend developers.

Data Formats: Parquet and Pickle because they are lightweight and easily adapted to pipelines. Requesst: Would love to see a taipy project video :)

It would be great to hear some ideas on how to write good tests for data related projects. Lately I've been using syrupy to write regression/snapshot tests to at least assure that the results don't change when I do a refactor. However this is not very robust. A challenge with all that is creating and managing good test data.

Great video 😁

Great content!!

For Tip 5, nbdev from fastai is a great package for exporting cells from a Jupyter Notebook to a script. From my notes, ymmv: At top: #| default_exp folder_name_if_desired.file_name Per cell exported: #| export To export, add below to same cell: import nbdev # For current directory path: nbdev.export.nb_export("Notebook Name.ipynb", "./")

can you go over using unit tests in data science projects? My team does not actively use them. We don't find a use when we add new features, change processing code very quickly

for caching and storing inermediate results, the fastest formats I've tried are msgspec for json‐like data and feather for table‐like data

Useful list of tips but I have additional tips we can derive by combining these tips. tip 1 + tip 6: Use a common way for externalizing configurations If each project externalizes configurations differently, for example, one uses a YAML file and another uses a .env file, it will be a nightmare for other people, particularly for engineers working on the deployment and the operation.

BTW jupyter notebooks in visual code with git are nice as you get simplicity of Notebook, performance data and versioning

The general point that you can learn from libraries is of course very good, and I've used sklearn as a design guide many times. That said, pandas specifically is probably a bad example for that purpose since a lot of its design is weird/messed up. For example, the default settings for reading and loading dataframes from csvs are all wrong -- it should be the case that if you write a dataframe and then read it, you should get the same thing -- saving stuff to disk should round-trip, right? Well, with pandas it doesn't, both because of indexing issues, and because it rounds stuff by default. Lots of little corner things in pandas like that, so IMHO while it's a powerful package that I probably couldn't live without, it's one best used _after_ you've acquired the relevant domain knowledge, not as a teaching tool for it.

Polars scales great. Read the CSV and query, lazily if needed. Parquet for intermediate file system storage, polars.write_database if needed. "If you have to ask, polars is enough".

for config, I like to have a centralised config.py with a Config class. It'll have attributes that try to get env vars (environ.get('myvar', 'some_sensible_default'), with a default.

6:50 to query a 'json file' or a collection of 'json files', there is lib tinydb

I love using Parquet as an intermediate format then use Dask to read them and do lazy processing as much as possible until the pipeline is forced out of Dask.

Is taipy running completely locally, or using their web services?

Can you do an in depth logging video? :)

Which video should i watch first?! there's no ordering in this playlist, it's confusing

I've been working on a project where files contaone raw numpy files and pandas df. I was really struggling to save the data to simple files as pandas is slow and I didn't know what to do with acquisition parameters. I first made my own serialised and deserialiser to store the data in huge JSON (using orjson for performance) but now I stumbled upon pydantic. It's relatively easy to implement compatibility for numpy and pandas and now it's still JSON files but the classes object are very compact and using ABC it's easy to create inheritance and apply these traits. But now I'm looking at alternative to make these files compatible outside of python using either or both SQL and hdf5. I'm quite supprise that this is not a solved problem. I found xarray that could use be it needs complete rewrite. So' yeah maybe another day

I have a Python code that use Tkinter to make a GUI. This code edit images. Now, I'm developing my webpage with Django, and I want to run the GUI code in my page. I hoped I could sove it with PYSCRIPT, but it's not compatible. I'm in a dead end. Can someone give me an advice?

It might be a silly question in this context, but does anyone know of a similar quality channel with a focus on Java? Arjan is such a great educator that I am genuinely considering switching to Python, but it's hard because I must learn Java too.

Can someone outline for me what benefits notebooks have over IDE development? I've recently switched from doing data science with an IDE in a typical software dev environment to using Databricks notebooks (due to a job change). I honestly can't see any benefit, but I can see a lot of drawbacks. In an IDE like Pycharm I can rapidly create experiments, I can visualise data AND I can write clean safe software. Notebooks put so many obstacles in the way of good development. What am I missing?

When would I rather use Taipy than Dagster?

Different approach to config variables, use a MyConfig class and store in a pickle file. this way the file isnt clear text when stored

HDF5 seems to be a rather useful data format, for both metadata and data.

Use schemas to describe and validate your data.

my tip for teams that run many ML pipelines in particular: don't make many projects based on a single cookiecutter. Instead, nest all of your pipelines in a monolithic repo which is ONLY in charge of writing ML code, and separate the projects by subfolders. That way, you don't have to maintain many different CI/CD pipelines and docker imgs (they tend to be large, but that's the dependencies, not the application code).

Notebooks are great for experiments.

Thank you, could please return to LLMs for short series with MemGTP, OS and Function calls (YT, v=rxjsbUiuOFo, robot to robot interaction), if time permits, could be able to come up with demo and thought process, how futuristic is the scenario, and will it be a cost effect consideration, on prem cloud platform..... Thank you

Redo the video and asd data version control dvc. That is a must once you work in an organisation. It has rudimentary pipelining for modeltraining as well. Everybody should know mlflow. And.... avoid tools like kubeflow if you do not have sufficient manpower to run it.

Taipy seems to have abandoned the pipeline as a user concept - it no longer appears in the docs. I assume it's still in the mechanism, but no longer explicitly exposed. Rather, the emphasis seems to be on building GUIs with data nodes. My experience with graphical programming like this has been that they are extremely difficult to review, as one has to unfold a lot of nodes to actually get to the code - maybe they've gotten around this somehow, or maybe they want you to assume their code is foolproof, a bad sign.

hey man, where tf you got that amacin tshirt?

A bear t-shirt and no mention of polars?

Aryan codes? Based.

pandas.. everytime i come back to an old (6M+) project it does not work because they changed the API. Never happened with any other library (numpy, matplotlib, scipi etc.) So I would only use pandas if it REALLY benefits you. otherwhise you'll have a guaranteed refactor in 6 months...

You are wearing a pandas shirt.

As a professional data scientist, I disagree with a lot of this advice. With exploratory analysis, you are often writing one-off notebooks that nobody will read or reuse. Spending the extra time to write tests in this situation wasted effort. A good middle ground is including `assert` statements in your functions to make sure they're doing what you think they're doing. Pull code you're going to reuse out of your notebooks, and then write tests for it then.

As always, start from scratch. It's from zero, nothing, etc.

By the time you said what i can do with tipy, i already was not interested. Perhaps tell people what tipy can do for you and then how to install it and shit...

Tip Number 0: Don't use Notebooks

Wow! Top of the comments sound like AI generated. Username like: name+numbers, and long text comment that noone asked for.

For point 5 I've started to get our teams to think: reusable code/functions go in .py files. Analyses go in .ipynb. they love Jupyter notebooks and this helps it be more readable. Bonus: the functions if used a lot can be packaged easier