You're very welcome!😎

You're very welcome! I'm excited to hear about what you will build with them 🙂

I appreciate the kind words 🙏 thanks for the support!

🤣

Hi tmb8807 :) I have followed a couple of tutorials on polars, but never used it on anything in a professional setting as of yet 🤔 I'll test it out more extensively. Any good tutorials you'd recommend? Typically, when I've worked on projects that needed high performance I've used Apache Spark - but Polars could be a nice in-between pandas and spark? Thanks for the support!

@@trentdoesmath thanks for the reply. There are a few tutorials on KZfaq, the one from Rob Mulla is what got me onto it. Because Polars can work with larger-than-memory data via the streaming API I’ve seen it suggested it could replace Spark on a single node for some jobs, although I’ve not done that first hand! But it could potentially expand the 'in-between' area, as you say. Main reason I like it is that I just find the syntax much more consistent and readable (and easier to write as a result). Your mileage may vary on that, though, especially if you're extremely comfortable with Pandas (it's a bit less "Pythonic", with more explicit methods for everything). Lazy evaluation and the query optimisation engine are a big selling point of it as well - can greatly improve memory usage.

Awesome! I'll check out the Rob Mulla stuff, thanks for the recommendation👍 For sure! It actually reminds me a bit of Scala 🤔... Very 'to the point'. Not sure if you have tried out Dask before? but it's yet another performance option out there.

Hi @kikiboy2545 🙂 thank you for your question. TL; DR - I chose to use jupyter as it is easier for me to demo with and record the video with. To your point on creating a .py file - I would recommend this if you are creating cleaning logic that is going to be re-used and shipped to 'production' as it is easier to test and maintain a straight Python script IMO. That being said, there is increasing support for the use of notebooks as the preferred environment - as examples, Snowflake, Databricks, Azure Synapse and more all support the use of re-useable notebooks to contain all of your logic. I've worked in teams where notebooks are preferred for all data pipeline code due to how intuitive and approachable they are - but as I say my personal preference is: use notebooks for exploration, and .py scripts for your production code 🙂 No need to apologize! I am glad to be part of your learning journey - keep pushing man! 😎