Exporting CSV files to Parquet with Pandas, Polars, and DuckDB

Рет қаралды 8,122

Күн бұрын

In this video, we'll learn how to export or convert bigger-than-memory CSV files from CSV to Parquet format. We'll look at how to do this task using Pandas, Polars, and DuckDB.
#pandas #python #polars #duckdb
Resources
► Blog post - www.markhneedham.com/blog/202...

Пікірлер: 15

@marycombinatoria7335 Жыл бұрын

saved my day, thank youuuuuu!

@learndatawithmark Жыл бұрын

You're very welcome!

@awe8401 Жыл бұрын

your content is great...

@learndatawithmark Жыл бұрын

Thank you!

@myyouaccounttube1024 Жыл бұрын

Many thanks for this nice Video. A question about the method you presented with DuckDB: When exporting a table from DuckDB into the disk with Parquet format using COPY, is it possible to have some partitioning parameter to specify keys (Hive style) based on which the data would be split?

@learndatawithmark Жыл бұрын

You can't specify a partitioning key at the moment, but I think in version 0.7 you will be able to do hive style partitioning. I'll make a video showing how to do that once the new version is released!

@guocity 3 ай бұрын

Pandas work much better in unclean data, how do you handle pyarrow headache in data conversion error?: ArrowInvalid: Could not convert '230' with type str: tried to convert to double make many dependencies unusable: to_parquet() convert pandas to polars open csv in data wrangle, save as parquet in data wrangle

@PYG143 Жыл бұрын

Many thanks, We have a requirement to convert huge csv file to Parquet . Is it possible using C# console program ?

@learndatawithmark Жыл бұрын

I think you should be able to do it using this library - www.codeproject.com/Articles/1145337/Cinchoo-ETL-CSV-Reader

@kpyoutuber4671 5 ай бұрын

Thank you, Mark!!. Can you also explain the parquet dataset? I used to create a partitioned Parquet dataset by using Pandas and Polars. But I want to know how to read data from such partitioned parquet datasets directly to Polars lazy frame (not to pandas as data size is larger than memory) to do some analytics. import polars as pl import pyarrow.parquet as pq # Read data written to parquet dataset pq_df = pq.read_table(r"C:\Users\test_pl", schema=pd_df_schema, ) pl_df = pl.from_pandas(pq_df.to_pandas()).lazy() Is there any better way to do this

@learndatawithmark 5 ай бұрын

Can you explain a bit more - m.h.needham@gmail.com if you like Do you mean that you have multiple Parquet files and you want to read them all?

@kpyoutuber4671 5 ай бұрын

@@learndatawithmark Thanks for the kind reply. I just came to know through a response received to my SO quiz that pl.from_arrow() or pl.scan_parquet() can be used to read from a partitioned parquet dataset. It worked. pl_df = pl.from_arrow(pq_df).lazy() and/or pl.scan_parquet(r"test_pl/*/*.parquet").collect(streaming=True)