Build a poor man’s data lake from scratch with DuckDB
Рет қаралды 25,045
Күн бұрынDuckDB is so hot right now. Could it replace our cloud data warehouses or data lakes?
Pete Hunt builds a data lake from scratch with DuckDB and Dagster. Follow the tutorial here: dagster.io/blog/duckdb-data-lake
Combined with Dagster, S3, and Apache Parquet, DuckDB can become a powerful, multiplayer data lake that can serve the needs of many organizations with very little effort. Think of it as a free, simple SQLite.
We can see the rise of DuckDB for subsets of workloads that don’t have massive scale and can take advantage of its simplicity and fast performance.
Give it a go!
Try Dagster for free for 30 days: dagster.io/lp/dagster-cloud-t...
@vikramtatke5930 2 ай бұрын
As a person with just 2 years of experience my mind was blown watching this. I am a single person writing code in my department so I don't have any seniors to learn from but I'm leading a data engineering project that deals with terabytes of data and each request is multiple times larger than the server's RAM and multiple such requests need to be processed in parallel to complete stuff in time. Also, we have the tiniest possible budget to aggregate 25 to 30 columns and billions of rows every day. Also, we need to cut down on costs. This was super helpful.
@michaelayoub2211 Жыл бұрын
Great video, thanks!
@marcosoliveira8731 7 ай бұрын
Really good stuff! A lot of good ideas.
@tobiaspucher9597 26 күн бұрын
Awesome!!! Please more!
@ImperialTerrain Жыл бұрын
thank you pete
@gw1284 Жыл бұрын
Thanks for this demo. Can you comment on what role polars may play in this?
@hwy9nightkid Жыл бұрын
polars is akin to pandas or spark dataframes.. a way to organize your tables of data , if im not mistaken
@marcosoliveira8731 7 ай бұрын
As pandas alternative.
@kalidsherefuddin 7 ай бұрын
Thanks for
@gauravlotekar660 Жыл бұрын
aawwwseome.
Skitsville
Рет қаралды 15 МЛН
CelerData
Рет қаралды 283
R Consortium
Рет қаралды 7 М.
TreshFilms
Рет қаралды 17 МЛН