What is a Data Lake?

Рет қаралды 225,545

Күн бұрын

Learn more about watsonx: ibm.biz/BdvxDa
A data lake is a centralized repository that allows you to store a vast amount of raw data in its native format until it is needed. Data-driven businesses often use this storage architecture to get more business value from their data assets.
Check out this quick overview video with Adam Kocoloski as he goes through a data lake architecture and explains how data lakes can empower you to collaborate and analyze data in different ways that will help you make make smarter and faster business decisions.
#DataLake #CloudStorage #IBMCloud

Пікірлер: 64

@MayankDhuria 2 жыл бұрын

I was way too distracted by the way he was writing in reverse. I was switching between being in awe as how well he was writing in reverse, and his explanation. Thus, I have to watch it again after accepting the fact that he is a genius, and then finally get to understand everything. Great explanation.

@aacastrocomandante Жыл бұрын

I think he was writing normally and the video was flipped/mirrored afterwards.

@dreamdrifter Жыл бұрын

He writes normally on a clear glass screen, with a camera on the other side recording it in reverse - then flips the image. Still pretty genius.

@karrcatmusic5360 4 жыл бұрын

Amazing that you can write mirroring letters ....

@IBMTechnology 4 жыл бұрын

Read this post on our community page that explains it all ibm.co/2SA1vGd

@hdebbache2000 3 жыл бұрын

You know you can invert in post processing right? Lol

@jpro2222 3 жыл бұрын

@@hdebbache2000 He is actually left handed and married in real life but not in this video jajaja

@leonardofriedrichmagro3785 3 жыл бұрын

@@hdebbache2000 But it would not have the same impact...for sure, all of viewers take a look on that and said: hes good!

@danzinde 2 ай бұрын

I think the concept of data lake should have just focused on explaining the "Store" box.

@miloslekic1 4 жыл бұрын

Great explanation, thank you!

@carthur 5 жыл бұрын

Very clear and helpful. He writes like Ira Joe Fisher

@ahmeddraz962 2 жыл бұрын

Simple and very informative. thx alot

@datasleek7950 Жыл бұрын

Interesting but Data Lake is not only used by ML. It usually used to store unstructured raw data. Some governance can be applied, however, you don’t build Dashboard out of the data lake. You first need to model that data into a Data Warehouse using dimensional modeling (allowing you to extract different dimension of your data). This multi dimension represented by few tables will allow you to slice the data in multiple ways, making reporting, thus dashboards easy to build. This is why Airbyte/Fivetran + Snowflake + DBT are the most popular data stack on the market right now.

@casuallistener618 Жыл бұрын

😯

@BlackLibertyGT 2 жыл бұрын

Need to explain this to Snr Management, this video is very helpful in breaking it down into something I can explain to others.

@meryemLux Жыл бұрын

Thanks, that was very informative

@bl8nc 4 жыл бұрын

Great presentation, thank you!

@IBMTechnology 4 жыл бұрын

Thanks for watching Malcom!

@kekuko 4 жыл бұрын

Very clear and helpful. Many thanks!

@IBMTechnology 4 жыл бұрын

Glad it was helpful Brandon!

@enyart91 4 жыл бұрын

You almost had me with the mirrored letters, until I realized- your wedding ring is on your right hand in this video. Very nice camera trick.

@b00psie 4 жыл бұрын

it could be on his right hand. not everyone wears it on their left.

@nIrUbU01 3 жыл бұрын

@@b00psie yes it could be, but it most likely isnt

@johnnytorres277 4 жыл бұрын

Why is it that every data lake explanation is full of theory without any concrete examples? Aren't all of us here because we're SQL or Cube programmers and want to know whats so great about Data Lakes? All I see is the same thing I do with sql databases: import the data, prep and transform it and then query it directly or create dashboard applications.

@joshuamintz8852 4 жыл бұрын

Hi Mauro, does this help? kzfaq.info/get/bejne/f7abhNNzycrJZY0.html

@kuppaigopuram8751 4 жыл бұрын

I agree. This video is very similar to what we do with SQL. He has not really told what data lake is. But, whatever he told is true about Data Lake. Here is a list of differences in Data Lake that are not possible in standard SQL based RDBMSs. 1. Big Data 2. On the Cloud (this is possible) 3. Separation of Data from Data Processing Engine 4. Self Service Model 5. ML (this can be done) 6. Data in native format (csv/parquet/json/avro/...) All the above are common to Big Data. Here is the list of data lake differentiator. 7. Central Repository; means single source of truth.

@Ali-ds5iy 4 жыл бұрын

Man this is an awesome idea to stand behind the mirror and write it..👍🏼

@sreenivasamadenahall 3 жыл бұрын

Good explanation, thank you. However, talk can get started with "Big Data" - which means data lakes are intended to store, manage and serve large Big volume, variability, velocity. Data is ingested in native format. It need to be kept organized, controlled and managed - governance. Data needs to be served in native or processed further for other needs - reporting and visualization, recommendations, process automations and more. Some real-life use cases to start the discussion. If the viewer already knows bits of data world (databases, datawarehouse, data lake etc), this helps to consolidate that understanding.

@surfh3r0 9 ай бұрын

data is really the new oil! nice explanation

@hariwaz100 3 жыл бұрын

good explanation

@ashtarathena488 4 жыл бұрын

Mirroring letters was a bit spooky/distracting at first. But great and simple content. Thanks.

@KuldeepSingh-cm3oe 4 жыл бұрын

Very good explanation

@IBMTechnology 4 жыл бұрын

Thank you, Kuldeep!

@jeroboam4486 6 ай бұрын

I still have no idea what concretely is a data lake!

@ajathaindira 3 жыл бұрын

awesome !!!

@pogococo2246 2 жыл бұрын

do you use data warehouse at the "store" phase?

@mzimmerman1988 2 ай бұрын

thanks

@justincooke5888 2 жыл бұрын

So does a Data Lake fall under Document Store due to it ingesting all types of Meta-Data such as Audio /Media , and text ?

@MasterofPlay7 4 жыл бұрын

ibm is still using the traditional rdbms right? what about hadoop?

@BourbonDrinker 4 жыл бұрын

Great video. I posted this to my LinkedIn.

@IBMTechnology 4 жыл бұрын

Hi Cotton Hollow Distilling! Thanks for the link love! -Sai

@DaRyteJuan 3 жыл бұрын

"Data Lake" a piece of unnecessary jargon that adds nothing to the conversation. We've been dealing with these principles for decades already.

@centurion09 3 жыл бұрын

What does “infuse” mean in this context ? I could not find an answer searching on the Internet.

@sase1017 2 жыл бұрын

Infuse to business decisions for managers (Dashboard), consume by other part of the service in an app(Application), or automate to make the entire process smarter with AI (Automation)

@Thee.Mighty 3 жыл бұрын

Left-handed.... It explains everything

@lounacrea8179 3 жыл бұрын

He s actually using his right hand to write

@adityaprasad465 4 жыл бұрын

Thanks, but I don't understand how this differs from data warehouses and ETL.

@RAJATTHEPAGAL 4 жыл бұрын

Actually it doesn't, what makes it largely different is the kind of features a data lakes gives. Its catalogues data and makes it more usable traceable for external data operations. So ya u can say I can simply extract data out of my warehouse/etl system and then operationile for my spark jobs .... Chances are in a data lake solution this solution is already inbread in it with it's own ui or api for easy operationalisation ( spark job related transformation of data , munging cleaning etc) ... A data lake is a full blown solution more importantly an overlay over the existing data infrastructure u have. Maybe an onprem hadoop, or clustered mongodb. A data lake software should primarily be able to create a single view of these and make sense of it. It's a thin line but the data lakes are supposed to be more organized.....

@David-2501 2 жыл бұрын

I would say it depends on the underlying tech. Data warehouses (DWHs) and Extract-Transform-Load (ETL) is focused on relational databases (Postgres/Oracle/Microsoft/MariaDB/MySQL/SQLite), whereas a Data Lake also includes "Not Only SQL" (NoSQL) technologies like Kafka (data streams), Hadoop (Document Store/csv file storage), Impala (SQL query engine for Hadoop), etc. When it comes to concepts, it *heavily* overlaps, IMO.

@sachinfulsunge9977 2 жыл бұрын

@@David-2501 I think there's a correction, DW are not focused of relational but dimensional databases

@bavuvan6298 3 жыл бұрын

Thank you! can you make video compare with data lake and data warehouse?

@IBMTechnology 3 жыл бұрын

We're glad you enjoyed it! 😃 We'll pass your feedback on to our team.

@shantnuchaturvedi5080 2 жыл бұрын

Nice Fancy Arrows on your diagram

@VaibhavPatil-rx7pc 3 жыл бұрын

E X C L L E N T !!!

@khalidjaradat 4 жыл бұрын

where the data lake ?

@scottswanson5358 4 жыл бұрын

Next time you do this you should write in the direction relative to yourself then just upload a horizontally flipped video.

@IBMTechnology 4 жыл бұрын

Hi Scott...actually that is what we did :) Check out this blog post for the details: kzfaq.info/love/KWaEZ-_VweaEx1j62do_vQcommunity?lb=Ugzf5SL_yh9NglCJzgF4AaABCQ

@shaunfrench5057 4 жыл бұрын

@@IBMTechnologyThe wedding ring on the "wrong" hand gives it away. Can you post-production CGI it onto the correct hand? ;)