Azure Data Lake Storage (Gen 2) Tutorial | Best storage solution for big data analytics in Azure

No video

Azure Data Lake Storage (Gen 2) Tutorial | Best storage solution for big data analytics in Azure

Рет қаралды 233,643

Күн бұрын

Data Lake Storage Gen 2 is the best storage solution for big data analytics in Azure. With its Hadoop compatible access, it is a perfect fit for existing platforms like Databricks, Cloudera, Hortonworks, Hadoop, HDInsight and many more. Take advantage of both blob storage and data lake in one service!
In this episode I give you introduction to what Azure Data Lake Storage is, how it works and how can you leverage it in your big data workloads. I will also explain the differences between Blob and ADLS.
In a short demo I will show you
- What is Data Lake Storage and how it works and why is it called Gen2?
- What does it mean being designed for big data analytical workloads?
- How does multi-protocol access work?
- What are key differences between ADLS and Blob Storage?
- Quick demo of creating ADLS in portal
- Quick demo of connecting from Power BI and using multi-protocol access
- How to use storage explorer with ADLS
- How do Access Control Lists work and how to manage them
- Demo with Databricks and ADLS
Sample code from demo: github.com/MarczakIO/azure4ev...
Next steps for you after watching the video
1. Azure Data Lake Storage documentation
- docs.microsoft.com/en-us/azur...
2. Transform data using Databricks and ADLS demo tutorial
- docs.microsoft.com/en-us/azur...
3. More on multi-protocol access
- docs.microsoft.com/en-us/azur...
4. Read more on ACL
- docs.microsoft.com/en-us/azur...
Want to connect?
- Blog marczak.io/
- Twitter / marczakio
- Facebook / marczakio
- LinkedIn / adam-marczak
- Site azure4everyone.com

Пікірлер: 249

@fernandos1790 4 жыл бұрын

I don't usually comment on youtube, but Adam I will make an exception for you. Your videos are easy to follow and educating, but most of all, straight to the point. The amount of time allocated to the video is perfect. Best wishes and please continue making training videos.

@AdamMarczakYT 4 жыл бұрын

Wow! Thank you Fernando for such a heart-warming feedback. More videos are coming!

@fsfs5665 3 жыл бұрын

I have been watching a lot of Azure videos. This one is the best and I will study more of your catalog. Thanks!

@AdamMarczakYT 3 жыл бұрын

Awesome, thank you!

@ajeetdwivedi2294 2 жыл бұрын

I don't know why I understand everything which you teach without even repeating the video twice. Its so much clear and to the point and especially demo part, starting from the background to the practical everything you present is just wow. God bless you !!

@shockey3084 3 жыл бұрын

Each and every second you took is informative. Great learning from you.

@AdamMarczakYT 3 жыл бұрын

I appreciate that!

@shailendersingh7093 2 жыл бұрын

This is first time i am commenting anyone on a KZfaq videos. I have seen thousands may be. "The best videos and so easy to understand" keep it up Adam

@adamolekunfemi6314 3 жыл бұрын

Excellent video. Taught with simplicity and clarity, without any noise.

@AdamMarczakYT 3 жыл бұрын

Glad it was helpful!

@carl33p 4 жыл бұрын

Your demos are fantastic. Love that you go step by step and don't skip things. Much appreciated.

@AdamMarczakYT 4 жыл бұрын

Thanks so much! :)

@CrypticPulsar 4 жыл бұрын

I spent days scouring the web for documentation and I was left out cold, then I thought I give KZfaq a try and I found yours up on the top, and I can't be happier.. thank so so sooo much Adam!! Keep up the excellent work.. very easy to follow, simple, rich content.. loved it!!!

@AdamMarczakYT 4 жыл бұрын

Awesome, thank you!

@sau002 2 жыл бұрын

I came to this video not expecting to learn much. I was wrong. Very useful.

@user-bh3gf9pp3w 2 жыл бұрын

Thanks Adam's wonderful video, it's really easy to understand the ADLS!

@AdamMarczakYT 2 жыл бұрын

Glad it was helpful!

@innnseeectooo 3 жыл бұрын

This is excelent. I'm preparing DP-200 and 201, and your videos have a lot of information concentrated, summarized and explained very simply. Thanks!

@AdamMarczakYT 3 жыл бұрын

Glad it was helpful!

@AdamMarczakYT 4 жыл бұрын

Please note that since the release of the video there were some changes made to the service. For instance an immutable storage feature is now in preview for ADLS :) azure.microsoft.com/fr-ca/updates/immutable-storage-for-azure-data-lake-storage-now-in-public-preview/?WT.mc_id=AZ-MVP-5003556

@avnish.dixit_ 3 жыл бұрын

Fabulous Work. Just one think always try to make you videos from production point of views. And it would be great it upload few new videos on "Data Mapping Flows" on Delta Lake, and Databricks features such such Mounting, Caching Streaming operations

@avnish.dixit_ 3 жыл бұрын

@AdamMarczakYT 3 жыл бұрын

I'm working on improving my workflow. By end of 2020 I want to have new streaming PC with better setup which would allow me to more freely create videos and reduce time required to make them. When this happens I will be able to make more videos faster and MDF in ADF is surely a big topic of interest to me. :) Thanks for tuning in!

@sammail96 Ай бұрын

@@AdamMarczakYT Bro Why you stop making videos?

@discovery-dx3ry 4 жыл бұрын

Your videos are very easy to follow. Many thanks for your effort to create all the azure videos.

@AdamMarczakYT 4 жыл бұрын

Thanks :)

@tarvinder91 Жыл бұрын

This is such a great tutorial especially that you share all teh difference with normal storage account. Its was extremely helpful for me

@lg061870 4 жыл бұрын

this is insanely complete! wow.

@AdamMarczakYT 4 жыл бұрын

Coooool! Thank you! :)

@alexandrupopovici2366 2 жыл бұрын

Great video as per usual, your channel is my go-to for preparing for Azure Certification Exams! I do have a question regarding the ADLS Gen 2 hierarchy, as one of my friends is preparing for the DP-900 exam (regarding a practice question). The question asks you to match 2 of the following 3 terms ([Azure Storage Account], [File share], [Container]) in the following hierarchy (only one answer is allowed): Azure Resource Group - [TERM 1] - [TERM 2] - Folders - Files The suggested correct answer is: Azure Resource Group - [Azure Storage Account] - [File Share] - Folders - Files But I don't see any reason why the following answer would not also be correct (besides maybe because containers being called file systems in ADLS Gen 2?): Azure Resource Group - [Azure Storage Account] - [Container] - Folders - Files What is your take on this? Thanks for taking your time!

@afzaalawan 3 жыл бұрын

What a great explantion with practical.. you are Star

@AdamMarczakYT 3 жыл бұрын

Thank you! :)

@javimaci4615 3 жыл бұрын

Adam you are a rock star. Your videos are extremelly well done. Thanks and keep up the good work!

@AdamMarczakYT 3 жыл бұрын

Glad you like this! Thanks!

@empowerpeoplebytech2347 4 жыл бұрын

Great explanation of many things together and also explaining the differences and linkage between ADL, ADB, PBI, etc. Thank you very much Adam for this one.

@AdamMarczakYT 4 жыл бұрын

My pleasure! Thanks!

@saurabhrai8817 4 жыл бұрын

Hey Adam !! You are becoming one of the BEST AZURE AUTHORITIES / SME on the Internet. Keep up the good work. Thanks for sharing your knowledge in such a simple way. Kudos !!

@AdamMarczakYT 4 жыл бұрын

Wow, thanks, that's a nice thing to say. I wouldn't say so since I'm just a trainer, but I love your enthusiasm and appreciation. Thank you kindly my friend :)

@yemshivakumar 4 жыл бұрын

Never seen this kind of KT video's to public. Thanks Adam for spoon feeding video to improve Azure knowledge.

@AdamMarczakYT 4 жыл бұрын

My pleasure!

@Haribabu-zj4hd 3 жыл бұрын

Very nice video explained clearly the concept, thank you so much Me.Adam🙏.

@AdamMarczakYT 3 жыл бұрын

Glad it was helpful!

@bethuelthipe-moukangwe7786 Жыл бұрын

Thanks for lessson, your videos are very helpful to me.

@emiliogarza6446 Жыл бұрын

Your content is gold, thanks a lot for making these videos

@anushamantrala5527 3 жыл бұрын

Your videos are really worth watching Adam , really thanks for the beautiful content 😁want many more videos from your side.Thanks in advance.

@AdamMarczakYT 3 жыл бұрын

Awesome! Cheers!

@Rana-zi4ht 2 жыл бұрын

Hey Adam! That was very informative and clear explanation about data lake👏 .Thank u a lot

@vijayt3678 4 жыл бұрын

Wow such a clear and simple explanation about Data lakes. Absolutely awesome thank you Adam for your great efforts for the community... More power to you...👍👍

@AdamMarczakYT 4 жыл бұрын

My pleasure! Thanks for watching ;)

@mgvlogs5948 3 жыл бұрын

you make very simple and easy explained videos well done Adam!

@AdamMarczakYT 3 жыл бұрын

I appreciate that!

@ANAND237 Жыл бұрын

Great demo. Thank you Adam

@GuilhermeMendesG 2 жыл бұрын

What a great video. Thanks Adam!

@vzntoup 3 ай бұрын

This tut was a blast! Thank you

@alperakbash 3 жыл бұрын

Unbelievable tutorial. Thank you so much for helping me to find everything I look for at one place.

@alperakbash 3 жыл бұрын

excluding Power BI of course. I am a tableau fan =D

@AdamMarczakYT 3 жыл бұрын

Awesome, thanks!! :D

@AdamMarczakYT 3 жыл бұрын

Hehe, thanks alright, we all have our preferences :)

@totanlj18 4 жыл бұрын

I really really thank you!! Your video makes me the week!

@AdamMarczakYT 4 жыл бұрын

You are so welcome!

@seagoat666 Жыл бұрын

Amazing Demo!! Many Thanks!!

@allanramos5721 3 жыл бұрын

Thanks for the contribution, Adam!

@AdamMarczakYT 3 жыл бұрын

Thanks! It's my pleasure!

@RohitJadhav-ik8gt 3 жыл бұрын

You are fantastic !!! Thanks for sharing valuable content.

@AdamMarczakYT 3 жыл бұрын

I appreciate that!

@GiovanniOrlandoi7 2 жыл бұрын

Great video. Thanks Adam!

@TechnoQuark 4 жыл бұрын

Hi Adam... Barvo. Excellent work. I recently watched few of your videos and they are absolutely fabulous... Thanks

@AdamMarczakYT 4 жыл бұрын

Awesome, thank you!

@mallikarjunap7302 4 жыл бұрын

Its excellent video for ADLS to connect to data bricks and power BI

@AdamMarczakYT 4 жыл бұрын

Thank you mate :)

@shantanudeshmukh4390 3 жыл бұрын

You are amazing Adam !! How can one know all these things, Azure, Power BI, ADF, Data Lake. You are genius. Thanks for knowledge sharing !

@AdamMarczakYT 3 жыл бұрын

Haha! Thanks :D

@zulumedia6374 9 ай бұрын

Fantastic and useful video. Thanks!

@sinyali8370 2 жыл бұрын

Very good and comprehensive tutorial, thank you!

@janvi_gupta_group 3 жыл бұрын

Awesome demonstration how to create and connect ADLS and running Scala code with databriks

@AdamMarczakYT 3 жыл бұрын

Many thanks!

@sammail96 Ай бұрын

4:52 ADLS Gen2 supports soft delete for blobs. When enabled, deleted blobs are retained for a specified period before permanent deletion1. However, soft delete for containers is not supported during the upgrade process2.

@niluparupasinghe7307 2 жыл бұрын

Excellent and very practical tutorial, thank you...

@frclasso 3 жыл бұрын

Very good tutorial, very helpful. Thank you.

@AdamMarczakYT 3 жыл бұрын

Glad you enjoyed it!

@pratimaprabhu2029 2 жыл бұрын

Very well explained 👍🏻

@constantini82 2 жыл бұрын

amazon tutorial, you explain so well, thanks

@max_frame 4 жыл бұрын

Excellent video, exteremly clear and concise. Thank you!

@AdamMarczakYT 4 жыл бұрын

Glad it was helpful!

@clapton79 3 жыл бұрын

For Databricks mounting: Please note that Azure version as of today will copy the secret ID and not the password itself if you hit copy at the end of the line just like Adam does. Copying the secret password seems only possible immediately after the creation of the secret by the copy button that appears right after the password. Took me some time to figure this out..

@AdamMarczakYT 3 жыл бұрын

Hehe! Good catch! Microsoft updated UI and now new keys have two columns Value and ID. Both have copy a button. Just make sure to use the copy button in the Value column :) Thanks!

@SatyaPParida 4 жыл бұрын

Fabulous tutorial.wish to see more like these. Informative content ✌️. I'll be using this knowledge in my project.Much needed video.Thanks

@AdamMarczakYT 4 жыл бұрын

Awesome, thank you!

@GG-uz8us 4 жыл бұрын

A quick correction, you should use select.write.csv(...) at 23:20, otherwise you would write all columns from original csv to the new csv file.

@AdamMarczakYT 4 жыл бұрын

Ah a good eye indeed. Coincidentally I noticed this as well yesterday as I was conducting training on this very topic. Cheers 😀

@michalhutny7356 4 жыл бұрын

Great work, as always!

@AdamMarczakYT 4 жыл бұрын

Dzieki Michał!

@noahmcaulay4420 4 жыл бұрын

Thank you! Extremely helpful video, and very informative. :)

@AdamMarczakYT 4 жыл бұрын

Glad it was helpful!

@ahsanijaz6318 3 жыл бұрын

great video...can you please also make a video of how we can move the Microsoft Navision data to the data lake

@AdamMarczakYT 3 жыл бұрын

Thanks! Unfortunately Microsoft Dynamics is not my field of expertise. Nav is pretty old system too so it's hard to find any useful examples :(

@snooprobbyrobb9026 4 жыл бұрын

Your videos are wonderful, sir! would love to see an in-depth one on Azure Monitor, perhaps how services such as these (storage/blob/data lake) can tie into it. I find the variety of monitoring options a bit overwhelming without knowing which are worthwhile. Have a great day! (please let me know if I just missed an Azure Monitor video somewhere in here)

@AdamMarczakYT 4 жыл бұрын

Great suggestion, I surely have monitor on thelist! Thanks for watching :)

@enavea 3 жыл бұрын

Interesante representación de contexto de como manejar un lago de datos que hoy tenemos y como estos los podemos transformar en información y prepararla para la inteligencia artificial sobre ellos.

@AdamMarczakYT 3 жыл бұрын

Gracias

@icici321 4 жыл бұрын

Great Video. Your explanation is very nice and easy to understand. Thanks very much.

@AdamMarczakYT 4 жыл бұрын

Glad to hear that, thanks! :)

@dianpriyambudi 4 жыл бұрын

Love it, many thanks Adam!

@AdamMarczakYT 4 жыл бұрын

My pleasure!

@selwynalexander9750 4 жыл бұрын

Super Adam!...Good for Analytical usecases

@AdamMarczakYT 4 жыл бұрын

Awesome!

@drScorp1on Жыл бұрын

Great video, just subbed.

@yuliyacher67 3 жыл бұрын

Thank you for information.

@AdamMarczakYT 3 жыл бұрын

You are welcome

@manishalankala1622 3 жыл бұрын

Well Explained

@AdamMarczakYT 3 жыл бұрын

Thanks!

@gurubazi 4 жыл бұрын

I really Haapy with your explanation and presentations..it helps me a lot

@AdamMarczakYT 4 жыл бұрын

Glad to hear that :)

@francisjohn6638 Жыл бұрын

Really awesome !

@mohitjoshi1361 2 жыл бұрын

At 10:30, I understand read and write, but how execute works here? What is execute permission in adls?

@seb6302 4 жыл бұрын

Love your videos Adam!

@AdamMarczakYT 4 жыл бұрын

I appreciate that! Thanks!

@SS-eu4eb 4 жыл бұрын

Very clear explanation. Thanks!

@AdamMarczakYT 4 жыл бұрын

Thank you so much :)

@nmhoang310 4 жыл бұрын

Good tutorial. Easy to understand.

@AdamMarczakYT 4 жыл бұрын

Glad it was helpful! 😀

@pdsqsql1493 2 жыл бұрын

Fantastic video

@puttarinkesh8535 2 жыл бұрын

Thank you, very nice video

@labelledamesansmerci8607 3 жыл бұрын

Can you make more videos on Azure Data Bricks calling multiple notebooks, making RDBMS calls, logging etc.,

@AdamMarczakYT 3 жыл бұрын

I will want to make some series in the future on databricks. :)

@venkatx5 4 жыл бұрын

Excellent Adam!

@AdamMarczakYT 4 жыл бұрын

Thanks as always, you are very active :) nice to see that.

@venkatx5 4 жыл бұрын

@@AdamMarczakYT Azure is Interesting + your videos are great as it has clear explanation with Demo.

@terryliu3635 4 жыл бұрын

Good hands-on intro, thanks!

@AdamMarczakYT 4 жыл бұрын

Thank you! :) Glad you enjoyed it Terry.

@sanniepatron8260 4 жыл бұрын

thank you for the videos! i am starting with databricks and it super clear! do you have some videos of delta lake databricks like merge things? it will be awesome to learn more about it!

@AdamMarczakYT 4 жыл бұрын

Glad to hear that! :)

@MrFromminsk 6 ай бұрын

Soft deletes are available in ADLS gen2

@GAMER-zz4cc 10 ай бұрын

Hi Adam Please provide the ADF series from basic to advanced level it helpful for me

@shantanuchakraborty4266 4 жыл бұрын

very good for beginners..Thanks to you.

@AdamMarczakYT 4 жыл бұрын

Thanks and welcome! :)

@ashseth7885 3 жыл бұрын

Hi Adam , thanks for your valuable time to create this video. I faced a problem while performing Add Role Assignment step , I saw that Azure has removed AzureAD from "assign access to" drop down list, . Please suggest any other approaches to mount data lake . Appreciate your efforts. Thanks

@agupta51 4 жыл бұрын

Excellent presentation.

@AdamMarczakYT 4 жыл бұрын

Thanks :)

@MohMOh-kv9gg Жыл бұрын

Hi Adam! Great video! 1 question: I have created an issue reporting, inspection and ideas apps on one of my team in Microsoft Teams. How do I export that data into Azure Data Lake?

@prashanthkommana7105 4 жыл бұрын

Wonderful demo. Can you please give us demo on Datafactory and API as well please.

@AdamMarczakYT 4 жыл бұрын

Thanks. Actually I already have few data factory videos (4) using blob storage and SQL, but blob and ADLS are so similar that if you would watch those and change connector to ADLS you wouldn't notice the difference. For API, what do you mean?

@prashanthkommana7105 4 жыл бұрын

@@AdamMarczakYT Hello Friend. Something on PAAS services. Also plz plz plz plz give full demo on Azure Site Recovery. Migrating OnPrem Infra to Azure. Please.

@hotkissfuck101 4 жыл бұрын

can u make video on service principle? with a sample demo

@AdamMarczakYT 4 жыл бұрын

maybe :)

@hotkissfuck101 4 жыл бұрын

@@AdamMarczakYT thanks. Also ACL :-)

@justair07 4 жыл бұрын

When creating containers, how do I know exactly how many containers I should make? For example, if I'm creating 5 apps that are completely independent of each other and the apps save pictures that the users take to the storage account, should I have 5 containers (1 for each app)? Or 1 container to support all apps?

@AdamMarczakYT 4 жыл бұрын

Hey Justin. There aren’t any specific limits scoped around containers so this is a design decision. There aren’t any specific guidelines so you should match what feels right for your organization and use case. But there are storage account level limits so those could be a deciding factor between one and many storage accounts, check those out in here docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits?WT.mc_id=AZ-MVP-5003556

@jurges8544 Жыл бұрын

Hi Adam thank you for the video, it was great. I have just one question in relation to Hadoop compatible access? This means that it can be connected with Hadoop, or it uses Hadoop every time it has some action inside the Data Lake . Thanks a lot once again

@CoopmanGreg 4 жыл бұрын

Excellent. Thanks!

@AdamMarczakYT 4 жыл бұрын

Glad it was helpful!

@satori8626 10 ай бұрын

If I want to use blob storage to store some files with low costs, and delta lake storage to store other files in a structured directory, do I need to create two separate storage accounts?

@shawndeggans 4 жыл бұрын

It looks like we don't necessarily need to use databricks, because ADF now support "Data Flows," which are a kind of no-code data transformation process. What are your thoughts on that? Is ADF a good substitution for Databricks (its actually using Databricks under the hood) for more advanced data transformation jobs?

@AdamMarczakYT 4 жыл бұрын

hey Shawn, very good question. Since Microsoft removed Data Flow step which allowed you to input your own script blocks I'd say for advanced scenarios I would use databricks since I would want to have full control. Microsoft also removed ability to provide your own link service for dataflow which in return also means if you want to connect to data sources within your private networks then public integration runtimes will not be able to connect (it can however connect to firewall vnet protected resources using managed identity), nor will you be able to add custom libraries to your dataflow (again, you don't own cluster so you can't control this) hence again narrowing down some scenarios. Net net, my opinion is that general directory of Data Flow is for simple cloud transformation scenarios at this point in time.

@MrLenzi1983 3 жыл бұрын

Adam your tutorials are amazing! is it possible to copy metadata and files from sharepoint and ingest into data lake using ADF?

@AdamMarczakYT 3 жыл бұрын

It should be possible using REST api, but I would advise against it. This is what Logic Apps were created for. Thanks for watching!

@princenwanguma3874 2 жыл бұрын

Thanks a lot

@AdamMarczakYT 2 жыл бұрын

Happy to help

@RaviKumar-op8gb 2 жыл бұрын

Is it possible to attach two data lake to single instance of databricks

@AdamMarczakYT 2 жыл бұрын

You can attach as many as you like. :)

@viral.vr2 3 жыл бұрын

thanks bro!

@AdamMarczakYT 3 жыл бұрын

No problem!

@kuldeepkumar-fp4tu 2 жыл бұрын

Hi Adam, I have a data Lake and I permit my collogue via RCAC, Now my collogue want to access this data lake via portal/storage explorer but he cant find it. any solution!!!!!!!!!

@eugenemagloire9456 3 жыл бұрын

Adam please what's difference betweem Blob and Data Lake(Blob storage VS Data Lake)

@AdamMarczakYT 3 жыл бұрын

This is covered in the video. Is it not clear? Do you have any specific questions?

@gobieee1 2 жыл бұрын

what is the max file size i can upload? is there any config in storage account where i can set the threshold size

@alfonsovillegas178 10 ай бұрын

You rock!

@gadankidevikiran 4 жыл бұрын

Superb

@AdamMarczakYT 4 жыл бұрын

Thanks 🤗

@chetangupta50 4 жыл бұрын

Great work

@AdamMarczakYT 4 жыл бұрын

Thank you!

@GG-uz8us 4 жыл бұрын

Very good introduction video, thank you. A quick question, why using AccessKey to mount Azure Blob Storage, using Service Principle to mount Data Lake? How do I use AccessKey to mount Data Lake?

@AdamMarczakYT 4 жыл бұрын

Good question! There is pretty much no difference, I just wanted to show both approaches, service principal is recommended but access key will work too. Check out how to use access key for data lake in the docs: docs.databricks.com/data/data-sources/azure/azure-datalake-gen2.html

@GG-uz8us 4 жыл бұрын

@@AdamMarczakYT Thank you. Another favor, will you be able to give a quick demo about Databricks? My impression about Databricks is all about in memory processing, good candidate for data streaming. Do you have a demo about from EventHub or ServiceBus to Databricks?

@AdamMarczakYT 4 жыл бұрын

Check this out docs.microsoft.com/en-us/azure/azure-databricks/databricks-stream-from-eventhubs I have only intro video on databricks as of now.