Microsoft Fabric Data Engineering [Full Course]

Рет қаралды 32,062

Күн бұрын

Download your certificate of completion after you finish this course:
📄 prag.works/data-engineering-i...
Student files
✔️prag.works/data-engineering-i...
Get ready for an in-depth exploration of Microsoft Fabric's data engineering capabilities in our upcoming Learn with the Nerds!
Dive into the world of data factory pipelines and Spark notebooks, where we'll unravel the secrets behind designing efficient data pipelines and leveraging interactive notebooks for seamless data processing.
Whether you're a seasoned data engineer or just stepping into the realm of data architecture, this will be your guide to mastering the tools that make data collection, storage, and analysis a breeze.
0:00 - Introduction to Fabric in Azure Data Factory
12:45 - Importance of Compute in Fabric
30:15 - Setting Up and Configuring a Fabric Workspace
48:10 - Working with Data Lakes and External Tools
1:08:55 - Running Pipelines and Understanding Performance
1:12:01 - Utilizing Spark Notebooks for Big Data
1:20:37 - Integration Possibilities with Snowflake and On-Premises Data
1:26:44 - Wrapping Up
Next step on your journey:
👉 On-Demand Learning - Start With The FREE Community Plan:prag.works/lwtn-trial
🔗Pragmatic Works On-Demand Learning Packages: pragmaticworks.com/pricing/
🔗Pragmatic Works Boot Camps: pragmaticworks.com/boot-camps/
🔗Pragmatic Works Hackathons: pragmaticworks.com/private-tr...
🔗Pragmatic Works Virtual Mentoring: pragmaticworks.com/virtual-me...
🔗Pragmatic Works Enterprise Private Training: pragmaticworks.com/private-tr...
🔗Pragmatic Works Blog: blog.pragmaticworks.com/
Let's connect:
✔️Twitter: / pragmaticworks
✔️Facebook: / pragmaticworks
✔️Instagram: / pragmatic.w. .
✔️LinkedIn: / prag. .
Pragmatic Works
7175 Hwy 17, Suite 2 Fleming Island, FL 32003
Phone: (904) 638-5743
Email: training@pragmaticworks.com
#Fabric #PragmaticWorks #AustinLibal #Training #Microsoft #Tech #FreeConference #LearnWithTheNerds
**Any sales mentioned in the video may no longer be valid. Offers are subject to change with/without notice and are for a limited time only.

Пікірлер: 24

@user-pd1hg1gh4w 3 ай бұрын

I paused in between once in a while to sip some water for you , Austin :D Great session! Cheers! :)

@austinlibal 2 ай бұрын

😅

@thetrue_fan_tv_0nline431 4 ай бұрын

Good job, definitely enjoying, thanks a lot for sharing. blessings.

@oluwafemifelix5929 6 ай бұрын

this is really commendable, thank you

@BidofVic 2 ай бұрын

Awesome.

@maxgaming8848 6 ай бұрын

I really like the part where you show how to build a dynamic pipeline with parameter!! I find that many tutorial only show basic function but not in depth or scalable solution which is what is required in real world environment.

@austinlibal 6 ай бұрын

Thanks! Glad you enjoyed!

@aluxhoj 3 ай бұрын

Thank you

@austinlibal 3 ай бұрын

You're welcome! Hope you enjoyed!

@austinlibal 6 ай бұрын

More questions from the live session that were not answered in the session. Q) Any Microsoft certification available that cover Microsoft Fabric? A) A new one has just been announced called DP-600: Implementing Analytics Solutions Using Microsoft Fabric! Stay tuned for content specifically around that in the future when it is generally avaliable!

@woliveiras 5 ай бұрын

A really fantastic session. Thank you. I have a question, and I don't know if it can be answered here. Is It possible to define a container to store my objects (Files, delta tables and others)? I understand that when you create your lakehouse, Microsoft provides you a specific location, but I would like to define my own area to store my objects. To be honest, I would like to use the concept of Medallion Architecture with landing zone, Bronze, Silver and Gold spaces. Thank you

@austinlibal 5 ай бұрын

You can absolutely define where you want to land your data in a Lakehouse. It's just a managed data lake so you still have the ability to create directories and subdirectories within the "Files" folder. I believe the best practice for leveraging medallion architecture in Fabric would be to have three separate workspaces for each layer. I would point you to Microsoft documentation for confirmation and their recommendations.

@brandonperks5877 4 ай бұрын

Are you able to use autoloader within your Fabric notebook to connect to a OneLake folder?

@juliekim6892 6 ай бұрын

I was wondering if we could take a shortcut to a folder in the gen 2 instead of using the full Data pipeline, given that our data doesn't need any transformation before it goes into the data lake? Also, how can I access data frames specified in the notebook for Power BI visuals or reports, or notebooks are merely another data pipeline to ingest (aggregated or transformed) data to the lake house?

@austinlibal 6 ай бұрын

Yes. You could have made a shortcut to the folder in the external data lake but the data still would only be in their native format as stored in the data lake. If you want to use the delta format to truly build a Lakehouse with ACID properties, you would need to write them to the Lakehouse Tables. Notebooks can be used as another ETL tool in Fabric and can also be scheduled with a data factory pipeline as a part of a larger orchestration. While you can visualize data in a Notebook, I do not believe there is a way to access Spark Notebooks created in Fabric from Power BI reports at this time.

@woliveiras 5 ай бұрын

Hi, I have another question regarding running pipelines. Can we run a pipeline based on file modification or new files that arrived in our file folder? I want to run my pipeline if we have a new file or update an existing one. In this case, it is essential to run the pipeline just for new or updated files. Thank you

@austinlibal 5 ай бұрын

This is totally possible through using metadata properties. If the size of a file increases, if the last modified date of a file gets updated. Many different ways to handle this based on the ingestion of your data into your source.

@user-ti5nj3xq4n 6 ай бұрын

My company has some data on Azure SQL that refreshes once every 24 hours. I can write dataflow to bring selected rows into PowerBI. However I want to create some reusable summary tables for which I think Python would be great. Is it possible that in Lakehouse, 1- I create Dataflow that brings data from Azure SQL into a table 2 - Create scheduled pipeline that runs every 24 hours and runs the Dataflow that overwrite my table 3 - Use Pyspark or something similar to create the summary table 4 - Write that table to Lakehouse I am not sure how step 3 and 4 will be automated on schedule? And I am not sure if above is possible at all? Can you please help?

@austinlibal 6 ай бұрын

Thanks for the question! There could be several ways to handle what you're wanting to accomplish with steps 3 and 4. You could as stated use PySpark to write some sort of summary table to the Lakehouse or potentially a different Lakehouse that is used as a serving layer for analytics, aggregations, and summaries. You could also consider just using T-SQL DDL/DML in a Fabric Warehouse to copy data from the Lakehouse to the same type of serving layer in the Warehouse. Lakehouse does not support DDL/DML but the Warehouse does. A SQL view might serve a similar purpose though in the Lakehouse. As far as scheduling and automating this, you could use a Fabric Notebook inside of a pipeline as part of your orchestration, you could run also run a Script activity or Stored Procedure activity to automate the SQL portion of dropping the old table and repopulating it or just updating it with new data. Hopefully this gives you some direction of where and what you need to look for, if you have further questions let me know!

@user-fu3cw5kx5b 4 ай бұрын

I want to learn MS Fabric, am I suppose to learn SQL and PBI before get in your MS fabric just want to check as I am coming from non technical background. If possible can you gude me with Road map.

@PatrickBateman12420 3 ай бұрын

If you want to get certified in Fabric (DP 600), you need to know quite a bit about Power BI including DAX, basic PySpark, and SQL. The following are NOT part of DP 600: Data Science and Real-Time Analytics.

@IGAGENT-sl8fe 6 ай бұрын

......

@drisselfigha3547 3 ай бұрын

You speak very fast. Not every one is fluent in english.

@austinlibal 3 ай бұрын

Thanks for the feedback! I get excited when I talk data! Take advantage of the ability to slow down KZfaq video to 0.75 speed and also turn on closed captioning and it might be a better experience for you!