Microsoft Fabric: How to load data in Lakehouse using Dataflow Gen2| End to End Flow

No video

Microsoft Fabric: How to load data in Lakehouse using Dataflow Gen2| End to End Flow

Рет қаралды 14,544

Amit Chandak Learn Microsoft Fabric, Power BI, SQL

Күн бұрын

Пікірлер: 64

@mohammadaamirkhan7830 3 ай бұрын

I learned so many new things on the Microsoft Febric Platform today. Thank you Sir for your detailed videos on this.

@AmitChandak 3 ай бұрын

Glad it was helpful!

@sudarsana1234 7 ай бұрын

Amit ,The way you are teaching, really easy to understand. will love to explore all 40 videos in this fabric Playlist 👍

@AmitChandak 7 ай бұрын

Thank you so much for your kind words! 🙏 I'm happy to hear that you find the videos easy to understand. Enjoy exploring the rest of the series, and feel free to share any questions or feedback you might have. Happy learning! 😊📊👨‍🏫 Mastering Microsoft Fabric 40+ Videos: kzfaq.info/get/bejne/ppOmY6xmuJPdkqc.html

@sudarsana1234 7 ай бұрын

@@AmitChandak Thank you Amit 😊. I might need your support and guidance for DP-600🙏

@SujathaS-sx3kg 4 ай бұрын

It was clear and concise explanation about fabric sir

@sanumpzha Жыл бұрын

Nice video Amit.👍Loved it.

@AmitChandak Жыл бұрын

Thanks. 🙏 Hope you will like the full series kzfaq.info/get/bejne/ppOmY6xmuJPdkqc.html

@sanumpzha Жыл бұрын

@@AmitChandak will watch the series..👍

@sohailansari9337 Жыл бұрын

Awesome explanation Amit bhai👏, I am little confused between the data lake and warehouse.. Can you do a detailed video about it would be awesome.. 😊

@AmitChandak Жыл бұрын

Please refer Microsoft Fabric: Various Components| Microsoft Fabric Create Workspace, Lakehouse, Warehouse- kzfaq.info/get/bejne/prymqcVesuDce6s.html Microsoft Fabric-What it is, Why Use, Who Should, When and How to use kzfaq.info/get/bejne/qciHga1-xtnUlXU.html

@sohailansari9337 Жыл бұрын

@@AmitChandak thanks

@riteshkumar4493 17 күн бұрын

Great explanation, thanks. Just one thing if we use fabric so in that case Power BI Desktop tool is required or not?

@AmitChandak 17 күн бұрын

Power BI Desktop tool is not required. But in Case you prefer to create report on Desktop, you can use that.

@riteshkumar4493 16 күн бұрын

@@AmitChandak Thanks. It means we can do all things in Microsoft Fabric, whatever we do in Power BI Desktop tool..

@saketsrivastava84 Жыл бұрын

Sir fabulous video, if possible create video on importing data from excel or migrating data from on premise data source into fabric.

@AmitChandak Жыл бұрын

Thanks. Check this video for excel - kzfaq.info/get/bejne/hs9lfNmcrd6ucqc.html

@adilmajeed8439 3 ай бұрын

Thanks for sharing. By loading the data through DFG2, the table names are appearing in proper case but if we do the same process through Apache Spark, the table names will always be in lower case? Any reference point ....

@AmitChandak 3 ай бұрын

Refer to this community.fabric.microsoft.com/t5/Data-Science/Notebooks-writing-tables-with-names-lowercased-how-to-fix/m-p/3753909

@rajushaikh888 7 ай бұрын

Amit, really appreciate your way of teaching... I am not able to find perfect video on Git integration in power bi, version control and multiple users working on same power bi report, how to merge and manage conflicts can you please make video on it.

@AmitChandak 7 ай бұрын

I need to create one yet. But please watch this from guyinacube - kzfaq.info/get/bejne/sNyppZWivqm1fqM.html

@SujathaS-sx3kg 4 ай бұрын

Sir I used create a folder for all measures, as they will be in one folder, whether I can create it fabric or I should create it in desktop

@AmitChandak 4 ай бұрын

You have create tables in fabric and use those as folders. Like we created a table with name measure with one column and no data and used that as measure folder.

@amaia9476 7 ай бұрын

Nice video. Is it possible to write data PARTITIONED through Dataflow gen2 into a Lakehouse? I know I can do it with a notebook after data is ingested from source. But what if I want to read data from source through DF Gen2 and write it into the Lakehouse partitioned at the same time?

@rayromych5534 5 ай бұрын

Thanks for the video I used dataflow to add data to a data lake from web api, which worked fine, but then I wanted to switch from "New table" to "Existing table" to add data every day by refreshing the dataflow, but when I go to edit or recrate destination - it doesnt see the table in the data lake that was created by initial refresh :(

@user-zv5id5kt4s Жыл бұрын

Hi Amit, please, will be dataflow gen 2 available also in regular Power BI or it will be exclusive to Fabric? Thank you!

@AmitChandak Жыл бұрын

I checked as of now it is not available. I have asked the question on forum. I will update once I see any replay.

@user-zv5id5kt4s Жыл бұрын

@@AmitChandakthank you very much!

@sonalimishra3356 9 ай бұрын

Instead of data lake house can we create all in ware house ?

@AmitChandak 9 ай бұрын

Yes, We can do an end-to-end flow on the Warehouse too. Dataflow Gen2, Data Pipeline sqn SQL can write to Warehouse. As of now, notebooks do not write directly to Warehouse. I have covered the flow in the series. Initially, the Warehouse destination was not available to I used an alternative. But now you can use warehouse destination Mastering Microsoft Fabric 35+ Videos: kzfaq.info/get/bejne/ppOmY6xmuJPdkqc.html

@JayPatel-hd8yz 6 ай бұрын

Hello Amit, I am a new member to your channel as I was following the same steps of loading csv file through WEB API options but its fails stating "Files contains corrupted data" can you please update the file. Thanks

@AmitChandak 6 ай бұрын

I tried this file again, and it worked for me. However, some users in the past have also complained that the Git file is not accessible, so giving file at Drobox too: github.com/amitchandakpbi/powerbi/raw/main/Sales%20Data%20for%20Fabric.xlsx Please download this file from Dropbox, and you can upload it to Fabric Lakehouse or OneDrive to use it. www.dropbox.com/s/z9d6qfah6451ct3/retailData_fabric.csv?dl=0 www.dropbox.com/scl/fi/ru5jusnq4c89bsbhganm0/Sales-Data-for-Fabric.xlsx?rlkey=e62al8qyleagpfm4ryggtwkvx&dl=0

@nissimelaluf1272 Жыл бұрын

Is it possible to Query the SQL Exten about calculated Measure as well like the actual columns ?

@AmitChandak Жыл бұрын

I doubt you can query measure from SQL as of now

@sonalimishra3356 9 ай бұрын

I have no data on cloud,what ever i am using that is on sql server ,and have created gateway connection for this,so when i am creating dataflow gen 2 through sql server ,using that on premise gateway getting this error " The selected connection has an on-premises data gateway that is different than the project's gateway. Committing will apply to all queries in this mashup" what to do next?

@AmitChandak 9 ай бұрын

Please check this video in the series. kzfaq.info/get/bejne/pat2YM15u9PalKM.html Always use latest on-premise gateway. If Dataflow uses already existing connection and not working, Then use new connection and update the details again.

@samirsahin5653 Жыл бұрын

Thanks, nice video. How to create a scheduled refresh? When I create a scheduled refresh on Dataflow gen 2, it creates new table every time it refreshs. Basically what I want is, import data via Dataflow Gen2 into Lakehouse, transform it (on SQL endpoint, notebook python or Power Query as you done) and then use this data set on PBI. Everything is okay but can't manage the refresh data. When I refresh dataflow gen2 it creates new table which power bi and transform process is not connected to this new table.

@AmitChandak Жыл бұрын

Refer if these two incremental approaches, if these can help Microsoft Fabric Part 15: Incremental Data Load(ETL) for Warehouse using Dataflow Gen 2, SQL Procedure, Data Pipeline: kzfaq.info/get/bejne/o9Z2hdGLp77XpoU.html How about incremental without Dataflow, Only using pipeline and SQL procedure for Microsoft Fabric Warehouse Microsoft Fabric Part 16: kzfaq.info/get/bejne/p9l_fMmF39Cng4U.html

@samirsahin5653 Жыл бұрын

@@AmitChandak Thanks for super quick replay, But I'm a bit confused. What I want to do is refresh table. Why should I go and deal with SQL? Is it not possible just refresh table in Data Flow Gen 2? When I run Dataflow Gen 2, it creates new table because Dataflow Gen 2 that we created is getting data from a file. I think we need to update data via Dataflow Gen2, not get data. Can we not do this without SQL?

@taniyaganguly 22 күн бұрын

can you help to write query to see datatype of columns from table in fabric

@AmitChandak 22 күн бұрын

In case of Lakehouse or Warehouse use SQL End Point and try this query SELECT top 100 * FROM INFORMATION_SCHEMA.COLUMNS

@taniyaganguly 22 күн бұрын

@@AmitChandak it worked thank you

@Akn876 Жыл бұрын

Hi Amit please make a video on how to create power bi data warehouse and difference between lakehouse and datawarehouse

@AmitChandak Жыл бұрын

Warehouse I will load tomorrow. For difference refer Microsoft Fabric: Various Components| Microsoft Fabric Create Workspace, Lakehouse, Warehouse- kzfaq.info/get/bejne/prymqcVesuDce6s.html Microsoft Fabric-What it is, Why Use, Who Should, When and How to use kzfaq.info/get/bejne/qciHga1-xtnUlXU.html

@mohammadaamirkhan7830 Жыл бұрын

Thank you Sir for making this video. However, I am getting error while at video 32:50 when I add implicit and explicit measure like Gross, Discount in visual in Power BI Web. It is showing error like "Can't display the visual - couldn't load the data for this visual. Please try again later or contact support" Could you please advise on this?

@AmitChandak Жыл бұрын

Please log an issue - community.powerbi.com/t5/Issues/idb-p/Issues. I faced that in past with couple of measures.

@mohammadaamirkhan7830 Жыл бұрын

@@AmitChandak Thank you once again for your advise.

@sonalimishra3356 9 ай бұрын

If we are using sql server to get data and if it have space then what to do?

@AmitChandak 9 ай бұрын

You can rename those columns in Dataflow gen 2(remove spaces), if destination is Lakehouse.

@sonalimishra3356 9 ай бұрын

@@AmitChandak even we can use ware house also right

@Laughing_out_loud_ Жыл бұрын

I was following this video till 23:03. Created a lake house but accidentally clicked on "Publish Later". Its been over 12 hours and I do not see the tables I added to the Lakehouse. These are the tables from my On Prem SQL server and are fairly large in size. Now I have a couple of questions: 1. What advantage I will have of keep these on the Lakehouse v Keeping them on SQL Server VM? 2. Can I also add Views from SQL server to Lake house and they will perform process data from their respective tables in the Lakehouse? 3. To me its seems like Fabric provides an alternative to everything I used to do with SSIS, SQL server, SQL job agents etc. So if my current system is working why do I need to shift to fabric?

@AmitChandak Жыл бұрын

On the three dots near to Dataflow gen2 name click and check Refresh history , if refresh is not running of failed. manually run it 1. What advantage I will have of keeping these on the Lakehouse v Keeping them on the SQL Server VM? Lakehouse is on the cloud and saves data in Delta parquet format, also it Read only SQL end point, you can use to query data in SQL. Power BI dataset do not copy data, so only one copy 2. Can I also add Views from SQL server to Lake house and they will perform process data from their respective tables in the Lakehouse? If you want to import yes. If you want a shortcut, no as of now 3. To me its seems like Fabric provides an alternative to everything I used to do with SSIS, SQL server, SQL job agents etc. So if my current system is working why do I need to shift to fabric It is a complete analytics system. It can replace many existing systems like SSIS, SQL server, ADLS gen2 etc

@SujathaS-sx3kg 4 ай бұрын

Sir can I create a new table for measures in power bi fabric lakehouse, if yes can you explain the steps

@AmitChandak 4 ай бұрын

Create an empty table. And move you measures there and hide the column of the table. For Lakehouse use Spark SQL for Warehouse you can Use SQL endpoint(Warehouse) Create table Measure (column1 varchar(5))

@SujathaS-sx3kg 4 ай бұрын

How to create empty table sir

@SujathaS-sx3kg 4 ай бұрын

In power Bi fabric I am able to create a measures, but not able to create new column as we create in powerbi desktop. Please explain how to create a column in power bi fabric

@ankittripathi8305 Ай бұрын

hello, I have added data into lakehouse through Gen2 from sharepoint and then want to model the tables, however for two tables, I need many to many relationship, but there is no option for this, how can i achieve this in fabric? Also if I connect the model to Power Bi desktop and want to change the relationship to many to many it would be converted to import locale mode, however i want it in direct/live connection, your inputs please thanks

@AmitChandak Ай бұрын

Please create a custom semantic model; you can have it there. But be careful, as many-to-many relationships sometimes do not work as expected. I have already shared this feedback with support team.

@ankittripathi8305 Ай бұрын

@@AmitChandak thanks for reply actually we are migrating old semantic models to fabric, yes we will try to be careful with many to many relationship as do we in past. Thanks amit

@Praying_Mother Жыл бұрын

Hi Amit, i am getting an error message "THE GIVEN DATA SOURCE KIND IS NOT SUPPORTED" can you help me out please?

@AmitChandak Жыл бұрын

Please share more details with me. You can ping me LinkedIn - www.linkedin.com/in/amitchandak78/

@user-pj5xt7zd1s Жыл бұрын

Hi Amit. Great Job. I have one doubt I am getting below error, can you help me with this? Error Details: Error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Microsoft SQL: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) Details: DataSourceKind = Lakehouse;DataSourcePath = Lakehouse;Message = A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server);ErrorCode = -2146232060;Number = 53;Class = 20 GatewayObjectId: 6309202d-8f81-4235-9651-68924de52acf. Request ID: 490256f4-86ec-4c9b-a71f-83515aa6f59f. (Request ID: c5b3731b-2a50-4ee7-a227-03045b44d390).

@AmitChandak Жыл бұрын

Is it Azure SQL or SQL server? If this on-premise SQL server Microsoft Fabric Part 17: Load local SQL server data in Lakehouse using on-premise Gateway and Dataflow Gen2: kzfaq.info/get/bejne/pat2YM15u9PalKM.html For last few days this one was not working (Using on-premise gateways)

@user-pj5xt7zd1s Жыл бұрын

@@AmitChandak Thank you so much, I am using an on premise gateway, and as you said its not working for last few days. I ll go through the video, Please do make more videos.