Understanding OneLake within Microsoft Fabric

Рет қаралды 68,633

Күн бұрын

The key to Microsoft Fabric is definitely OneLake! OneLake enables the reuse of data within Fabric to help drive all of the engines. One copy of the data! Josh Caplan joins us to help get an understanding of the power of OneLake.
OneLake, the OneDrive for data
learn.microsoft.com/fabric/on...
OneLake shortcuts
learn.microsoft.com/fabric/on...
OneLake File Explorer
learn.microsoft.com/fabric/on...
Josh Caplan:
/ joshcaplan1984
/ josh-caplan-2139a3a6
📢 Become a member: guyinacu.be/membership
*******************
Want to take your Power BI skills to the next level? We have training courses available to help you with your journey.
🎓 Guy in a Cube courses: guyinacu.be/courses
*******************
LET'S CONNECT!
*******************
-- / guyinacube
-- / awsaxton
-- / patrickdba
-- / guyinacube
-- / guyinacube
-- guyinacube.com
**Gear**
🛠 Check out my Tools page - guyinacube.com/tools/
#MicrosoftFabric #OneLake #GuyInACube

Пікірлер: 55

@lwhieldon1 10 ай бұрын

This is so cool! I can see this being a quick but powerful win for small businesses that want to harness their data without having a ton of overhead in getting services up & running!

@lopypop Жыл бұрын

Wow! This was the best explainer so far! I'm interested in tracking when other external sources will be available via shortcuts because my organization keeps data warehoused in other (non-Amazon) sources.

@wmcnabb Жыл бұрын

Adam and Josh - thank you for the demo! As always...awesomeness has been extracted.

@hurikhan Жыл бұрын

It has not been extracted, It was just linked. :D

@jenlam4466 Жыл бұрын

Hi Adam, it would be great if you can do an end to end demo on how to use Fabric based on a company scenario. 😉

@GuyInACube Жыл бұрын

We are working on putting some things together. stay tuned.

@anuragbawankar685 Жыл бұрын

Thanks a lot Adam and Josh !!!

@Bharathrs1985 Жыл бұрын

What's the performance impact of having data as shortcut compared to having these in one lake when we use it in different engines especially direct query on power bi

@phyberoptx Жыл бұрын

Following.

@marienaalvarez8235 8 ай бұрын

I have the same question. Have you find a response?

@stevenar5316 7 ай бұрын

Same question.

@jamieclarke2694 2 ай бұрын

At a guess, copying the data into one lake is better for data manipulation. Transformations, and selects etc. Referencing the tables as a shortcut is akin to using a server less SQL instance with external tables. Physical data moving, indexing, and keys may not be possible. I'm guessing here though based on limited knowledge currently.

@jason.campbell474 Жыл бұрын

Can a shortcut be established to an on premise SQL server? Or, is it possible to sync OneLake with (on-prem) SQL server tables?

@chrisjm77 Жыл бұрын

Looking forward to the response to this question.

@jamieclarke2694 2 ай бұрын

There's an on prem SQL Server option to copy data into a lakehouse in fabric. I'm not sure how that relates to one lake as I've just started this video and still getting used to Fabric terminology

@jamieclarke2694 2 ай бұрын

I guess using a data gateway like you would from Power BI service.

@vantuandang7761 Жыл бұрын

Does anyone know if DirectLake is as performant as the import mode? As far as I know, each workspace is located in one region: do we pay cross regions transfer costs when using a shortcut between two workspaces located in 2 different regions?

@kingnemmi9686 Жыл бұрын

Thank you so much for the demo!! I have a question regarding a potential use case for this. I have a client that has multiple data locations, they're still moving into the cloud scene however they currently utilise an Access Db for a particular set of data. If this db was hosted on a vm within azure, could OneLake potentially reference data from Access Db?

@bladerunnerisback Жыл бұрын

Great stuff, thanks for sharing. Any chance to have a shortcut for Snowflake coming soon?

@sandeepbarge4699 Жыл бұрын

Is my understanding correct that structured data is stored in Onelake in .Parquet files? If so how efficient is it as compared to storing data in RDBMS like SQL server?

@zubingidwani998 Жыл бұрын

Do you see anything on the horizon to allow an ISV to provision a onelake, setup domains and grant external guest users (the ISV's customers) access to different paritioned datasets? In otherwords is there anyway for an ISV to grant lakehouse to their SaaS customers?

@FredericLEGUEN-Excel 8 ай бұрын

It looks fantastic. I have a question. I use to store my data in Azure SQL. Is it better to migrate my data from Azure SQL to OneLake or it's better to stay like it is now.

@inkuban Жыл бұрын

If we use a different source other than S3 like ADLS for example can we still load the datasets directly from those external sources within our Power bI REPORT.

@mnhworkdev1652 Жыл бұрын

Can a on-prem SQL server be set up as a shortcut by using a remote gateway? This would ensure that our data at rest would still be on-prem.

@phyberoptx Жыл бұрын

Do we have an eta on the shortcuts for Dataverse? Will there be shortcuts for Business Central as well?

@martinbubenheimer6289 8 ай бұрын

I like the shortcut table feature! Is there also a way to create shortcut tables to REST APIs or OData web APIs so that I can get the latest data from source systems like Sales Force or Business Central without copying the data on a schedule into OneLake where it will eventually get outdated or is missing columns I need at all?

@michalskiradek85 Ай бұрын

Hey Adam, does using one lake and direct lake storage mode in power bi semantic model allow to connect model to really big tables (hundreds of millions of records) while keeping visuals and dax fast and efficient? Thx in advance!

@brenp403 7 ай бұрын

Amazing vid. Very informative. Question: Is it possible to have a deeper dive into the inner workings? All the Marketing and Sales sessions/materials/etc. repeat the line (or something similar) that: "it's one OneLake and you can have one Fabric on top". Josh mentioned in this video that whoever owns (uploaded) the data, say to your store DB, continues to own it and the OneLake consumer gets a copy. I'm curious on how this is actually working. The last thing I want is our data and analytics team "connecting" to our application data stores and running crazy queries on our operational data. Given Josh said it's copied data, I'm curious to know more and what that means for debugging missing data, eventual consistency, atomicity, etc. Thanks!

@adolfojsocorro Жыл бұрын

Hi, I understand Premium gives you access to Fabric's features. But is OneLake priced separately?

@ncbshiva Жыл бұрын

What is the difference between data in onelake and data in adls gen2 datalake ?

@azilyso Жыл бұрын

When clients (say power bi) consuming a shortcut data source, data is transferred to one lake first, then forwarded to clients, right? If it's true, it sounds like copy the data eveytime consumes.

@joanguerette Жыл бұрын

What if we have a DataHub, can we connect the MS Fabric One Lake to our DataHub ? Do we have to do data governance in 2 places ?

@kr6so 3 ай бұрын

Once you connected a data source in OneLake data hub, how do you disconnect and connect to data source not in OneLake?

@KichereTheDataScientist 28 күн бұрын

what tool do you use to bluehighlight in your videos?

@kb1629 Жыл бұрын

One lake to rule them all!

@mmastro96 Жыл бұрын

This is great

@GuyInACube Жыл бұрын

Glad you liked it! 👊

@user-hu5mv6oq7l 8 ай бұрын

can we create a dashboard of real-time streaming data without using network (internet)

@how3recordings Жыл бұрын

So this is like synapse, whats the used case for One Lake rather than Synapse?

@dagg497 3 ай бұрын

All files are converted to parquet and then delta parquet. So basically you have to build an ELT framework for handling this... Really anoying it isn't done automatically and then generate scripts.. Like 10% extra developer time comñared to a storage blob into sql database - Sigh 😢

@rickybobby6548 Жыл бұрын

How about cost and license structure?

@datadevlab 11 ай бұрын

Hi everyone! I really didn't get the idea of OneLake? Isn't it another Sharepoint?

@sakurai2601 Жыл бұрын

what's the difference between onelake and onedrive ?

@tomfisher1099 Жыл бұрын

wait, this video showed there were GIAC socks for sale, but searching all over the store I see no socks. That's bananas.

@BobWilliamson-to3xp 3 ай бұрын

So how much does this cost and can I get free datasets

@narayanaou Жыл бұрын

One Lake seems like Data Virtualization

@nigelnaicker7948 Жыл бұрын

not convinced about this yet, shortcuts are nice in theory, but performance will never be as good as if the data was imported.. secondly, look at the video at 9 min and 25-30 secs , the insert took about 5 seconds. i know we are nerds and we like new shiney things, but lets balance that with being practical.

@jacksorjacksor Жыл бұрын

Slightly out of sync "Yo" at the beginning of the video. Literally unwatchable. p.s. brilliant content, thank you so much for providing so much insight and material into such a new product

@Phoenixspin Жыл бұрын

Who died on Adam's shirt? I'm sad.

@amithmihiranga1913 Жыл бұрын

F this ran over my head. I get the dataflow but not data lake. Hmm..😢

@dagg497 3 ай бұрын

Datalake and delta lake sucks imo. It's not drag and drop really. If you load incremental changes from a source like SAP or Oracle or API. You have to build an integrator in Data Factory, SQL scripts in Synapse to convert to Parquet files, do Transformations, Implement a Delta logic and Save in Delta Parquet, Then transform the data again, Build views, and then you can consume the data. I much prefer Data storage gen 2 or Storage Blobs and a regular SQL Database and do the transformation in Synapse.. Now you are reliant on Data Factory+Synapse+Parquet logic+OneLake+Synapse+Synapse or SQL serverless And It is super slow to read Parquet since It's basically thousands of KB size files to merge like heap pages in an SQL database.