Would love a comparison with "the regular way" with an azure sql and adf in terms of performance and cost. highlighting which cost drivers will lead you down the lake house route and which performance considerations you'd have with power BI on top

Yoooooo! I didn't know you guys did Synapse videos too! I been making the shift from developing on prem DW to trying to implement a more modern cloud data lakehouse architecture. What I was hoping to see was how to build out the different layers... and how to implement the delta loads on that. Trying to wrap my head around all of this. It is a paradigm shift to say the least.

Thank you for this video. It's been very helpful to see these possibilities using parameters! I'm looking forward to the rest of the series with Stijn!

it is really nice, but really some of the workflow for ADF "Pipeline" is really unintuitive. I mean creating the same parameter 3 times !!?

Wicked video 👍👍. A great intro into how easy DF can be. I'll be watching the rest in this series for sure!

Wow your enthusiasm is infectious Patrick! I am not familiar with Azure but your clear explanations help decipher this new maze of content. Thank you both. It all helps.

I really like the new content. Please keep it coming.

Definitely looking forward to this series.

"Hey this thing is really quick and easy".... 3000 settings and configuration changes later.... "See!"

I’ve been happy doing this with SQL Server and SSIS and loading everything in a tabular model for analysis. But ever since we implemented Dynamics 365 in the cloud the only way out for the data from Dynamics is into a data lake. This series has come at the right time cause it’s exactly what we’re trying to do.

Amazing. I truly Appreciate the content.

Thanks for choosing the use case demonstrated. I am sure it would help a lot of us in building generic pipelines. In your next video, it would be nice if you could highlight advantages of using Data Lake v/s RDBMS. I mean why would someone give up RDBMS for data warehouse and go for data lake, which stores data in file system in .parquet format. Sorry if you covered it and I missed it.

Hi can you make a video on how to capture the data from powerbi service to SharePoint list

Amazing content!

all about pumping more data from on-premise into the cloud, is it not? but why?

First - I love you guys, great video as always but....as they say, devil is in the details so just wanted to provide some context for these technologies. Yes this is the new landing area pattern and it's wonderful for dynamic source acquisition. No more BIML to do this! And just want to say that the dynamic landing area ELT is also of course wonderful. But (so many buts here)....just wish there was more context provided for this as it could be misleading on what it should be used for. First, you can do the same dynamic landing using ADF so this is not unique to synapse pipelines. And the dynamic capability in of itself has nothing to do with datalake per se. The data lakehouse is really the whole synapse serverless sql pool, not synapse pipelines. Two completely different things. And finally, in reality is if you try to use datalake as your final landing area you will get a real nasty surprise when you try to join datalake files together when loading integrated targets such as DW (whether you're reading straight from the datalake or via serverless sql pool) as the performance is bad....I mean really bad....compared to using Sqlserver tables. Remember, in the end these are just flat files in the datalake not indexed tables. Why oh why would you take sqlserver source and put it in flat files? If you have a compelling reason fine but as general rule that's a pretty bad idea folks. We are using data lake only because D365 sources now support sync'ing with datalake files. But then we pull it into our sqlserver landing area where the real work is done. It's SUPER cool with synapse sql pool ability to run Sql queries against flat files and treat it like regular database source. But it's not practical from performance standpoint to actually use that for anything other than a source for the real sqlserver landing area as it just can't scale well as flat files.

Q Hello Patrick how can I delta load from multiple on Onprem SQL Tables in to Azure data lake

How's the performance on the sql server source? For big instances, I'm imagining some long run times. Also, how are deltas and change handled? Sorry to inundate with questions but it also looks like there's a prerequisite for having an on prem integration runtime as well?

Nice! How did it build out fact and dimension folders in the lake? How would it know what table (dim or fact) to put under each folder?

I am not sure if you covered the difference between Data Lakehouse and Data lake . Would love to know about the difference especially in Azure environment

Thanks for this video and blog post, very usefull. One question about the medallion arquitecture. Is mandatory that gold layer resides in the lakehouse? In a "smaller" project I want to send data from silver in lake to gold in azure sql database. Technically, I know is possible. But I want to hear opinions from the design side. What do you think?

I've followed the steps exactly until 6:05 but then the "databasename" parameter doens't show up. It still looks like 5:42. What have I missed? I guess there are some extra steps that are not shown here or the cutting was bad. The source dataset suddenly has another name at 6:05 - what's going on there?

Quick question, for the merging strategy do we need to use it on the bronze or silver layer?

I'm a bit confused by the multiple files per table and how these would be used. At 13:54, you show three files in the Customer folder. Given the dates, I assume each is a complete snapshot. You remark that you don't care about the file names. But when it's time to use the file, you obviously only want to use a single file so as to not duplicate data. Programmatically, this would seem to be difficult without fixed file or folder names. For example, if you needed to build a pipeline to copy data into silver, how would you select the most recent file? Or if you built a notebook to analyze the data, how would you reference the most recent file regardless of when you used the notebook?

If you're connecting to an on premise db, aren't you supposed to connect to sql server as your source dataset?

Is the copy activity going to support "Delta" as a sink?

How would you handle having to connect to multiple SQL Servers as your source data though?

Like Patrick, my mind has just blown!!! WOW!!! I'm assuming the same can be done in ADF and a basic ADLS v2 storage account if you can't afford Azure Synapse?

Great for the first cut of Data Source ... what happens when a column is added removed or modified in source. Select * gets the data but won't the schema be different?

to make the example work you needed to install azure integration runtime and java runtime environment on the computer, is that correct?

amazing

Way too fast guys... what's the rush?

I would like to see user access grant on Sql server end.

8:13 Batman is that you?

How you’ll version control this and deploy from one env to another?

I wish MSFT would stop calling Synapse and datablake a lakehouae. it's obvious it wants to take Databricks' market share. But synapse and a data lake is not a lalehouse. It's a deliberate attempt to confuse.

This is nothing new. You have been able to do this before with an on-prem file systems and an on-prem sql server -> it's the tail of "the emperor's New Clothes". You will also loose all meta data from sql server when you put the data in the cloud file system.

Surely not the best video you have produced mate. Expectation was higher . Not saying that complete waste of my time though!!

Too complicated, not good for beginner.