Yep, absolutely - I've built way too many SQL frameworks myself. But I differentiate between the code generators (my engine builds SQL at runtime/deployment), dynamic SQL (my code cannot be followed without lighting a candle and performing a ritual sacrifice), and actual dynamic, parameterised workflows in a programming language (scala, python, C#, whatever). I'm likely using the wrong terms, but I think it's important to differentiate the step change of how we think about our ETL between those approaches.

I having meta data driven framework as well. It make maintaining the pipeline way easier for me.

@@AdvancingAnalytics This "my code cannot be followed without lighting a candle and performing a ritual sacrifice" made me Literally roll on the floor laughing, Black magic and all taken into consideration ill jump back to my DOS scripting days %1 🤣

Could you explain how databrick could be used instead of fabric? Sounds really interesting, but I don't know that framework...

It’s something I’ve leveraged in snowflake in the past to be able to modify two tables in a transaction and have a rollback if it fails. In reality though it’s not that hard to live without it.

It should be so, it is just an abstract layer on top of Gen2. Also the storage is probably created on the fly, how you used data, same as databricks, but when you do pricing, you can estimate Gigs necessary and see. It is very cheap, like 0,024 for 1GB.

Oooh, that's a Microsoft question. I can take a guess - given SSAS is largely used to get around model size limits in Power BI, and DirectLake pretty much removes any model size limits... The argument for SSAS doesn't look great. But I honestly don't know, there might be other factors!

I agree - we havent carried out an SSAS implementation for a couple of years. PPU removed any need for it. Fabric seems to put the nail in the coffin. The only potential for consideration is where microsoft offered x number of pro licenses as a consequence of another microsoft implementation (dynamics).

Count me in as a Fabricators!!

Mount up ....

Yep. Each of the experiences can read data from the other. So you can read from a warehouse and merge it into a Lakehouse, as it's only one of the types where the write is being performed. What you can't do, is write into both a Warehouse and a Lakehouse at the same time, using the same workload.

That's a full migration path to help move your code from Dedicated into Fabric Warehouse, from what I've heard it's a big ambition to make the migration as smooth as possible. And the Delta from Warehouse is full Delta, with all the trans log

​@@AdvancingAnalytics So sun is setting on dedicated then. This is not unexpected tho, giving that MS rarely put any update on dedicated SQL Pool lately, but having to migrate out of Azure entirely is quite hard sell. Will there be equivalent product in Azure release as well? Coz the way PBI governance and permission control is not the best way to apply with IT infrastructure IMHO.

Exactly, so if you're happy with Spark SQL and don't need full T-SQL, the Lakehouse side of the fence does what you need

Hola! We'll do a video digging into how capacities are managing but at the simplest level - you pay for a certain size of SKU, similar to Power BI Premium. P1 now being F64. This gives you a certain number of v-cores you can use, averaged out across the day. For an F64 that's 128 cores (edited: originally said 32 cores, forgot it was double not half!) So you might use a mix of spark, SQL and power BI throughout the day. Or you might use a huge amount of spark for an hour or two, then nothing for the rest of the day. The capacity is measured out across 24 hours to determine that you're within your capacity. You can put guiderails in to determine how many executors you're using in spark etc, but it all goes against your central capacity allocation. You can also separate capacities out - give the business one capacity for their reporting etc, whilst the engineers use a different capacity to produce the datasets. There's more to it, with other consumption methods and configuration, but that'll wait for the video!

Quick update - edited the previous reply, I'd put F64 as having 32 cores, it's actually 128, ie: double cores not half! Serves me right for answering questions from on holiday :D

@@AdvancingAnalytics Thanks, Simon. I really appreciate your (your team's) dedication, professionalism in dealing with details, and straightforwardness in commenting, along with the humor you bring to it :) Also, the fact that you go back to correct statements in your comments shows that you care about quality and brand. Hats off to you. I will definitely try my best to understand the "power" and division of this "power" in terms of the entire capacity. Benchmarks will be crucial in assessing what we can achieve with each SKU and how long it will take to process this and that. And, of course, how to design mixed workloads to avoid interference, queues, resource QoS battles, and similar issues. There's still a lot ahead of us, and there's still a lot ahead for Microsoft. We'll see how it ends :)

Hah, nope, nothing deliberate, just didn't have time to go into connectivity for each object :)

@@AdvancingAnalytics Okay :). As you probably already figured out, I just find it to be another point of confusion for people trying to make out a difference between the warehouse and lakehouse workloads.

As far as I know the sql endpoint in the lakehouse is read only, while the one in the wharehouse is read and write

I was expecting something new for dedicated sql pool but didn't expect this big that I have to move out of Azure.

@@chasedoe2594 to be fair, the dedicated pool technology base is from the MPP Datallegro appliance......and going back the early to mid 2000s. Simon could keep me honest on that date range though.....with Teradata I believe being the originator of the mpp tech

​@@NeumsFor9 I'm fairly sure Tandem NonStop got in there first with MPP. During the 90's, it already had disk processes that performed SQL predicate filtering (selection & projection) using dedicated memory cache and it an X & Y communications bus fabric for dual path reads and writes. That was baked in for fault tolerance, but came to the fore when the SQL engine was deployed into the OS post flat file (Enscribe, the equivalent of VSAM). All of this gave it the unique ability to hot swaps CPUs and add them in, without the need to reboot. You could have 16 locally configured paired CPUs andif I recall it was technically feasible to run that out to many hundreds of distributed nodes. This is the timeline, not sure where Teradata fitted in, would be good to know. 1970 - Information Retrieval whitepaper written by EF Codd published ​ 1970 to 1980 - Ingres research project at University of California, Berkeley. The codebase, via BSD license, spawned Sybase, Microsoft SQL Server & Tandem NonStop SQL​ 1979 - Oracle was first commercially available product​ 1981 - IBM SQL / DS, now known as DB2, spawned from System / R​ 1987 - NonStop SQL released by Tandem Computers, ft. parallel processing & fault tolerance​ 1989 - SQL Server 1.0 released by Microsoft, Sybase & Ashton Tate​ 1995 - SQL Server 6.0 first version without Sybase involvement

And that's a perfectly valid approach! Hoping we'll see closer integration with Unity & OneLake as both sides mature!