Hi @bumdinh9911 per Bradley, "Hello! That is a great question, sadly I could not find a way to do it at this time. The closest way I found was to trigger the Job scheduler API for Microsoft Fabric and you could run a job from a pipeline, but there are limitations and one listed was passing a parameter via an API. Now, just to play the other side. While it is not possible you could write the notebook information to a DW or to an Azure SQL Database table, and then do a look up operation to get the data. So we could accomplish the task of getting data from a notebook back into a pipeline, but it is not as straight forward as I was hoping. If that changes, or I should say 'when that changes', I promise you I will make a video on that!"

Hello Garima.. thank you for reaching out.. you can go to Fabric capacity under your Azure portal to create one under the same subscription & Tenant...you can create in any of the subscription type

Hi @dankboii per Bradley, "Not yet. Yet being the key word. But as of right now this is specifically for Azure Blob Storage only."

Thank you for watching @robcarrol!

We do don't we? I did a more recent video on Elastic Jobs was a wee bit less when it comes to moving pieces.

Gotta love that Fabric Magic!

Thank you Kirby! We really appriceate you and all your great work!

Glad you liked it my friend. Keep come back to the channel for more great content by the team.

Thank you @TheSQLPro!

Hi @claudiovasquezcampos9558 are you looking for this in a Spark SQL Statement in a Notebook or a T-SQL Statement as part of a data pipeline task against an Azure SQL DB, Azure SQL MI, SQL Server, or Fabric DW?

Hi @fazizov per Bradley, "Hello sir! If I'm correct in my understanding you have a Fabric / Power BI Tenant and the Azure Account is not associated with that through the M365 Tenant. The Azure Account that you want to Mirror must be associated witht he M365 account (not an M365 expert, but this is what I've been told). If that is the case the Mirroring should not be limited to Trial capacity, you should be able to Mirror into it. The Trial capacity is equivalent to an F64 so you should have the functionality of an F64 other than those items that have been called out as not included in the trial capacity like private endpoints and managed identity. If your Azure Account is associated with the M365 account your tenant is in please let me know, Mirroring should not be facing any other restriction."

Thank you very much for detailed explanation,I will definitely try that advice!

Absolutely love this suggestion @ismailbartolo9741 , we will get Bradley to start working on this!

​@@Tales-from-the-Field🔥🔥🔥 thanks 😊

Hi @abeerahmed5634 could we get a little bit more information. Are you building text from Lakehouse fields, or is it plugging in numbers. You may not be able to get to specific, but trying to understand what parameters we need to define to send from Notebook1 into the child notebook.

Your very welcome. TY for coming to the channel. The team does utilize Camtasia for all our video editing. If you keep an eye out they drop good deals around holidays.

@@danieltaylor6623 Thank you very much Daniel!

Hi @777bashir Yes this is completely possible! Bradley released a video on this just 11 days ago, check it out here: kzfaq.info/get/bejne/obOGp6ulxNecoH0.html

Hi @terryliu3635 per Bradley, "Yes Terry, you are correct. This is just for the Data Flow Gen 2 process in this video. Ok, scratch my previous comment. I realized that link goes to the data factory page. On the Public Road map it says 'In the future (Q2 CY2024), we'll also introduce Copilot for Data Factory in Data Pipelines. ' here's the link to that: learn.microsoft.com/en-us/fabric/release-plan/data-factory#copilot . The good news is we are currently in Q2..... so hopefully we will have this soon!"

Hi @zongyili569 per Bradley, "Just want to make sure I understand this correctly. We've Registered the Microsoft.PowerPlatform resource provider, provisioned a Virtual Network and set up a subnet for the Microsoft Fabric tenent to use and we delegated the subnet to the service Microsoft.PowerPlatform/vnetaccesslinks, went to our Fabric Tenent and when we selected Virtual network data gateways, we had a License capacity, then you could select the Azure Subscription and the resource group, and the Vnet was provisioned in and it didn't show up? My initial thought would be that it could be a tenant where they Azure Subscription is not attached to the Fabric Tenent, but if you can see the subscription and the resource group, then you may need to call support. I'd love to hear from you on this to see what the resolution is. "

@@Tales-from-the-Field Hi, it was my mistake. Fabric pipeline doesn't support vnet data gateway right now. That's why I can't see those vnet data gateway connections.

Hi @OneNI83 per Bradley, "Look at this much like Azure Data Factory. Use this for your initial fork lift, based on your comfortability with this and other technologies. If you are a T-SQL Warehouse person, land it in files and do a CTAS to load or load it directly to a warehouse and then use T-SQL to Transform your data. If you are spark person, land it in files and use a notebook to transform and load your data. This is a powerful tool, but there's multiple ways to ingest data after it lands. So super long answer for a short questioni, Yes you can use this to bulk load. Not sure, there's not really a limit and you could scale the parallelism to increase throughput for VLDB workloads."

Hi @adilmajeed8439 per Bradley, "Hello sir, it's a little hard to troubleshoot this on KZfaq. Could you hit me up on Linkedin and we could DM with screen shots?" www.linkedin.com/in/sqlballs/

Hi @mathiassaint per Bradley, "Sure thing. So for this demo what I wanted to showcase was that no matter what format the data is stored in with Microsoft Fabric we can access it using any Engine we would like. So why did I use KQL DB instead of the SQL Endpoint? I could do that. However, this data set is very linear in nature and I'm using Direct Query. The KQL Engine works really good with time series data, I could create T-SQL stored procedures to accomplish something similar but KQL's query capabilities for a specific fraction of time that grows as I write more data to the database allow me to get this quickly in a way that will not deteriorate over time with volume and remains lightning fast. Could be nothing more than preference on my part, but if I were building this for custmer use that's the engine I would recommend for this workload. Thanks for the questioni!"

Hi @michamatracz706, "Thank you! We've actually got some content we are planning to do with SQLalchemy. That package has a lot of great stuff to it. Hope you stay tuned!"

Bradley how did you get the max bubble in your Power BI working like that?

Hi @NicolasPappasA per Bradley, "I'm breaking building this report a bit more in my next video. For the max size I used a measure to make the most recent bubble bigger than the others. I put that measure in the size option for the ArcGIS Map. The colors were a conditional statement I did not cover, but I used the values to make the trailing dots yellow and the most current dot Red. I will try to expand on this the next time I'm putting a video together on the ISS. I'm not done, hopefully, I've got some fun stuff planned for this!"

@@Tales-from-the-Field amazing... looking forward to it...I created a measure too with a max latitude but couldn't get the way you did...sounds like I am missing the conditional statement, going to try that...Thanks again for replying to this

Thank you @pawewrona9749! We are BIG fans of the KQL DB over here. Agreed, it doesn't get nearly as much love as it should!

Hi @acxhgg per Bradley, "I hate to start off with saying it depends, but kinda. You only have 4 data types for parameters that can be passed from Pipeliens to Notebooks (currently). Int (integer), Bool (boolean), Float, or String. You would need the data type of the parameter is string. However you could pass some type of small JSON array in text, and then have the notebook parse the string to make it an array, but it wouldn't be native. I hope this helps!"

@@Tales-from-the-Field Thank you for your answer ! I tried implementing a logic that would convert data from an array to a single String variable. My current use case is to retrieve latest JSON files from a specific folder of a lakehouse, and to call a notebook with these latest files as parameter, in order to do certain actions on my JSON files. Hopefully it'll be native soon !

Hi @dagg497 per Bradley, "Hi dagg497, not sure what this comment is to in regard to the video. For the Data Flow Gen 2 you wouldn't really use this to load ADLS Gen 2 as this is a transform mechanism to land data into OneLake, Data Warehouse or Data Lake. Pipelines do have ADLS Gen 2 as a destination and they are used more for data movement."

Hi @andreablenxcacopardo5359 per Bradley, "Do you have customer managed keys set up? By default all databases in a Azure SQL MI Instance use a Microsoft Managed Key for TDE encryption. Copy only backups cannot be taken using a Microsoft Managed Key. So you must do one of two things: 1. Create an Azure Key Vault, Create a Key, and then associate that Key with the Managed Instance. or 2. (WARNING NOT THE BEST PRACTICE!!!!) Unencrypt the database and drop the TDE Key. If you have done those things and you are still getting an error review the following documentation to make sure the Azure Key Vault can communicate with the instance. I hope this helps!" learn.microsoft.com/en-us/azure/azure-sql/database/transparent-data-encryption-byok-overview?view=azuresql

Thank you for watching @Temidayoomoniyi!

Thank you @crax2000! Really appriceate you watching!

@bornluthor this is DBABulldog(Daniel), you have the ability to setup 3 different signals. I did only for successful completion of the job. However, you have 3 different signals(alerts) you can choose from the dropdown. You can choose to fire an alert on success, on failure, and on timeout.

Hi @gopalannasundaram9989 per Bradley, "Great question, this functionality is on the road map but it is not currently in Azure SQL DB Mirroring in Microsoft Fabric. Right now there are some limitations with regards to data masking or encrypting the fields and it being brought over into Microsoft Fabric. That functionally is coming but I'm not sure how soon it will be here."

Hi @vishwassee per Bradley, "This is a really interesting question. I don't want to say no, but it may not be. The only possible way I could see this working would be if you have an a site to site VPN connecting your on premises to Azure, and there is a network that you have a network gateway set up to that main network, and you have an F64 provisioned and a Managed private endpoint for your workspace that is attached to that network. Provided that you have end to end connectivity and can hit a source using jdbc or pyodbc (probably utilizing SQL Authentication), maybe this would work. Keep in mind I'm not a networking expert, and I do not have the means to set up this envrionment and proof it out..... but MAN do I WISH I could, because if it did work it would be really cool!"

Thank you Will!! Always appreciate your support my friend!

@hnedrawan11 this is DBABulldog(Danile) in this particular scenario I was using Azure SQL DB. I am assuming that you are referring to Azure SQL Hyperscale when you mention 10 TB. This is supported; however, you may want to review the following information in the provided link: learn.microsoft.com/en-us/azure/azure-sql/database/database-copy?view=azuresql&tabs=azure-powershell#database-copy-for-azure-sql-hyperscale

Hi @user-gf4oy6qo4u sorry we do not have any videos planed on HCI devices at this time. I checked with the team and we do work against a variety of services but not that one.

Hi @babakbarzegari9228 per Bradley, "Sorry it took so long to get back you you @babakbarzegari9228 ! That was the Service Principal that I had created. You have to create an Azure Entra ID Service Principal in order to add it to the Microsoft Fabric Workspace and then you can use that to create the linked service in Azure Data Factory to the Microsoft Fabric Workspace. I hope this helps. Here's a tutorial on creating a Service Princiapl : jiasli.github.io/azure-notes/aad/Service-Principal-portal.html "

Thank you for watching @rossgh76 !

Thank you my friend. I also presented and additional option recently using Elastic Jobs. I would check that one out as well so you have options.