Deploying Azure Data Factory With CI/CD Using Azure Pipeline

Рет қаралды 74,815

Күн бұрын

This video will focus on how to use Azure Data Factory CI/CD Pipelines and promote ETL Pipelines from Dev environment to UAT then production. How to create an Azure DevOps pipeline to deploy Azure Data Factory pipelines, link services, triggers, datasets, etc to several Azure Data Factories.
Learn a complete tutorial for creating Azure Data Factory CI/CD pipelines with Git enabled that use different branches for developing new features for ETL pipelines and other Azure Data Factory components.
This will cover, creating three Azure Data Factories, two Azure Storage accounts, Azure Release pipeline with Azure Artifacts based on Git repo and deploying Azure Data Factory ARM templates files to another environment.
It will show how to develop all the Azure Data Factory components in the feature branch. Then how to raise a pull request to merge those changes with the master branch. After that publish Azure Data Factory development which will create adf_publish branch which will include the ARM template that describes the current Azure Data Factory Environment. And by using this ARM template, we can deploy it to UAT and Production environments.
Commands:
Stop trigger
---------------
-armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" -ResourceGroupName "[RG name]" -DataFactoryName "[Target ADF Env]" -predeployment $true -deleteDeployment $false
Start trigger
-------------
-armTemplate "$(System.DefaultWorkingDirectory)/[artifact source alias]/[SC folder]/ARMTemplateForFactory.json" -ResourceGroupName "[RG name]" -DataFactoryName "[Target ADF Env]" -predeployment $true -deleteDeployment $true
Stop trigger
---------------
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $false
Start trigger
-------------
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $true
-factoryName "[target ADF name]" -AzureBlobStorage1_connectionString "" -AzureBlobStorage2_connectionString ""
-factoryName [target ADF name] -AzureBlobStorage1_connectionString "" -AzureBlobStorage2_connectionString ""
-factoryName "ADF-Radwan-UAT" -AzureBlobStorage1_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan1;AccountKey=xx;EndpointSuffix=core.windows.net" -AzureBlobStorage2_connectionString "DefaultEndpointsProtocol=https;AccountName=storageaccradwan2;AccountKey=xxx;EndpointSuffix=core.windows.net"
-armTemplate "$(System.DefaultWorkingDirectory)/_adf_publish_source/ADF-Radwan-Dev/ARMTemplateForFactory.json" -ResourceGroupName "adf-rg" -DataFactoryName "ADF-Radwan-UAT" -predeployment $true -deleteDeployment $false
About the Author
-----------------------------
Video:
• Mohamed Radwan is a Pr...
Blog:
mohamedradwan.com
Linked-in
/ mohamedahmedradwan
GitHub
github.com/MohamedRadwan-DevOps
Twitter:
/ mradwan06
DevOps #Azure #Azure_Data_Factory_tutorial #Azure_training #Azure_DevOps

Пікірлер: 113

@jasonbougas2999 3 жыл бұрын

Thanks for this detailed walk through. I got up and running with my Dev to UAT deployment really fast following your demo. Really helpful!

@MRadwanMSF 3 жыл бұрын

Glad it was helpful!

@msvasankv 3 жыл бұрын

It's a great presentation and a nice walkthrough on creating all the stages. Thanks, Mohammed!

@MRadwanMSF 3 жыл бұрын

Glad it helped! and thanks for your comment!

@riojedac 3 жыл бұрын

Good to find you here Mo! This is a great and didactic video, thanks!

@MRadwanMSF 2 жыл бұрын

Glad you liked it!

@bonnevilleedits2303 11 ай бұрын

There's a slight mistake in the values for Start and Stop Trigger. In the Video Mohamed sets the delete deployment flag to true for clean Resources and Start Trigger. But you also need to set the predeployment flag to false for it to start the triggers again as this is the flag to distinguish between pre and post deployment.

@HemantKrSahu 2 жыл бұрын

Bravo Man !!! Thanks for making this helpful video.

@empowerpeoplebytech2347 3 жыл бұрын

Fantastic! Thank you very much for this detailed video on ADF CICD. I was stuck at an initial point where my account and project were not visible in ADF, it helped. Also, now have a clear understanding how the flow from various branches takes place. This is a very big thing which every ADF developer must know. Thank you very much. Wish you lots of success!

@MRadwanMSF 3 жыл бұрын

You're very welcome!

@alphacharith 3 жыл бұрын

Thank You Mohamed Radwan. Excellent Video

@MRadwanMSF 3 жыл бұрын

Glad you liked it

@NSys-ru3ix 10 ай бұрын

Excellent tutorial, thank you.

@minineng21494 2 жыл бұрын

Thanks so much for the video, really helpful!!

@irammapatil9957 Жыл бұрын

This is really helping for me...i got so many things which i was looking for..Thanks

@MarimuthuSA 3 жыл бұрын

Great Video; Detailed explanation. It helped a lot.

@MRadwanMSF 3 жыл бұрын

Thanks Mari, I am happy you like it!

@sanjithkumar7600 Жыл бұрын

thanks for the end to end demo :)

@terryliu3635 3 жыл бұрын

Thanks! This is what I’m looking for.

@MRadwanMSF 3 жыл бұрын

Glad I could help!

@bharathsairam443 3 жыл бұрын

Fantastic...Very good explanation...Thank you Mohamed..

@MRadwanMSF 3 жыл бұрын

Glad you liked it Thanks for your comment!

@CoopmanGreg 3 жыл бұрын

Excellent Video! Thanks!

@MRadwanMSF 3 жыл бұрын

Glad you liked it!

@UmerPKgrw 2 жыл бұрын

Very Well Explained

@lifechamp007 2 жыл бұрын

Super helpful - Thank you so much Radwan !! #StayBlessednHappy

@Mramaamo 3 жыл бұрын

Thanks Mohammed, its a very good one

@MRadwanMSF 3 жыл бұрын

Glad you liked it! Thanks!

@sriramudhyab 3 жыл бұрын

Excellen video and very thorough explanation..Thank you

@MRadwanMSF 3 жыл бұрын

Glad it was helpful! Thanks for your comment!

@saikarthik4a9 3 жыл бұрын

Thanks Radwan, It's a clear explanation. Looking forward from you to get more videos

@MRadwanMSF 3 жыл бұрын

Thanks!

@syedfarhaan-e9k 6 күн бұрын

thank you so much bro. bless you bro for this! this is really helpful.

@lukeroberts70 6 ай бұрын

Fantastic walkthrough, helped a lot, thank you so much!! One thing I did need to do for the ci-cd.ps1 to run was under Use Latest PowerShell Version, was to choose Use PowerShell Core under the advanced option below. Otherwise it didn't like the format of the script.

@gustavoortega4361 2 жыл бұрын

Thanks for the video :)

@MRadwanMSF 2 жыл бұрын

My pleasure!

@gobindamohan 3 жыл бұрын

This is really awesome

@MRadwanMSF 2 жыл бұрын

Thanks!

@swapnilwagh7515 3 жыл бұрын

Nice one!

@MRadwanMSF 3 жыл бұрын

Thanks!

@sjitghosh 3 жыл бұрын

Excellent

@MRadwanMSF 3 жыл бұрын

Thank you so much 😀

@LandscapeInMotion 2 жыл бұрын

Thanks for not speeding through this demo!

@gameimdum7396 3 жыл бұрын

Thanks mate

@MRadwanMSF 3 жыл бұрын

Happy to help

@AndersKnudsen93 3 жыл бұрын

Thanks for the great video, Mohamed! I was wondering how this would work if the ADF kept on growing and more and more pipelines along with more linked services would be added. Would that require to keep adding connection strings and alike to the "override template parameters" section of the release pipeline?

@user-jf1mz2ds6b Ай бұрын

This information is very informative and has been incredibly helpful, especially due to its hands-on nature. Instead of creating a single resource group for all three environments (dev, UAT, and prod), I believe it is more practical to create separate resource groups for each environment. This approach better separates the environments. What do you think?

@hamzamanouzi4937 2 жыл бұрын

Hi, Radwan. Thank you very much for this! It saved me a lot of time. I have one little question: what if the data factorys are in two different resource groups? Thank you very much again.

@galilioXX 2 жыл бұрын

Thanks a lot Mohamed for the great presentation, very informative and straight to the point! I configured the CICD pipelines for my organization to auto-deploy ADF changes, however I ran into an issue in which the deployment messed up my scheduled triggers start times as it used the start times from the source code which is an old value. I was wondering if there is a way to deploy all the changes without changing the triggers start times? if no, how we can keep the trigger's start time value from being overwritten?

@arupnaskar3818 Жыл бұрын

Hi Nemairi, Do u know is there any way to release pipeline with disabling adf triggers.. ?? Please, I am stuck 😔

@colinrippey1723 3 жыл бұрын

Great presentation. In a scenario where you have say two developers and each developer works on their own dev branch, would you have two ADF-DEV instances or would the two developers share the single ADF-DEV instance and work on their individual branches via the git integration? I am trying to get my head around what happens in Data Factory as you switch git branches, does all the associated artefacts “swap themselves out”? Does the Data Factory git integration really imply you just have one ADF-DEV and one git branch being developed at a time? Any thoughts you have on this would be most appreciated, thank you again for taking the time to put together such a useful video.

@harishchalla268 2 жыл бұрын

Thanks for the great video, Mohamed Radwan. The pipelines which you are pushing to the destination(UAT), it reaches to UAT live mode. Is it possible to move pipelines from DEV to UAT git repo mode instead of livemode ? (in which both DEV and UAT are connected to git.)

@chennareddy2770 2 жыл бұрын

Thanks Mohamed, this is really great. I have a question on stop and start trigger , if i dont include the script which you have added before and after the ARM template deployment, will it just copy the datasets and pipelines but will not trigger the pipeline to copy the blob content to source to destination correct? I have multiple Data Factories in my resource group and is it safe to use the stop and start trigger script, hope that it will not disturb the other Data Factories

@Andyfshito 3 жыл бұрын

Hi! I would like to know if there's a way to avoid clicking on Publish every time you perform a pull request to the master branch. Is that possible? Thank you!

@sabirpatel3555 2 жыл бұрын

Thanks for the video, Mohamad Radwan , Can you please create a video or guide us how to create a CI-CD pipeline using Azure SQL table. how should we proceed for that.

@abidnadia6382 Жыл бұрын

Thank you for sharing this relavent information however I would like to ask about data flow and where to make the data transformations ? Thank you in advance :)

@debbieedwards7267 3 жыл бұрын

So helpful. Thank you. My only problem is that my dev and prod are in different subscriptions and resource groups so I don't quite know how to tie this into your where your dev and prod are in the same subscription. for example. At 40.51 when setting up the ARM template. Should the resource group be your dev or Prod one if they are different?

@MRadwanMSF 3 жыл бұрын

Glad it was helpful! This is just a small example but in reality you should parameterize everything and most probably we will use variable group docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml

@bitips Жыл бұрын

Thanks for that, But in different environments we have different DBs (connection strings), so how can we manage this dynamically?

@bharanidharanm2653 3 жыл бұрын

Hi Mohammed, Thanks for the video. I am using managed identities to blob storage and database. How can i form the variables for "override template parameters" could you pls help clarifying. thanks.

@abreezy1st 2 жыл бұрын

what do we do if we have copy pipelines in dev and uat but we dont want them in prod? is there a way to specify which environments to pass copy pipelines into?

@tipums123 3 жыл бұрын

Thanks Mohamed, it was a great presentation. One question - Instead of Master can we add Powershell script files to adf_publish branch? Is there any pros or cons adding them in adf_publish?

@MRadwanMSF 2 жыл бұрын

Thanks!

@amitsinhaism 3 ай бұрын

but how we would change the storage accounts for UAT and prod as they would be different for UAT and prod ?

@camassasantiago933 3 жыл бұрын

great video, I wonder can we just get changed files in repository and only deploy these changed files without all files. I stuck in how to do that

@MRadwanMSF 3 жыл бұрын

You can use another way that only deploy the ADF pipelines that you want, see this link: marketplace.visualstudio.com/items?itemName=liprec.vsts-publish-adf

@andychiarwantopranata6579 3 жыл бұрын

Hi, Radwan. Thanks for the detailed explanation! I wonder if we can change the dataset source programmatically in release pipeline? For example: in ADF-Dev I use Azure SQL Database as dataset source, but when publishing to ADF-UAT it automatically changed to SQL Server On-Premise Database (assuming both of data source has the same table structure).

@MRadwanMSF 3 жыл бұрын

Yes you can do that, you can use YAML and replace function. See the following. devblogs.microsoft.com/scripting/use-powershell-to-replace-text-in-strings

@andychiarwantopranata6579 3 жыл бұрын

@@MRadwanMSF Thanks!

@rodrigueschetan Жыл бұрын

Hi Radwan , Can we parameterize self hosted run time in Azure data factory while doing azure devops ?

@kamblekiran786 2 жыл бұрын

Very nice video @mohamed, Can you please elborate more on powersell script we are using it. For what purpose it is used?

@AxL28AxL 3 жыл бұрын

For the Data Factory Scheduled Trigger, in my Dev the Trigger status is Stop and I would like to ask if I deploy it to Test or Prod would the "Stop Trigger" and "Clean Resource Resource and Start " task would start the status of the Data Factory Scheduled Triggers?

@MRadwanMSF 3 жыл бұрын

The script should start all triggers

@ketanshah9082 3 жыл бұрын

Awesome work Mohamed, you explained it so well in your video. thank you so much. Mohammed is there anyway you can give access to this presentation(ppt) you prepared for this demo?

@MRadwanMSF 3 жыл бұрын

Thanks, I am happy it was helpful. Sorry, I can't find the ppt.

@ketanshah9082 3 жыл бұрын

@@MRadwanMSF no worries Mohammed. Please keep up this great work you do in teaching some of us so easily and with step by step guidance. I am looking forward for Azure DevOps with Databricks, Synpase, HDInsights would be great to learn.

@jubatistim 3 жыл бұрын

Great presentation man, thank you! Where I can get the ci-cd.ps1 file content?

@MRadwanMSF 3 жыл бұрын

I am happy it helped. Here is the link where you can find the file docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment

@adamkhouiy2719 3 жыл бұрын

Excellent video Mohamed. Thank you!!. I have just some questions about deployment in different resource group (Dev, Uat, prod) and each time with different resources names. For example using dev-storeaccount in Dev resource group and uat-storeaccount in Uat resource group and so on ... My first question is : do we use the target resource groupe name in start/stop trigger script ? . Second question how we can handle the parametrization of resource name from an environement to an other ?

@MRadwanMSF 3 жыл бұрын

Thanks! You should parameterize everything and most probably we will use variable group docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml

@adamkhouiy2719 3 жыл бұрын

@@MRadwanMSF Thanks !

@BharathKumar-zq6lm 2 жыл бұрын

Great presentation Sir I have one question - we have 3 different self hosted integration runtime with different names for each environment dev, uat and prod respectively. how to parameterize it while deploying to uat and prod env?

@rodrigueschetan Жыл бұрын

Hi Bharath, did you parameterize self hosted IR now ?

@minufrancis222 Жыл бұрын

I see a task - publish data factory - is this the new form of deployment or not? It has checkboxes which says 'StartStopTriggers' as well

@DINESHKUMAR-cu2db 2 жыл бұрын

Unable to connect the AAD in Azure devops. Error is the current organization policy does not allow guest users to access the organization . Change the policy setting to external guest access

@antongolubev3511 2 жыл бұрын

Can I use multiple ADF for developing one pipeline in one repository?

@kannan114 2 жыл бұрын

What's the difference between adf and master branch? Is master a staging and adf actual master in each environment?

@prabhurathinam 11 ай бұрын

Hi Mohamed, Thanks for a great video. Appriciate it. One of my linked service is conencted to Dataverse. In the template parameter i'm unbale to pass Service uri for the uppper environements. i can only able to pass Service connection and its secrect. is there a way to pass the Service URI as well in the parameters? Please advice. Thank you

@rameshbaddam5465 2 жыл бұрын

I just have a question . Where is the build pipeline ?. Don’t we have build pipeline like other services for adf?

@amitghotikar 3 жыл бұрын

Predeployment should be set to false for starting triggers , isnt it?

@bhavyavful 3 жыл бұрын

Hi Mohamed. This video was great. I have a question for you regarding ADF. I am facing a problem in creating a single pipeline. Pls reply if you are interested to sort out my problem.

@HanumanSagar Жыл бұрын

Hi can you help me in stop trigger task while selecting Azure subscription I'm not able to see the subscription list..what should I do if want to see list of subscriptions?

@MrDwikipedia 3 жыл бұрын

Thanks for this it's been very helpful! I got it working after a lot of trial and error. One question and maybe I'm misunderstanding, but shouldn't the start trigger be -predeployment $false ?

@MRadwanMSF 3 жыл бұрын

Glad it helped! Yes, for the start trigger and if I didn't do that, it's my mistake :-)

@ravindranathmetla8180 2 жыл бұрын

Mohammed, having issues to deploy data factory with global parameters.. can you send any link or info regarding this..

@SynonAnon-vi1ql 11 күн бұрын

31:29 You see everything is great in this video but I am not understanding one thing. I would really appreciate if you could please explain. My question is why did you merge the code from feature branch to master without first testing it in the UAT environment. This is opposite for me. I was thinking you will show the automated mechanism to first merge the dev code to "UAT" branch, then deploy it in UAT env for testing purposes. Once the testing is successful, the code from UAT branch will be merged into Master branch and from there it will be deployed into production. But you first merged the Dev code into master without verification (UAT). Now if the UAT fails your master is no more good and is not representative of the prod. Can you please explain why my thought process is wrong? Thank you!

@AxL28AxL 3 жыл бұрын

So the UAT and Prod don't need to be associated with Azure DevOps Git?

@MRadwanMSF 3 жыл бұрын

Yes, only Dev assoicated with Git and the rest of envs are not

@hemanthkh Жыл бұрын

Please share the yaml pipeline

@user-um1bd3zz9f 4 ай бұрын

هل ممكن حضرتك تعملنا فيديو بالعربي شرح ل azure data factory

@RajeevKumar-zf8ox 3 жыл бұрын

Hi Radwan, I have one Databricks. How to deploy it using CI/CD

@MRadwanMSF 3 жыл бұрын

You need to use this tasks marketplace.visualstudio.com/items?itemName=riserrad.azdo-databricks

@shwetasaronja2110 2 жыл бұрын

We need the powershell script that you copied from microsoft website at the time of creating a file.On microsoft website it is not available. Please watch video at 34.26 time and guide me

@irammapatil9957 Жыл бұрын

Hi, can you send powershell script link which you have copied from microsoft website

@snad256 2 жыл бұрын

Hi I'm unable to add custom parameters for Snowflake. Could you please get back to me?! Great video man! Thanks

@abhijnanbp1397 2 жыл бұрын

Hi Radwan, This is a great video! I have a question - in this video, you have demonstrated that all those changes made in Dev will be deployed on UAT and Production in the same CD pipeline. However, in the real time scenario, if I make 10 changes in Dev, all those 10 changes should be deployed on UAT. Imagine, out of those 10, we got an approval to deploy only 4 changes onto Production, and the rest of the 6 should still be in UAT for extended testing. I am curious to understand how can we deploy only those 4 changes onto production. I look forward to having answer, and that will be of great help. Thanks in advance!

@TheMadhan1988 2 жыл бұрын

Did you get a solution for this?

@mohammedalbosifi3975 3 жыл бұрын

بدنا دورات بالعربي

@MRadwanMSF 3 жыл бұрын

احضر المحاضرة الجاية kzfaq.info/get/bejne/mqqKqpR8nNXbZmw.html

@zabeehullah3673 4 жыл бұрын

Hi, I have a scenario and need community advise to manage it in best possible way. Due to compliance issues, I am not allowed to save our source code in any Geo location supported by Azure Devops Services. so under repo when we create a repo, by default its created somewhere in Geo location, ideally where azure devops service is hosted.But this approach is NOT-OK for us. And I want to use our own server "hosted in MS-azure" as repo server.so whenever, we create a repo using create repo under the project repositories this should be stored in that particular server.. please advise how can i achieve this? our own server is having Visual studio 2019 and connected to azure devops also.

@shakechen7944 3 жыл бұрын

you can use tfs server deploy local.

@NitinMathewGeorge Жыл бұрын

But now the arm approach is having limitations with parameter limit of 256. Secondly the concept of branches in this approach isn't ideal and acceptable from MS.

@saisylaka955 8 ай бұрын

Hi Mohamed, I have follwed the same setps which you explied in the video. I'm getting below error while running the pipeline. Please help me out in this. Error: 2023-11-03T06:31:42.8563878Z ##[error]Could not find the modules: 'Az.Accounts' with Version: ''. If the module was recently installed, retry after restarting the Azure Pipelines task agent. 2023-11-03T06:31:42.9672510Z ##[error]PowerShell exited with code '1'. 2023-11-03T06:31:45.6177222Z ##[command]Disconnect-AzureRmAccount -Scope Process -ErrorAction Stop 2023-11-03T06:31:45.7413326Z ##[warning]Run Connect-AzureRmAccount to login.