Azure Synapse Analytics | Continuous Integration and Continuous Delivery (CI/CD)

Рет қаралды 26,728

Күн бұрын

With ever changing business dynamics and ever evolving technical landscape, for IT teams being agile and nimble is no longer a choice but a necessity.
This video talks in detail about Azure Synapse’s native integration with Azure DevOps and GitHub and demonstrate how you can setup git repository and build pipelines for continuous integration and delivery of synapse workspace’s artifacts (or application) from one environment to other.
00:00:00 Introduction to CI/CD
00:01:25 CI/CD in Synapse
00:02:41 Azure Synapse - DevOps Lifecycle
00:05:06 Demo - Setting up Git Repository for CI
00:27:16 Demo - Generating ARM Template for deployment
00:31:42 Demo - Release Definition and deployment to UAT
00:51:20 Demo - Deployment Automation, deployment to production and conditional deployment
01:03:18 Best Practices and Wrap-up
Thank you once again for watching, please do like, subscribe and let me know your feedback or any specific topic you would like me to cover next. Looking forward to see you all soon in the next video!
GitHub Repo to download deck and script used in the video:
github.com/AasTrailblazers/Az...
Source control in Synapse Studio
docs.microsoft.com/en-us/azur...
Continuous integration and delivery for an Azure Synapse Analytics workspace
docs.microsoft.com/en-us/azur...
How to use CI/CD integration to automate the deploy of a Synapse Workspace to multiple environments
techcommunity.microsoft.com/t...

Пікірлер: 59

@ContainsAll 7 ай бұрын

Excellent and very well structured video. Very helpful. Thanks for Uploading and sharing the knowledge !

@SinghalPuneet Жыл бұрын

I am a regular follower of your videos and these put be in better state and built my confidence high to deliver.

@alexcord 2 жыл бұрын

This is the best explanation about Synapse CI/CD process, even better then Microsoft I only miss the configuration of OverrideParameters, I guess it is the same as we do on ADF pipelines. Tks again!

@ParveenKumar-oc3np 6 ай бұрын

Thanks for this video. This is one of the best video to understand synapse ci cd deployment.

@johnpatrick4185 2 жыл бұрын

Great!... What hidden gem about Synapse on KZfaq! Only 14 comments!? Good stuff!... good job! Thank you!

@aniruddhaparwekar737 2 жыл бұрын

Thank you Arshad. This is one of the best videos for synapse deployments. It was very well structured and logically flowing and the introduction section of DevOps was very useful.

@getsid25 Жыл бұрын

Arshad sir.... amazing series...sincere thanks for making the effort to educate all

@muhammadrahman3510 2 жыл бұрын

Dear Arshad, I followed this instruction vidoe and it worked just perfect to deploy CI/CD pipeline for my large scale Synapse Data Warehouse solution which includes, many Pipelines, Data Flows, Linked Services and Datasets for my ADLS Gen2, Serverless SQL Pool and Dedicated SQL Pool, SQL Script Files (for both Serverless SQL pool and Dedicated SQL Pool), Please note that, I also follow your other valuable videos on Synapse SQL Pool and I learned a lot. Thank you so much and I appreciate it. Please continue your effort for the Azure Synapse community!

@PontyclunBosomPals 2 жыл бұрын

How ? this doesn't copy across serverless or dedicated pools -this just moves the scripts

@cliffordraygentiles 2 жыл бұрын

oww, cool! I would like to know how you did that, basically my challenges are 1. how to add other parameters on the ARM template if I have a lot of linked services and datasets for my pipelines 2. How to choose specific artifacts only to be released on the uat/prd

@PontyclunBosomPals 2 жыл бұрын

@@cliffordraygentiles I had to get around it through power shell. I added numerical values to the start of the script names and using an invoke sql command which iterated through the scripts based on the numerical value assigned. I am sure there are easier options using an ARM template but poweshell gave the most flexibility

@Amin-tx4ku 2 жыл бұрын

Hands down! This is the best explanation ever. Thank you for sharing your knowledge.

@andersbergmaal 2 жыл бұрын

Awesome! Thanx for sharing, and for the thorough walkthrough. This helped me alot in understanding both Synapse version control and Azure DevOps!

@themaligatorr 2 жыл бұрын

this is a masterpiece, by far the best Synapse version control video out there. Again, thank you

@kenpoken1 2 жыл бұрын

Fantastic video. Keep up the great work. Very valuable

@AhmedMohamed-kr8hf 2 жыл бұрын

this is by far the best session that i had seen in synapse CICD, thank you arshad. can you please add another session for the DACPAC implementation

@PontyclunBosomPals 2 жыл бұрын

yes please- considering most orgs will use a serverless instance it would be good to move automate the serverless databases and deploy the scripts.

@rupalibolke1586 Жыл бұрын

Thank you so much for very detailed explanation of Azure Synapse CI CD. This is definitely going to help many.

@sumanbiswas2995 2 жыл бұрын

Thank you Arshad, the content and the demo was absolutely the best!

@ebadoshio Жыл бұрын

Thanks so much for this video!

@mohammadal-aqrabawi7145 2 жыл бұрын

Great demo!

@jb-eh2fp 2 жыл бұрын

Thank you Arshad Ali, it was very helpful. Your explanation is perfect as always. You have the best Synapse Analytics channel on KZfaq. It would be great if you make videos on designing Data Lakes (Delta Lake), folder structures, creating different zones etc.

@f4bglv Жыл бұрын

Excellent and useful video. Thank you for sharing your experience!

@teamlorio Жыл бұрын

Great tutorial!

@sbining Жыл бұрын

Brilliant - really well put together. Thanks Arshad.

@phranger 2 жыл бұрын

Thank you, this video has been very helpful with Synapse!

@sergeiillarionov4057 Жыл бұрын

It was wonderful. Pretty clear. Thank you.

@azurelearner4055 9 ай бұрын

Clear and simple explantion !! kudos

@user-eg1ss7im6q Жыл бұрын

thank you, i have learned something.

@YanliangSong Жыл бұрын

Thanks for the sharing. it is helpful.😀

@dileepananayakkarahiniduma6390 2 жыл бұрын

Thank you Arshad, your explanation is very clear. Videos related to designing Data Lakehouse would be helpful to everyone.

@LifeOfDreams0 Жыл бұрын

wondefull .. great job

@xuananh431 11 ай бұрын

Thank you !

@suniguha Жыл бұрын

You are the best!!

@KathyLoisAmores 10 ай бұрын

Thanks so much

@ashutoshmishra287 Жыл бұрын

Thank you

@yuanyuan39 Жыл бұрын

The best of the best session on Synapse CICD. Could you also explain have to handle parameter overriding for other linked service, such as Azure KeyVault and Linked Storage Account.

@cliffordraygentiles 2 жыл бұрын

Thank you for this clear demonstration, I learned a lot since I don't have experience on the CI/CD yet. Can you help me with the following. 1. how to add other parameters on the ARM template if I have a lot of linked services and datasets for my pipelines 2. How to choose specific artifacts only to be released on the uat/prd Keep up the good work!

@softwaretitbits5849 2 жыл бұрын

Very nice presentation. Please create something similar to build and deploy scala jar files.

@rafaelveloso6531 Жыл бұрын

@darta1094 Жыл бұрын

Very good session, just please, don't forget to zoom to make small details visible. 🔍🙏

@garymelhaff7779 Жыл бұрын

Very complete coverage good job. Curious for those of us with existing ADF devops projects, would it be wise to create a new project for synapse artifacts and not mix ADF with synapse artifacts

@shaanproms2101 2 жыл бұрын

Thank you for sharing this knowledge Arshad. Can we call the source control and CICD the same? And, if the artifact is a sql script would a component overwrite its version in production DB? Thanks a lot!

@yuanyuan39 Жыл бұрын

Hello Arshad, Thank you sooooo much for your awesome tutorial on setting up CI/CD for Azure Synapse! I have one question regards to setting the Pipeline variable for CI/CD, would love to hear your ideas and thoughts. I need to set the value as ‘extsynapsecicdadlstest’ for ‘synapse-cicd-demo-ws-test’ Workspace, but I need to change the value to ‘extsynapsecicdadlsuat’ for ‘synapse-cicd-demo-ws-uat’ Workspace. How can I do it in the ADO release pipeline so that I can update the parameter value when deploying to a different Synapse Workspace? Thanks in advance!

@100sangeetharajan 2 жыл бұрын

Hi Arshad, great video, I would like to know if the same solution will work when the test & Prod workspace are present in a separate subscriptions for each one ??

@naveengaddameedi7193 Жыл бұрын

Thank you for the cool demo, it is very much useful.. one quick question on the storage account when you have created 3 synapse workspaces. in my case I want to use a different storage account names for each workspace individually, does it still work without parameterising the storage account name ?

@krishnapanduri 2 жыл бұрын

Thanks for sharing, how do we connect Azure DevOps in different tenant?

@rohitpitre954 2 жыл бұрын

Hi Arshad. Is there any tutorial/documentation around how to deploy hotfixes for Synapse in production ? There is a Microsoft documentation around ADF but not for Synapse.

@pylanookesh8227 2 жыл бұрын

@arshad Ali : I am trying to use the same process but in my workspace there are alot of notebooks, pipelines and scripts which resulted in arm workspace template exceeding more than 20 mb size which resulted in an error related to size

@giridharpunnamaraju4202 2 жыл бұрын

Hi, a quick question please. While we connect Azure Repo to Synapse workspace, is there a way I can give a custom commit message when I add new changes to Azure repos from Synapse workspace?

@AvinashReddy21 2 жыл бұрын

Hey Arshad, great demo. I get this error and unable to deploy any notebooks ##[error]Encountered with exception:Error: Failed to fetch the deployment status {"code":"400","message":"Failed Component = DataFactoryResourceProvider, ErrorCode = 400, Error = BadRequest "} any help is appreciated !

@anjanay9089 Жыл бұрын

There is a different situation I need to handle currently. Pls help if any alternative than manual approach. The process has been implemented same as mentioned in the video. All good and happy path. There were some data issues reported and my development team had directly implemented the changes on PROD instead of Dev. Noe how to get all those changes from Prod to Dev. Pls help if any way.

@user-bi9nv6ev2h 9 ай бұрын

Hello good day. We have 2 existing synapse workspaces, dev and prod. After setting up git configuration on dev workspace, I noticed that all artifacts (pipelines, storage, datasets, scripts, notebooks) from prod are now existing on the master branch. How is this possible to see prod artifacts when I haven't explicitly connected prod into our dev workspace? Thanks

@youngbloodofindia3334 Жыл бұрын

Very Impressive for one time migration from Dev to Test or Test to Prod. Looking for changes in the pipeline and over and over will it migrate everything based on the ARM template or will it take only the modified components. Pls let me know if there is any way we can enable or disable. Thanks

@LifeOfDreams0 Жыл бұрын

Hello sir, I have a question , after successful deploy in UAT/PROD , is there anyway like date/time stamp to verify changes in upper environment ??

@douglaspiresmartins2955 Жыл бұрын

Hi Arshad Ali. Awesome content, as usual, you rock. I got it to do the same steps, however, I am stuck on one question. If, we are gonna use it for a CI/CD with Synapse, how could I parametrize Notebooks for Spark Pools? For example, one issue I faced, was the name of the Storage Account, if they are different, on our dev, it could be st_DEV but for uat can be st_UAT. If we hardcoded the name into our notebooks, to set the path, the next environments will be wrong. I tried using parameters in my pipeline and setting my cell as toggle parameters, but I think this is not the way. For example, with linked services with the same name, changing into the TemplateParameters the name of Storage Account, works well across different stages, but I search here and I couldn't find a way to use pyspark using linked service.

@levidevos15 Жыл бұрын

@@kunaldasofficial Use parameters in your notebook. You can pass a value in a synapse pipeline to a notebook (lookup activity --> notebook activity). Since there are no global variables in Synapse yet, hard copy a dev/production value in your datalake to call upon.

@asatech92 Жыл бұрын

Hi Sir, can you make a video on the requirement - Developer wants to create pipelines in Synapse usng json files rather than using GUI of Synapse analytics and clicking on publish button. Rest env is same as in this video i.e Build > artifact > release

@jamieg1802 Жыл бұрын

i have been pushed out of my field and old coworkers destroyed my life ; harassed for 2+ years, waiting to hear from authorities, hope they will help me