Azure Synapse Analytics - Source Control with Git Integration

Рет қаралды 14,301

Күн бұрын

The BIGGEST missing feature since we got our hands on the Azure Synapse Analytics workspaces preview has been the ability to have any kind of source control. This week we've seen a whole buzz of excitement as finally source control was added and it's very much in line with the existing Data Factory experience.
In this video, Simon takes us through the initial setup of source control using Azure DevOps, from linking projects, committing changes to feature branches and deploying it to the development environment.
As always, don't forget to hit that subscribe button, and drop by the Advancing Analytics website if we can help you on your Azure Synapse journey - www.advancinga...

Пікірлер: 32

@hamzah1738 2 жыл бұрын

It is the first time that someone can clearly explain the difference between commit and publish

@christofferp 3 жыл бұрын

Your channel is golden! Really amazed by your broad understanding of analytics pipelines :D

@MrWaxxx22 2 жыл бұрын

I was about to comment that you were talking too fast...then i noticed i was watching on 1.25 (facepalm). Great content keep it up!!!!

@touheedahmed6663 2 жыл бұрын

Great Content. Clear my concepts. Thanks.

@l_combo 3 жыл бұрын

Loving your content, loving the idea of the git integration. Keen to see it maturing compared to more data science type platforms (Databricks / Domino Data Lab / DataIku etc...)

@baklava2tummy Жыл бұрын

Thanks, this is really helpful. What I don’t understand though is how to include the tables, views etc I’ve got within my dedicated pool (but aren’t visible in my workspace).

@akhilannan 3 жыл бұрын

I can see all the contents are in json format. Even SQL scripts are appearing as json. Do you see this as a road block in doing development from an IDE like ADS or SSMS?

@AdvancingAnalytics 3 жыл бұрын

Potentially - I had assumed for anything following the traditional SSDT / Database project approach, you'd have a separate database project in the same repo (hence why I created Synapse in a subfolder to segregate). I think we'll end up thinking of "workspace objects" and "database objects"... maybe? Honestly haven't even begun to piece together what a full devops story across all areas looks like yet!

@pankajsingh23UTube 3 жыл бұрын

Looking forward to video on Programmatically/Manually deploy ARM template in other environments (UAT, Prod etc.). One thing is strange that unlike ADF Microsoft has not given Import/Export ARM template functionality for Synapse.

@AdvancingAnalytics 3 жыл бұрын

Yeah... I tried to get a good ARM deployment demo working last week, but honestly there were too many bugs in the deployment process. I got it working with quick a few hacks and workarounds, but with the general availability rush, I'm assuming it's something that'll be fixed over the next week or so, then I'll pop a proper video together! If it's still broken next week, I guess I'll do a tour of all the bugs and how to work around :) Simon

@kinnajluxen8836 3 жыл бұрын

@@AdvancingAnalytics Hi, thanks a lot for your insightful videos. I am very interested in that follow-up as well! Keep up the good work :)

@richpslk 2 жыл бұрын

New to ADF and Git, but familiar with source control in general. Can you clarify what happens when you click publish when the selected branch is something other than the main branch (EG as at 21 minutes in the video). Does it publish to the live workspace the contents based on the main branch, meaning you still need to create a pull request from development back to the main branch to get changes live, or is this done in the background?

@yadhurammc 3 жыл бұрын

Looks great. This gives one more reason for all those data projects to move in synapse where they get everything in one place. Now the interesting thing will be on the CI/CD part. Do you see an option to move database projects to synapse? Traditionally we build a dacpac and do the deployment. Any such approach possible here? May be we have to still maintain a separate DB project. Also, would it be possible to deploy individual changes. For example, what if one want to deploy only the pipeline changes to prod keeping other things untouched. May be we have to maintain a 'Prod' branch, do change there then do a pull.. lot of push pull merges :P Such scenarios will be interesting I guess.

@AdvancingAnalytics 3 жыл бұрын

Hey! Yeah, I think it's currently exactly as you said. An SSDT database project separate to the Synapse folder for deploying Dedicated SQL Pool projects via the usual dpac route. Deploying specific changes is definitely a separate branches & pull requests mechanism - same as ADF currently! Simon

@monikaazure4563 Жыл бұрын

Hey..After following all the steps my team is not able to make changes into git.How to provide them access?

@pedroduque1407 Жыл бұрын

Hi! ¿Does anybody know if the Serverless SQL Pool is included in the CICD, if the database, table, external resources scripts can be integrated in the CICD?

@crazybauns Жыл бұрын

any advice on how to deploy serverless objects?

@marcocaviezel2672 3 жыл бұрын

Great Git is finally here 💪🏼 Great video Simon! Thanks a lot! To get it right, is the code pushed to in the main branch what’s visible in the live mode? Best regards, Marco

@AdvancingAnalytics 3 жыл бұрын

Nope - in order to see it in "Live" mode you need to publish code from whatever branch you have designated as the "collaboration branch". In the vid, I was using main as a separate code line that would be used to trigger a DevOps pipeline to the next environment. I'll see about doing a proper multi-environment demo soon! Simon

@LexisStv 2 жыл бұрын

@@AdvancingAnalytics What's about demo? Is it in plans?

@Ravi-gu5ww 3 жыл бұрын

Looking for synapse ci/cd

@soluconsultor1511 3 жыл бұрын

Hi, Great Video. Is it possible to use Git Repo for db (SQL pool) DDL (tables, procedures, views, ...)?

@AdvancingAnalytics 3 жыл бұрын

The dedicated SQL pools still have the same accessible SQL endpoint, so I assume you can deploy SSDT projects from a git repo as with the standalone SQLDW. Technically the same approach should work for views in serverless SQL Pools but I've not tried! Simon

@sid0000009 3 жыл бұрын

Hi Simon, If i recollect right in ADF, we could only publish from the Main Branch and not a feature branch. It meant that we had to move the changes first to main branch ( pull request ) and then create the ARM template in the adf_publish branch. Here I see we could publish from the Dev branch as well :) Or I am getting too old to recollect it right :(

@AdvancingAnalytics 3 жыл бұрын

Close, but it's whatever branch is designated as the "Collaboration Branch", in this case I set it to the Development branch, so it's only that one you can publish from. I think I messed around with the terminology a bit in the vid!

@sid0000009 3 жыл бұрын

Also would like to know if we can make a selective sync while publishing..example I only want a X script to be updated and not others which might have been changed? thank you for your quick win videos!

@sid0000009 3 жыл бұрын

@@AdvancingAnalytics got it thanks!

@AdvancingAnalytics 3 жыл бұрын

@@sid0000009 you can only do a selective commit (ie: save X object to the branch) but publish puts everything in the collaboration branch to live. If there are pieces you don't want to publish, keep them in a separate branch!

@arsenaler308 3 жыл бұрын

How do we keep our "feature branch" up to date with the collaboration branch, like a "git pull"?

@AdvancingAnalytics 3 жыл бұрын

Yep, just create a pull request and complete it. You can create a pull request either from Synapse, or from git/DevOps directly!

@arsenaler308 3 жыл бұрын

@@AdvancingAnalytics also, I notice that Synapse ignores Spark pools when publishing to workspace_publish branch. It declares my dedicated SQL pool in the TemplateForWorkspace.json but completely forgets about the Spark pool. Do you experience the same thing?