How to Use Git w/ Azure Data Factory

No video

How to Use Git w/ Azure Data Factory

Рет қаралды 10,656

Күн бұрын

►► See how I’ve helped 10+ small data teams add more structure using modern tools, w/o making a mess (Free Workshop) → www.kahandatas...
Data factory is one of the best at combining the user-friendliness of a drag and drop tool w/ the versioning and automation of a code based tool.
One of the best features of data factory is the ability to connect directly to the get provider of your choice.
In this video we will see how this process actually works in Data Factory.
More about Source Control on Data Factory:
By default, the Azure Data Factory user interface experience (UX) authors directly against the data factory service. This experience has the following limitations:
- The Data Factory service doesn't include a repository for storing the JSON entities for your changes. The only way to save changes is via the Publish All button and all changes are published directly to the data factory service.
- The Data Factory service isn't optimized for collaboration and version control.
- The Azure Resource Manager template required to deploy Data Factory itself is not included.
To provide a better authoring experience, Azure Data Factory allows you to configure a Git repository with either Azure Repos or GitHub. Git is a version control system that allows for easier change tracking and collaboration.
Timestamps:
0:00 - Intro
0:26 - Create New Repo
1:20 - Configure Git
3:18 - Review Updated Repo
4:02 - Data Factory Workflow
6:56 - Live Mode Setting
Title & Tags:
How to Use GIT and GitHub on Azure Data Factory | Azure Tutorial for Beginners
#kahandatasolutions #dataengineering #azure

Пікірлер: 13

@KahanDataSolutions 2 жыл бұрын

►► Establish a Well-Structured Data Warehouse for Your Small Team In 90 Days (Free Guide) → www.kahandatasolutions.com/guide

@ajgiliana Жыл бұрын

Solid video and straight to the point. Thank you

@MathewBurford 2 жыл бұрын

Thanks for this i have been meaning to try it out

@KahanDataSolutions 2 жыл бұрын

You're welcome. Hope it helps!

@edemfromeden5432 Жыл бұрын

any chance for creating a CI/CD tutorial covering ADF ?

@ihafidh 2 жыл бұрын

Nice video to showcase ADF and easy git integration in a development environment. However, ADF is not as straightforward when you have to do continuous integration and DevOps and have to promote to higher environments.

@KahanDataSolutions 2 жыл бұрын

Thanks for the feedback! Sounds like you have some great experience using ADF in a production environment. What type of issues have you run into with this process? Any suggestions for others?

@ihafidh 2 жыл бұрын

@@KahanDataSolutions The main issue that I ran into is that when you Terraform ADF, you can only hook up a Git repo to a dev environment because linked services properties are baked into the pipelines. The workaround for other Terraformed environments, is that you have to import ARM templates. In terms of automating continuous integration, Microsoft has documented the process. Best practice is to use Key Vault for your linked service properties and you also have to connect to an ARM template as part of the DevOps pipeline. It's doable but not very straightforward. An idea for a video :) I think that if ADF uses environment variables and have the ability to connect to different repo branches (dev, qa, prod), then it would make continuous integration easier to set up. It would also add capability to Terraform ADF into other environments without needing to import ARM templates.

@VijayKumar-my6mr Жыл бұрын

Hi, I would like deploy feture branch in to test data factroy and tester test it, and then merge to master for final tested code change... How to dothis ? Kindly help.

@HasithaChinthaka Жыл бұрын

This is really helpful, do you have any tutorials on how to use GitHub actions to deploy Azure Data Factory to different environments (dev, uat and prod)

@KahanDataSolutions Жыл бұрын

Thanks Hasitha! Unfortunately I do not have a tutorial for that at this time.