Azure Data Factory - Incremental Data Copy with Parameters

No video

Azure Data Factory - Incremental Data Copy with Parameters

Рет қаралды 18,589

Күн бұрын

This Topic covers the following:
1. Incrementally Copy Data from one Database to Another
2. Parameters to Support - Can be run for multiple tables using the same pipeline
Other Videos on Azure:
Subscribe:
/ @azurebitsandbytes5804
Azure Portal:
• Azure Portal - Sign up...
Storage Account:
• Azure Storage Account ...
SQL:
• Azure SQL - Introducti...
Data Factory:
• Azure Data Factory - I...
• Azure Data Factory - ...

Пікірлер: 25

@namangarg7023 20 күн бұрын

Tough to achieve in different real time scenarios but still a masterpiece activity explained for new users..tks

@Rafian1924 3 жыл бұрын

Lucid explanation. Good job brother.

@chandrasekharnallam2578 2 жыл бұрын

nice explanation with easily understanding the concept. thank you

@shaileshsondawale8765 Жыл бұрын

Awesome Content.. Great work done Sir....

@pushpakbansal7070 3 жыл бұрын

Very good explanation, helped me in getting the job done.

@kolliuma9509 3 жыл бұрын

Great Explanation Sir with good content

@Lihka080 3 жыл бұрын

@ Azure Bits and Bytes Hi Sir very useful and excellent, could you please add update logic, so if there is any change in source table records it should update the records in sink table then need to take latest from source to sink table. Thanks in advance

@vishwanathr3507 3 жыл бұрын

Hi How to Filter multiple values in Parameter like SELECT * FROM DimCustomer WHERE EnglishEducation IN (@'{Education('LookupEducation').Output}') not sure it is correct way to pass multiple values in to parameter in adf paramter. Thank you

@preetijaiswal9089 3 жыл бұрын

Hi, This Really worked and helped me out . Now I need to do CDC based on other column like Id or other unique column so how to implement this as our watermarkcolumn is of datetime type. Please Help me out. Thanks in advance

@azurebitsandbytes5804 3 жыл бұрын

in your case, you can make use of Id value as the watermark column, and change the logic to extract a id range(from and to) and use that to fetch data from source system. Hope it helps.

@preetijaiswal9089 3 жыл бұрын

@@azurebitsandbytes5804 I need to do Capture Data Change and do the update only on the basis of UpdateTime so how to do that. I really appreciate for your assistance. Thanks.

@RajeevRanjan-nf6xc 2 жыл бұрын

Thanks a lot sir!!! Can I get ddl scripts for practice

@sirigiriharikrishna1772 Жыл бұрын

@@preetijaiswal9089 @azurebitsandbytes5804 Please help how cdc will work in this case?

@ashwinds6293 3 жыл бұрын

Am getting an error in pulling the data through ADF for Oracle as a Source . Could you please help me in this

@krishnakumar-vi3tf 3 жыл бұрын

It was nice,but one question how can i handle multiple tables passing dynamically to pipe line,right now you are entering table names at pipe line level manually but my requirement is i have to pass table list for both lookups dynamically

@krishnakumar-vi3tf 3 жыл бұрын

i got to know there is another video where we can pass tables dynamically with master pipe line..excellent..its an urgent requirement for me..thanks a lot

@nagarajupinnu3621 Жыл бұрын

why can't we update watermarkDate in a table directly with query instead of Stored procedure which is extra object

@azurebitsandbytes5804 Жыл бұрын

The script activity was not present at the time of making this video. You can use this definitely.

@hungerspoint7358 3 жыл бұрын

How it will handle duplicate data?? If pipeline runs twice at at time the. It will again copy the data??

@azurebitsandbytes5804 3 жыл бұрын

making use of tumbling window trigger will help avoid overlapping runs.

@juzerpatanwala5133 2 жыл бұрын

where can we find this dataset of customer orders

@dataengineerazure2983 3 жыл бұрын

PLEASE HOW GET FILE SOURCE FOR TRAINING? TKS

@wheretastesdivedeep 3 жыл бұрын

How to handle if the table don't have data?

@azurebitsandbytes5804 3 жыл бұрын

Which table?

@rohitsethi5696 Жыл бұрын

no this is not useful.The table should be manually enter. This is 200% end developer can end wrong table or misspell or space in table name end or before. The best way to store all the table name in the Table then for each loop process