Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using Watermark

No video

Azure Data Factory - Incrementally load data from Azure SQL to Azure Data Lake using Watermark

Рет қаралды 7,231

SoftWiz Circle

Күн бұрын

Пікірлер: 12

@palyamsudheer Жыл бұрын

Great content, Thanks. How do you deal with deletion of records? I have an OLTP server (SqlServer) where application can delete records, so trying to understand 1. how to identify delete 2. Treatment for deleted records in the late (something like making them inactive or highenddated etc)

@SoftWizCircle Жыл бұрын

There are multiple way to handle this. Some of following way generally use. 1. Instead of Hard Delete use soft Delete with flag 2. if soft delete is not possible then you can set delete trigger and keep record in another table and use that table to update OLAP. 3. we can also enable audit or CDC to find out deleted records and using audit or CDC record we can update OLAP. let me know if this make sense

@SandhyaVeda Жыл бұрын

Can u please mention the syllabus, which is sufficient to get job in ADF. Also kindly order ur playlist

@SoftWizCircle Жыл бұрын

sure. i will do in sometime. apologies for delay

@souranwaris142 Жыл бұрын

Hello, I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table & store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.

@SoftWizCircle Жыл бұрын

watermarking is one of easiest and widely used solution where we do store last successful push time for given table and then use store proc to get and update this table. only catch here that table which you have to push must have a timestamp column to compare and find change after last run.

@souranwaris142 Жыл бұрын

@@SoftWizCircle I am pushing table directly to blob storage account I don't have azure SQL and I am getting table from on premise server SQL so that's why I have a confusion where I need to create this. Please help me for this. Is it possible can give me 5mnt on call show I can show you my problem

@varunvengli1762 5 ай бұрын

Even I'm stuck on this issue

@ranjansrivastava9256 Жыл бұрын

Good Article, Lack of Visibility of the Video.

@SoftWizCircle Жыл бұрын

Sorry for that I tried to record in HD this abut still seems somehow blur.

@mk-ho9iz 9 ай бұрын

what the fuck....how can lower lastmodifytime can pass through it..meaning the copy condition gets false and data should not be copied

@SoftWizCircle 9 ай бұрын

Please try once. last modified date is a column in sql table in this demo and we can parametrize sql statements or stored proc etc and pass value from adf