Great content, Thanks. How do you deal with deletion of records? I have an OLTP server (SqlServer) where application can delete records, so trying to understand 1. how to identify delete 2. Treatment for deleted records in the late (something like making them inactive or highenddated etc)
@SoftWizCircle Жыл бұрын
There are multiple way to handle this. Some of following way generally use. 1. Instead of Hard Delete use soft Delete with flag 2. if soft delete is not possible then you can set delete trigger and keep record in another table and use that table to update OLAP. 3. we can also enable audit or CDC to find out deleted records and using audit or CDC record we can update OLAP. let me know if this make sense
@SandhyaVeda Жыл бұрын
Can u please mention the syllabus, which is sufficient to get job in ADF. Also kindly order ur playlist
@SoftWizCircle Жыл бұрын
sure. i will do in sometime. apologies for delay
@souranwaris142 Жыл бұрын
Hello, I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table & store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.
@SoftWizCircle Жыл бұрын
watermarking is one of easiest and widely used solution where we do store last successful push time for given table and then use store proc to get and update this table. only catch here that table which you have to push must have a timestamp column to compare and find change after last run.
@souranwaris142 Жыл бұрын
@@SoftWizCircle I am pushing table directly to blob storage account I don't have azure SQL and I am getting table from on premise server SQL so that's why I have a confusion where I need to create this. Please help me for this. Is it possible can give me 5mnt on call show I can show you my problem
@varunvengli17625 ай бұрын
Even I'm stuck on this issue
@ranjansrivastava9256 Жыл бұрын
Good Article, Lack of Visibility of the Video.
@SoftWizCircle Жыл бұрын
Sorry for that I tried to record in HD this abut still seems somehow blur.
@mk-ho9iz9 ай бұрын
what the fuck....how can lower lastmodifytime can pass through it..meaning the copy condition gets false and data should not be copied
@SoftWizCircle9 ай бұрын
Please try once. last modified date is a column in sql table in this demo and we can parametrize sql statements or stored proc etc and pass value from adf