Great share and great content with detailed steps on how to achieve the desired results from scratch. I like your videos where you share the knowledge by executing each and every steps involved. Although it is perfect video but there is room for the improvements. What I felt is missing in the explanation is a one pager diagram or the solution design. at one point viewer may loose the context at which stage what is happening. Great if you can corelate the steps you are performing with one pager slide in between so that ppl can sense at what stage what is happening and why is it important to perform certain steps.

if we re-run the pipeline for modified data will be duplicated?

Great video, thank you. Do you have a video to connect the latest modified file to Azure SQL from the output folder

very complex process.....ADF should have easier option for such activity.. its very common requirement to have incremental load.. should be more easier to implement...

is it so complicated to pick the last modified file in the folder?