5. How to load all the foldernames and count of files inside each folder into SQL table using ADF

No video

5. How to load all the foldernames and count of files inside each folder into SQL table using ADF

Рет қаралды 9,154

Күн бұрын

In this video we learnt how to get all the foldernames and count of files inside each of the folders and load it into SQL table using ADF pipeline
#adf
#azuredatafactory
#datafactory

Пікірлер: 38

@mybinotes3294 Жыл бұрын

.. I learned ADF watching your videos .. thank you so much

@azurecontentannu6399 Жыл бұрын

Glad to know that 😇 All the best for future

@avinashthakur4744 2 жыл бұрын

Nice nice Explanation is very good Annu Keep up the good work ✌️😊

@azurelearner4055 8 ай бұрын

hello after 15 min this video is getting stuck and not able to follow afterwards please check

@rajnishrajput2367 Жыл бұрын

Thanks Annu, very nicely explained I wanted to loop through 5 layers of subfolders and copy the jason to azure sql db, this example was useful

@azurecontentannu6399 Жыл бұрын

Very glad to know it was helpful 😊 Thanks for watching and commenting

@HugoAraujo 10 ай бұрын

Great! This was what I was looking for.

@Akashthakur-pu1ps 11 ай бұрын

Very well explained

@user-cp9xm6mw2c 9 ай бұрын

Hi Annu, thanks for your videos and sharing knowledge ,could you please make videos on incremental load, scd1,scd2

@azurecontentannu6399 9 ай бұрын

Kindly watch out part 9 to 12 of synapse playlist, similar implementation can be done in ADF: kzfaq.info/get/bejne/fL6nZMqBxseWXYE.html

@user-cp9xm6mw2c 9 ай бұрын

Thanks Annu

@hritiksharma7154 2 жыл бұрын

Very nice video

@kishorkumar007 Жыл бұрын

Thxs for the video this has made my job simple. i am really thankful mam..

@azurecontentannu6399 Жыл бұрын

Thankyou for watching the videos. Glad to know it was helpful

@OnlineForward Жыл бұрын

This is impressive use case, but now consider everything until FILTER then copy to a blob storage all the files you get from filter.

@emmanuelokwoli4069 8 ай бұрын

Great video! Can you please show how you can copy this multiple files into a database? not just the number of files but the con tent of the files

@Tillutelugu 2 жыл бұрын

Thank you

@ashokreddykunduru5038 Жыл бұрын

can you please explain when we use Get Metadata activity and Lookup activity & deference between the Get Metadata activity and Lookup activity

@chittaranjanpradhan5290 Жыл бұрын

HI any, first of all good job on making such videos. One thing I didn't get it in the lookup activity you have used for folder name as '@{item().name}.But how the pipeline will come to know that is from GEtMetadata1 because we have one more fetMetadat2 also present where @{item().name} also present. Please explain

@chittaranjanpradhan5290 Жыл бұрын

Can u please explain

@azurecontentannu6399 Жыл бұрын

Hi Chittaranjan, thankyou for watching the video.. So, getmetadata2, filter and look up activities are inside for each . Foreach is the place where we have defined item as getmetadata1 output that means wherever inside foreach you refer item().name it will refer to foldernames one by one for each iteration. Now, if u see filter activity expression, there we are refering getmetadata2 output as item. So in filter expression when we are refereing item().name it means getmetadata2 output . But then again coming to lookup, item().name means getmetadata1 output. Now, suppose your requirement was to take all the file names coming out of getmetadata2 and copy all files as tables into sql , then we would add another foreach connected to getmetadata2 which is not directly possible to have foreach within foreach, so u gotta add execute pipeline and add foreach in child pipeline, and then pass getmetadata2 output as item in foreach2 and add copy activity within foreach2 and in copy activity if you refer item().name then it wl refer to getmetadata2 output not getmetadata1 output because you are within direct umbrella of foreach2 which means getmetadata2 output.

@twinklebatham1725 3 ай бұрын

Thank you so much🤗

@SantoshKumar-yr2md 5 ай бұрын

This is good but if we need to copy Folders with file inside folders, how to do that please suggest

@SaurabhMenon2897 2 жыл бұрын

Hello, as an alternative to the filter activity, could we instead use the length function over the childItems array of the second get metadata to get the number of files inside the folder?

@azurecontentannu6399 2 жыл бұрын

Yes that can be another possible solution 😊 Thanks for sharing

@SaurabhMenon2897 2 жыл бұрын

@@azurecontentannu6399 You’re welcome. Nice video! This is a good use case for metadata management.

@somily800 Жыл бұрын

Hello, I have one similar case but I need to iterate each container and each subfolder something like recursively, so I need to identify all the subfolder nested

@azurecontentannu6399 Жыл бұрын

You can try wildcard file path too

@DR-go3uy 9 ай бұрын

Hi, thank you for your video, could you tell instead of count how do i get actual folder path (example: 2023/11/23....n subfolder and actual file name) how to output the actual folder path instead of count of files.

@azurecontentannu6399 8 ай бұрын

you can use $Filepath option in additional column in copy activity but that will just give file name as the output

@nagababukamatam6192 Ай бұрын

if i hava a empty folder before deploy,it is showing folder 1 -count 8,folder 2 - count 5,folder 3 - count 0 after that if i copy a file in folder 3 the it is again show double time show's folder 3 - count 0 & folder 3 - count 1..please help me.

@stangoodvibes Жыл бұрын

Ok ive found several examples of this, NONE of which show a kind of recursion that then goes back into the level 2 folders to get or count the files in the subfolders. The use case in have is to start at a parent folder and extract the path and name of every file in all the subfolders. The folder structure is such that there may be any number of subfolders with files in the last child folder. The subfolders may be 6 or 7 levels deep 😑