In this video we learnt how to get all the foldernames and count of files inside each of the folders and load it into SQL table using ADF pipeline #adf #azuredatafactory #datafactory
Пікірлер: 38
@mybinotes3294 Жыл бұрын
.. I learned ADF watching your videos .. thank you so much
@azurecontentannu6399 Жыл бұрын
Glad to know that 😇 All the best for future
@avinashthakur47442 жыл бұрын
Nice nice Explanation is very good Annu Keep up the good work ✌️😊
@azurelearner40558 ай бұрын
hello after 15 min this video is getting stuck and not able to follow afterwards please check
@rajnishrajput2367 Жыл бұрын
Thanks Annu, very nicely explained I wanted to loop through 5 layers of subfolders and copy the jason to azure sql db, this example was useful
@azurecontentannu6399 Жыл бұрын
Very glad to know it was helpful 😊 Thanks for watching and commenting
@HugoAraujo10 ай бұрын
Great! This was what I was looking for.
@Akashthakur-pu1ps11 ай бұрын
Very well explained
@user-cp9xm6mw2c9 ай бұрын
Hi Annu, thanks for your videos and sharing knowledge ,could you please make videos on incremental load, scd1,scd2
@azurecontentannu63999 ай бұрын
Kindly watch out part 9 to 12 of synapse playlist, similar implementation can be done in ADF: kzfaq.info/get/bejne/fL6nZMqBxseWXYE.html
@user-cp9xm6mw2c9 ай бұрын
Thanks Annu
@hritiksharma71542 жыл бұрын
Very nice video
@kishorkumar007 Жыл бұрын
Thxs for the video this has made my job simple. i am really thankful mam..
@azurecontentannu6399 Жыл бұрын
Thankyou for watching the videos. Glad to know it was helpful
@OnlineForward Жыл бұрын
This is impressive use case, but now consider everything until FILTER then copy to a blob storage all the files you get from filter.
@emmanuelokwoli40698 ай бұрын
Great video! Can you please show how you can copy this multiple files into a database? not just the number of files but the con tent of the files
@Tillutelugu2 жыл бұрын
Thank you
@ashokreddykunduru5038 Жыл бұрын
can you please explain when we use Get Metadata activity and Lookup activity & deference between the Get Metadata activity and Lookup activity
@chittaranjanpradhan5290 Жыл бұрын
HI any, first of all good job on making such videos. One thing I didn't get it in the lookup activity you have used for folder name as '@{item().name}.But how the pipeline will come to know that is from GEtMetadata1 because we have one more fetMetadat2 also present where @{item().name} also present. Please explain
@chittaranjanpradhan5290 Жыл бұрын
Can u please explain
@azurecontentannu6399 Жыл бұрын
Hi Chittaranjan, thankyou for watching the video.. So, getmetadata2, filter and look up activities are inside for each . Foreach is the place where we have defined item as getmetadata1 output that means wherever inside foreach you refer item().name it will refer to foldernames one by one for each iteration. Now, if u see filter activity expression, there we are refering getmetadata2 output as item. So in filter expression when we are refereing item().name it means getmetadata2 output . But then again coming to lookup, item().name means getmetadata1 output. Now, suppose your requirement was to take all the file names coming out of getmetadata2 and copy all files as tables into sql , then we would add another foreach connected to getmetadata2 which is not directly possible to have foreach within foreach, so u gotta add execute pipeline and add foreach in child pipeline, and then pass getmetadata2 output as item in foreach2 and add copy activity within foreach2 and in copy activity if you refer item().name then it wl refer to getmetadata2 output not getmetadata1 output because you are within direct umbrella of foreach2 which means getmetadata2 output.
@twinklebatham17253 ай бұрын
Thank you so much🤗
@SantoshKumar-yr2md5 ай бұрын
This is good but if we need to copy Folders with file inside folders, how to do that please suggest
@SaurabhMenon28972 жыл бұрын
Hello, as an alternative to the filter activity, could we instead use the length function over the childItems array of the second get metadata to get the number of files inside the folder?
@azurecontentannu63992 жыл бұрын
Yes that can be another possible solution 😊 Thanks for sharing
@SaurabhMenon28972 жыл бұрын
@@azurecontentannu6399 You’re welcome. Nice video! This is a good use case for metadata management.
@somily800 Жыл бұрын
Hello, I have one similar case but I need to iterate each container and each subfolder something like recursively, so I need to identify all the subfolder nested
@azurecontentannu6399 Жыл бұрын
You can try wildcard file path too
@DR-go3uy9 ай бұрын
Hi, thank you for your video, could you tell instead of count how do i get actual folder path (example: 2023/11/23....n subfolder and actual file name) how to output the actual folder path instead of count of files.
@azurecontentannu63998 ай бұрын
you can use $Filepath option in additional column in copy activity but that will just give file name as the output
@nagababukamatam6192Ай бұрын
if i hava a empty folder before deploy,it is showing folder 1 -count 8,folder 2 - count 5,folder 3 - count 0 after that if i copy a file in folder 3 the it is again show double time show's folder 3 - count 0 & folder 3 - count 1..please help me.
@stangoodvibes Жыл бұрын
Ok ive found several examples of this, NONE of which show a kind of recursion that then goes back into the level 2 folders to get or count the files in the subfolders. The use case in have is to start at a parent folder and extract the path and name of every file in all the subfolders. The folder structure is such that there may be any number of subfolders with files in the last child folder. The subfolders may be 6 or 7 levels deep 😑
@harshbrahmbhatt3727 Жыл бұрын
Can we load the data of files in different folder into the azure sql table?
@talamanchisudheernadh698 Жыл бұрын
Hi Maam i have 10 files like file1,file2,.... i want to load first 5 files to sql can you do one video on this scenario