No video

2. Get File Names from Source Folder Dynamically in Azure Data Factory

  Рет қаралды 95,554

WafaStudies

WafaStudies

Күн бұрын

In this video, I discussed about Getting File Names Dynamically from Source folder in Azure Data Factory
Link for Azure Functions Play list:
• 1. Introduction to Azu...
Link for Azure Basics Play list:
• 1. What is Azure and C...
Link for Azure Data factory Play list:
• 1. Introduction to Azu...
#Azure #ADF #AzureDataFactory

Пікірлер: 83
@rbor-xb5eg
@rbor-xb5eg 9 ай бұрын
thank you sir, youre legend, respect from brazil
@GunNut37086
@GunNut37086 Жыл бұрын
Expertly done!! You explained this perfectly for me. Thank you for sharing your expertise.
@WafaStudies
@WafaStudies Жыл бұрын
Thank you 😊
@damayantibhuyan7147
@damayantibhuyan7147 4 жыл бұрын
Nice videos, Clear Steps.. Please keep uploading. Thank You !
@debasisroy9625
@debasisroy9625 4 жыл бұрын
Excellent content !! Thanks Mate for taking out your time. Could you pls do a series on Data Factory DevOps integration. Building a CI / CD pipeline using library variables
@rakeshupadhyay1
@rakeshupadhyay1 2 жыл бұрын
Very good content, practical scenarios are helpful
@WafaStudies
@WafaStudies 2 жыл бұрын
Thank you ☺️
@empowerpeoplebytech2347
@empowerpeoplebytech2347 3 жыл бұрын
Good one, very helpful and practical scenario. You made it exactly as it is needed!
@aruntejvadthya1309
@aruntejvadthya1309 3 жыл бұрын
Very good explanation👌 . Its very helpful
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you 🙂
@tipums123
@tipums123 4 жыл бұрын
Very good explanation. Thank you for this video!
@sabinsunny655
@sabinsunny655 3 жыл бұрын
Good content , really helpful, clear steps , thanks for the video
@gubscdatascience1805
@gubscdatascience1805 2 жыл бұрын
very helpful sir
@WafaStudies
@WafaStudies 2 жыл бұрын
Thank you 😊
@MarceloSilva-us1gh
@MarceloSilva-us1gh 3 жыл бұрын
Amazing! Thank you my friend!
@WafaStudies
@WafaStudies 3 жыл бұрын
Welcome 🤗
@abnavalagatti
@abnavalagatti 3 жыл бұрын
Very good explanation
@siddheshamrutkar8684
@siddheshamrutkar8684 2 жыл бұрын
What a content boss.. Really very impressive.. May I know which videos should I refer to get started with Azure Cloud as I am relatively new to this.. I know MSBI and wants to get upgrade myself to Azure Cloud.. Kindly suggest and Your Contents are awesome.. Hats off to You.. 🤟👏
@WafaStudies
@WafaStudies 2 жыл бұрын
U can start with Azure data factory and Azure Synapse Analytics
@bhavnakamble975
@bhavnakamble975 3 жыл бұрын
Amazing and really helpful
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you 🙂
@SatyaPParida
@SatyaPParida 4 жыл бұрын
Wonderful content ✌️. Really helpful
@WafaStudies
@WafaStudies 4 жыл бұрын
Thank you 🙂
@subhashkomy
@subhashkomy 3 жыл бұрын
Can you please create a video, how to upload multiple Excel data in Sql Server using Data Flows and please also used data conversion. It doesn't seem to be as easy as we do in SSIS.
@deepaksahirwar
@deepaksahirwar 2 жыл бұрын
It would much helpful if uploaded
@deepaksahirwar
@deepaksahirwar 2 жыл бұрын
Thank you for great explanation. Please could we expect learning videos on Azure Synapse Analytics?
@WafaStudies
@WafaStudies 2 жыл бұрын
I created a playlist on Synapse analytics. Kindly check it. Link below. kzfaq.info/sun/PLMWaZteqtEaIZxPCw_0AO1GsqESq3hZc6
@deepaksahirwar
@deepaksahirwar 2 жыл бұрын
@@WafaStudies Tons of thank you,dear Sir . Much much helpful
@WafaStudies
@WafaStudies 2 жыл бұрын
@@deepaksahirwar welcome 😁
@sirisiri3797
@sirisiri3797 2 жыл бұрын
Very usefull videos pls make data bricks videos also
@WafaStudies
@WafaStudies 2 жыл бұрын
Hi I am already doing databricks playlist. Kindly check it. Currently that playlist is inprogress
@sirisiri3797
@sirisiri3797 2 жыл бұрын
If possible can you make in telugu also
@YanadireddyBonamukkala
@YanadireddyBonamukkala 3 жыл бұрын
If possible try a video on crating Global parameters and pass the values dynamically with different DataBases
@venukumar1094
@venukumar1094 2 жыл бұрын
Hi Maheer, Thanks for the detailed explanation. For this topic the scenario should be "Read Files from Source Folder Dynamically in Azure Data Factory" instead Get File Names.. We are not reading/getting "filenames" right, the files were being just copied from source to Sink.?
@tallaravikumar4560
@tallaravikumar4560 Жыл бұрын
First he is copying file names n then using those file names he is copying the files.
@anujgupta-lc1md
@anujgupta-lc1md 4 жыл бұрын
amazing plz add some incremental load handling data & how to check whether files are present in blob or not from validation or getmetda actitiy.
@WafaStudies
@WafaStudies 4 жыл бұрын
Sure will add videos on those scenarios. Thank you
@Kannan-lt1ud
@Kannan-lt1ud 3 жыл бұрын
thank you, this helped me...
@lib133
@lib133 10 күн бұрын
what if one wants to to a similar things but with .txt or .sql files ( stored in a ADLS Gen 2 container ) ?
@sandeep5996
@sandeep5996 4 жыл бұрын
Nice work! Could you please make a video on. How to check 0Kb csv files / zero row record from source. If zero Kb file/Zero records in source trigger an email, in azure Data Factory. Thanks in advance.
@yusufuthman8571
@yusufuthman8571 3 жыл бұрын
Hi, This Tutorial is really helpful but I find it difficult to point the files to a database instead of the same directory as the source file as is the case in most real life scenarios, please help on how to get these files into an SQL server database
@pradeepert
@pradeepert 2 жыл бұрын
Can we pass file path dynamically? I have sql table from there I can take the file path. This file path needs to be passed to getmetadata and list the files. Looking for your help. Thank you so much!
@rajeshgowd
@rajeshgowd Жыл бұрын
Hello Maheer, could you please make a video about copying excel files. if we implement as like above video, in excel files its asking sheet name.
@esrasultan8963
@esrasultan8963 10 ай бұрын
Hi Maheer, do you video, where we copy csv file from dynamic folder in adls to new folder in adls and store it as parquet.
@ArabaEfsanesi
@ArabaEfsanesi 2 жыл бұрын
Thanks for video first. My question is; what should i do if i want to copy only a specific file types from my input folder (ex. just csv files) in this foreach loop example?
@viveknimmagadda2397
@viveknimmagadda2397 Жыл бұрын
At minutes 4:27 in the video, we can see the recursive property, however, that's not the case for me. The software has been updated and the video might be outdated. Can you please help me with this as I cannot find the recursive property?
@mobinbagwan5747
@mobinbagwan5747 3 жыл бұрын
Can i use the items().name inside activity which will be inside a if else loop activity which is inside for each activity ? I hope my question makes sense
@WafaStudies
@WafaStudies 3 жыл бұрын
You can use if that expression is within foreach
@itech7313
@itech7313 Жыл бұрын
Hi please tell us how do we import multiple files from different sources in ADF (this is interview quesion)
@princyshinas8740
@princyshinas8740 2 жыл бұрын
Hi... I could nt see recursive option in synapse analytics. can u suggest some ways to get subfolder?
@mohangonnabathula2261
@mohangonnabathula2261 2 жыл бұрын
I can't find the "recursively " option anymore in Get Metadata activity. Can you please let me know how to get all the files recursively in GetMetadata acty ? Thanks.
@mathangiananth6599
@mathangiananth6599 3 жыл бұрын
Hi , thanks for uploading amazing content . i have one question here , what if we hav different file types in the source blob container like .TXT,.CSV,parquet files , orc files and I want to copy all of these to a different path . This was asked in one of my interviews. Can you suggest what can be done here .TIA
@sumitbarde3677
@sumitbarde3677 2 жыл бұрын
in this scenario you can use a lookup activity. Create a config file store all the dynamic parameters you want to pass to a pipeline, after lookup use foreach and copy activity. Sorted
@sid0000009
@sid0000009 3 жыл бұрын
the data set which you use to get the file names is not the same what you select inside the copy activity. Ideally both should be same ?
@manognadamacharla3346
@manognadamacharla3346 3 жыл бұрын
Hello thanks for your wonderful videos, can you please give an idea on pushing all folders along with files in data lake with one go.
@kunalratnaparkhi828
@kunalratnaparkhi828 3 жыл бұрын
Could it be possible to create a destination folder structure same as the source folder structure automatically while uploading files in a data leak using a data factory? The destination folder structure should be created automatically. Example - I have 3 files in the "C:\Test\Upload" folder. This is on-premises. Now I want to upload those files in data leak using the data factory and destination folder structure should be C:\Test\Upload", which should create automatically. Please advice.
@sureshch8328
@sureshch8328 3 жыл бұрын
Hi I am using pkg parameter Startdate is false but it was not picking parameter due to UTC time zone how can we change this dynamically.
@vinayraghuwanshi9419
@vinayraghuwanshi9419 4 жыл бұрын
In your metadata, there is an option of recursively but in my metadata, I did not see this option
@srinivasu5984
@srinivasu5984 3 жыл бұрын
nice
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you 😊
@anupgupta5781
@anupgupta5781 3 жыл бұрын
If want to copy different files into different output sub folder how could I do it
@rakeshmishra4650
@rakeshmishra4650 3 жыл бұрын
Nice video but if I'd like to store/capture the filename into a sql table column then how can we do that? Could you please update us here. That means load the three files into sql table with respective filename into sql table column. Thanks
@itech7313
@itech7313 Жыл бұрын
He please us how do we import data from different sources for multiple files this is interview quetion
@ravisamal3533
@ravisamal3533 3 жыл бұрын
if i want to copy multiple file from blob to sql db , how can i do so
@vijaysagar5984
@vijaysagar5984 2 жыл бұрын
Hi Bro, Any workaround for CSV files which has multiple headers and we can merge them as one Header ? Source is FTP and some files are good and some files has multiple headers.
@krishnakuraku6853
@krishnakuraku6853 4 жыл бұрын
How can we execute dataflow tasks parallel in ADF..? Any idea please..
@WafaStudies
@WafaStudies 4 жыл бұрын
To execute data flows in adf u need to use data flow activity. You may have your data flow in foreach and execute it for multiple iterations in parallel
@krishnakuraku6853
@krishnakuraku6853 4 жыл бұрын
@@WafaStudies Thanks bro....
@krishnakuraku6853
@krishnakuraku6853 4 жыл бұрын
if the schema varies how foreach detect metadata of different tables or files and load data into different tables or files.
@WafaStudies
@WafaStudies 4 жыл бұрын
@@krishnakuraku6853 u need handle that. Allow schema drift option allow you to handle that. Pls check my vidoe on schema drift concept in data flows.
@fannymaryjane
@fannymaryjane 3 жыл бұрын
Can I use with the configured wildcard?
@YanadireddyBonamukkala
@YanadireddyBonamukkala 3 жыл бұрын
Is there any possible to change the linked service dynamically at triggering level, if yes please share your video here
@WafaStudies
@WafaStudies 3 жыл бұрын
kzfaq.info/get/bejne/g5higM1h18mmk6c.html
@bandarurohithkumar439
@bandarurohithkumar439 2 жыл бұрын
How to contact you?
@ayubshaik2415
@ayubshaik2415 3 жыл бұрын
Hi.......
@stangoodvibes
@stangoodvibes Жыл бұрын
ANOTHER video that shows the same thing - getting files from a singlr folder level. Too easy. How about getting all the files from a nested folder structure where the actual files may be n levels down (n is unknown)???
4. Delete old files from storage using Azure Data Factory
16:39
WafaStudies
Рет қаралды 43 М.
Алексей Щербаков разнес ВДВшников
00:47
👨‍🔧📐
00:43
Kan Andrey
Рет қаралды 10 МЛН
UNO!
00:18
БРУНО
Рет қаралды 4,7 МЛН
Azure Data Factory | Copy multiple tables in Bulk with Lookup & ForEach
23:16
Adam Marczak - Azure for Everyone
Рет қаралды 188 М.
Azure Data Factory - Dynamic File Names with expressions
14:32
MitchellPearson
Рет қаралды 24 М.
1. Handle Error Rows in Data Factory Mapping Data Flows
31:48
WafaStudies
Рет қаралды 153 М.
5. Copy Activity in azure data factory | Copy data activity in ADF
14:02
learn by doing it
Рет қаралды 4,6 М.
Azure Files vs SharePoint: Best Document Storage Option?
14:58
Jonathan Edwards
Рет қаралды 24 М.