Azure Data Factory | Copy multiple tables in Bulk with Lookup & ForEach

  Рет қаралды 184,483

Adam Marczak - Azure for Everyone

Adam Marczak - Azure for Everyone

4 жыл бұрын

With Azure Data Factory Lookup and ForEach activities you can perform dynamic copies of your data tables in bulk within a single pipeline.
In this episode I will show you how to perform bulk copies with ADF.
Source code for demos: github.com/MarczakIO/azure4ev...
In this episode live demo of ADF lookup activity in SQL to Blob export scenario and how to control this using metadata table.
Next steps for you after watching the video
1. ADF bulk copy scenario
- docs.microsoft.com/en-us/azur...
2. Lookup Activity docs
- docs.microsoft.com/en-us/azur...
3. ForEach Activity docs
- docs.microsoft.com/en-us/azur...
Want to connect?
- Blog marczak.io/
- Twitter / marczakio
- Facebook / marczakio
- LinkedIn / adam-marczak
- Site azure4everyone.com

Пікірлер: 356
@rapchak2
@rapchak2 2 жыл бұрын
Cannot thank you enough for your incredibly well laid out, thorough explanations. The world needs more folks like you :)
@quyenpn318
@quyenpn318 2 жыл бұрын
I really really like how you guide step by step like this, it is quite easy to understand. You are the best “trainner” I’ve seen, really appreciated for your time on creating those useful videos.
@albertoarellano1494
@albertoarellano1494 4 жыл бұрын
You're the best Adam! Thanks for all the help, been watching your tutorials on ADF and they're very helpful. Keep them coming!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
My pleasure! Thanks Alberto!
@ahmedroberts4883
@ahmedroberts4883 Жыл бұрын
Excellent, Excellent video. This has truly cemented the concepts and processes you are explaining in my brain. You are awesome, Adam!
@genniferlyon8577
@genniferlyon8577 2 жыл бұрын
Thank you Adam! I had been trying to follow some other written content to do exactly what you showed with no success. Your precise steps and explanation of the process were so helpful. I am successful now.
@apogeeaor5531
@apogeeaor5531 Ай бұрын
Thank you, Adam. I rewatch this video at least twice a year, Thank you for all you do.
@shaileshsondawale2811
@shaileshsondawale2811 Жыл бұрын
What a wonderful content you have place in social media.. What a world class personality you... People certainly fall in love with your teaching..
@gunturulaxmi8037
@gunturulaxmi8037 Жыл бұрын
Videos are very much clear to the people who would like to learn and practice.Thanks alot.your hard work is appreciated.
@rajanarora6655
@rajanarora6655 2 жыл бұрын
Awesome explanation, the way you teach assuming in layman terms is pretty great, thanks!!
@pratibhaverma7857
@pratibhaverma7857 2 жыл бұрын
Your videos are great. This is the best channel on KZfaq platform to learn about ADF. THANKS 🙏😊
@RavinderApril
@RavinderApril 3 ай бұрын
Incredibly simplified to learn. .. Great!!
@satishutnal
@satishutnal 3 жыл бұрын
You are the example for how teaching should be. Just awesome 👍
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Wow, thank you!
@elisehunter3424
@elisehunter3424 2 жыл бұрын
Brilliant tutorial. Easy to follow and it all works like a charm. Thank you!!
@markdransfield9520
@markdransfield9520 Жыл бұрын
Brilliant teaching style Adam. Very watchable. I particularly like how you explain the background. I've subscribed and will watch more of your videos.
@jakirajam
@jakirajam Жыл бұрын
The way you explain is super Adam. Really nice
@hollmanalu
@hollmanalu 4 жыл бұрын
Adam, thanks for all your great video's! I appreciate your work very much! Keep up your great work!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
My pleasure! Thanks!
@santanughosal9785
@santanughosal9785 2 жыл бұрын
I was looking for this video. Thanks for making this. It helps a lot. Thanks again.
@AVADHKISHORE
@AVADHKISHORE 3 жыл бұрын
Thank you Adam!! These videos are really very helpful and builds the foundation to understand ADF.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
My pleasure!
@anacarrizo2209
@anacarrizo2209 11 ай бұрын
THANK YOU SO MUCH for this! The step-by-step really helped with what I needed to do.
@radiomanzel8570
@radiomanzel8570 Жыл бұрын
it was so perfect , I was able to follow and copy data in first attempt .thanks
@leonkriner3744
@leonkriner3744 Жыл бұрын
Amazingly simple and informative!
@anderschristoffersen2513
@anderschristoffersen2513 3 жыл бұрын
Great and simple walk through, good job Adam
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you, I appreciate it! :)
@sumanthdixit1203
@sumanthdixit1203 3 жыл бұрын
Fantastic clear-cut explanation. Nice job!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad it was helpful!
@pdsqsql1493
@pdsqsql1493 2 жыл бұрын
Wow! What Great video, very easy way step by step tutorials and explanations. Well done!
@ericsalesdeandrade9420
@ericsalesdeandrade9420 8 күн бұрын
Amazing video. Complex topic perfectly explained. Thank you Adam
@paullevingstone4834
@paullevingstone4834 3 жыл бұрын
Very professionally demonstrated and very clear to understand. Thank you very much
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
It's my pleasure Paul! :)
@diatprojects7220
@diatprojects7220 Жыл бұрын
Very well explained. Thank you so much!
@CoolGuy
@CoolGuy 2 жыл бұрын
You are a legend. Next level editing and explanation
@Cheyenne9663
@Cheyenne9663 Жыл бұрын
Wow this was explained so well. Thank you!!!
@priyankapatel9461
@priyankapatel9461 3 жыл бұрын
You have depth knowledge in every service. I learn from scratch using your channel. Keep posting Thanks you and God bless you.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Awesome, thanks!
@garciaoscar7611
@garciaoscar7611 11 ай бұрын
This video was really helpful! you have leveled up my Azure skills, Thank you sir, you have gained another subscriber
@amoldesai4605
@amoldesai4605 4 жыл бұрын
I am a beginner in Azure Data Engineering and you made it simple to learn all the tactics.. thanks
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Glad to hear that!
@xiaobo1134
@xiaobo1134 3 жыл бұрын
Thanks Adam, your tutorials are very useful, hope to see more in the future
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you like them! Will do more!
@e2ndcomingsoon655
@e2ndcomingsoon655 2 жыл бұрын
Thank you! I really appreciate all you share, it truly helps me
@maimemphahlele1102
@maimemphahlele1102 3 жыл бұрын
Hi Adam Ur videos are just too brilliant. This is subscription I wouldn’t mind paying to support. Ur lessons are invaluable to learning.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Awesome, thank you!
@deoroopnarine6232
@deoroopnarine6232 4 жыл бұрын
Your videos are awesome man. Gave me a firm grasp and encouraged me to get an azure subscription and play around some more.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
That is amazing to hear! Thank you!
@amtwork5417
@amtwork5417 3 жыл бұрын
Great video, easy to follow and to the point, really helped me to quickly get up a running with data factory.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad it helped!
@anilchenchu1017
@anilchenchu1017 Жыл бұрын
Awsome adam there cant be a way to explain better than this
@amarnadhgunakala2901
@amarnadhgunakala2901 4 жыл бұрын
Thanks Adam, I'm waiting like this video on ADF, Please do regularly...
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
You got it!
@avicool08
@avicool08 2 жыл бұрын
very simple yet powerful explanation
@AdamMarczakYT
@AdamMarczakYT 2 жыл бұрын
Glad you think so!
@wouldyoudomeakindnes
@wouldyoudomeakindnes 3 жыл бұрын
your skills are in the tops thanks, love to see your channel grow
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
I appreciate that!
@NewMayapur
@NewMayapur 4 жыл бұрын
fantastic video Adam!! Really helpful to understand the parametrisation in ADF.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Great to hear that!
@waseemmohammed1088
@waseemmohammed1088 3 жыл бұрын
Thank you so much for the clear and nice explanation, I am new to ADF and learning a lot from your channel
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Great to hear!
@ashokveluguri1910
@ashokveluguri1910 4 жыл бұрын
You are awesome Adam. Thank you so much for detailed explanation.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
My pleasure!
@jacobklinck8011
@jacobklinck8011 3 жыл бұрын
Great session!! Thanks Adam.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
My pleasure!
@eversilver99
@eversilver99 3 жыл бұрын
Excellent video and knowledge sharing. Great Job!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you enjoyed it!
@naseemmca
@naseemmca 3 жыл бұрын
Adam you are just awesome man! The way you are teaching is excellent. Keep it up.. you are the best...
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks! 😃
@Lucas48288
@Lucas48288 3 жыл бұрын
These tutorials are so useful!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you like them!
@eatingnetwork6474
@eatingnetwork6474 3 жыл бұрын
Thanks Adam, amazing workshop, very clear and easy to follow, thanks for helping, i am wiser now :)
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Perfect! Thank you!
@ElProgramadorOficial
@ElProgramadorOficial 3 жыл бұрын
Adam, You are the best!. Thanks man!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you :)
@sameeranavalkar9352
@sameeranavalkar9352 Жыл бұрын
Thank you so much for this. It helped a lot
@TheSQLPro
@TheSQLPro 3 жыл бұрын
Great content, easy to follow!!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you think so!
@nathalielink3869
@nathalielink3869 3 жыл бұрын
Awesome. Thank you so much Adam!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
My pleasure!
@GaneshNaik-lv6jh
@GaneshNaik-lv6jh 3 ай бұрын
Thank You so much.... Very good explanation, Just Awesome
@Vick-vf8ug
@Vick-vf8ug 3 жыл бұрын
It is extremely hard to find information online about this topic. Thank you for making it easy!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad it was helpful! Thanks!
@geoj9716
@geoj9716 3 жыл бұрын
You are a very good teacher.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you! 😃
@dev.gaunau
@dev.gaunau 3 жыл бұрын
Thank you so much for sharing these valued knowledge. It's very helpful for me.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad it was helpful!
@southernfans1499
@southernfans1499 2 жыл бұрын
👍👍👍 very good explanation.. 👍👍.
@kamalnathvaithinathan5737
@kamalnathvaithinathan5737 4 жыл бұрын
Awesome Adam!! you are the best. Thank you so much
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
My pleasure! :)
@satyabratabarik49
@satyabratabarik49 Жыл бұрын
Great Explanation !!!!
@prasadsv3409
@prasadsv3409 4 жыл бұрын
Realy great stuff sir.this what am looking in youtube
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks a ton!
@shivangrana02
@shivangrana02 4 жыл бұрын
You are really best Adam! Your tutorial helped me a lot. Thanks
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Happy to hear that!
@shivangrana02
@shivangrana02 4 жыл бұрын
@@AdamMarczakYT You are welcome. Please keep up the good work.
@jac94
@jac94 4 жыл бұрын
Thanks Adam! very clear!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Glad it was helpful!
@abhishektrivedi3406
@abhishektrivedi3406 4 жыл бұрын
You're awesome Adam, thanks for such a great tutorial. I also tweeted this video. Thanks.!!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Awesome, thank you!
@mosestes9417
@mosestes9417 4 жыл бұрын
Really helpful! you made it very easy!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Glad you think so!
@dataisfun4964
@dataisfun4964 8 ай бұрын
Thanks, this is beautiful.
@ivanovdenys
@ivanovdenys 4 жыл бұрын
It is really cool that you make it so simple :)
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thank you! 😊
@verakso2715
@verakso2715 2 жыл бұрын
Thanks for your awesome video, it helped me out a great deal
@AdamMarczakYT
@AdamMarczakYT 2 жыл бұрын
Glad I could help
@jozefmelichar5960
@jozefmelichar5960 3 жыл бұрын
Fine tutorial. Thanks.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad it was helpful!
@krzysztofrychlik9913
@krzysztofrychlik9913 3 жыл бұрын
dzieki! bardzo pomocne filmy!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Dziekoweczka!
@rubensanchez6366
@rubensanchez6366 3 жыл бұрын
Very interesting video Adam. I found quite enlightening your idea of storing metadata. probably it could be maintained separately tracking last record loaded so we could use it as an input for delta loads through queries instead of reloading the full table on each run.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
You can either use watermark or change tracking patterns check this out docs.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview?WT.mc_id=AZ-MVP-5003556
@pavankumars9313
@pavankumars9313 2 жыл бұрын
You are very good 👍 explained well thanks 😊
@arjunsaagi590
@arjunsaagi590 4 жыл бұрын
Thank you Adam, Very much informatics video
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
My pleasure!
@RajivGuptaEverydayLearning
@RajivGuptaEverydayLearning 3 жыл бұрын
Very nice video with good explanation.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you liked it
@Ro5ho19
@Ro5ho19 3 жыл бұрын
Thank you! It's under appreciated how important it is to name things something other than "demoDataset", but it makes a big difference both for understanding concepts, and maintainability.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad it was helpful! You are of course correct, if it's not demo then take care of your naming conventions.
@BACHARBOUAZZA
@BACHARBOUAZZA Жыл бұрын
Thank you! well done.
@matthewmark7224
@matthewmark7224 6 ай бұрын
amazing work. thanks.
@surafeltilahun7404
@surafeltilahun7404 3 жыл бұрын
Excellent!!!!!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Many thanks! Cheers!
@nalinbuddhika
@nalinbuddhika 4 жыл бұрын
Thanks for the vidoes. I am letting the adds runs till the end :)
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Awesome thank you for your support! ♥
@MrPadmanabham
@MrPadmanabham 4 жыл бұрын
Very good and clear..
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Glad it was helpful!
@leelaagrawal6375
@leelaagrawal6375 3 жыл бұрын
great teacher
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you! 😃
@hamzaehsankhan
@hamzaehsankhan 4 ай бұрын
BRILLIANT
@gouravjoshi3050
@gouravjoshi3050 3 жыл бұрын
Good one adam sir .
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you :)
@praveensharma-uw8ds
@praveensharma-uw8ds 2 жыл бұрын
thanks for the video
@marvinvicente3138
@marvinvicente3138 2 жыл бұрын
Hi Adam, I really appreciate you video. Thanks for your videos! I hope you can also create a video for ODBC as data source.
@vicvic553
@vicvic553 3 жыл бұрын
Hey, one thing about English - please guys correct me if I am wrong, but I am pretty sure what I am talking about - you shouldn't say inside a sentence "how does it work", but "what it works". Despite that, the content is awesome!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
You can if you ask a question. "How does it work" is a question structure, not a statement. it should be "how it works" if I'm stating a fact. You wrote "What it works" but I assume that's a typo. It's one of my common mistakes, my English teacher tries to fix it but it is still a common issue for me ;) Thanks for watching!
@BijouBakson
@BijouBakson 2 жыл бұрын
Thank you.
@aks541
@aks541 4 жыл бұрын
Very well explained & succinct. One request - if possible create a video for loading ADW (Synapse) data-warehouse by ADF
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks! I'm waiting for synapse new workspace experience to be released to make video about it ;)
@veerboot81
@veerboot81 4 жыл бұрын
Hi Adam, very nice work this, I made this for a client of mine and found out one important thing: within the For each not all blocks are executed as if they are working together atomically. What I mean is that if you start two thinks in parallel using the For Each block and within the for each block you have two blocks - say block A and B - connected using parameters (item()) within these blocks say X and Y. Block A starting with item X will not necessarily be using the item X in block B although connected! So I want to suggest one extra advice to use only one block in a For Each block at max if using parameterazed block within or if you need more than one block start a separate pipeline within the For Each block which will have the multiple blocks. These pipelines will be started as separate childs and to the work in the correct order. With kind regards, Jeroen
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Hey, not sure I understood what you meant here. Using parameters is not making any connection between the actions.
@veerboot81
@veerboot81 4 жыл бұрын
@@AdamMarczakYT I'm using a for each loop to load tables with dynamic statements, if I need more than one block (like a logging call to sl server, a copy block to load the data and a logging block after being done with loading these blocks can be in the for each loop itself, but if you start in parallel multiple times different load of tables the blocks will not follow each other sequencially, but will be running through each other, so the logging will not belong to the copy block for example. I will see if I can make an example if I find the time. To solve this I start always another pipeline within the for each and put the blocks in this pipeline. This will create child pipelines in the for each loop ensuring the right order of execution of the blocks (logging start, copy and logging end)
@feliperegis9989
@feliperegis9989 3 жыл бұрын
Hey Adam, awesome work and explanation! Do you have another video explaining how to deal with massive data copies from tables in bulk using ADF and that may resolve issues with maximum data or rows of data? Can you make a video with a demo explaining how to deal with this kind of scenarios that you mentioned that's the story for another day? Thanks a lot in advance!! =D
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks. Well, Lookup shouldn't be used for data, but for metadata driven approach, so 5000 rows limit is very good here. It is rare when you will copy over 5000 tables/files with different structure)/etc. If you do you can do different techniques but in those cases I probably would shift approach entirely. Will think about this.
@alikh1984
@alikh1984 Жыл бұрын
Greate one!
@pedrovasconcelos3204
@pedrovasconcelos3204 Жыл бұрын
Very Good
@CoopmanGreg
@CoopmanGreg 4 жыл бұрын
... GOLD !!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Awesome!
@RobHeim
@RobHeim 7 ай бұрын
Thanks!
@szymonzabiello2622
@szymonzabiello2622 3 жыл бұрын
Hey Adam. Great video! Two questions regarding the pipeline itself. 1. How do we approach Source Version Control of the pipeline? In SSIS we could export a package and commit to Git or use TFS. How do we approach versioning in Azure? 2. What is the approach to deploy this pipeline in upper environment? Assuming that this pipeline was created in dev, how do I approach deployment in i.e. UAT?
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
I think this page describes and answers both of your questions. docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment?WT.mc_id=AZ-MVP-5003556 thanks for watching :)
@simonscott1121
@simonscott1121 2 жыл бұрын
Reminds me so much of SSIS.
@paulh1720
@paulh1720 Жыл бұрын
awesome
@vinayrajm.u847
@vinayrajm.u847 3 жыл бұрын
very nice
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks
@maranp6618
@maranp6618 4 жыл бұрын
Please do a video on Parameterized pipeline in detail with different parameters
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Did you had a chance to check my video on ADF parametrization? kzfaq.info/get/bejne/pq-Ddcqo18rcgIE.html
@joncrosby1
@joncrosby1 2 жыл бұрын
Thank you for your great videos, they have been super helpful. I'm working on a proof of concept similar to this video, however it's SQL to Azure SQL. Any links or references you can offer to help me with parameterizing the Azure SQL sink side?
Azure Data Factory Parametrization Tutorial
22:07
Adam Marczak - Azure for Everyone
Рет қаралды 172 М.
Please be kind🙏
00:34
ISSEI / いっせい
Рет қаралды 191 МЛН
3M❤️ #thankyou #shorts
00:16
ウエスP -Mr Uekusa- Wes-P
Рет қаралды 10 МЛН
Always be more smart #shorts
00:32
Jin and Hattie
Рет қаралды 43 МЛН
Vivaan  Tanya once again pranked Papa 🤣😇🤣
00:10
seema lamba
Рет қаралды 29 МЛН
Azure Data Factory Tutorial | Introduction to ETL in Azure
24:59
Adam Marczak - Azure for Everyone
Рет қаралды 831 М.
18. Copy multiple tables in bulk by using Azure Data Factory
18:27
Data API builder is now Generally Available | Data Exposed
9:39
Microsoft Developer
Рет қаралды 6 М.
Process Excel files in Azure with Data Factory and Databricks | Tutorial
34:14
Adam Marczak - Azure for Everyone
Рет қаралды 114 М.
50. Copy Incremental data within Azure SQL DB - Multiple Tables
32:59
CloudAndDataUniverse
Рет қаралды 8 М.
Azure Data Factory Custom Email Notifications Tutorial
29:22
Adam Marczak - Azure for Everyone
Рет қаралды 82 М.
Azure Data Factory Mapping Data Flows Tutorial | Build ETL visual way!
26:25
Adam Marczak - Azure for Everyone
Рет қаралды 221 М.
После ввода кода - протирайте панель
0:18
Up Your Brains
Рет қаралды 1 МЛН
Main filter..
0:15
CikoYt
Рет қаралды 14 МЛН