No video

18. Copy multiple tables in bulk by using Azure Data Factory

  Рет қаралды 60,763

WafaStudies

WafaStudies

Күн бұрын

In this video, I discussed about Copying multiple tables from SQL Database in bulk to Azure blob storage using Azure Data Factory
Link for Azure Databricks Play list:
• 1. Introduction to Az...
Link for Azure Functions Play list:
• 1. Introduction to Azu...
Link for Azure Basics Play list:
• 1. What is Azure and C...
Link for Azure Data factory Play list:
• 1. Introduction to Azu...
Link for Azure Data Factory Real time Scenarios
• 1. Handle Error Rows i...
Link for Azure LogicApps playlist
• 1. Introduction to Azu...
#Azure #ADF #AzureDataFactory

Пікірлер: 62
@diogodallorto1
@diogodallorto1 3 жыл бұрын
I've whatched so many classes of him that now I can perfectly understand his English. Thank you bro!
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you 🙂
@varunvengli1762
@varunvengli1762 5 ай бұрын
If we want to copy only few tables and automate it..whats the method? No one is KZfaq has explained this real time scenario.. Kindly explain
@nigereast3796
@nigereast3796 3 ай бұрын
I think you can use a stored procedure for this
@samrattidke896
@samrattidke896 3 жыл бұрын
Awesome...you deserve 80k+ subscribers...👍👍
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you 🙂
@sudheerjajjarapu6876
@sudheerjajjarapu6876 3 жыл бұрын
Scenario1: Copy all table data from SQL DB to Azure BLOB storage. Output file name will be Schema name ‘_’ table name .csv 1. Copy all tables in the Azure SQL DB 2. Put a condition if the table is having only one row or zero row then data will not be copied for that table. 3. Copy data in a BLOB storage path as CSV file in this scenario how , where and how we put 2nd condition
@bobbyj2492
@bobbyj2492 2 жыл бұрын
use lookup inside foreach table, to count num of rows in that table, if it matches the condition, then copy
@patrickmuaenah3733
@patrickmuaenah3733 Жыл бұрын
Thanks so much for this, Saved my week
@WafaStudies
@WafaStudies Жыл бұрын
Welcome 🤗
@RahulKumar-jg5ly
@RahulKumar-jg5ly 3 жыл бұрын
Superb explanation and presentation.... I watching your all the videos. 👍👍👍
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you ☺️
@rk-ej9ep
@rk-ej9ep Жыл бұрын
Clear explanation.👍👍Many Thanks
@sudipmazumdar5358
@sudipmazumdar5358 2 жыл бұрын
Great topic & very helpful for understanding several key activities. Thanks a lot :)
@WafaStudies
@WafaStudies 2 жыл бұрын
Thank you 😊
@MsWishnew
@MsWishnew 3 ай бұрын
Please discuss about the restartability, say for example out of 100, if 10 of the tables fail, I need to restart only the failed one automatically. How this scenario will be handled. BTW thanks for this tutorial :)
@Rafian1924
@Rafian1924 3 жыл бұрын
Really good man. Thanks a lot.
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you ☺️
@acverse
@acverse 2 жыл бұрын
Very helpful! Thank you :)
@WafaStudies
@WafaStudies 2 жыл бұрын
Welcome 😊
@atiry839
@atiry839 3 жыл бұрын
good one, thanks for sharing.
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you 😊
@MaheshReddyPeddaggari
@MaheshReddyPeddaggari 3 жыл бұрын
Will u pls make more on real time videos that would be very helpful to the new people
@WafaStudies
@WafaStudies 3 жыл бұрын
Sure
@ajaiar
@ajaiar 2 ай бұрын
WHen i click preview data it says enter the value for schema name and table name
@codeworld8981
@codeworld8981 3 жыл бұрын
Hi i tried this but problem is only one table copied... When I run the PL it's prompting for table name.. But I have parameterized the table .. Pls can u clarify
@narra_rakesh_2021
@narra_rakesh_2021 3 жыл бұрын
Bro Make a video on '' Copy data From SAP to Azure Data lake using ADF"
@vijaybodkhe8379
@vijaybodkhe8379 9 ай бұрын
Thanks for Sharing🎉
@vishalshrivastava1073
@vishalshrivastava1073 Жыл бұрын
once our pipe line start if any of table throwing error while copy then how to make sure it wont stop others to copy ?
@ABDULSAMADKHAN2007
@ABDULSAMADKHAN2007 2 жыл бұрын
Rather than Blob storage, Can we Insert the same table record into another table directly, without using storage service?
@harrydadson7467
@harrydadson7467 Жыл бұрын
Yeahhh same process for a sink DB
@saketsrivastava84
@saketsrivastava84 9 ай бұрын
How can we use similar approach if we want to copy data from snowflake to azure sql db?
@user-ct1ip9yk5z
@user-ct1ip9yk5z Жыл бұрын
How can I also get column headers in csv files. I am able to get all data but first row is not column header
@thesujata_m
@thesujata_m 3 жыл бұрын
How to copy folders and files without changing hierarchy structures, also am not aware what is the depth of hierarchy?
@rathikavenkatesh2510
@rathikavenkatesh2510 2 жыл бұрын
Have a doubt. Support if any table deleted in source database, how do we handle.
@prajvalsingh810
@prajvalsingh810 2 жыл бұрын
At 11:00 I am getting only the schema parameter while I have created table parameter also.
@himanshutrivedi4956
@himanshutrivedi4956 3 жыл бұрын
Good 1..
@WafaStudies
@WafaStudies 3 жыл бұрын
Thanks 😊
@himanshutrivedi4956
@himanshutrivedi4956 3 жыл бұрын
@@WafaStudies I have just replied on your scd2 video if you add those 2 date columns it would be great..
@Offical_PicturePerfect
@Offical_PicturePerfect 2 жыл бұрын
Copy data from different multiple sources to ADLS by using single pipeline ?
@TechnoSparkBigData
@TechnoSparkBigData 2 жыл бұрын
Suppose I have some 200 tables in on premise sql server and want to migrate some 100 out of those to Azure sql server then how can dynamically create Table in Azure sql server with different table schema, because each table will have different columns so how to dynamically create these tables and copy data?
@souranwaris142
@souranwaris142 Жыл бұрын
Hello Sir, This is really nice. I have done my pipeline perfectly, Thanks for this. but actually, I would like to ask one more thing. I want to add a trigger after this pipeline that will work if data will update in any tables then it will update in our tables as well but just new data not change past rows. Could you help me for this
@priyaperfect9485
@priyaperfect9485 3 жыл бұрын
LOOKUP has a limitation of fetching 5000 rows. what can we do to fetch more than 5000 rows
@WafaStudies
@WafaStudies 3 жыл бұрын
I will try to create a video in this soon and post 🤗
@priyaperfect9485
@priyaperfect9485 3 жыл бұрын
@@WafaStudies 😊
@nivethashrishabhat2610
@nivethashrishabhat2610 7 ай бұрын
Hi sir, i have seen your video of getting external table from the linked data(eg. blob). my question is ' can we insert data to the external table'?
@nivethashrishabhat2610
@nivethashrishabhat2610 7 ай бұрын
The use case here is whenever the blob file gets modified, i need to insert it in the external table using serverless sql pool
@mrpoola49
@mrpoola49 3 жыл бұрын
That's really cool....I have a similar requirement but I need only subset of columns from the source to destination. How can I do it? For e.g tbl_employee has 100 columns but I am interested to copy only 10 columns to the destination and also I need only few employees (where clause) from the source to destnation. How is that possible. Thanks
@swagatmohanty5818
@swagatmohanty5818 3 жыл бұрын
Follow the above steps to get all the columns and then under mapping tab of Copy activity, you can import schema and choose only the required columns.
@anujgupta-lc1md
@anujgupta-lc1md 3 жыл бұрын
same situation please explain incremental load or delta load
@WafaStudies
@WafaStudies 3 жыл бұрын
Will do soon
@anujgupta-lc1md
@anujgupta-lc1md 2 жыл бұрын
@@WafaStudies till now no update for delta load how to do from azure sql tables to blob
@chandruk3729
@chandruk3729 2 жыл бұрын
Hi Bro, I wanna do this in reverse way.. Need to copy excel sheets from blog storage to different sql tables.. can you please suggest ?
@RKTECH1021
@RKTECH1021 2 жыл бұрын
Hi Maheer, I found your channel looking for ADF help, Great content here, sequential series one by one for each area of ADF. I have been continuously watching ADF videos one by one. learning a lot. I have one question. we have Dataverse folders in Datalake and I want to load those folder based files to Azure SQL database. each folder has csv partitioned by monthly based csv
@rathikavenkatesh2510
@rathikavenkatesh2510 2 жыл бұрын
Hi I have doubt, if some tables get deleted in source db, will it impact in destination. or will it create only the existing tables in source only. Since I am daily delete db and do the copying activity from source as the source gets changed daily basis. please confirm
@bmaheshwarreddy4911
@bmaheshwarreddy4911 Жыл бұрын
Hi Maheer. Your channel really worthy one for ADF Developer community. Thank you for providing videos and suggesting developers. I have a small doubt. As per 18 scenario you are moving sql tables to blob. but i am trying to snowflake tables to blob which causes ab error " ErrorCode=UserErrorInvalidCopyBehaviorBlobNameNotAllowedWithPreserveOrFlattenHierarchy,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot adopt copy behavior PreserveHierarchy when copying from folder to a single file.,Source=Microsoft.DataTransfer.ClientLibrary,' can you please help me in this regard
@padmajakotturi239
@padmajakotturi239 2 жыл бұрын
is it possible to copy multiple files form blob storage to sqldb without using foreach loop(this one i got in interview question)
@WafaStudies
@WafaStudies 2 жыл бұрын
Use wild card file paths
@amanahmed6057
@amanahmed6057 2 жыл бұрын
Hello I need your help I am having interview on Azure Can you guide me 🙇🙇🙇🙇
@bhawnabedi9627
@bhawnabedi9627 3 жыл бұрын
👍🏻👍🏻
@WafaStudies
@WafaStudies 3 жыл бұрын
Thank you ☺️
@user-yj9rv7us4x
@user-yj9rv7us4x Жыл бұрын
👍👍👍👍
КАКУЮ ДВЕРЬ ВЫБРАТЬ? 😂 #Shorts
00:45
НУБАСТЕР
Рет қаралды 3,2 МЛН
Они так быстро убрались!
01:00
Аришнев
Рет қаралды 3,2 МЛН
Challenge matching picture with Alfredo Larin family! 😁
00:21
BigSchool
Рет қаралды 41 МЛН
50. Copy Incremental data within Azure SQL DB - Multiple Tables
32:59
CloudAndDataUniverse
Рет қаралды 9 М.
Azure Data Factory | Copy multiple tables in Bulk with Lookup & ForEach
23:16
Adam Marczak - Azure for Everyone
Рет қаралды 188 М.
Scenario-10: Copy multiple files from blob storage to azure Sql database
14:20
23. Send Email alert when Pipeline fails in Azure Data Factory
21:59
47. Migrate multiple tables from local SQL DB to Azure SQL DB
23:39
CloudAndDataUniverse
Рет қаралды 7 М.
КАКУЮ ДВЕРЬ ВЫБРАТЬ? 😂 #Shorts
00:45
НУБАСТЕР
Рет қаралды 3,2 МЛН