Azure Data Factory Mapping Data Flows Tutorial | Build ETL visual way!

  Рет қаралды 219,338

Adam Marczak - Azure for Everyone

Adam Marczak - Azure for Everyone

Күн бұрын

With Azure Data Factory Mapping Data Flow, you can create fast and scalable on-demand transformations by using visual user interface. In just minutes you can leverage power of Spark with not a single line of code written.
In this episode I give you introduction to what Mapping Data Flow for Data Factory is and how can it solve your day to day ETL challenges. In a short demo I will consume data from blob storage, transform movie data, aggregate it and save multiple outputs back to blob storage.
Sample code and data: github.com/MarczakIO/azure4ev...
Next steps for you after watching the video
1. Azure Data Factory introduction video
- • Azure Data Factory Tut...
2. Check mapping data flow documentation
- docs.microsoft.com/en-us/azur...
3. Helpful tips and samples
- github.com/kromerm/adfdataflo...
Want to connect?
- Blog marczak.io/
- Twitter / marczakio
- Facebook / marczakio
- LinkedIn / adam-marczak
- Site azure4everyone.com

Пікірлер: 304
@Haribabu-zj4hd
@Haribabu-zj4hd 2 жыл бұрын
So nice of your talent explaining the data flow in simple way. Thank you so much Mr.Adam.
@bifurcate1788
@bifurcate1788 4 жыл бұрын
Adam, I have been watching many of your videos. As someone new to Azure, i find your videos immensely valuable. Keep up your great work, really appreciate!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Awesome, thank you!
@waklop4384
@waklop4384 4 жыл бұрын
Just discovered the channel. Your material is hight quality. It's excellent work. I will go watch more. Thank you Adam !
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thank you. This means much :)
@pradeeps3671
@pradeeps3671 3 жыл бұрын
Hello Adam, pls let me know how to connect to dynamic crm .. Pls send detail to pradysg@gmail.com
@joshuaodeyemi3098
@joshuaodeyemi3098 Жыл бұрын
I love you, Adam! I have been struggling with using expression builder in Data Flow. I can't seem to figure out how to write the code. This video just made it look less complex. I'll be devoting more time to it.
@subhodipsaha7608
@subhodipsaha7608 3 жыл бұрын
I just find your videos while searching for ADF tutorials in youtube. The materials are fantastic and really helping me to learn. Thank you so much!!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Happy to help! :)
@dimitarkrastev6085
@dimitarkrastev6085 Жыл бұрын
Great video! Most videos seem to focus mostly on the evertisement material straight from Azure. At best they show you the very dumb step of copying data from a file to DB. This is the first video I saw where you actually show how you can do something useful with the data and close to real life scenario. Thank you.
@hovardlee
@hovardlee 2 жыл бұрын
-1979 and ,12 This is why complex logic is needed. Nice tutorial :)
@yashmeenkhanam3451
@yashmeenkhanam3451 4 жыл бұрын
Outstanding !You just made Azure easy to learn. Thank you.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Awesome, thank you!
@johnfromireland7551
@johnfromireland7551 2 жыл бұрын
ADF is but just one part of about 100 significant tools and actions in Azure. :-(
@CallousCoder
@CallousCoder 2 жыл бұрын
Hi Adam, is it possible to create these pipelines as code as well? Or somehow create them from my actual Azure pipeline? It would be sheerly insane (but it is a Microsoft product) to require and maintain two pipeline one that’s yiur Azure pipeline for CI and CD and one for the ADF. I really would want the Azure pipeline to be able to fill/create the ADF pipeline. But I haven’t found anything yet.
@hassy9118
@hassy9118 3 жыл бұрын
Excellent presentation of ADF Data Mapping...we love to see more Data Factory ETL videos. Thank you Adam!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks, feel free to check my ADF playlist.
@fadiabusafat5162
@fadiabusafat5162 2 жыл бұрын
Nice one Adam. Cool one. Keep doing fabulous videos always fella. Many THanks.
@alanzhao8074
@alanzhao8074 3 жыл бұрын
The best video about Azure Data Flows I can find. Thank you Adam!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Wow, thanks! :)
@Raguna
@Raguna Жыл бұрын
Very good explaining the Data Flow. Thanks Mr.Adam.
@subhraz
@subhraz 2 жыл бұрын
Very well explained and demonstrated. Really helpful to get started with Data flows.
@hrefname
@hrefname 3 жыл бұрын
Thank you so much for this. I subscribed immediately, very informative and straightforward azure info. Will definitely recommend your channel. Keep up the great work!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Awesome, thank you!
@omarsantamaria6871
@omarsantamaria6871 4 жыл бұрын
Awesome video. I've seen a lot of site & videos and they are so complicated, but all yours are very crystal and anyone can understand.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks Omar :)
@gunturchilli767
@gunturchilli767 3 жыл бұрын
Thanks you so much Adam. I was able to crack an interview with the help of your videos. I prepared notes according to your explanation & 3 hrs before the interview i watched your videos again it helped me alot
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Fantastic!
@Lego-tech
@Lego-tech 4 жыл бұрын
Very crisp and clear information, I watched many videos but Adam's contents are awesome!! Thanks dear!! All the best for future good work!!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thank you so much 🙂
@sameerdongare1113
@sameerdongare1113 4 жыл бұрын
Another awesome video. The best part of Mapping Data Flow was the Optimization...where we could do Partitioning.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thank you! Glad you like it :)
@abhijitk7363
@abhijitk7363 4 жыл бұрын
Adam, Thanks for this excellent video. You explained almost every feature available there in data flows. Looking forward a video on Azure SQL DWH. I know it will be great to learn about it from you.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Glad it was helpful! I'm just waiting for new UI to come to public preview then the video will be done :)
@SIDDHANTSINGHBCE
@SIDDHANTSINGHBCE 3 жыл бұрын
These videos are great. Helping me so much! Thanks Adam
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you like them!
@arulmouzhiezhilarasan8518
@arulmouzhiezhilarasan8518 3 жыл бұрын
Impeccable to know reg Mapping Data Flow, Thanks Adam!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
My pleasure!
@aarontian5979
@aarontian5979 2 жыл бұрын
Your channel is totally underrated, man
@BijouBakson
@BijouBakson 2 жыл бұрын
It must be very challenging to do all this thing in English for you I imagine, Adam! Congratulations for pushing through despite the difficulty. 🙂
@skillT01
@skillT01 Жыл бұрын
👍 Its amazing , Practical implementation of Data Flow.
@sarahaamir7457
@sarahaamir7457 3 жыл бұрын
Thank you so much Adam! this was very clear and great video and a big help for my interview and knowledge.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Very welcome! Thanks for stopping by :)
@icici321
@icici321 4 жыл бұрын
I am new to data and ETL stuff but your video's are too good. Excellent examples and very clear explanation so anyone can understand. Thanks very much.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thank you, always happy to help!
@soumikdas7709
@soumikdas7709 3 жыл бұрын
Your videos are very informative and practical oriented. Keep doing .
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you, I will!
@rqn2274
@rqn2274 3 жыл бұрын
Nice video Adam. Professional as always
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Wow, thanks!
@ngophuthanh
@ngophuthanh 2 жыл бұрын
Thank you, Adam. As always, you rock.
@lowroar5127
@lowroar5127 2 жыл бұрын
So helpful! Thank you very much Adam!
@eramitgoswami
@eramitgoswami 3 жыл бұрын
Your way of explaining is outstanding, after watching it feel like Azure is very easy to learn. kindly keep sharing good videos Thank You..
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks a ton :)
@susanmyers1
@susanmyers1 3 жыл бұрын
@ Work I'm having to build out a Data Mart with no training on my own. You are literally saving my hide with your videos. THANK YOU!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad to help! :)
@bharatruparel9424
@bharatruparel9424 4 жыл бұрын
Hello Adam, I just finished this video. Very well done indeed. Thanks and regards. Bharat
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks Bharat :)
@NehaJain-ry9sr
@NehaJain-ry9sr 4 жыл бұрын
Awesome videos Adam, your videos are great help to learn Azure. Keep it up :)
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks, will do!
@avinashbasetty
@avinashbasetty 4 жыл бұрын
Features are very interesting. want to try with the different partitioning techniques. Thank you for sharing such amazing stuff
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
My pleasure! Thanks! :)
@wojciechjaniszewski9086
@wojciechjaniszewski9086 4 жыл бұрын
very well done on explaining principles of mapping data flows!!!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks a lot!
@chandrasekharnallam2578
@chandrasekharnallam2578 3 жыл бұрын
excellent explanation with simple scenario. Thank you.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad it was helpful!
@sapecyrille5487
@sapecyrille5487 Жыл бұрын
Great! You are the best Adam.
@generaltalksoflife
@generaltalksoflife 3 жыл бұрын
Hi Adam, Thank for helping us in learning new technologies. You are awesome 👌🏻👌🏻👌🏻👏👏.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
My pleasure!
@achraferraji3403
@achraferraji3403 2 жыл бұрын
Amazing Video, we want other parts !
@grzegorzz4025
@grzegorzz4025 3 жыл бұрын
Adam, great tutorial! Kudos!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you liked it!
@shantanudeshmukh4390
@shantanudeshmukh4390 3 жыл бұрын
Wow ! Fantastic explanation.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you liked it!
@GiovanniOrlandoi7
@GiovanniOrlandoi7 2 жыл бұрын
Great video! Thanks Adam!
@AdamMarczakYT
@AdamMarczakYT 2 жыл бұрын
My pleasure!
@balanm8570
@balanm8570 4 жыл бұрын
As useful another Awesome video Adam !!!. Excellent. It was to the POINT !!!. Keep up the good work which you have been doing for plenty of users like me. Eagerly waiting for more similar videos like this from you !!!. Can you please have some videos for Azure Search ...
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thank you so very much :) Azure Search is on the list but there is so many news coming from Ignite that I might need to change the order. Let's see all the news :).
@KarthikeshwarSathya
@KarthikeshwarSathya 3 жыл бұрын
This was explained very well. thank you.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
You're very welcome!
@mohmmedshahrukh8450
@mohmmedshahrukh8450 Жыл бұрын
best video on azure I have ever seen❤❤
@549srikanth
@549srikanth 3 жыл бұрын
I would say this is the best content I've seen so far!! Thank you so much for making it Adam! Just wondering, is there a Crtl+Z or Crtl+Y command in case we did some changes in the dataflow and restore it to previous version?
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Awesome, thanks! Unfortunately not, but you can use versioning in the data factory which will allow you to revert to previous version in case you broke something. Highly recommended. Unfortunately not reverts for specific actions.
@549srikanth
@549srikanth 3 жыл бұрын
@@AdamMarczakYT Excellent!! Thank you so much for your reply!
@johnfromireland7551
@johnfromireland7551 2 жыл бұрын
@@549srikanth I publish each time I create a significant new step in the pipeline and I use data preview before moving on to the next step. Also, you can , I think, export the code version of the entire pipeline. Presumably you can, then, paste that into a new Pipeline to resurrect your previous version.
@eddyjawed
@eddyjawed 3 ай бұрын
Thank you Adam Dzienkuje, this is a great tutorial.
@valentinnica6034
@valentinnica6034 4 жыл бұрын
that was actually not so hard. thanks man, you're awesome.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
No! you are awesome! :)
@rajanarora6655
@rajanarora6655 2 жыл бұрын
Your videos are really great and helped me understand lot of concepts of Azure. Can you please make one using SSIS package and show how to use that within Azure Data Factory
@Cool2kid
@Cool2kid 3 жыл бұрын
Your video content is awesome!!! Your video is very useful to understand Azure concept specially for me who just started Azure journey. I would like to have one video where we can see how to deploy code from Dev to QA to Prod. How to handle connection string, parameter etc during deployment. thanks again for wonderful video content.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
ADF CI/CD is definitely on the list. It's a bit complex topic to get it right so it might take time to prepare proper content around this. Thanks for watching and suggesting ;)
@biswajitsarkar5538
@biswajitsarkar5538 4 жыл бұрын
Thanks Adam !! very informative video.Liked it a lot..
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks and you are most welcome! Glad you hear it.
@javm7378
@javm7378 2 жыл бұрын
I really like your tutorials. I have been looking for a "table partition switching" tutorial but haven't found any good ones. May be you could do one for us? I am sure it'll be very popular as there aren't any good ones out there and it is an important topic in certifications :-)
@MrVivekc
@MrVivekc 3 жыл бұрын
very good explanation Adam. keep it up.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks, will do!
@MrVivekc
@MrVivekc 3 жыл бұрын
@@AdamMarczakYT Adam do we have trail version of Azure for Learning purpose?
@anubhav2020
@anubhav2020 2 жыл бұрын
Hello Adam, thanks a bunch for this excellent video. The tutorial was very thorough and anyone new can easily follow. I do have a question though. I am trying to replicate an SQL query into the Data Flow, however, I have had no luck so far. The query is as follows: Select ZipCode, State From table Where State in ('AZ', 'AL', 'AK', 'AR', 'CO', 'CA', 'CT'...... LIST OF 50 STATES); I tried using Filter, Conditional Split and Exists transforms, but could not achieve the desired result. Being new to the Cloud Platform, I am having a bit of trouble. Might I request you please cover topics like Data Subsetting/Filtering (WHERE and IN Clauses etc.) in your tutorials. Appreciate your time and help in putting together these practical implementations.
@atiry839
@atiry839 3 жыл бұрын
Wow,I like your video, I did it today. and I had good result. thanks for your good explanation.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Great job! Thanks!
@RahulRajput_018
@RahulRajput_018 3 жыл бұрын
Thanks buddy ...Great work
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
My pleasure
@JoeandAlex
@JoeandAlex 3 жыл бұрын
Brilliant way of explanation
@JoeandAlex
@JoeandAlex 3 жыл бұрын
Subscribed to your channel
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you, appreciated 🙏
@horatiohe
@horatiohe 3 жыл бұрын
thanks for the great content!! you are the man :)
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
I appreciate that!
@rahulkota9793
@rahulkota9793 3 жыл бұрын
Very useful. Thank you so much.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad it was helpful!
@nick6s
@nick6s 2 жыл бұрын
Excellent tutorials
@skybluelearner4198
@skybluelearner4198 Жыл бұрын
Good explanation there.
@isurueranga9704
@isurueranga9704 2 жыл бұрын
best tutorial ever... 💪🏻💪🏻💪🏻
@yashgemini4024
@yashgemini4024 3 жыл бұрын
Appreciate you content. Thanks.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
My pleasure! :)
@dataintelligence6262
@dataintelligence6262 4 жыл бұрын
Great job! Thanks for all
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thank you too!
@niteeshmittal
@niteeshmittal 4 жыл бұрын
Adam, Your content is always easy to grab, excellent work mate. Could you please explain how to create a pipeline which has a copy activity followed by a mapping data flow activity.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks, just drag and drop copy activity and data flow blocks on the pipeline and drag a line from copy to data flow activity.
@DrDATA-ep6mg
@DrDATA-ep6mg 3 жыл бұрын
Very nice tutorial 👍
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you! Cheers!
@dintelu
@dintelu 3 жыл бұрын
Wow..lucid explanation..
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you think so!
@jayong2370
@jayong2370 Жыл бұрын
Thank you Adam.
@arun06530
@arun06530 3 жыл бұрын
nice & detailed video.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you!
@abu1479
@abu1479 3 жыл бұрын
Adam, excellent presentation of ADF concept. I find all your videos really helpful in understanding the ADF concept. One question in regards to the sink dataset in dataflow, how can I create dynamic folder in my blob storage based on the year, month and day when this dataflow was triggered?
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Depends on what do you want to achieve. You can either set partitioning by date column which will split the data by date. Or if you want to put entire dataset in one folder using date then use formatDateTime expression like formatDateTime(utcNOw(), "yyyy/MM/dd") as path.
@jayakrishna9153
@jayakrishna9153 4 жыл бұрын
very good explanation..keep doig
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks. Will do.
@carlossalinas8497
@carlossalinas8497 4 жыл бұрын
Adam you have an ability at explaining complex things, this tutorial made my day, thanks
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Glad it helped! Thanks!
@Rafian1924
@Rafian1924 3 жыл бұрын
Lovely bro!!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks 🔥
@sudarshanbhattacharjee4411
@sudarshanbhattacharjee4411 Жыл бұрын
Thanks for such good video
@thisiszico2006
@thisiszico2006 4 жыл бұрын
Awesome again.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Glad you think so!
@harshapatankar484
@harshapatankar484 3 жыл бұрын
Amazing videos.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Glad you think so! :)
@adiky
@adiky 3 жыл бұрын
Hi Adam, that's a great tutorial, many thanks for it. I have a question that can we write the transformation functions in different language like Python or R instead of Scala? If yes can you please share some details on it?
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Unfortunately not right now :( If you need those then use Azure Databricks instead.
@rubendarksun6691
@rubendarksun6691 2 жыл бұрын
Or a python/R script in a batch process, right? Databricks would be better option of you need spark, since its also more expensive than batch
@desparadoking8209
@desparadoking8209 3 жыл бұрын
Thanks for the informative and detailed video adam, 😊👌. Your content is practical. Can you make a video on how load the data from Oracle table to azure data factory? It would be helpful for audiences.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks! That's the plan :)
@mikem8915
@mikem8915 4 жыл бұрын
Outstanding.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks Mike ;)
@chsrkn
@chsrkn 4 жыл бұрын
Features are very interesting and thanks for your clear explanation , could you explain bit more reusable data flows , I mean to use same data flow for multiple tables/files like reusable pipelines.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks for suggestion, I will try to make video on data flow parametrization in the future. Thanks for watching :)
@erick1992santos
@erick1992santos 4 жыл бұрын
Excellent content!! I just have one doubt, where I can find documentation about the functions of scala? I do not know anything about it. Just have subscribed your channel! Thanks a lot!!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Tank you Erick! I think the list of available functions are standard for Spark. I never checked if they all match but you can find them here spark.apache.org/docs/2.4.4/api/sql/
@RC-nn1ld
@RC-nn1ld 3 жыл бұрын
Love these videos so easy to understand, do you have a video on new XML connector
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Great, thanks! Not yet, maybe in near future :)
@samb9403
@samb9403 2 жыл бұрын
Great video. Question: Under "New Datasets", is there a capability to drop data into Snowflake? I see S3, Redshift, etc. I appreciate the video and feedback!
@gursikh133
@gursikh133 4 жыл бұрын
Adam, FOr using transformation do I need to learn scala. Or just refer the documentation you specified for scala functions and write the transformation?
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Documentation should be enough. MDF is targeting simple transformations so in most cases documentation alone will suffice.
@gbw5679
@gbw5679 4 жыл бұрын
Thanks Adam!
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
My pleasure!
@JoshuaDHarvey
@JoshuaDHarvey 3 жыл бұрын
Great video thank you
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thanks again! :)
@RahulJain-wk2wv
@RahulJain-wk2wv 3 жыл бұрын
awesome!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Thank you! Cheers!
@nidhisharma-rb7nx
@nidhisharma-rb7nx 2 жыл бұрын
Adam, great video.I m new to Data Flow and I have one doubt, I want to implement File level checks in Data Flow but not able to do it. All tasks are performing data level checks like exist or conditional split. Is it possible to implement File level check like whether file exist or not in storage account?
@khurramshehzad980
@khurramshehzad980 Жыл бұрын
very nice
@jg5399
@jg5399 4 жыл бұрын
Thanks for the nice video. Do you know if there is anyway to connect to Dynamics F&O or D635/CE yet to use Data Flow as my source?
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Hey, thanks for watching! Not that I'm aware of, I would assume this would be the case for Logic App to fetch data to Blob Storage and then trigger ADF processing. docs.microsoft.com/en-us/azure/connectors/connectors-create-api-crmonline
@TheLastBeat2
@TheLastBeat2 3 жыл бұрын
Hi Adam, so glad I found your channel. Your videos were a big help for achieving the AZ900 certificate. Now I am studying a lot to uplift my knowledge and get the Azure data engineer certificate. However, I have an important question! Data flows are expensive, sometimes clients don’t want to use this, are there alternatives to achieve the same result in azure data factory? Thank you very much!
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Well you can't have the cookie and eat the cookie :) In my opinion it's not that expensive compared to other available tools.
@TheLastBeat2
@TheLastBeat2 3 жыл бұрын
@@AdamMarczakYT True! I am currently struggling with csv files that sometimes have extra spaces after the words in the header, this then gives error when doing a copy activity to Azure SQL Database. Do you have any idea to make my flow a bit more flexible so that it can deal with this? It needs some trimming in the header
@TheLastBeat2
@TheLastBeat2 3 жыл бұрын
I thought of doing a SELECT in a dataflow to then change to the correct header titles, but for this I need to know where the spaces will be in the future. So also not flexible.
@mustafakamal5945
@mustafakamal5945 2 жыл бұрын
Hi Adam, Thanks for making this videos, very clear and concise. I have a question (sorry not related to this video) regarding Conditional split - Can the output stream activities, run in parallel ?
@AdamMarczakYT
@AdamMarczakYT 2 жыл бұрын
They typically run in parallel as it's Apache Spark behind the scenes.
@mustafakamal5945
@mustafakamal5945 2 жыл бұрын
@@AdamMarczakYT Thank you !
@sharmilashrestha5449
@sharmilashrestha5449 3 жыл бұрын
Adam, your tutorial are very simple to follow and provides lot of insight. However, I could not set up my data flow in the way I wanted. What I want to do is -> Get a list of integer from Table 1 (on-prem SQL server) -> Use this list of integer to query a cosmos structured stream -> Then I want to perform some transformation with the data returned from this cosmos stream -> sink transformed data back to Table 2 (on-prem SQL server). However, I do not see my source datasets on the data flow drop down. Any help would be appreciated.
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Make sure to check if you use supported data source types docs.microsoft.com/en-us/azure/data-factory/data-flow-source?WT.mc_id=AZ-MVP-5003556
@MrSARAZZ
@MrSARAZZ 4 жыл бұрын
Hi Adam, just watched two of your videos on Azure Data Factory, nice work. Any chance you can do one on ADF using REST API as a data source with a JSON output, then store in a SQL Server sink?
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Great suggestion! I'll add it to the list of potential topics. :) thanks for watching ;)
@gift2299
@gift2299 4 жыл бұрын
@@AdamMarczakYT Please, a tutorial on this would be amazing!
@omarsantamaria6871
@omarsantamaria6871 4 жыл бұрын
Hello Adam. Your video is impressive, as always, but I'm concerned about the source dataset. Question: Does the DataFlow activity only work if the datsource are connected to Azure SQL? I tried using a previous dataset connected to the local server, but this dataset does not appear on the Source settings / Source options / Source dataset in DataFlow activity option. I tried with New option and it is only enabled to select the AZURE dataset. All options in the database are disabled. So I couldn't create a data set for SQL Server neither.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Hey mapping data flows currently support 6 data services for both source and sink. docs.microsoft.com/en-us/azure/data-factory/data-flow-source#supported-source-connectors-in-mapping-data-flow I'd check if you can trick data flows by using Azure SQL connector to connect to on premise SQL server, but I never personally tried.
@tenghover
@tenghover 2 жыл бұрын
Would you plan to make video for introduction of each transforamtion components? Thanks
@JohnJohnson-bs4cw
@JohnJohnson-bs4cw 3 жыл бұрын
Great Video. Can you use data from a REST Api as a source for a Mapping Data Flow or does the source have to be a dataset on Azure?
@AdamMarczakYT
@AdamMarczakYT 3 жыл бұрын
Here is the list of supported data sources for MDF docs.microsoft.com/en-us/azure/data-factory/data-flow-source?WT.mc_id=AZ-MVP-5003556 . Just copy data from REST API to Blob and then start MDF pipeline using that blob path as a parameter.
@khushbookumari-pc2xw
@khushbookumari-pc2xw 4 жыл бұрын
Please also explain how to use data analytics in pipeline flow. This is best content . Thank u so much
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thanks you. As to your question can you elaborate on data analytics part? What exactly would you like to see.
@khushbookumari-pc2xw
@khushbookumari-pc2xw 4 жыл бұрын
Anything like how to make function or procedure and how to use this in pipeline to execute .basic flow of pipeline by using analytics.
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Hey, I do plan to have implementation videos like this in future. Although pipeline of videos is long so I can promise anything right now. I added this to the list of potential topics :) thanks!
@khushbookumari-pc2xw
@khushbookumari-pc2xw 4 жыл бұрын
okay, Thanks
@velupalani2496
@velupalani2496 4 жыл бұрын
Velu well done
@AdamMarczakYT
@AdamMarczakYT 4 жыл бұрын
Thank you!
Azure Data Factory Tutorial | Introduction to ETL in Azure
24:59
Adam Marczak - Azure for Everyone
Рет қаралды 821 М.
Azure Databricks Tutorial | Data transformations at scale
28:35
Adam Marczak - Azure for Everyone
Рет қаралды 371 М.
ELE QUEBROU A TAÇA DE FUTEBOL
00:45
Matheus Kriwat
Рет қаралды 23 МЛН
CAN YOU HELP ME? (ROAD TO 100 MLN!) #shorts
00:26
PANDA BOI
Рет қаралды 36 МЛН
Super sport🤯
00:15
Lexa_Merin
Рет қаралды 20 МЛН
Exploring the preview of datamart in Power BI!
10:19
Guy in a Cube
Рет қаралды 90 М.
Creating your first Data Warehouse in Microsoft Fabric
6:56
Guy in a Cube
Рет қаралды 51 М.
Azure Data Factory Parametrization Tutorial
22:07
Adam Marczak - Azure for Everyone
Рет қаралды 170 М.
Processing Slowly Changing Dimensions with ADF Data Flows
1:10:53
Pragmatic Works
Рет қаралды 28 М.
Azure Data Factory | Copy multiple tables in Bulk with Lookup & ForEach
23:16
Adam Marczak - Azure for Everyone
Рет қаралды 181 М.
Nokia 3310 versus Red Hot Ball
0:37
PressTube
Рет қаралды 3,8 МЛН
Теперь это его телефон
0:21
Хорошие Новости
Рет қаралды 1,7 МЛН