Data Engineering Was Hard Until I Learned These 5 Secrets!

Рет қаралды 42,879

Darshil Parmar

Күн бұрын

Пікірлер: 100

@DarshilParmar Ай бұрын

1500 Likes and I will upload Fundamentals of Data Engineering Masterclass here

@sumithraj7954 Ай бұрын

For free?? Darshil

@tech-n-data Ай бұрын

1500 on the video? Because you are about there in one day or that comment? Y'all smash the like on both, we need our masterclass!

@deadkachu7678 Ай бұрын

Can u make videos on these topics u have said so that we can understand properly.

@RamakshaBirajdar Ай бұрын

@DarshilParmar done, we want the masterclass 🤝

@Userabj001 Ай бұрын

I am among 1.5k .❤

@ansonnn_ Ай бұрын

0:23 1st mistake: understand the purpose of building the data pipeline, ask the why instead of straight dive into the technical details 2:02 2nd mastering sql is non negotiable 3:08 3rd automate everything 4:36 4th mistake: did not learn about debugging, maybe try Grafana / Prometheus, with an alerting mechanism 5:54 5th Learning is a continuous process

@srimathiganesan1516 Ай бұрын

YOU VIDEOS ARE REALLY AMAZING!! I never comment usually but your work is really inspiring. Data always excites me and being a data research analyst now I wanted to shift to data engineering as I'm in love the work flow of it. Since I knew fundamentals in MYSQL and python for data analysis which includes fundamentals like loops, functions and conditional statements then forwarding to hands on numpy and pandas. Is it enough or is there something I should learn apart from this for data engineering ?

@shivangishrivastava9096 Ай бұрын

Could you please make a video on Debugging concepts, especially w.r.t. data engineering?

@DarshilParmar Ай бұрын

Will do

@xx-pn7it Ай бұрын

How to understand the flow of data for the specific project or work

@DarshilParmar Ай бұрын

It starts from asking questions to business people, what are they really expecting and do with the data, then from there you can start thinking about how to execute it

@AtiaNasr Ай бұрын

@DarshilParmar ... Is data Engineering is a suitable for someone who is love coding very much and don't like work with visualization stuff like dashboard, who like to focus on distributed systems, scalablility performanece and need to heavy logic thinking to do

@vikrantshelke2418 27 күн бұрын

Eagerly waiting for the Fundamentals of Data Engineering Masterclass! PS: Could you please make a video on Microsoft Fabric? Is it helpful as a data engineers? Should we learn this tool?

@manojthalari397 Ай бұрын

I just Googled that how to like a video 1500 times with single account(Unfortunately No option), Please upload Master class

@DarshilParmar Ай бұрын

Will do it soon

@MeanLifeStudies Ай бұрын

Thank you for great contributions to the community and make me realise to contribute something to the community. So I started recently contributing to the community with SQL interview preparation, problem and solution series. Soon I will contribute many Thank you.

@tejasvijb4228 Ай бұрын

Waiting for your Fundamentals of Data Engineering Masterclass 💙

@DarshilParmar Ай бұрын

Coming soon

@aalamaram7680 Ай бұрын

Great video Darshil...If i carrier change to data engineer to learn is difficult to get job ?

@jayanthgajula Ай бұрын

Best video editing

@gokul2555 9 күн бұрын

Am a ETL/Data guy for 14 years... trust me he is giving the best advice which no one will say... DEBUGGING man I wish some one would have told me when I was fresher

@prasadjoshi007 2 күн бұрын

DEBUGGING is mostly skipped all are focusing on building while most of the time you to debug.

@lectures3727 Ай бұрын

Hi bhaiya Needed a laptop in college and programming i started learning data science and ml but dont do gaming so i didnt take any GPU laptop I have 2 options First lenovo ideapad pro 5 with 14" ultra 9 H processor 32 gb ram 1 tb ssd and 84 wh battery Second is lenovo yoga slim 7i with ultra 7 processor 32 gb ram 1 tb ssd and 65 wh battery Both price same Which one should i go for?

@Vitali-pe3wg Ай бұрын

I am struggling to become a Data Engineer. In few months I will write my Bachelor thesis and it should be about data engineering. I hope with this thesis i will be able to enter the job market. Any recommendations for a thesis to enter the market?

@DarshilParmar Ай бұрын

I'd say learn high-demand tools, try to get any job in data, doesn't have to be DE only and make your way from there

@sugirthachandrasekaran3523 Ай бұрын

Thank you Darshil. How to learn about debugging?

@l_drago2875 28 күн бұрын

Timelines for next set of cource launch kafka airflow and projects

@krrishshylock6984 Ай бұрын

Hi darshil, want to know can data engineerin work on model development as well . can they switch to mlops roles .

@saachisrivastav605 Ай бұрын

Can you please upload some thorough projects along with the business mindset behind them and also how we should build further projects

@DarshilParmar Ай бұрын

I have lot of them on my KZfaq - check KZfaq analytics

@kanishcktewatia597 Ай бұрын

can you please share how you create these animations in your videos?

@mazharsayed7161 Ай бұрын

Is ml ,dl is important in data engineering???

@RobinHood-yn3bb 30 минут бұрын

no DE is Another sector

@AbhinavPeddemoni Ай бұрын

Bro how can we create a code snippets like you showed in thumbnail? Which tool it is I saw many people use this to post their code on LinkedIn also, That one with 3 color dots on top left?

@DarshilParmar Ай бұрын

Just write code to image on google

@AbhinavPeddemoni Ай бұрын

@@DarshilParmar thank you so much bro!

@sandhya_608 Ай бұрын

Plz ,anyone can tell me which CSE specialization should i choose for data engineering ??

@kakahalwaipunawale7991 Ай бұрын

Darshil now that you have more than 1.5k likes Could you please upload the data engineering fundamentals masterclass

@DarshilParmar Ай бұрын

I need to build a PPT, record, and edit video, it takes time! will upload once it is done

@kakahalwaipunawale7991 Ай бұрын

@@DarshilParmar Hey Thank You for your reply. And Thanks for all the efforts you are taking. I look forward to the masterclass 👍

@izhanabdullah Ай бұрын

Which tool are you using to create architecture diagrams?

@DarshilParmar Ай бұрын

Google Slides

@ezeefied Ай бұрын

Hey mate , great video. Thank you. Would you let me know what software do you use for video editing

@DarshilParmar Ай бұрын

I used to use FinalCutPro but now I have an editor who edits my videos

@user-hb2kh3br9u Ай бұрын

Sir can u pls tell me that whether Information and Data Security subject or Essentials of Data Collection and Ethics subject will be the better option for data engineering career. This is an elective subject in my 3rd sem

@prasadjoshi007 Ай бұрын

Really Helpful!!!

@DarshilParmar Ай бұрын

Thank you

@sudarshanmhaisdhune1039 Ай бұрын

Thank you Brother!

@DarshilParmar Ай бұрын

You are welcome

@mansinayak3360 27 күн бұрын

Hello guys, Can anyone send me link to the lecture where Darshil sets the AWS pricing alarm. It would be really helpful. I found the where he created the account then IAM user but not the one where he sets the alarm. It is there but I've lost it.

@mansinayak3360 27 күн бұрын

@DarshilParmar Hello, tagging you if you can help with the same.

@DarshilParmar 27 күн бұрын

Check cloudwatch - docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html

@mansinayak3360 27 күн бұрын

@@DarshilParmar Sure, Thanks!

@SurprisedGlobe-yq5uh Ай бұрын

I completed my b. Com computers and later i have competed my masters in marketing domain. Is am a eligible for a data engineer

@tanny_tales Ай бұрын

Bro in second part of youtube analysis video s3 trigger is not working properly and the data(parquet files) is not getting uploaded and no graph is visible

@DarshilParmar Ай бұрын

Check IAM permission

@tanny_tales Ай бұрын

@@DarshilParmar i checked can it be related to the thing that data types have to be same when creating etl jobs, interface has changed completly,kindly help

@kirankumar3536 Ай бұрын

Hi Darshil, I have some experience on Azure adf. This combo will help me to go as a azure data engineer or only for AWS engineers. Have you used Azure cloud in this combo ?

@DarshilParmar Ай бұрын

Yes, we have project in Spark course on Azure

@kirankumar3536 Ай бұрын

@@DarshilParmar what about pipeline implementation by using Azure ?

@DarshilParmar Ай бұрын

Yes, it's building data pipeline only!

@_DEBANJANDHAR Ай бұрын

I had created an account on 24 June, From today total 8 days are remaining I.e till 25 july for my free trial period. But it gets deactivated and asking my credit card details, WILL THEY CHARGE ME ??

@lectures3727 Ай бұрын

Hi bhaiya Needed a laptop in college I chose 2 options hp pavilion plus 16gb ram fixed 1 tb ram fixed i7 13700h rtx3050 6gb 1.01 lakh Ideapad pro 5 16gb ram fixed 1 tb ssd ultra 7 155H rtx 3050 6gb 1.13 lakh Which one should i prefer and if i want to go more low which can i take

@DarshilParmar Ай бұрын

Don't go HP

@lectures3727 Ай бұрын

@@DarshilParmar bhaiya actually i am very confused on which laptop to take and dont want to overspend I dont do machine learning now but may do so i dont know if i should go for a dedicated graphics card and which processor to go i7 or ultra 7 and ram Can you please suggest me which configuration should i go for and what price range I am getting lenovo ideapad same laptop with 13700h online for 94k but parents are saying not to buy online so i cant decide what to do but i need a laptop now

@DarshilParmar Ай бұрын

@@lectures3727 If you play games then graphic card is good else just go for other option Basic config - 16GB ram, 500-1TB SSD is good, any latest processor. If you can afford maybe try Microsoft surface or macbook

@lectures3727 Ай бұрын

@@DarshilParmar no i dont play games Can i go for any latest processor I am learning data science machine learning now Will that laptop be enough if i take ultra 7 or i7 I am mostly getting p type processor

@gauravg2359 Ай бұрын

Helpful

@shreyaupadhyay1447 Ай бұрын

Sir, I'm a third year cse student. I want to be a data engineer and could you please suggest resources for that. And, do freshers get job in this field?

@DarshilParmar Ай бұрын

Check out my roadmap

@Tanishq_1803 Ай бұрын

Bhaiya how it is data engineering professional certificate by ibm on coursera

@DarshilParmar Ай бұрын

I suggest you to do GCP one

@Tanishq_1803 Ай бұрын

@@DarshilParmar and for data analytics and data science how is ibm data analytics and data science course?

@uzaifquadri Ай бұрын

Is dsa compulsary to crack an product based company , if yes then how much , any suggestions from where to get this skill set

@DarshilParmar Ай бұрын

I have a video on this kzfaq.info/get/bejne/naebbNCQyZibqZ8.htmlsi=fJOrV6Gzxa8Ecyov

@zeeshansaud9535 27 күн бұрын

Data Engineer Job Role and AI Contribution: kzfaq.info/get/bejne/hLhnf5OFuKzYg3k.htmlsi=kj4d_8rCdbQSCpPc

@hemanthvarupula9873 Ай бұрын

Can a person from outside india . Can take data vidya combo course

@DarshilParmar Ай бұрын

Yes you can - datavidhya.com/combo-course-info

@hemanthvarupula9873 Ай бұрын

@@DarshilParmar and the subscription is for life time

@DarshilParmar Ай бұрын

@@hemanthvarupula9873 Yes

@OppoTeam-u7s 8 күн бұрын

How much you making per month

@mian7104 Ай бұрын

❤

@vedantterse8859 Ай бұрын

can we get more discount on your combo course for limited period like it is still expensive for students who are just starting off in the field , hope you understand and you'll do consider it

@DarshilParmar Ай бұрын

I have 50% offer going on, if you want you can start with one course first and then go one by one

@avinashvyas1597 Ай бұрын

@@DarshilParmarI was trying to buy your sql course but there is no discount even the coupon code is also not valid Is there any other way around ?

@DarshilParmar Ай бұрын

@@avinashvyas1597 Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Purchase Individual Courses with 50% off: 1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50 2. SQL for Data Engineering - pay.datavidhya.com/cop/x4UpBcaKsi?code=SQLT50 3. Data Warehouse for Data Engineering with Snowflake - pay.datavidhya.com/cop/BpfGFSahpQ?code=DWT50 4. Enroll in the Apache Spark Course Here - datavidhya.com/courses/apache USE CODE: EARLYSPARK for 50% off

@avinashvyas1597 Ай бұрын

@@DarshilParmar Thank you Darshil Also you’ll be mentoring me now is that correct ?

@DarshilParmar Ай бұрын

@@avinashvyas1597 There are no 1 to 1 session but we have dedicated discord for this so if you have questions I will help you out