Check out ProjectPro for DE, DS, ML projects. This link gives you additional discounts: bit.ly/3GmbSq9 Thank you for watching. What are you thoughts on these topics? Let's talk! 😀

Thank you Jash for this frank video, this puts alot in perspective. I am currently transitioning from Warehouse executive to Data sphere and training to improve soft skills in visualization tool (Power Bi) and query language (MySQL). I end goal was and still is to be a Data Engineer, before now I had no idea these three things you mentioned 1. Second tier folks in most compnay 2. Dependency (bridge between producers and stake holders) and 3. Be Update with technology. This is most appreciated. Thanks also for the link to Project Pro cause I know for a fact, a ceritficate will not get me a job its practice and documentation of work done.

Your opening skit got me. My team is basically data engineering, but also core development in our small company too. And once was in a project planning meeting that had some managers, project owners, and marketing/sales in it. The main project was just parsing out text from some documents to check for some sort of risk assessment compliance. And near the end after I just briefly explained my thoughts what doing this would entail for our team which was JUST importing data, parsing for certain identifiers, and generating reports; one of the marketing/sales guys asks, "So when we have this can we advertise this as AI and machind learning because that's big right now". And I basically facepalmed(good thing it was a zoom meeting with my camera off) and got flustered and started say explaining why this wasn't AI and it would be a whole other thing to use that for this project. Which was apparently the wrong thing to say because he just asked why can't we do that? And I'm thinking, "nobody on our team has any experience with that so how can we build that product in your timeframe?" Luckily my boss jumped in and saved me to explain that we couldn't do that now and that it would be a huge time commitment and would derail the whole thing.

The problem with the Business Analyst specialism is [as an intern/junior], employers expect you to have an MBA and/or prior business experience, a solid interest in business/finance in addition to the technical skills, which creates a significant barrier to entry. As a learner looking to change careers at a mature age, data engineering is far more attractive to me, because the emphasis is primarily on the technical skills.

Excellent video, important points well-presented with real-world examples! The ‘Dependencies’ section (6:00) illustrates why data (or software or most any other type of) engineering will not be eliminated by ChatGPT. AI will get better and better at dealing with the “this ain’t right” complaints, but it will be a while.

As a data engineer, I agree with your points but many of them are barely even negatives, at least for someone who would actually enjoy this kind of job. I enjoy being on the back end and a "second class citizen" as nobody is expecting me to pump out software like they do for the software engineers. The dependencies can get annoying but honestly the investigational work of this job is some of the most fun and keeps you engaged. Learning new software all the time is fun and it's cool to find better ways to do what you were doing before. Overall this job is a privilege to be able to work in as it has amazing WLB, fantastic pay, and low stress compared to 95% of other jobs in the world. Yes it has its challenges but it is nothing compared to other careers, the only hard part really is the initial learning curve.

Brilliant, making these types of videos help with communication skills, stuff that I have been told I lack of. One thing that you have mentioned is the very broad tech skills we have to deal with, but you haven't said that sometimes it requires to switch from one to another with a snap of a finger. Some pipelines are failing and have to fix them, and at the same time doing your normal development work. Good stuff, I suppose your work is key and haven't been affected by Google laid offs.

Thanks Jash for bringing up much much valid situations and scenarios for data engineering.. 👍

I'm a business analyst at a small company so some of the data engineering needs will be falling on my shoulders. This was a very informative video with great perspective!

My sister is data engineer and I am very proud of her. She works 10 Hours a day. Some times i feels sad for her for her long working hours and it is taking serious toll on her health.

Awesome !! love you bro.... your content is really helpful.

Introductory part was pretty awesome. Good improvements in content presentation. Keep it up so we will also keep supporting you to grow.

Thanks for the video Jash.. you literally spoke my mind about DE being just the support function in Product Based and captive Industry. There is always a cap on budget and most of the stuff we do as a DE goes under the rug as the work wont get enough visibility to stakeholders.

Coming from a datawarehouse background and I hear and see DE and DS I remember the need of data architecture and the use of a ETL tool for datawarehousing and BI. DS and DE give me the impression of anything goes and quick and dirty and after a while these projects tend to go to more structured, scd and maybe a need to use an ETL tool to realise that. So I miss a bit the added value of DE and DS over old school datawarehousing.

This video is so on point! data engineering is such fast-paced field that by the time you learn something, something else gains momentum and everyone starts following the trend. It is a never-ending process of learning and upgrading.

Isn't that true for the DevOps/SRE roles too? Being a DevOps Engineer myself, I always feel like a "Second class citizen" in the sense that my work has little to do with the aspects of the application that are driving the whole business and is mainly about just getting it available to the customer. Sure we're the ones to ensure the availability, redundancy etc and help the organization get rid of mundane manual stuff but i never felt that sense of longing. @Jash Whats your take on this

Great video, really appreciated the insights on the challenges of being a data engineer

3:35 the Witcher shirt already had my attention but you really had my attention when you said you were a former Zser. I’m a current Zser who’s frantically watching these videos to understand the scope of growth of my role. Literally have a connect with my PD coach this week 😂

After that opening skit, I love this guy!

Very precise. By asking to fix the data sources you must rely on different team and if they are slow / unresponsive it may take months to get the results...

Hi Jash, Nice and very informative video. Your consistency of quality is superb!! I am from a service based company working as a DE. I can totally relate to the pain points in service based you were talking about, but that also makes it a thrilling job. Isn't it? I learned today to continuously update myself in the technological aspect. Thanks for that. Please continue bringing this quality content in the future as well!

lol I have been working as a data engineer and I didn't even know. I created and maintain our current ETL pipeline. It is pretty simple but as I get data from various sources it gets complex. I can relate with the issues that can arise in such cases

Great perspective Jash. You definitely laid out the important aspects of the job. One part you missed out was that in many companies you may have to be on call to help with pipeline break/fix if reports/dashboards cater to critical business functions and it is a thankless/low visibility job with nice pay. Google or top tech companies may have SRE teams but that may not be the case with non tech companies. So if you don’t like to do on call/break fixes then don’t be lured by the money. I recently left this field due to on call burnout and I work for a fortune 100 company. Good luck!

Yeah us Data Engineers are like doctors who people think, know the answer to all questions. I get question as well like "Why cant you do this? Why are we paying you all this money?" and my answer is usually "This is the job of an architect or a data scientist or ML engineer" and this infuriates them to the core.

Regarding scd type II, how do you explain them to users or business analysts. I had a lot of problems with analysts just using needless transformations to visualise such a tool, as opposed to giving them a long format table that they can visually transform easier. I know that a slider can solve the problem, but I found it that it can take more time to find the solution for them, than just give them what they want and accept my faith. Just curious

It is very well explained, thanks for your effort 👍

Cool on the point view! Love it :)

The Intro 🤣👌🏻 Very well made video!!! Great Job Jash... ✌🏻

as an aspiring data scientist this was very informative

Ohhh so that's why Data engineers move to Data Manager or Data Scientist role after a few year.... I always kept wondering why such good talents choose a different track ... Changing technologies is the most important factor here... unless someone is very passionate about learning new things Data engineering is going to be trouble for them

Spot on from a fellow data engineer.

loved the video and explanation

Thank you it was a great informative video, I would like to make a request to understand how a data engineer will approach a problem of the sql database if it has 40-50% is a legacy data and what approach you will take to document it in absence of any formal documents.

I'm trying to become a business analyst for a medium sized company and it seems like it's a hybrid of a regular business analyst and a data engineer/scientist (Knowledge of R, SQL, and other software was a requirement) and I think that's exactly what I want.

Hi josh, thanks for your videos they're very informative and we appreciate the effort . In your Q/A video u mentionned hiw the second step after following the roadmap or acquiring the necessary skills would be to apply it through projects :as a DE student i find it hard to do that since it always seem too big of a task for a beginner/student . Are there websites or sources that contain projects ideas or guided ones either in DE or DS ?

Most projects in I have interviewd for in GCP end up using DataFlow(Apache beam) instead of Spark for distributed workloads. Could you please provide some usecase that can be used to have an actual hands-on witth spark or maybe some reference. I would want to apply for roles that use spark next. Thanks!

Hi Jash Radia, I am From Nepal 🇳🇵. A 19 y/o boy having a curiosity to learn Data Science. Can you make a video on the courses that are available online for data science step-by-step. If you can !! It means a lot to us ❤

Hey Jash! I'm a recent high school graduate with a keen interest in pursuing a Computer Science or Information Technology course at an Indian engineering college. Before diving into college, I've decided to take a proactive approach by focusing on learning Java programming and Data Structures and Algorithms (DSA) first. I believe mastering these foundational concepts will provide me with a solid groundwork for my future studies. After gaining proficiency in Java and DSA, my goal is to explore the exciting fields of Data Science and Machine Learning. I'm eager to embark on this learning journey and would appreciate any guidance or advice on the best approach to achieve my goals.

as a fellow data engg, working in service based company, i can relate with you. stress is high and WLB is low also pay is less :( we are literally treated as second class citizens and other people will not give us the same respect like SDEs get. also learning part is so much correct. i literally spend some extra hours everyday to learn new tools and tech that i dont get time to spend on other things. starting to regret in data space. its better to be frontend or backend engg who has great pay, respect, high wlb and most imp less learning than DE.

Your skit at the beginning was funny. I am an ML Engineer who now works as a Data Engineer. So many people have no idea that the two roles are different things!

Hi Jash, thanks for providing such an informative source. Especially, the last part about tech tools transitions (Hadoop to Spark) blew my mind because I was about to take the Hadoop course for this spring semester! Now, I would like to ask you if Hadoop's skillset applies to Spark or Snowflake. If so, I'd keep my Hadoop course registration, otherwise, I'd drop it and will take the Spark course instead.

I liked the topic about "being near and away from the business outcome" I am an ex - frontend developer doing master's in data science now. I have this same question for about 2 years now as I try to figure out my true interest in this big IT industry. So there are 4 factors - 1 salary, 2 significance (how close is our work to the business outcome), 3 dependency (how crucial is our work and to what degree can it drive the stakes), 4 work-life (+ how much can we delegate later in the career) I do not want the best of all. But what I want is maximum value of this fraction: (salary * good work-life) ÷ (dependency or stakes). What role would allow me to maximize this fraction and where will I need to put the 2nd factor significance in this fraction? And somehow dependency and significance also affect number of jobs and their growth rate and salary growths too. What would be the best role if you want the best optimal point from all the above factors? I will consider focusing on that role and it's tech stack and skills.

Then what are the jobs that fits into "go to starbucks and open up the laptop" profile??

Hi jash, it's difficult to manage DE techstack, companies still don't understand what should they expect from employee, even in interviews they will ask OLTP, OLAP, Hadoop suite of applications, Spark, Snowflake, Databricks with any cloud services, even for them it becomes difficult to chose right candidate.

Can you comment on project pro.. Does it really help.. Does it have real time projects which industry uses, what did u feel personally and do you recommend it

Hi Jash, I've been wanting to choose data science as a career and stumbled across your channel. It's been really amazing and informative learning from you. I am 20 years old in college and have already started learning python after watching your roadmap video. Anything I should be beware of for the future scope in this industry?

Hi jash, missed the 2 hour mark but got a question. Is it better to focus on one thing of data engineering or to know everything a little bit for me to get a job. Please let me know since am starting a bootcamp in a few days :)

Hi jash...which cloud will create more data engg jobs going forward?

Very well explained brother🫡 all points I can relate 😅

what bothers me the most is multiple tech stack doing the same thing, not getting fit because of it, and the job opportunities are still less.

I can totally relate to this points.

thanks buddy!!

Hi jash Serious question I am currently in support role Wanted to switch my career to data engg. I am avg in coding will it be suitable for me Please answer as soon as possible. I want to make this decision quickly

I have total 8+ of years expereince in IT and I have more working expereince on sql and working exp of 3 yrs in informatica. Whaat do you suggest for me to learn new tech is it data engineer or devops is better ? Please suggest😢

Hi Jash, Thanks for the info. Which one is good for a Masters student in Data Science, Data Engineer or Data Scientist?

Hey Jash! i'm shortlisted for the role as a btsa at zs associates. So just wanted to know if there are any similarities in the work that a btsa does to that of what a DE does. Thank you ❤

How I can move from database developer to data engineer. With an experience of 3.5 yr in(database developer )

Thanks a lot Jash for the contents. Can you put a highlight on the differences in the use cases of Dataframe transformations and spark SQL.

I have a confusion, should I focus more on MySQL or PostgreSql ? I know that most of the syntax are same.

Another way to think about it is that DE's are the ones who actually deliver something whereas most of the work DS' do goes absolutely no where.

the amount of times i had talks with C-level managers expecting something like a datalake solving aaaall their analytical problems ... completely oblivious about it's not more than a tool .. enabling fast processing of large data amounts and grab and forget data collection .. but it's not providing value .. (yet) all the work has to be done anyways ..no matter which tool is used ..

Hey brother, I'm looking for internship in data engineer or ml engineer role in ahmedabad. Do you know which companies hier data engineer in ahmedabad for internship and full time job?

I have begineer level of knowledge in Spark. I want to know from where I can prepare or solve interview related Spark questions. Can you provide suggestions for improving my spark skills and any websites or resources to practice?

Please help!! very confused now a days. Im mechanical engineer having 4 years of experience. I quit my job and learn Data analytics skills like powerBI, python,sql.. now someone suggest me go for DE . Great opportunity in future. DA job are less according to him.. wht can i do.please help. Should i starts learning DE? Or focus my existing skill for DA?

Bro, how does Google prescribe Dataflow for new data pipelines and it does not even get a mention from their own engineer? Is it really a niche?

For becoming a data engineer do we need to have good knowledge in maths or maths is very important would you clarify this please? Because I'm good at maths but I'm aspiring to become a data engineer and this is the drawback I have

I have been working as Data scientist for 2 years but not getting much work on Data science things instead I get other things. Recently I started working on DE tools it caught my interest, was planning to switch from DS to DE What are your suggestions?

please make a video covering how to get job at good place with good package as a fresher,companies target,skills,interview prep and what mistakes we should avoid

I just found your channel. This is awesome. Can you do a top ten onlilne resources (free or not) for learning DEng?

I'm really interested to become data engineer in future but I really dk which btech degree I choose to pursue first bcuz I'm not that good at studies so Cse degree will be hard but I can get IT or data science

Hi i am trying to switch my career from mechanical to data engineering Can you sugesstes please for the streaming which tool to learn is apache kafka is enough ?

Hey great video, helpful information. I currently work as Data Scientist for 2 years, I thinking of exploring Data Engineer career path, what should be my initial learning in terms of technology or tool?

Hi @JashRadia Thank you for the videos. Im trying to find a course where we have a structured system design course which is sufficient for Sr DE interviews.

Jash i am interested in DE but i am in big confusion to choose this field.lot of people are going in to this and lot of are coming so it difficult to stay in this field for a longer period? .

Its funny that data engineers feel second class citizen when compared to managers n ds, truth being told all these data centric manager n technical roles have evolved after recent big data boom where companies wants to make most out the data that they have or wants third party company to help them with it. Guess what during the hard times it was these same roles or associated data product teams that were kicked out first, be it during covid pandemic or recent layoffs. If you really want secured n stable job in tech industry you better associate with tech stack onto which the core business runs 24*7, cool tech buzz might give shoot your salary up in corporate world in very short period but it can never ever guarantee stability.

I have been a data analyst for 6 years now, current in a lead role, currently earning over 25 lpa (fixed). Have super high interest in this role but changing careers is pretty daunting. How much would does a DE role get paid for someone starting afresh?

Great video

Thanks much Jash for the details . Iam currently working as oracle dba in rotation shifts.Could you suggest some job with less coding skills

I have been in Data space for more than 1.5 decades. My advice is, if u r not good in math, then pls pls dont go for DS. Bcoz I have been seeing so many primary school mistakes from Data scientists (those who have completed a DS course and claim that they are data scientists). Start ur career with DA/ DE, then parallelly build a good foundation in Math - then slowly switch to ML/AI... As of now, most of the Data scientists are just configuring the Black Boxes (ML algos) - not creating or contributing to the ML algos... I m not kidding... In an interview, I asked a guy to write the equation of a straight line and asked to explain it in simple words... Responses were awesome

Is it possible to get remote jobs in data engineering for freshers?

Then which is the best domain to work in? I am interning as a DE and all these points kinda demotivated me to leave this ASAP and switch to SDE/Developer roles...

Hi Jash. I am a developer and currently I am confused to opt whether Data Engineering or Cloud Computing, Could you please tell which of these two has a better scope and salary in future.

Is that a #YNWA in the badckground? and thanks for the insights btw!

All valid points. In cost centers, or CoC, where you are not under the CIO organization, your direct lead is non technical person and you are limited to tools associated with no or limited cost, then you are Data Engineer without described harsh reality, its even worst, all is hyped to Data Science, no one want to invest time to data governance.

I was too late to see this video. still, I'm eligible to comment here! please suggest, when we should think move of data engineering jobs people who want to survive in IT 45 age+?

Completely Relatable Bro....

I landed a Data engineering Internship and I am hoping it turns into a job after I graduate. The company has been awesome far

What about the job security in of a data engineer in product based fintech company when the revenue of company is reducing. So will they fire data engineers or the Java developer.

Hi @JashRadia I'm working in QA as automation testing I have opted course for Data engineering Do you think good idea to switch from QA to DE? Are pays are better in DE then QA?

I have learnt python and it’s libraries for ML I have learnt MYSQL and tableau ! What else is required for a data engineer role

Hello sir. Sir I m from commerce background currently I m working as MIS accounts and want to shift in data engineer line could u plz guide me?

Business request changes and so data request will change and complex business logic will have to be implemented. As far as I am concern No-Code Technologies can not meet the expectation of business 100%. There will still be some element of development and embedded coding

off-topic: dude what an epic t-shirt you have there. Nice!

Thanks for the video Jash, I kinda stumbled in as a fresh data engineer myself, and I do believe I am going to stay. Hope we can connect.

Need advice everyone For someone who isn’t a big fan of coding, which role will be most suitable data analyst or data engineer (data scientist is coding algo heavy so not going that side)

I have 4.3 years of experience on GCP, mainly Big query and Composer.. I was laid off last month.. I'm planning to upskill now.. Which skill or tool is the hottest in demand in market right now for me to get a job and also secure next few years of relevance..

is there any upper hand for data analyst while transitioning and giving interviews for DE role?

I used to think Machine Learning was the brilliant thing in the world of data science, but after doing data science course i have great respect for Data engineers. ❤.

You are my inspiration sir

Hi Jash i am good at writing sql queries and have keen interest in data engineering,, coming to coding part I can do easy and manage things but feel demotivated when can not solve difficult questions like software engineer s do i am looking to make long career in this what do you suggest,,,