What is Data Observability?

Рет қаралды 34,661

Жыл бұрын

Learn more about Databand → ibm.biz/IBM-databand
Check out IBM Analytics → ibm.biz/ibm-analytics
Data observability can help data teams and engineers better understand the health of data in their system and automatically identify, troubleshoot and resolve issues in near real-time.
In this video, Ryan Yackel talks about the difficulty in managing the health and quality of your data sets, what data observability is, and how it can help you fix issues with your data before it impacts your bottom-line.
Get started for free on IBM Cloud → ibm.biz/ibm-free-cloud-tier
Subscribe to see more videos like this in the future → ibm.biz/subscribe-now
#databand #observability #DataScience #MachineLearning #Deeplearning #ArtificialIntelligence #BigData

Пікірлер: 21

@Flankymanga Жыл бұрын

Hello guys. Thank you for this blackboard session. I have a suggestion: Theory is nice and it is the first step to get into the specific IT topic. But as they say: seeing is believing... could you please make a video with practical use case, how data observability practices are being utilized to solve data pipeline issues? In other words: How data observability looks and works in real life on real tools. That would be awesome to see. And if you could merge this topic with Data Governance that would be totally awesome! I am loving these blackboard explanations! Thank you very much!

@dineshshekhawat2021 Жыл бұрын

Very insightful, learned a very useful concept on a high level. Keep bringing such videos!

@AISmallBusinessHub Жыл бұрын

Thanks for breaking this down clearly. It makes it easy my product designer brain to understand the concepts

@mustufabaig1483 Жыл бұрын

What I understand from this video is that data observability is just a grouping of "data quality", "data lineage", "ETL Logging" etc. into one umbrella. These are common concepts within data warehousing/engineering. Some teams solve them with frameworks/vendor tools, some through custom development and some use both approaches. Data observability is just a new name for it. good presentation though.

@ladonwilliams1333 Жыл бұрын

Absolutely great overview. Appreciate the train analogy. I got a good understanding of the premise of Data Observability, plus the bonus of a concise understanding of data engineering and pipelines. Need this gentleman explaining more concepts!

@MsVikas99 4 ай бұрын

Really great overview. Thank you

@billlabranche Жыл бұрын

Great illustration to the concept of Data Observability - thanks for making it simple enough to understand. PS... I'm very impressed with your "backwards writing skills"!!!!

@alexpishvanov736 Жыл бұрын

I really like this idea to shift thinking paradigm from been software engineer to data engineer:)

@robfletcher6274 Жыл бұрын

Great explanation, nice one

@thedatawhisperer Жыл бұрын

Great job!

@nikitashrestha684 Жыл бұрын

Hello, thanks for this explanation. I would like to see more videos on NLP domain. To be more specific using pre-trained tranformers for text analytics ( Transfer Learning )

@jorge-hernandez-ramirez Жыл бұрын

Good!!

@rajarshisadhya8174 Жыл бұрын

Thanks much for the really cool explanation. Wondering whether there is any tool available that provides the capability to really ‘Observe’ any changes/incident in the data pipeline?

@paddyarunachalam7057 8 ай бұрын

Thank you. Great session. Could I ask what technology you use to put this together?

@SoniaRaval 11 ай бұрын

Great overview and love the train analogy

@ARATHI2000 11 ай бұрын

Understood in theory..Thanks! Several years ago, one issue we used to run into was some of our Enterprise customers were missing sending data on defined schedule into our SaaS env. Our ETL data ingestion job (sort of data pipeline tool) won't help since it won't be kicked off. So we wrote something outside of the ETL job to ensure we can catch such data misses by the customer...could/should this be part of data observability?

@silicon6086 Жыл бұрын

Is Data Observability same as DataOps ?

@ladonwilliams1333 Жыл бұрын

I think that Data Observability is more of a technical implementation to identify/remediate issues in the data pipeline that is managed by Data Engineers and others potentially involved in the process due to cross-functional responsibilities. Whereas DataOps is a methodology to foster a culture of effective communication and collaboration between data stewards, data users, and executives/business process owners. So, Data Observability helps to ensures quality data so that DataOps can glean business value from data presented. In essence, Data Observability integrally folds into DataOps.

@davidmurphy563 Жыл бұрын

I think it would have been a better presentation if he had stuck to concrete examples more; like with the betting company. The train was cute, and I did get it, but he explained something abstract by making it even more abstract. Heading in the other direction and injecting some hard reality would have been more instructive. At the end of the day, this is not an academic exercise, it's there to solve real world problems.

@vio4jesus Жыл бұрын

SoooOoooooo we've been moving data for years with ETL tools. Tools that generally don't require a super coder. Some of these tools have some level of monitoring built in, and some have data quality modules. BUT NOW..... let's go back to hand-writing code to move data. Hhhhhhmmm seems like a bad idea.