What Is Data Mesh, and How Does it Work? ft. Zhamak Dehghani

Рет қаралды 9,249

Күн бұрын

cnfl.io/podcast-episode-175 | The data mesh architectural paradigm shift is all about moving analytical data away from a monolithic data warehouse or data lake into a distributed architecture-allowing data to be shared for analytical purposes in real time, right at the point of origin. The idea of data mesh was introduced by Zhamak Dehghani (Director of Emerging Technologies, Thoughtworks) in 2019. Here, she provides an introduction to data mesh and the fundamental problems that it’s trying to solve.
Zhamak describes that the complexity and ambition to use data have grown in today’s industry. But what is data mesh? For over half a century, we’ve been trying to democratize data to deliver value and provide better analytic insights. With the ever-growing number of distributed domain data sets, diverse information arrives in increasing volumes and with high velocity. To remove the friction and serve the requirement for data to be consumed by operational needs in various use cases, the best way is to mesh the data. This means connecting data through a peer-to-peer fashion and liberating data for analytics, machine learning, serving up data-intensive applications across the organization, and more. Data mesh tackles the deficiency of the traditional, centralized data lake and data warehouse platform architecture.
The data mesh paradigm is founded on four principles:
1. Domain-oriented ownership
2. Data as a product
3. Data available everywhere in a self-serve data infrastructure
4. Data standardization governance
A decentralized, agnostic data structure enables you to synthesize data and innovate. The starting point is embracing the ideology that data can be anywhere. Source-aligned data should serve as a product available for people across the organization to combine, explore, and drive actionable insights. Zhamak and Tim also discuss the next steps we need to take in order to bring data mesh to life at the industry level.
To learn more about the topic, you can visit the all-new Confluent Developer course: Data Mesh 101. Confluent Developer is a single destination with resources to begin your Kafka journey.
EPISODE LINKS
► Zhamak Dehghani Kafka Summit Europe 2021 Keynote: How to Build the Data Mesh Foundation: • Zhamak Dehghani | Kafk...
► Data Mesh 101 Course: cnfl.io/data-mesh-101-course-...
► Saxo Bank’s Best Practices for a Distributed Domain-Driven Architecture Founded on the Data Mesh: cnfl.io/distributed-domain-dr...
► Placing Apache Kafka at the Heart of a Data Revolution at Saxo Bank: cnfl.io/placing-apache-kafka-...
► Why Data Mesh?: developer.confluent.io/podcas...
► Join the Confluent Community: cnfl.io/join-confluent-commun...
► Learn more with Kafka tutorials, resources, and guides at Confluent Developer: cnfl.io/visit-confluent-devel...
► Live demo: Intro to Event-Driven Microservices with Confluent: cnfl.io/event-driven-microser...
► Use PODCAST100 to get an additional $100 of free Confluent Cloud usage: www.confluent.io/confluent-cl...
► Promo code details: www.confluent.io/confluent-cl...
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#datamesh #apachekafka #kafka #confluent

Пікірлер: 16

@mainajnabee 9 ай бұрын

Thank you for asking right questions and picking her brain on this topic. She's a treasure trove! 👏👏

@mailmahee 2 жыл бұрын

not all data-lakes are necessarily orthogonal - s3 bucket access policies and IAM can accommodate most of the points Zhamak is making but would involve a platform paradigm shift as well as a cultural shift in {k}ontainerized ops practises.

@brookster7772 Жыл бұрын

I would be very interested in learning more about this new data, mesh architectural layer, and the connection to the AI engines such as Palantir for example… Palantir has an abstraction called the ontology… how does all of this shit together? I am an architect, tasked with implementing these new technologies, and I am very interested in learning more if somebody could make a recommendation

@mehdismaeili3743 2 жыл бұрын

Excellent.

@brookster7772 Жыл бұрын

My first thought is the data mash is just another layer on top of everything else that has no API and dozens of client applications pulling data directly from the tables in the data mesh….? Almost takes me back to the 90s where we just had a massive oracle database and every department had their own clients getting SQL connections and running SQL queries… later of course we close the database behind web services and now we seem to be going back to the open massive data mesh? Still a little unclear to me, but it’s some thing I need to understand and learn and looking forward to it.

@manishchandrajiit 2 жыл бұрын

Thanks. This video gave me really good amount of idea on what it is, why this concept came about and challenges ahead.

@michaelburton8024 2 жыл бұрын

Tim I have a free Confluent acct so how might I use the code for extras? Thank you

@Confluent 2 жыл бұрын

Hi Michael, this document may be helpful in answering your question: docs.confluent.io/cloud/current/billing/overview.html#promo-codes

@michaelburton8024 2 жыл бұрын

@@Confluent Thank you so much for the prompt help

@dlinanhu 2 жыл бұрын

I dont get the idea of Data as a Product. Can you give some examples please?

@Confluent 2 жыл бұрын

Hi David! This resource on data as a product from the Data Mesh 101 course on Confluent Developer might be useful: cnfl.io/data-mesh-101-module-3. If you have additional questions, feel free to post them at forum.confluent.io as well. Hope this helps!

@dlinanhu 2 жыл бұрын

@@Confluent thxs!

@ankitlakum1 2 жыл бұрын

🙏

@sweeper240 2 жыл бұрын

Complimentary to any warehouse lake, lake house, or mesh... or whatever new concept of data storage governance and security model is thought up in the future Our AI/ML effortlessly harmonizes and contextualizes all data across all silos even third-party or unstructured content like handwriting, images, etc If you want to achieve mesh at any scale, you'll want to talk to me.

@michaeldemarco82 2 жыл бұрын

This is fine but truly what she is talking about is one of the benefits of Microservices anyways so nothing new from my perspective just evangelizing it is what is happening here. There are two problems with Micro services depending on where they are located. Data across the wire is latent so Microservice does not solve that. This is why compute and storage is great when they are colocated (but there are drawbacks to even this). Data latency is one of the biggest problems in performance so how is that solved? Also, another problem with Microservices in a purist sense is that when your doing reporting and you need data aggregated Microservices is not the solution which is why there are hybrid solutions that have Microservices but then data is aggregated in the repository and indexed for quick response by a push to the data mart which updates in the background. Try to do a query across Microservices data. The core problem with data will always be that the goals of singleness of purpose and distribution of that data will always be in conflict with speed and performance which is why you need a hybrid solution with data pulls from Microservices to fulfill consumer needs but for reporting and analytics you need data aggregated and delivered fast which is where the updates to the data mart come into play. you can lessen the forking effects of distribution and performance with query engines like hyperscale or big query but your going to pay dearly for it

@reactiveland3111 2 жыл бұрын

It's just another microservice, another node in the streaming topology ... These "new" microservices own/process/publish analytical data. And bringing all the data HOME (domain team) from AI team