@Confluent Жыл бұрын
To see the entire course and the immersive hands-on experience, check out 👉cnfl.io/data-mesh-101-module-1
@marcello4258 Жыл бұрын
This serious is gold. Although it’s obvious that it is to promote confluent it is very accurate in the real world. What caught me attention was him using the right words like instead of saying „data warehouse does not work“ (what is very wrong) saying „on a non-trivial scale) I’ve been working in the analytics department for many years so I feel like getting bullshitted by many self claimed experts talking about data mesh/fabric lately
@chrismolanus 2 жыл бұрын
The teams(domains) that manager operational system(Order management, CRM) would not consider making their data available as a clean, governed product as "added value" to the company since they would already be pushing Their data to a datalake, of some sort, and someone else(another domain) cleans and governs it. Since the raw(est) data an organization has is the data that comes from operations systems (Order management, CRM) for a transition to datamesh to work these team would have to consider the work needed to clean, govern and publish their datasets as adding value to the company. In my experience this is why most of these project stall of fail. What change management strategy would you recommend to improve the success rate of such projects? Can you elaborate on what "incentives" you have seen work?
@adrienyaho 2 жыл бұрын
Look at this project stratos it's a decentralised approach of a data mesh.. 😁😁.. thank yu for your videos.. it's amazing .
@diogennesthesinope61 2 жыл бұрын
Thanks alot! Does this mean that each local domain has its own data store (data base)? Or I can use this Kafka/Confluent to read data from third party so that either Alice or Joe can consume and use the data locally as they desire? Thanks
@ryanhaney 2 жыл бұрын
It's a mesh, so you are free to have local databases if you wish. If it's a third-party and you have some way to get data into the mesh (direct connection to the third-party db, or an API), then do that. If you need to connect a legacy project to the mesh, it probably has it's own data source. You'll want to figure out where it makes sense to inject code that produces events into the mesh. If you can't modify the code, perhaps it makes sense to integrate the database via a connector instead. And finally, if it's a new project, consider using confluent platform as a native data source. You can produce data into topics for sharing via the mesh while reading data via ksqldb.
@diogennesthesinope61 2 жыл бұрын
@@ryanhaney I really thank you for this reply. I also have a follow up question: In case I want to deploy a microservice to confluent cloud, will using the free confluent tool help in any way? I am currently developing a microservice and decided to use confluent tool (community via docker I suppose). I will want to deploy the final application in the could. Say this microservice app consists of several spring boot services, and python; do you have any idea of any performance overhead which can hinder these services? For instance, running spring boot locally consumes lots of RAM and other resources. Thanks.
@ryanhaney 2 жыл бұрын
@@diogennesthesinope61 You host your own microservices wherever you want (locally, in your cloud provider, wherever). Think of Confluent Cloud as a managed data store. If you have an app (e.g. web/mobile), you'll want to create an API for the app that proxies calls to Confluent Cloud. Your self-hosted microservices can produce data into topics, consume data from topics, or issue push/pull queries from ksql clusters hosted in Confluent Cloud. You can think of KSQL as providing a cached view over one or more topics. Kafka topics are the commit logs and ksqldb caches the most recent updates in a table. I prefer using ksql because it's easy to execute a query inside the microservice as a stream or a one-time fetch. KSQL clusters come at a cost (currently $0.22/hour/node). I think you can host your own and connect them to your managed kafka instance as well. So the flow is: Your App Your API (kafka or ksqldb in Confluent Cloud)
@diogennesthesinope61 2 жыл бұрын
@@ryanhaney Thanks alot. I am trying to do this the first time. I am also reading the docs but you really helped me with this explanation. So you mean if I decide to host my App in some way, I can also decide to choose a PaaS which provides Kafka and KSQL etc to write to topics or read from them (in case I decide not to use Confluent maybe because of cost)? Then I can make my API calls as I want. But I saw that Confluent is working with some cloud providers recently. Thanks again.
@ryanhaney 2 жыл бұрын
@@diogennesthesinope61 Regardless if you are hosting kafka/ksql on your own or via Confluent Platform, you're going to want to treat kafka/ksql as you would normally treat a relational database in a traditional web application. Just like you wouldn't open up a port directly to SQL Server from the browser, you're going to have to build an intermediate API that controls access to your data. In the case of Confluent Platform, you host kafka and ksql locally. In the case of Confluent Cloud, kafka and ksql are hosted for you (in a supported cloud of your choice). Either way, you will not be able to host your custom APIs, consumers, producers, websites, etc. in either Confluent Platform or Confluent Cloud. Treat it like a database. Hope that helps.