Change Data Capture + Event Driven Architecture

Рет қаралды 12,078

Күн бұрын

Change Data Capture (CDC) is a way to monitor and capture changes in data so other systems can react to those changes or stay up-to-date with the data. It isn't new, but it's been gaining popularity around event-driven architecture. Let me explain the benefits of CDC and some best practices, so you don't shoot yourself in the foot.
🔗 Solace
solace.com/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to a private Discord Server and any source code in my videos.
🔥 Join via Patreon
/ codeopinion
✔️ Join via KZfaq
/ @codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
📧 Weekly Updates: mailchi.mp/63c7a0b3ff38/codeo...
0:00 Intro
0:44 CDC
2:06 Tooling Example
2:50 Inside vs Outside
4:00 Translation
#eventdrivenarchitecture #softwarearchitecture #softwaredesign

Пікірлер: 32

@CodeOpinion Жыл бұрын

Are you using CDC? What's the use-case?

@LilPozzer 11 ай бұрын

@viktorshinkevich3169 Жыл бұрын

4:19 and CDC itself can also capture specific views instead of tables directly so it would be another layer of separation. As a service B owner I can change internal of schema whatever I like as long as I comply with my contract so having CBC targeted to db view could ease that process

@bayonnesensei8281 Жыл бұрын

I agreed with you, I am using the same approach in production which I've do CDC at View Level instead of making CDC in a database table.

@evanhobbs Ай бұрын

This was an excellent explanation and very timely. Thanks!

@rz5541 Жыл бұрын

Always a pleasure watching your videos - many thanks for sharing all this good knowledge

@CodeOpinion Жыл бұрын

Thanks. Appreciate you watching.

@thomas-sinkala Жыл бұрын

Thank you for making this.

@CodeOpinion Жыл бұрын

My pleasure!

@pieter5466 4 ай бұрын

"Data on the inside is different than data on the outside." Under-appreciated point, had not yet heard it articulated thus.

@CodeOpinion 4 ай бұрын

I also often say Private vs Public to get across the same gist.

@Vale-ck9rz Жыл бұрын

Perfect timing 😂. This Saturday I will present my thesis that shows IDM solutions that use event-driven architecture and Derbezium software.

@shahrazkl Жыл бұрын

Amazing video

@ArtemKhodos Жыл бұрын

Thank you Dereck, awesome explained as always! Right now I am working on decoupling my own domains from monolithic application to own service. Some tables can be just moved, but looks like a have to copy a few tables as they are used in infra layer in "my" domains. I think CDC might be a good option to keep the data consistent.

@CodeOpinion Жыл бұрын

Depending on the usage, another option is providing a API that exposes that underlying data so you aren't accessing it from a replicated table (assuming that's what you mean).

@pradeepsahoo7243 6 ай бұрын

I get the usecase of CDC that should have different structure which describes a business event instead of a CRUD event, but that translation from database logs seems challenging, however, what could be easier to do is if a transaction happens at Application A, we don't care what it persists to its own state ful data source and it can very well publish the transaction to an event streaming platform and the whole usecase for CDC can go away.

@ilijanl Жыл бұрын

Great video, like neal already mentioned, CDC probably shouldn't be used for integration events. I wonder how you would integrate a warehouse inside a business, especially having multiple (external) services. Would you use CDC, ETL or ELT or some other methods?

@CodeOpinion Жыл бұрын

Not mutually exclusive, could be a combination.

@coolcoolj0102 Жыл бұрын

Can you share some available open-source CDC tool/library?

@dbpieter Жыл бұрын

Debezium, mentioned in video

@neal321 Жыл бұрын

I understood CDC to be more for integration at the data layer, for replication, data warehousing, ETL etc. In the example of Service B using CDC to go though a translation layer to write a message to the message broker, would it not be simpler for the service layer to publish a message to the broker itself, since it would have the business context at this stage and not introduce CDC and a translation for event driven architecture? It seems like a complication to me since all database updates should be going through the service anyway as you point out? Great channel by the way!

@CodeOpinion Жыл бұрын

Yes, while that would be ideal, if it's a legacy system, most start using CDC as away to derive events because there may not be a single point of truth for making state changes. Meaning you have multiple applications or processes that might be changing data without having any control over if they are publishing the required event.

@neal321 Жыл бұрын

@@CodeOpinion Ok yes, that makes sense for a legacy application or proprietary system where you don't have access to change the code, now that I think of it I have used this strategy in the past for an old school POS systems on their MSSQL database, thanks for the reply

@MrSpyTubes Жыл бұрын

@@CodeOpinion But is this translation layer a Service's B responsibility ? Because it might be difficult for a legacy system to do it, and cdc is seen as a way to gather data from that system without making changes to it.

@pradeepsahoo7243 6 ай бұрын

@@MrSpyTubes In this particular usecase, Service A is legacy, so, no control over which process is writing to DB and how, Service B is a newer more flexible service and needs real-time events to update DWs, Caches, etc. The CDC process needs to have some of the business logic to not only read the CRUD events from DB logs, but transform it into a business event. Now, that's only possible to do if the underlying data structure is understood and some business logic from the Service A application is applied to transform the CDC event before publishing to the event streaming broker.

@NervousNab Жыл бұрын

is outbox pattern a specific implementation of cdc?

@CodeOpinion Жыл бұрын

You could use CDC in that way to publish events. I would persist the actual events in the db and have cdc process publish those rather than trying to infer and transform a data change to an event.

@shahrazkl Жыл бұрын

We have high number of tnx on sql server, Could enabling CDC affect overall SQL performance?

@Variant55 Жыл бұрын

Late reply, but yes. CDC adds overhead and this could negatively impact the server.

@haraheiquedossantos4283 Жыл бұрын

It seems like CDC is a great choice in scenarios that we are using event in data centric way to do some data distribution/propagation or something like that, so I mean, in event carried state transfer. Do you agree?

@CodeOpinion Жыл бұрын

Yes, I'd say CDC is closer to event-carried state transfer than anything. I'm always very cautious with it because sharing data between services is not something I advocate. It depends on what the data is for and how it's being used.

@pradeepsahoo7243 6 ай бұрын

@@CodeOpinion So, I understand not sharing data between services means not sharing the underlying data structure or data storage system between services which couples different services together and creates complexity which can't be easily managed or broken down. So, sharing the business events as CDC is more desirable approach.