In my experience, the unnecessary complexity is due to the boredom of principal devs, tech leads, or CTO's that have been pulled away from direct implementations. Or leadership that just wants to be able to put tags on a resume so they can tell the next company that they "lead X company into a new technical era", but leave out the fact that their support engineers are now having to work 3X as much to maintain their forced garbage.

"The elephant in the room"

Which smart person was it who said, "Haven't we learned not to use databases for message passing?"

indeed, the friendly blue elephant got your back, friend!

I think the problem nowadays for the over engineering dilemma, is that everyone became programmers instead of engineers. They just jump in, use whatever technology available and new to implement their task. Days when engineers sit down and think the problem with pencil and paper is no longer among us. In my career, every boss wants to see me writing code, while I'm thinking using pencil and paper, and because of that pressure you finish picking up ready made general solution tweak it and show it to them, no matter how much the cost of dependency. Well...🤷🏻‍♂️, what can you do!

Sorry, but the message queue being a simple table, it has missed so many quirks that people really should have regardless of how scaled you are: retries and dead-letter queues are just some features that you’re code should not have to be in charge of. Not to mention that it would probably require polling for new messages

What do you mean by "using apache airflow as a data warehouse"? Apache Airflow is mostly just a feature rich job runner, I don't think it stores any data (apart from metadata it uses to run the jobs)

Im new to this and i think this is what i was looking for . A simple database system to start learning about .

This is a very interesting concept but actually older than most of the developers today: don't over-engineer it. In my software engineering career, I was young and naive too, and tried to bring the latest shining tools whenever I can, without really considering the downside: additionally dependencies that could break and the cost of them. When I think back, most of the time I didn't need the extra capacity/scalability, but I did spend thousands of hours fixing the bugs and issues coming from those unnecessary complexity. I get it we developers are secretly trying to use employer's resources to learn and gain experience in some new tools, languages, libs, that's probably the main motivation behind it.

Or you can use Redis for everything too, but love the pitch, great points

I've been on the both sides of equation. Database doing everything was fine at the start, but made life a living hell for everyone when company grew... No kind of money spent on servers, or outside consultants managed to solve the problem, only evolving into that overengineered solution helped to move forward. Yes, you can do basically everything with Relational databases, but bottleneck can be reached much sooner than expected, and not having plan B could mean failure of business

Please listen to this video and put postgres everywhere so when I join the company for tons of money, I'll look like a superhero proposing specialized tools that work out of the box.

Ironically it is simpler and less buggy to use AWS services for this than using a db. Take the queues for example: you aren’t driving that data to be the input of a function, and as a consequence have no scaling concurrency, it would require polling to work, and has no failure mode in the same way a DLQ would work.

I agree in principle with keeping things simple, but I've been bit before by not using specialized tools that are available when the demand for certain features or certain levels of performance grows beyond what a simple solution can handle. Even if you're very good at abstracting the functionality into a limited interface and can come up with a drop-in replacement, you're going to run into issues, but there are also cases where that just isn't possible at all because the better solution fundamentally requires a different API. Postgres is incredibly flexible as a programmer and can do almost anything you want it to. But that's dangerous, because it's not actually good at everything.

Crazy this channel is so small, really good content.

Also a big player in the software tweeted this for career advice “do not accept tradeoffs a lot of advice out there is not real advice, it's the author trying to cope with their own shortcomings by claiming some fundamental rule you will hear things like "while you obsess over tech i shipped a product" or "being a good engineer and being a good manager are different skills" the truth is you can be good at both and there's no reason you can't - there's always an example of someone who is do not limit yourself out the gate by defining some narrow role/identity for yourself - all of that is fake and you should strive to be great at programming, product, design, marketing, business, etc you'll fail but you'll realize it's all the same and land somewhere pretty special” That being stated, you might not actually be over engineering your problem. Also in big tech, we make sure that all items scale independently and are cost effective. That way we can write off a feature and not worry about it as much anymore. Especially with spikes

Duuude. THANK you. If I have to look at one more over-engineered architecture diagram that's really just a desperate justification for deploying Kafka where it isn't needed, I'm going to S C R E A M

This video is gonna pop off!!! It's because it's a great insight.

This approach doesn't solve anything. It just moves complexity from one place to another. In order to handle all the different edge cases each of those systems handle and in some cases even to handle the main usecase, such as in the case of airflow for example. The engineers would have to writhe the entire base of logic by themeselves, this could accumulate to hundreds or thousands of hours of developing something you can have off the shelf. So even thought there is a complexity to learning and running those systems (in part can be automated), you can be up and running very quickly with hight quality systems developed by devoted people. By developing everything by yourself you will need to handle all of the bugs and issues that the original developers of those systems already handled. So in the end does it worth it? Not sure at all!

if all you have is a hammer, everything looks like a nail... there is a good reason why specialized services like a message queue exist. i've met developers who knew one thing only and used it for everything. they are a disaster to any project. following your idea who needs postgresql when there are good old files and sockets?

Hey great video! I’m curious what’s your point of view on using AWS DynamoDB? People say it’s great for scalability, but I worry about things like costs, vendor lock in, and query limitations

It seems that by the comments people are generally opposed to this. I personally like the idea of a "jack of all trades" database like postgresql because it keeps everything in one place. No need to orchestrate a bunch of different services. Ome good example is saving a record from the front end and not having to then call a separate message queue. The database takes care of it for you. It also ensures that no matter what or who saves something to the database the message always gets cued on the database side.

Using kafka add's a buffer to the database, but I get your point

I think the use postgres as a memory database by increasing its cache is not a good take, since the cache size would apply to every table in the database and could make other big tables "steal" the cache from the table that you actually need the cache. Instead (if your RPS is low) I think it would be a good idea to use the resources of the language (or simple external deps) as a cache, an example would be expiring_dict library in python that it makes very easy to make a lru cache, however this would work only in server environments with a single instance, which is common in low RPS usecases

yup, Temporal and Postgres are the only 3rd party systems you need

postgres is the single best piece of software ever

Making sure i understand correct: For 1. Basically create a cache table and enable shared buffers to make it happen prior to a DB call by keeping the table in memory, and implement the logic to check if the cached entry exists on the server side, if its a hit read the cache and if miss, execute your query and write server code to store result in cache table as it returns response to front end, then write a CRON job to delete old entries. For 2. Create a queue table, insert incoming entries for processing and on the server it seems vague how exactly these will be processed but I'm guessing you'd have a long running service for dequeuing that locks the table, attempts to process, if fail, leaves the entry in DB (not sure how delete works here, it sounds like you'd have to reverse the transaction if it fails server side which seems complex) if success, deletes the entry and moves on to next. My concern here is how to write this service so it does not slow down other server operations, and make it generic enough to use for more than 1 use case. For 3. Not really sure what 3 is? You mentioned timescale DBs in the intro, and im not sure it makes sense to store time series data in a SQL DB unless you have a very small amount of entries because lets say your users submit time series data in only one request but it contains tens of thousands of points, then even with a small userbase youre putting the whole application at risk by building this inside a SQL DB. I'm no backend expert, I'm frontend focused, so maybe I'm missing some things but I'd definitely like to see more elaboration on the queue service and handling large quantities of data (again this isn't even large scale, it might only be a few thousand users but if they all create 10,000 new rows in your table every day I would imagine this would cause problems?) Thanks for the video.

The queue part is a bit strange to be honest, i think listen and notify are the most powerful feature in postgres you did not mention. In my opinion, people overuse q's as a whole.... Listen and notify give you reactive programing using data as originator of function and that is the biggest gain for 99.99999% of systems in the world

if you delete the enqueued item in the database, you wont be able to reprocess items if something happens to the node, losing jobs in the process. Updating the queue with a timestamp to lock an item, sending a heartbeat from time to time to keep it lock, and then deleting it when it's done is safer imo.

I kinda tried like even having postgresql like a redis cache but the more you write to postgresql the more complex and hard to manage it is, you have to deal with garbage but yea i prefer to keep things simple

After all the BS that there is over there, your video is more than welcome, thanks a lot for it!

hmm ... good idea to combine the delete with the "select for update skip locked" in one go.

I use Kafka at work and it has been a painful experience for the most part. Way too much complexity around it for something that can be done well by something like Postgres I am learning.

Cool video, take my sub!!

Postgresql is great but try reading or writing to/from a table while writing 80k rows per-second to another table.

I mean, MySQL is simpler and adheres to SQL guidelines more closely

I get your point.. but as a Data Engineer I really can't agree with building a DWH on a row DB. Aaand Airflow is a job handler, not a DWH

it's just easier to use the other services because they have everything you could need out of the box with all the language parts good to go. no design work needed

Implementing a queue in a relational DB is the bad youtube advice I was looking for. I can't wait to destroy a multi-million dollar project armed with this knowledge. Seriously, just because you can does not mean you should.

Thank you

im a startup CTO with 12 years in the industry and i approve this message

Thats so clever!

reinventing the wheel is over-engineering.

For cache, maybe MySQL/MariaDB memory table would be better?

Imagine trying to do this thru some ORM

If all you have is a hammer, everything looks like a nail.

Whenever MongoDB is part of your stack, it needs simplification! (And removal of that cancer!)

Postgres is best but it's overrated, at my work we just use Postgres we have a lot of JSONB columns that are ugly AF especially when it's 3lvls deep. It can become such an anti pattern trying to be a Nosql with heavily unstructured data. My work colegue switched to Mongo/reddis from Postgres as they mainly had unstructured data thus postgresql is an overkill.

Why do I need pg at all? I can do anything pg does in my own code, and then I don't need to be an expert in pg.

I agree things are usually too complex, but I orefer MongoDB as my mains database. It can do everything you described, everything Postgress does and more. No need to use Blobs or weird Json fields, or even create a schema (you can create it if you want). Finally, the aggregation pipeline is crazy powerful.

This advice can potentially shoot yourself in the foot. everything is nuanced and you can do everything right in a Postgres database can’t even scale to one user. (Been there, had a Django-Postgres server that couldn’t scale to even 5 users.) A better way to think of feature development is to use the best tool for the job. You wouldn’t use Postgres for graph problems (well you can but I don’t know if you should).

Simple, yet crazy effective

KISS ❤

Yeeess, I'm not alone who sees this, like kids din't read documentation and create own new NOSQL xD!!!!

Don't use PostgresDB for queuing

why db just use plaintext

Yes keep things small but build it to be able to scale, to a point where a partial redesign will be needed, putting it all in postgres is also a problem if postgres is the fault, where a msg queue and a cache could have helped and lowered costs initially.

You are saying everyone problems look the same to yours; that's where the video did not holds. The video is based on lots of assumptions that won't applied to real-world situations. However, if I were to pick one, it will be MongoDB over Postgres. I'm not assumed that everyone's problem are the same to mine. MongoDB solved ways more common issues than Postgres. People who bashed MongoDB just want to applied relational database design to MongoDB and found that it is a bad idea; instead of reflect on their design, they blamed MongoDB. It is easier that way, isn't it? 😢

Any third party dependency is a liability. An ideal system is the one that does not have third party dependencies. In practice this ideal is impossible to achieve, but everyone must try to get as close as possible to it.

This makes me angry but isn’t entirely wrong. Lots of quirks would come up, leading to tribal knowledge of how patches/hacks are added, but not philosophically dumb approach

This goes too far, yes you can use postgres as your primary and only database for everything, but no you should definitely use a dedicated message queue service, it's easier and far more robust than trying to write your own. Also, there's a fine balance between keeping architecture simple and reinventing the wheel.

This is a great example video of how not to do sytem design. Just the fact that you're using one single point of failure for 3 different systems should be enough but making a video that inexperienced ppl may use as a "good" example is absolutely mind bogggling to me. PPL please use the individual systems instead of a database engine to do the work that caching or queueing systems are designed for. Generally each system is designed specifically for that task and should perform much better than botching together a similar solution using an outdated relational db with less features and less stability.

Deleting the enqueued item is very, very naive and irresponsible! Everyone who just implements this from the video will be very badly surprised when see how the workers for Many reasons cannot execute the jobs, but they already deleted the record and you cannot try again. For everyone who wants to implement it the right way with a database, you must read and implement principles that SQS from AWS used in their service.

Sqlite

place some hate speech into this comment when you fail your first latency, scalability, or any other NFR. Only then you'll understand why.

I'd be inclined to argue you should pick mongodb over postgres. Spares you any database design issues and will most likely hold up for a very long time before you hit any sort of design limitations. Storing objects is much more convenient than relational data when you start a new project from scratch.

get rid of that music

Please do NOT follow this video's advice. There is a reason specialized solutions exist. While you can do almost anything in postgresql, this is akin to saying you can drive any type of screw with a flathead driver. Using sql as MQ is not something you should do, ever. There is a reason brokers exist: performance, interoperability, just to name two. Using sql like that is an insane overhead. Use the right tool for the job. This guy tells you one screwdriver is enough. Don't follow his advice.

Nice! Can't wait to take the Dr. Chuck Postgres course 🦾🥳