Stonehenge is a distributed monolith, everyone loves it. But I build a distributed monolith, I get fired. It's not fair.

Personally i love it when my boss forces me to spend 5 days to develop something using microservices, when i could do it in 5 hours in Django, and the app only has 1,000 internal users.

Here is the problem Primeagen: > Netflix has a specific issue > Netflix comes with a solution for their issue > 0.25% of the world companies benefit from the solution the same way cause only a handful of companies in the world at a certain scale can face the issue. > blogposts and vlogs on youtube starting flying around about it cause INFLUENSOOOOOOOORS have to cREatE cONTent to live. > Mediocre companies adopt the Netflix solution pitching something resembling this at an abstract level: "hei boi, we use what netflix is using, come claim clout for your CV with us." > 99.75% of the companies worldwide adopt the same complex solution for their CRUD app cause they need to aTTrAcT tALenT > ????? > rekt

I've been thinking lately, that literally every approach is shitty, and there is no holy grail. Your role as an architect is to pick the least shitty option, not the best one.

Just remember, nothing is quite so permanent as a temporary solution.

Every microservice adds on a layer of monitoring, alerting, logging, retry processes, synchronization, idempotency requirements, authorization, deployment, and dead letter queues to every “micro” service. Worth it sometimes but go in with eyes open.

Holy shit, that time format rant turned from joke to a full blown PTSD episode real quick.

Breaking a system into distributed systems is arguably the best possible way to make it complex and hard to maintain.

"There could have been an Amazon competitor back in the days that, instead of building a monolith and solving customers problems, they build a distributed system (a micro service architecture)... That might be the reason, you have never heard of them..."

Build modules until you need microservices, for scaling, then the modules can be jettisoned as a microservice with minimal changes in code as necessary.

Thanks on reading all these articles and putting it on youtube. Your personal take stemming from experience, as a many times more experienced engineer than I am, is truly the value I am looking for in youtube tech content.

You need a microservice to get time

I read Sam Newman’s monolith to microservices. He actually recommends starting with monolith with logical services that could be broken out if it needs to independently scale.

In my team we have more microservices than developers and nobody outside our team uses it

The whole industry is driven by trends. Everybody is doing it - so it cannot be wrong. This ends up in those waves of going into one direction and then in the other. We should focus us on the use case, take over the responsibility and find the best approach for our solution.

Scalability is a performance optimization. Premature optimization is usually a big mistake. Effective performance optimization starts with performance data. So if you are building out a microservice infrastructure for scalability and you don't have real world performance data, then you are doing a premature performance optimization. If you want to organize your code for development purposes, you should do so in a way that doesn't have significant performance implications. If you want scalability, you should start with real performance data and use that to inform your design decisions.

For us the biggest issue by far has been network latency. Even the most basic read operations take hundreds of milliseconds to complete due to having to fetch data (often sequentially) from different microservices. Also 90% of our alerts are now request timeouts between microservices. My stance today would be that microservices only make sense for fire-and-forget applications. Anything that needs to respond to a user request should be a monolith.

The major part why I have found some form of "services" are valuable (how "micro" they should be is a question very specific to every particular project and company) is independent release cycle. If you have a modularized monolith, you still are tied into common release cycle, and it tends to be quite heavy, reducing your ability to release as often as you like. Dealing with release cycles if you're in the same process as 20-30 other teams often ends in a situation where your release cycle is "maybe we'll get something on prod in a week or so, depending on the volume of issues". Now, this assumes that you can release as often as you like. If you have external limitations to your release frequency (certifications, security/compliance audits, if you're tied to particular firmware patch releases etc.) that math changes a lot.

The biggest problem for me with microservices is the persistence/data layer, there are a lot of queues, a lot of streams, a lot of replication, a lot of inconsistent data and so on. Apart from a whole tooling structure that has to be built to sustain production. I don't, sometimes i just believe that a monolith well written, well modularized, works well well damn.

I converted a micro-services system to monolith, it resolved about 90% of their issues. The company went on to being acquired for $500 mill

Many years ago, I worked on a system that hot-swapped DLL libraries from disk gracefully when they were updated with new versions without restarting the application. Now I think about it, that was pretty damn close to micro-services - only faster due to being in memory communication, not via the network stack!

I've been working for quite some time in the banking industry and I remember a peculiar example of a project some colleagues of mine worked on: a big bank commissioned a multi-million dollars architecture based on microservices and events, at the end of a 3 years period of development they decided to throw away the whole thing and go back to a monolithic non-event architecture. Maybe it was just badly designed, but I think the problem is the very nature of event-driven stuff and I say that because I've worked on another big bank repository a couple of years ago with a similar kind of architecture and it was an event spaghetti and each request caused and untraceable amount of data retrievals and it was ugly to work on and difficult to maintain. Hundreds of people worked on that repository over a 20 years period and it was a huge pile of crap.

My hot take is that people want the flexibility to compose a system from smaller components, so they have a separate repo per component. It seems easier to make each component a service and compose those services to avoid cross-repo dependencies. Monorepos -- the /new/ (relative to microservices) buzzword for many of us non-googlers -- make composing components into a single service much easier, and as more orgs adopt them, people are less incentivized to ignore the downsides of making every component a service and therefore are presented with the decision to compose subsets of components into a service or a distributed system on a case by case basis. Don't write eventually consistent distributed monorepos

I love how the chat was gaslighting the shit out of you while trying to explain gaslighting hahahaha

the real solution here is have a hybrid mixing serverless, monolith ,microservices but not so separated that it takes 3-4 steps to explain something

Definitely think whether you go monolith of microservice depends on the project. Recently I was only in an ancillary capacity involved in a project where I strongly advocated for a microservice system. My major reasoning was that certain components in the project will have to be replaced in a one or two year horizon. My advice was to have an API to handle these system components with a clear plan to have it replaced with a newer system component when that replacement horizon arrives. But, having a well planned API will mean that the component can be replaced without impacting the rest of the architecture. Also, by having it this way, my argument was that I could scale the system between on-prem (no internet connectivity needed) to hybrid or cloud.

I asked ChatGPT about the fallacy and it said it was the "hasty generalisation" fallacy, which is also known as the "unrepresentative sample" fallacy. The wiki entry for it appears to be "Faulty_generalization".

I think about how to modularize my app code often especially for my hobbyist projects as a game dev. Unity uses (a mangled) C# and I can achieve isolation by declaring assembly defs. However; I often discover I’m bad at setting boundaries. I get into circular dependency arguments with my compiler. It’s fun stuff to consider.

Unison language nailed this, by rebuilding a way to achieve dependency or module injection through ability or algebraic effect. All your needed implementation became libraries or modules. And the deployment process they are building in immaculate.

cargo workspaces for rust are a god send for this. Keep your data layer (module) as a agreed upon ORM or Raw SQL standard how to call your database. After that for each workspace member imports the shared domain. After that if you want scalability just docker container each set of routes. For example you have 3 workspace members for a twitter clone auth, status and timeline. Each one has the domain workspace and can just call what they need from the database. This allows for code reuse, scalability and low latency. There are somethings I suggest to clients to microservice out. Auth system are usually a good idea and even email/sms message relay servers. 9 times out of 10 thought I don't recommend it. I have found no reason for it, keeping teams in separate bubbles is just as easy as well. As long as you group each set of routes correctly you shouldn't have confusion on what team 'owns' which data. I do this by doing status_media or status_comment in the domain tables. If you need a change ask them to update the layer so their code still works or you can add a new function to the layer and send it over for approval.

Enforcing module boundaries in a monolith is incredibly difficult imo.. so I do like some type of “microservice” structure to enforce them. But that doesn’t mean you need to separate things in such a way that you incur such huge dependency chains - you can still have big modules

A few weeks ago I spoke to a guy building a startup, building microservices for months. I told him that he should not be doing it this way. He wanted to keep code "modern". What he should actually be doing is to build the core functionality and ship the code fast. Writing a modular monolith is usually a better idea for startups. A big mistake they do is to think that they know the requirements.

When he did the Seinfeld impression I CACKLED in the middle of a coffeeshop and got weird looks from strangers.

The article thesis is defended and correct. One repo, one tested/compiled/coupled unit. Incompatible interfaces must fail to compile. Modules can be separately scaled and deployed as microservices, but maintained, verified, compiled, and tested together.

In the film, Gaslight he was upstairs rooting around in a boarded up attic, using the gas lights upstairs which caused the lights downstairs to dim. When his wife brought this up, he convinced her that she was going mad. He also hid things and accused her of hiding them in the pursuit of getting her getting her locked up in an asylum and getting the house.

People hate large monoliths because they take forever to build, test, and deploy. Often if one person messes up, it blocks everyone. Modules offer a better authoring experience than microservices, but microservices have great CICD. I think the ideal setup is a monorepo with shared tooling+packages, and granular build/test/deploy.

I only just compiled ffmpeg from source for the first time last week. I felt so proud I went for a walk just to strut and smirk in public lol

Actchually, they are pretty hard to maintain if you are dropped into a new project. Deploying the app is actually deploying 50 different apps which can all fail separately.

What about API Gateways to manage interactions between various versions? Conceptually this allows for multiple parallel versions to operate simultaneously and allow other service teams to iterate at their own pace.

Go has been getting daily love and I'm here for it😂

I present to you the monolith: POSTFIX if only it was modular and almost never broke.

Date - Time is living hell on Earth

Awesome video man, can you do a full video about your editor setup?

For serverless your codebase functions like a monolith for the developer working on it, however the deployment chunks out your defined functions into individual microservices. In your regression test harness you can capture the behavior of each service post-deployment. IAC in a single system is the best of both worlds.

~ 30:30 about the git submodules… I feel your pain 😂 We have a Yocto build setup using submodules, and it sounded like a fantastic idea when I was reading up on it getting started, but in practice it’s definitely had some pain points…

I see few reasons for microservices: 1. Legal constraints. If you are working with an external software development company to integrate their software into your system. 2. if the service has already been developed because it is a common problem, such as authorization, reporting, user management, etc. - everything except business logic. You could even go so far as to title backup and monitoring as a "microservice".

"My tool should not exist in a year"... there's nothing more permanent than a temporary solution.

I would argue that microservices is more about optimizing for development process speed (by promoting more losely coupled and high cohesion modules) at the cost of less optimal technology choices imposed by the distributed system that follow by it. Short Time to market is more important today than optimal technology choices. We trade off one upside for another downside here. And regardless of any current regime, nothing can replace strong engineering practices in the organization. Without it you will eventually end up with big ball of mud regardless of whichever architectural regime you follow. The return of the monolith will hence solve nothing on it self. It is just another technology choice. Strong engineering practices is key for any architecture to work and evolve over time in respons to changes in requirements.

Me being ahead of the curve and writing my code modular for over 3 years

That isUTCDate function looks a lot like my function I used at work to determine if the material number for a pick order was provided as: an SAP material number, an *outdated* SAP material number that needed to be translated to the newest version (cause they were too dumb to use the versioning system in SAP) the document number that actually appears on blueprints, the customer document number, or some terribly formatted version of any of the above where periods were changed to underscores or someone forgot to change their excel files so they don't drop trailing zeros or just some made-up-ass number that didn't exist in any of our warehousing systems at all. Glad to know that despite the fact that I am very much an armature programmer I can hang with the best of them in the pro world ;)

7:14 change management is always hard, that's one of the most hard things ever in software. software ate the world because it can change and adapt, but on the other hand, its also its biggest disadvantage, sometimes its too easy to change, then things break.

Usually we associate the monolith with the concept of "big ball of mud" while microservices are tidy, tiny little cute (or at worst case, the are "small" balls of mud) One may wonder why it's not possible to achieve a tidy monolith, and in fact there is no theorethical limitation that prevents to fulfill this ambition.... But try to work in a monolith and explain to some manager/stakeholder or even some engineer that "I can't just query that table straight away....IT'S OUTSIDE MY BOUNDARIES", while in a microservice architecture it's easy: "do you want me to add one week of work to set up a a new endpoint, a client a circuit breaker a subscriber and a projection? Well so don'ask me again to read data in someone else's table"

Chat: "Oh no, he remembered to disable alerts this time. Lets repeatedly try to break his concentration with highlighted questions completely unrelated to the post he's reading!"

At my job the management has us just kicking the can down the road, with sufficient force to ensure it won't be our can anymore by the time we get back to it.

I'm convinced microservices exist (partially) because JavaScript has no concept of "visibility" or access modifiers. One of the things microservices offer is a well-defined boundary between "modules", which is obviously useful for large projects, making up for a shortcoming of the language. What should be a very lightweight function call, becomes a very expensive remote/network call.

You know, there are micro services we all sort of agree upon, Like data storage on a database and files on something like s3/minio, payment systems (not sure if it can be considered "micro" and we often outsource it but still) maybe even that an identity provider should be a separate service as authentication can be a cause for security concern which you might just want up and ready as a service.

Microservices isn't real they can't hurt you ❤ Meanwhile microservices at work:

As a dev and architect I will argue for distributed monolith or microservices as the best way, because the targeted scaling you can do is great

in. five years we'll be circling back to microservices again

7:28 go to your LB and add a redirect, keep doing that until your LB dies or you loose your sanity.

“Date is just a number” -prime

5:36 bro's face is so symmetrical

Best advantage of microservices is the ability to deploy changes without relying on another team and also not needing to work in whatever service language + frameworks another team chose.

You know that image of Homer looking great, but all the fat flaps were tied up behind his back? That's UML for me.

9:39 linkerd is an excellent, open-source service mesh that gives observability and control across the entire call stack. it essentially runs an in-pod, transparent proxy for k8s service traffic.

I’d subscribe to more insightful tech talk discussions. I understand the funny guy live takes but I’m genuinely interested in these topics but it’s hard to process the randomness and sidebars.

And today I learned about numeric separators in javascript- 2023_0000 - neat!

One detail I also believe is often overlooked, microservices does not mean communicating over a network using HTTP, for latency critical a sidecar strategy using pipes could also be used if the code separation is worth the effort.

"The key is to establish that common architectural backplane with well-understood integration and communication conventions, whatever you want or need it to be" -> *Solid foundation for the tech* "The key is to give the team the direction and goal, the autonomy to accomplish it, and the clarion call to get it done" -> *Solid foundation for the people*

Galactus knows all, you must not have kept him happy

I work with hundreds of git submodules. You have to write automation scripts to make it manageable. When sharing libraries across projects, it's faster to develop with submodules than republishing npm packages, so they have a place.

Timezones is the black hole of programming. Nothing escapes that level of complexity.

Probably not in Python or somethin that manages dependencies like Python :)) At one of jobs, I saw how folks made modules with C# instead of microservices. I liked it. They had projects organized like multiple libraries (some of them were kinda old, even in older versions of the C#), they could manage every librarie's dependencies separately. Even have "libraryA 1.0" in one place and "libraryA 1.1" in another (I am NOT saying they did it for some purpose, I am just saying that that it caused no trouble at all). Correct me if I am wrong, but when I think about the same approach for Python - I think it will be a dependency hell, because when project becomes large, one module obligates all the other modules be in the same context and fit into dependencies.

I think half the problem is people dont know where to start/end a microservice's purpose. I like modules and pulling out as you see one part needs different scale

Libraries are a good thing and have existed forever, and yes, adding dependencies to many projects where you need it, will serve the need. But next time you want to make a change to the library source code, you are now stuck updating 100 other projects that depend on it, or worse forget to update a project that depends on it. So instead of writing a service with code, you can just wrap the library with a lightweight server (netty, micronaut) and host the library on it instead. You get best of both worlds and only have 1 place to update the code and others will just fall into place as long as the serving API is still the same, ofcourse.

hey, sorry about the question, do you have somwhere the list of plugins you use for your Vim? (autocompletion and so on) thanks and great video!

I suppose the modular monolith is becoming the "new trendy thing".

I think the following is more reasonable than Micoservices even for larger corps: 1. one team, one module 2. a team can own dedicated datastorage if they have to deal with write heavy traffics. More specifically, imagine we are building KZfaq. There would be 'chat team' who's developing backend system for chat. It's a very bad idea to let them use RDB for developing chat functionality since RDBs are not scalable for write heavy tasks. Instead, they should use scalable NoSQL or NewSQL like DynamoDB or CloudSpanner to store chat data. Using their own datastorage doesn't mean they have to own dedicated codebase (repository) or application server. They can just develop a module or client library or API endpoints which utilize the datastorage while sharing codes and application servers with other teams.

I enjoy the arguments about all these companies switching to microservices, kafka, and mongo as magic on improving their applications... when it was actually the fact they took 10+ years of lessons learned to just rewrite the application. Switching stacks and patterns was an excuse. They would have had just as good performance improvements had they taken the same stack they were already on and used lessons learned to get rid of all the old garbage.

Microserves would be a great solution to many problems if communication wasn't limited by the speed of light and physical interconnects. When you physically separate all the computation, you are forcing processing to happen in serialized form and overall latency for a given request gets worse.

"Who writes bad monoliths?" - John Carmack.

The company I work at attempted using submodules...and later abandoned it for NPM modules approach

Touching Some Fuking Grass Dude

Only because you mentioned obs, this is better than the obs overlay imo. White text overlaid on often other white text in the blogs being read is basically impossible to keep up with.

27:52 Just here to clarify that n orders of magnitude is 10^n , so "five to seven orders of magnitude" is 100000x - 10000000x, not merely 10x (which is ONE order of magnitude)

The saying you're thinking of is something like "creating the disease to match the cure", I think. "We discivered a cure for the disease we created, " or something?

Really it just comes down to how easily you can make changes. A real monolith is simple, but can be difficult to deploy when lots of people are working on it and they don't always guarantee that it's in a deployable state, and that can lead to long lived feature branches, painful merges, slow release cycles, etc. Microservices solve an organisational problem, not a technical problem. They allow pieces of the software to be deployed quickly and independently, and allow teams to focus on their own areas without having to be an expert in the entire system, but that comes at the cost of many network calls that may fail. You accept that complexity in order to scale the number of people that can work on the system. A distributed monolith with shared modules is the worst of both worlds. You make a change in a module and then have to deploy multiple services that use it.

Strawman, but also, people are looking for a silver bullet. If there's a problem with something it's obviously not a silver bullet, so time to use something else.

Basically if a number is larger than 2,023,000, it's a UTC number

I'm unsure of where to draw the line for when a product uses microservices. My perception has previously been that if my TEAM only has a single service - I am not developing any microservices. But articles argue that microservices is just when "each team can work on isolated parts of a system". I work for a frontend web team. We use many services developed as separate codebases by separate teams (content-API, auth, tracking, ads, payments, personalization, and a website-team) some would say we are using microservices since each team can work independently on their own codebase. However from the viewpoint of one dev who's only concerned with the work within my own team I sure don't feel like I'm working with microservices if I decide to add a brand new feature like a game into the same codebase and service that normally consists of a news website.

Code and it’s consequences have been a disaster for the human race.

It sounds like NetFlix had a time assignment problem then, and need a standard library for each language used that is then used by all devs so time stamps are consistent. I guess there would need to be a company wide developer update in it, but it would be worth it long run.

25:25 English, the pseudolanguage that has a totally random writing-to-speech system. In other languages, a ten year-old can read and pronounce words of which he doesn't even understand the meaning.

I could swear I learned about gaslighting being akin to 'telling a white lie' which ironically turns out to be the most damaging lie, when it's consistently enforced and ends up making the person doubt their own perspective, inevitably leads the person to experience gang-stalking or eye-gouging. It's why I tend to admit to anything small if someone 'genuinely' asks, or agreeing with someone 'forcefully wrong' they could be right, which I now realize could be my egotism refusing engagement. I think I might've retrofitted the word to past experiences by cultural osmosis, cause I can't figure out where I heard it, but I remember distinctly feeling relieved in an asylum and finally being able to conceptualize what someone made me, do to me.

gosh hitting up 100 services to deliver one request?! That's one fine technical debt! chop chop.. 10:22

when MS were in vogue, I attended an MS focus conference. Even the people that presented MS solutions you could feel they knew they were selling horsesh&t with all the hacks you had to do just to make simple things work. Do people really still think this is a good idea?

The article mentions microservices bring re-use. But isn’t the whole idea of microservices that you DON’T re-use… microservices style optimizes for fast changing environments. The more consumers a service has, the harder it is to change. So when you have > 2 or 3 consumers, split of a new service instead of bloating the purpose of one service.

Left a comment for the future him in the codebase then quits Netflix, absolute chad

that is the wrong definition of scalability, lol :D 0:50

It's called the "Nutpicking fallacy"