So, I stumbled by a YouTube video titled “Microservices are Technical Debt”. A guy with an unmaid bed in the background was interviewing Matt, a principal engineer from DoorDash. Let’s glance over the mess in the life of the interviewer and focus on the mess presented by Matt.
Well, where do I start. Let me jump at the deep end right away and put out a clickbait statement. It would seem that Matt has some misconceptions about how to build a highly scalable, highly distributed software system consisting of highly autonomous components. Ouch, right?!
Am I a world super expert in those kinds of systems? Nope, I just have been doing these distributed systems for about a decade so I know a thing or two about them. Watching that video made me suspect that Matt has some architectural misconceptions related to distributed systems and misses critical pieces required for successful development of such systems. After watching the video I was surprised that Matt didn’t ask himself a question – what am I doing wrong because my product clearly has serious architectural / design issues?
What issues you might ask? Well, for one, Matt mentions that most frontend initiated interactions in DoorDash result in 1000+ chain of RPC calls between microservices. Hearing that I had a single reaction – WTF?! How can someone think that is acceptable? Well, it seems Matt blames those shortcomings on DDD, without really understanding it because if he understood DDD, then he’d known about bounded contexts, aggregates, boundaries, responsibilities, etc? His condescending comments at 21:30+ seem to indicate that he considers DDD a nuisance. That definitely came as a surprise to me.
I don’t usually bother commenting on or responding to articles or videos or whatever else, but this time around it was past midnight and I was stunned by what I’ve witnessed, so I couldn’t resist and commented on the video!
Once again the “microservices” misconception strikes and makes a killing! For some reason, after all of these years, people continue to believe that “microservices” is something more than a simple deployment strategy! Few loud people online made their names by promoting “microservices” idea as an architectural style and, as always, loads of software dudes jumped on the bandwagon of a “new shiny toy” without trying to grok the concept and what it really means. I bet you many wouldn’t be so excited to join the movement if they realized they need to learn and apply a mix of Service Oriented Architecture, Domain Driven Design and Event Driven Architecture at the very least if they want to even attempt to be successful at building highly distributed software systems!
Let me repeat what I’ve been saying for years (whispering in tiny circles, I should say instead, haha!) – “microservices” idea is nothing more than a deployment strategy.
Preposterous! Such an idiot! Who does he think he is?! Smart people on the InternetS say it’s the holy grail and it’s a kind of software architecture!
Well, if you are still here – let me repeat. A microservice is nothing more than a deployment artifact within a largely distributed software system. “Microservices” is NOT an architectural style.
What I’ve been seeing over years is that a lot of engineers, and not stupid ones, who never built highly distributed software systems, don’t think twice when they start on such systems. And there lies the biggest issue – being a decent software engineer but not having experience of building highly distributed systems won’t help you. Matt and his team, it seems, is a case in point – he himself says at around 8:50 mark in the video that DoorDash realized that they no longer can do with a big monolith, so they set out to build a distributed system (based on “microservices”) – “people who never done it said we are the only ones here so we’ll build it”.
Now, don’t take this as a criticism of the engineers. This is a criticism of the management. Even if you have a great team who’s built you 10 or 20 single family homes, you don’t hire them to build a skyscraper. By all means – use them, they are amazing domain experts, but get few leaders who know how to approach building something that you’ve never done before and let those guys teach the rest of the crew a new way of thinking.
I’ve written on this subject before in few of my posts (see few previous posts from few years ago), but since we’re here let me re-iterate some of the points that one would need to adopt in order to attempt being successful at building a highly distributed software solution:
- Forget all the “wisdom” you’ve read on “microservices” – a microservice is simply a deployment artifact containing one or multiple DDD Aggregates.
- You need to understand the foundation of a distributed system architecture. In my view this includes 3 pillars that have been around in one form or another for good 20-30 years
- Service Oriented Architecture
- Domain Driven Design
- Event Driven Architecture
- SOA and DDD, in my understanding, compliment each other very well (do I dare say they are different ways of describing the same concepts? Nope, I’m not Eric Evans or other smart guys behind SOA)
- SOA, to me, is way more foundational than what you can read in some online resources – it’s all about ownership of data and business logic, boundaries of responsibilities and transactions orchestration. Not so much about P2P communication that emerged as part of SOA implementations put forth by big software shops back in late 90 / early 2000.
- In my view a DDD Bounded Context and an SOA Service are quite similar conceptually
- EDA is all about publishing notifications when system state changes. In the case of mixing the big three (SOA, DDD, EDA) we’re going to be talking about DDD Aggregates taking a responsibility of managing state mutation and, as a result, publishing notifications into the greater system for whoever might be interested.
- If all communication you have is notifications of a state change, then you’ll have to change how you think about interactions and dependencies.
- Use message brokers as infrastructure to transmit EDA events and SOA commands
- An event is a notification of a state change that already occurred. There is always single publisher of an event.
- A command is an instruction to perform state change. There is always a single consumer of a command.
- As I said in the first bullet point, a microservice is simply an executable component containing one or more DDD Aggregates.
- Build out microservices based on non-functional concerns – load profiles, scaling profiles, security concerns, resource requirements, etc.
- Either include multiple DDD Aggregates into a single deployment unit if they are not going to interfere with each other operation
- Or package a single DDD Aggregate into its own deployment unit (a microservice) if this aggregate has to deal with high load, has special scaling requirements, etc.
- To be successful with this, you need to start understanding DDD Aggregate concept. Well, ideally, you want to keep thinking about DDD as a whole too.
- Think about transactional consistency. This is your boundary of data change.
- Every piece of data must have a single owner – eg. a single DDD Aggregate must be responsible for changing this piece of data.
- A hint – this is the only constraint related to data stores and distributed systems. Forget the “a data store per microservice” nonsense!
- You also need to start understanding boundaries of responsibilities. Enter DDD Bounded Contexts, which are a bit more than responsibilities. Go read up on them.
- Build out microservices based on non-functional concerns – load profiles, scaling profiles, security concerns, resource requirements, etc.
- From high level autonomy is one of the conceptual cornerstones – all these Aggregates or “microservices” should be autonomous and be able to carry out their duties on their own.
- This has very far reaching consequences – if components are autonomous, their operation is much more stable and predictable
- Teams owning specific components are more autonomous within an organization
- Microservices / DDD Aggregates should not do P2P communication almost ever. This creates coupling. Temporal and spatial. And it’s bad – it leads to all kinds of failures and issues.
- Accept eventual consistency between DDD Bounded Contexts or Aggregates.
- Eventual consistency should never be present within a single BC or an Aggregate
- Bounded Context B needs some data from Bounded Context A? Consume event type of messages emitted by Bounded Context A and build a “view model” of data that Bounded Context B might need.
- P2P / RPC communication between your “microservices? Hrmm, not good. Something is wrong with data, logic and transactional boundaries.
- UX design is a very powerful influencer
- I don’t see this mentioned often, but in my experience UX design choices have power to massively affect underlying distributed architecture.
- Involve UX designers from day 0 and work with them to ensure UX choices are not going to force system architecture into bad position
- Work with PM/POs to communicate complexities that arise from UX design choices, try to influence those to ensure underlying architecture isn’t compromised
Now let me add this. As long as you’re able to ensure autonomy of your transactions, avoid coupling, then nothing prevents you from deploying a single executable with all of your aggregates in it. Sure, you’ll need to deal with various scalability and load concerns, see how to communicate between these aggregates deployed within a single executable (not a rocket science, tbh), etc. This way you can avoid the complexity of multiple deployment artifacts for some time but still have an option to chisel out pieces into separate deployment units when / if needed. Just make sure to avoid any kind of in-process coupling between those autonomous components otherwise you’ll quickly end up with a proper monolith.
There are plenty other consideration, but I’ll let you think about it all and then, if you have questions, feel free to ask.