 So before we start let me to get to know you a little bit more So could you please raise your hand if you kind of a software engineer? Okay, like the most of the audience and could you please raise your hand if you consider yourself on the management position? Yay, so not so much Okay, so we will try to explore the problem of over engineering in today's talk But let me first introduce myself. So who am I? My name is Alex. I'm with Althoros and my primary expertise is web development infrastructure automation and machine learning So what's the problem? Well, you know over the last several years. I my friends and my colleagues were involved in designing and implementing several complex systems and Each of those systems actually incorporated microservices architecture, but you know, not all of these projects were successful So I tried to classify my experience and experience of my peers and friends to kind of get and Understand what did the team did wrong? Actually, what mistakes they had in their process and why their systems became over engineered So yet again, sorry, I just like to make people to raise hands So could you please raise your hand if you felt that the project you're working on was over engineered? Okay, yet again like the most of the audience I could actually, you know Like merge the questions like are you a software engineer because probably every software engineer felt bad and You can't actually, you know define probably what over engineering is, you know It's not easy at least some people say like, okay if It is if you're a project does something that you don't need it to do like has some advanced capabilities But they're actually more than that over engineered solutions are actually harder to maintain and they tend to cost more and We are in a silicon welly in the land of startups So imagine how it connects with the startups if you have a limited funding You actually can't allow yourself to over engineer to be honest So let's go to the first problem probably the most common one It's about ignoring the data. So, you know over the time when I was speaking with the guys who actually designed those systems I was asking them like, okay, so how do you actually, you know make decisions? How should you split your system into microservices? What drives you and they're like, well Okay, I know these hotspots like image processing or something and I just need to scale them So I kind of extract them But they sometimes just don't think about how often these features going to be used in their system So they ignore any kind of I don't know usage data From their application and you know on some stage I mean when you just starting to develop your solution, you just don't have that data, right? So probably probably you shouldn't do that early, right? If you don't know what you're doing Then what's the point? So I kind of try to classify like the four primary sources of data You can use when you kind of try to design your application So the first data source is actually user experience analytics So how to use that so you implement the application at least the beta or prototype and you start gathering the data as Early as you can That way you can understand what pages do your users use and how long do they stay on those pages If we are talking about web applications, you can do the same in mobile apps or even desktop apps So it's not really a problem So another data source is actually performance and we engineers I mean we actually tend to overuse this one because we kind of expect that some parts of the system will be slow and we Try to use Exclusively this reason to drive our decisions, but that's not sometimes true, you know So that leads us to another data source which is called well use cases Sometimes engineers think that the business is something they shouldn't care about they don't try to you know analyze and classify What exactly the application should do But probably the information about the distinct group of users or applications or sorry, sorry But maybe the information and how you should design your system is right there in use cases in those docs So for example when I design something I like to think can I sell it separately? Try to think about this phrase. So is there a separate product inside of my application? Can it work like separately? Can I bill for it? Is it useful actually because you know when you create some reporting monitoring something microservice? It's not exactly a product, right? And I'm not sure if it can you know developed independently from the rest of the system And well, that's kind of one of the key points of microservices at all So if the microservice doesn't have any kind of business value Why should you do that? Why should you extract it? And so the next data source? Well, actually it's so common in the way that no one thinks about this is the source code So yet again if you have your prototype application and they think about well So we have this beta prototype. I need to split it. I need to refactor it I need to make it. I don't know scalable and then guys starting to look into the other metrics But you know you actually can data mine your source code You can get the idea of what parts of your code change together Because you know yet again engineers sometimes create the code Which is so intertwined and connected that it might not be obvious Which parts of your system are actually dependent on a shell or so you have some kind of I don't know price in service And every time you update prices in your application. You suddenly need to change the UI so probably you did something wrong and You can actually extract that information by analyzing the pull requests the branches It's actually pretty easy to do You know if you know some scripting language like shell or you have some package Which has the API for the git or whatever source control system you use you can actually do that and Visualize that well and no one does it for some reason. So this is crazy Well, let's let's check this out. So can you raise your hand yet again if you actually did this at least one time? Okay, well Bravo Bravo this one man in blue shirt. He has a beard So that's probably the reason why he did it Okay, so I was talking about this information and stuff so You probably got the idea that okay, if I don't have the data, what should I do? Well, let's let's cover the example first so my friend was working on the banking app and This is kind of a slice of the architecture there So they have like much more microservices, but here we see the documents the documents basically at PDFs with multiple pages So they have this system which employees use and they upload the PDFs into that system The PDF goes to API gateway the gateway then sends the data to PDF processor and What this service does? Well, he basically processes the PDFs They extract the information from the documents like names contacts and something like that Obviously it uses some bit of machine learning. So it's not so, you know precise They just gather stats using that system then the result is going to streaming API It's basically a microservice with web sockets I don't know endpoint and the web app is actually connected to the stream API So let's kind of cover it yet again You as employee upload the PDF and you can wait on the page to on the web page To see how the progress goes, you know, how the data is extracted from PDF PDF, sorry. So if you have like 90 pages, you're going to wait quite a bit, right? So the architect decided well Okay, I want to update the page in real time That's actually quite useful feature and the development team like the dream team. They're like, okay, we can do that So that was cool. They kind of added Well, nifty web framework to the web app. So they connected to the web sockets They extracted the microservice the code was good. I'm not talking that it was crap, right? So they had the tests so they had integration tests for the web app, which actually launches it Kind of uploads the document and waits in the loop until the document that's processed to check how the web page is Well, he's actually updated. So that was terrific. And then This new information bit came in You know, the point here is that document will actually take some time to be processed, right? so at least several minutes and the total time actually depends on how many pages do you have in your document and So employees they weren't actually waiting on the page You know the usage data show that the users spend at most about two seconds on that page You know what's scary here because you know, how many services were involved in this? I mean the PDF processor needs to somehow notify the streaming API that you know You have you have some progress on this document. You need to update the web page So also they have this web application with the web web framework and they needed well kind of a new guy to write it Because well, these were kind of smart backend guys. They didn't know anything about JavaScript so and the streaming API web service which actually came only Because of this use case So yet again the code wasn't so bad It was great and they had tests and all that stuff But well at the end they didn't even even need that, you know users never saw those updates They were just waiting for the document, you know, they go grab some coffee They go back and they see okay your document is processed So why should we care about that and that leads us to Another problem when people split their systems or designed them from the ground up Using microservices architecture so too early, you know without heading their data And that's actually well pretty common mistake because they think like okay I spoke with those sales guys and they told me like microservices are cool So we need to do that so we can put the label on the box and say like we are scalable We are reliable. We are so so so cool But you know in the end for example, if you look at the backlog of these projects The backlog is dominated by the technology Like how can microservice a communicate with B to do implement a synchronous communication, you know It's it's not related to the business at all. So it just you know, it just siphons the money Out of the company siphons the budget So it hurts the business and it's more complex because you know, if you have those these juniors I mean in your in your corporation and they come to work on your project They kind of need to go through unboding procedure So they need to know how to launch your system how to configure it and you can't imagine How it hurts the development process if the developer just can't deploy this stuff It's so complex. They just can't analyze how to work with it They just don't know where the errors are going from they need to look at all microservices And this is daunting, you know in yet again It kind of kills the whole purpose of the microservices architecture where you can develop the system in independent manner like every microservice is an application every microservice has its own release cycle its own Technologies its own way of configuring things So well bottoms up it just slows you down and costs more. So this is pretty obvious Let's see at another side of this problem and I could call it I don't know too late, but I decided to call it huge rewrite So sometimes in large organizations that tech depth Accumulate so quickly on the projects that they decide like okay, we we need to throw it like away We need to rewrite it. So yet again time to raise your hand. So guys How many of you actually wanted to rewrite the system you're working on from scratch? Okay, okay, I think you're not being honest to be honest Because I actually think that every engineer will have this kind of feeling Otherwise you're a pessimist, you know Well, especially at the beginning of my career, I was thinking like I can't rewrite anything, you know Just give me a night. I will do it But you know some projects like I don't know windows try to create operating system overnight Well, you can't do that, right? You just can't but they try anyway so we writes they usually don't and well in my experience because The way I like to explain it is this when you develop a software project You kind of try to accumulate the knowledge about the domain Like okay, if this is a banking app, so watch our clients are what do they do? How can I like make their lives easier and when you rewrite stuff? You just try to solve the problems created in your code created by the engineers in some way So if you were hasty you have some crap you need to fix it So if you didn't think about the design you have some crap yet again you need to fix it so Usually when it's all stake game, it's kind of risky because sometimes businesses you might not believe it But sometimes businesses they actually agree for that. It's like oh our application down again. Oh those engineers They kind of killed it. We need to rewrite it We can we can actually allow them to do that and it can actually hinder the business too In what way so for example business stops altogether stops the updates. They are not moving With the market speed, you know, they don't add new features They just wait for you to rewrite the crap that you've created and that's problematic to be honest Because yet again if it fails, they just burned down the resources and they didn't fix anything, right? And sometimes when engineers solve the problems created by other engineers They tend to over engineer things like they add more patterns more abstractions They kind of enjoy themselves in the way that okay, this is the castle I've built But the point is sometimes the systems who became over engineered and so abstracted that you can't actually move further again You need to rewrite it yet again. Sometimes you don't have tests. So people who call Refactoring this process refactoring if you don't have tests. That's not refactoring. There is another word for that I'm not going to say it So, yeah, let's move to another problem here So I have this friend, you know, I like to say I have this friend So just no offense in DA and all that stuff. So I have this friend and he was working on the news portal app and these news portal They actually grew in audience quite quickly. So they decided like, okay, we need Microservices like right now we need to scale we need to kind of improve ourselves and so yet again the dream team Was kind of created the most expensive engineers like from local market We're kind of gathering and forming the team to rewrite that stuff and implement microservices But you know the thing was what was there is that they didn't think about, you know, ramifications of transferring application to microservices So for example This is how I could summarize the development process So at the first step you kind of design stuff or think about what are responsibilities of the service You try to analyze the requirements, then you think about how to implement that then you actually do that That's development step or implementation step Then you kind of test it probably and then you deploy it if it's a web app or mobile app I don't know publish it to the app store and something and it kind of happens again and again and again in the loop You can argue that well, that's not how I do things I have like 80 additional steps But to be honest even if we have this argument I can pretty much convince you that it's just the same loop all over again And so the ramifications of microservices are here on every step So for example during the design step you need to think well What microservice will do you can't just you know go to the one folder of your monolith and start writing code You just you just can't you need to create a new one or you can need to at least think About the responsibilities which microservice will do that and sometimes you know meetings people tend to argue with each other Like okay, which microservice should do that So another part of this is the development remember that junior I was talking about because if that junior cannot launch the code I mean the whole system or at least the part of the system he needs and If he can't like clearly understand how to debug it You know how to kind of reload the changes what he should actually update when he changes Well, that becomes a bit of a problem So the process needs to be obviously documented and then there is testing if you have this large monolith You actually well if you're talking about web applications, okay, so you have this web app You run the test on this web app. You have a single endpoint and you just don't think about it You know, it's it's kind of easy, but in case with microservices Probably you want to test them independently You need to have some continuous integration in place because if you don't well, you're crazy And you're going then you just won't understand which services failing during the release and yet again the release or the deployment step What microservice should I update if I change something? So it's it's not clear because you know the release process for the monolith It just let's push the monolith let's deploy the monolith so it doesn't change at all But with microservices you might have older dependencies like what should I deploy first if you change the API? You need to make sure that some of the microservices are actually backward compatible with it So new problems you should probably think about them beforehand And so this news portal stuff the guys actually had some bottlenecks here They made this crucial mistake, which I would call. I don't know. Let's not call it by the sport So they had this team lead guy. He was really smart. Well, he is actually pretty smart But but he was responsible for the deployment of the application. He became kind of a release manager And so without him knowing not even one microservice could be deployed into the system And they couldn't just they couldn't just analyze what microservices need to be redeployed after the changes were made Because they just didn't have a process, you know for separate or independent testing of that So yet again, I'm not saying that the code was bad or tests were bad. No, no, no These guys were actually amazing, but they just didn't think that well If we split our application to microservices if we will create like, I don't know 10 or 20 separate applications they tend to develop independently and Probably if you have I don't know if you have some nuclear plant software stuff, you don't want to update it all at once Right, so maybe you should start with the simpler stuff not not just you know the whole applications in one run So this is actually pretty complex so let's let me finish with the well with the latest problem and Yet again, I like to say that it's so common in in the way that no one does it. I Would call it cost-benefit analysis So how to better describe this well we engineers are actually hired because of our expertise, right? So we are kind of experts in our field and yet again It kind of intersect with the fact that engineers don't usually think about the business like what feature will do What consequences will it have for my client? How it will improve the life of my client and how much it's going to cost because yet again If you will burn all the money and not deliver anything. Well, they won't be company next month, right? So I cannot tell you exactly how you should do the cost-benefit analysis in your projects But you surely need to understand that when you add the microservice Someone has to manage it right someone has to monitor it someone has to deploy it and Clown foundry itself is not going to do it for you I mean that's just the platform and you as engineers are using it, right? So you need to kind of take the responsibility for the actions you do What features do you implement? Because sometimes well management when I am leading I'm just asking like okay How how much time will it take to do that like estimates give me the boundaries like management likes to ask that stuff and They sometimes do not accept no as an answer For the reason that engineers don't usually you know involve money in the process So if you say something like okay, it's going to cost us like a couple of thousand bucks on the infrastructure Then maybe they will understand that So you I can't like define the formula for every project But you surely need to find a way to kind of count the money you spend on the development The money you spend on maintenance the money you spend on the deployment stuff So it will be clear for you if do you want to play this game with microservices or not because you know for the small and medium-sized projects sometimes these costs they just you know they just You don't need them you won't be able to succeed with your projects with it so with that let me finish and Mention a couple of guys I actually stole some ideas from So first guys named Robert Martin So the guys from I don't know Ruby community might know him better as Uncle Bob And he gave a couple of talks named architecture the lost years and the reasonable expectations of your CTO So they're kind of inspiring in the way that they speak about you know taking the responsibility for the code You write thinking about what ramifications These changes have Well, and he mentioned cost benefit a little bit But wreck yarn the guy from the dot net and the main driven design community actually speaks about it all the time so he gave a couple of talks called stop over engineering and The last one the lawn set story of microservices So you might not like how he describes the architecture. He may sound a little bit too harsh But you know sometimes it's useful to see like This world from the both points, you know to analyze it well what you can do to in order to improve So I don't know if you want to write it down or take a picture you can do that so with that let's go with the questions and Actually, you know in order to stimulate you I have Some USB stick drives. I think I have four of them. So let's play a game who asks the questions first gets the USB stick Okay, your gentleman. Oh, can we pass the mic? I don't know Okay, maybe I should do that. Yeah Give me a sec Okay, okay, so the short answer is don't That's that's probably the simplest one because you know The only the only cases where I've seen this to work is when the the application itself was more You know the business requirements were more complex than the overload coming from microservices So if we are talking about pizza delivery, for example, pizza delivery is probably one of the simpler domains, right? Well, what pizza to deliver do you want some additional ingredients and compare it to the Microservice architecture like communication patterns how you should deploy it and all that stuff So I usually measure it by that. So if for example, we're talking about some banking application Where I don't know the loan process or if we're talking about insurance something with money involved like large money involved These applications they tend to be like much more complex than Microservices stuff and this is where you can actually like implement it without any issues. Okay, so I promised the USB stick Sorry, I didn't gave it to you the first time but you can have it So the next question Okay, okay. Okay guys, let us try to optimize it Can you pass the mic to the next person after you finish? It doesn't okay. Where is the organizer guys? The overengineered the microphone Now it works, okay great from the experience the scrum approach in different flavors Think this problem or even make it worse to be honest it sometimes make it worse It's it's not like I hate any kind of agile process. No, no, no, no, that's that's not the talk I would like to have but you know To be honest well in these kinds of processes sometimes when people kind of design those things they tend to collaborate You know and sometimes people try to balance the architecture So if you have like several different people working on that problem You actually tend to kind of go away from overengineering the problem is with you know highly charismatic individuals which can actually sell microservices to your company make you do that But in the end they won't be responsible for the decisions you make in the process So you will fall into overengineering category. So I hope I answered your question Okay, so use the stick for you. Let me take it Let me take it and then so who's the next one to ask the question Okay, okay, I'm just collecting the sticks From your experience, how have you dealt with avoiding overengineering and converting a monolith to a microservice tail? So sorry, can we come again? Sure from your experience. Have you ever faced in a scenario when breaking down a monolith? How do you do your void? Okay, okay, so the question is when I have the monolith already How can I try to avoid overengineering when splitting this model with the my microservices? So, okay, let's let's think about that. Well, actually the application is really complex and we have to do that, right? Well, I prefer refinement over the huge rewrites So yet again, well, let me show you the first slide So when you have an existing system in place, you can start collecting data. You have the source code You can analyze. I don't know the classes or parts of the system which we're changing together Then you can find those hotspots. They will be pretty obvious, you know, like I have this I want to extract that there's a microservice But the right way to do that is not to extract everything in one blow You need to extract it one by one and continue to collect the data and see how it affects your system Does it help you to perform better? Does it help you to kind of, you know, implement changes in your system better? Only that way you will understand if your decision was right or wrong So if you see what I'm saying, well, okay, the stick for you These are actually, you know, big ones, I don't know, 30 gigabytes or stuff So if it will sparkle your imagination about the questions Here you go Okay, I have the last one Okay, okay, so Just take the stick, take the stick So the last stick was given away and now the question was about the persistence level layer So how do we fix complexity on the persistence layer? So how many of you actually heard about the term like domain-driven design? Okay, okay, so lots of hands like around half So the thing is there is a pattern called the read model And so what this pattern talks about? Well, you will you will find like different explanations. I actually have this one So This model doesn't actually map to one business entity, you know The read model is something you can use To collect the data from multiple databases multiple models into one so if we think about I don't know some kind of a page and Our project has users and I don't know let's call it cars So you can create a read model called user car report And this model is going to be built from Oh, sorry, it's going to be built from multiple data sources So you write a separate micro service or you or you have this one micro service which is responsible for building those read models and Then then you can actually query this read model very small read model is not You know, you don't need to actually to be restricted in what database exactly you want to store it Because it's actually going to depend on your use case Like do you have one of a kind of this model like in the whole application like a single annual report or something? Or do you need some kind of filtering by ID or flavor or color? I don't know so basically that's how I do it I try to kind of find those models in my application and extract them as a read models So I can kind of abstract the way I can read or query for these models And how I can actually build them, you know from multiple microservices probably So this is also also called like query commands segregation if you heard the term so Oh actually yet again it depends on what is your domain about you know If your domain is kind of complex on its own It's it actually fixes the problem because imagine if you have a separate microservice I don't know with the front end just to show that data and it has like pretty simple back end to supply the data to the UI So if you make this back end microservice to connect to all the databases in your system or in your project It's going to complicate the things for the guys who actually support that back end And you know sometimes the teams of UI engineers. They actually you know support their back end I mean the simplest version like a gateway or something So you can actually hinder the process for them if you make them to know too much about Outline microservices so that way you kind of provide for them An exclusive gateway I would say in a sense that you kind of hide the complexity of the information for them You just speak with them like in what form do you want to use that data? How do you want us to collect it then you build that model and you're done Surely it's a microservice, you know, you know that that actually depends, you know Well, look at the large like enterprises or well if I can call them enterprises like Twitter or Facebook They have this one holy endpoint right which are going to so obviously this is kind of top level gateway API Public facing API so you may call it a monolith in some sense But it can actually have you know some advanced routing inside of it So it's going to talk to different API gateways in your application So I usually like to do it this way if I have like several teams in my project and in complex projects. That's true They can implement their own API gateways and then we're going to measure them together I mean in in one top level one if you get what I'm saying So sorry, I don't have an additional stick for the second question or the third one So that's not going to work guys. So, yeah Any other questions for now? Okay, so if I'm not blind from the projector, I just don't see any hands So well, let me thank you again for attending to my presentation Well, I will find you. I don't know I will be on Altra's booth So if you didn't hear something you wanted to hear you can just drop by and we are going to talk about it So yet again, have a nice rest of the summit and well goodbye