 Thank you dear friends. It's so nice to see you. Let's get right to it Did you notice that executing the same actions in different order can often lead to different results? Even if we take very very correct actions that normally get us to beer or some other wonderful results and do that in The wrong order We can get into a disaster How often have we heard oh shit? I should have done that first The famous thing and it especially applies in the software development because as we all know premature optimization and what is premature Where is this point? It is pretty hard to predict sometimes So what we're going to do today is we're going to talk about some not so obvious aspects of building the Distributed system of microservices in Python and as well as in other languages Without failing much because I know that's like yes if if you are full-time employed getting fired is pretty hard If your freelancer is gets easier Fortunately, we have now this whole microservice hype makes things much easier. You can fail in so many ways So getting fired is is really getting there. So I will be also Sarcastic sometimes so if you recognize yourself, it's not you. I'm not I don't mean anyone I mean myself in the first place because I went through this whole thing and let's start simple We start with a stage zero before even the whole microservice thing starts Imagine that the big boss is calling you. Hey, we have to talk you think am I already fired or what's happening? No, you're not he says it's time. I Was thinking I was reading report. It's time that we go microservices At this point you might think hey, but sir. We're just a hairdresser saloon We accept booking something on the phone anyway, like what are we going to split into microservices? Don't do that Don't do that. That's humiliating the boss. He perhaps read so many Gartner reports about success stories He was thinking how he will brag in the bar with his boss friends. I hey, oh Jimmy Microservices now. Oh, what do you have and you offer him some Django thing? No, let's just do what he wants because otherwise if you don't then you fail of the stage zero You will not even get that job as the freelancer is particularly relevant And now you think I'm talking a bit extreme maybe but I have a real-life case So I worked for one big and famous company which will I will obviously not name they hired me to fix their architecture and When I presented my first draft they said alright, but where are microservices so What microservices is that a requirement to say of course? I mean looking to the future now Should supposed to be an expert. It's a good well should understand that Microservices are not so much a tech solution as an HR solution when you have Thousands of people and they have to cooperate on the same project efficiently That's a very good way to go or if the Lord is has such a difficult nature But you guys you have like one request per second big time They were disappointed We thought we were hiring an expert and you're offering us a monolith Lost the job So let's just agree microservices take it as granted. It's hype. They will come don't resist it accept it and Start forming your happy and motivated team that is just eager to jump right into that right into immigration How would you normally do such a big new beginning? You would perhaps read a book and there are many good books You have a good selection also of the cloud container patterns available now published everywhere even for free But the book I mean The book is boring. Who reads the book it's 7 19 for god's sake. We have google google is a new book Google is all we need. Why would you read 500 pages in advance? Your team will be so upset. So if the team Wants to just prototype, you know Freedom of choice and the freedom is a basic human rights They will be upset if you don't let them prototyping you might lose your job So don't do that let them prototype because google driven development is actually not that bad It's not as bad as stack overflow driven development. Who knows the difference Difference is that stack overflow is when things are already on fire So if it's google they are sort of planning it in advance All right, now let's go a bit more serious now. Let's go deeper down to the stack stack You have to design it from scratch and here to be correct I have to say stacks because we have a new fancy polyglot infrastructure We have many languages and it's not just about the languages Because every language has own stack every language has own tools or on build process And now we have many of this wonderful You know when you push python code you perhaps do pilings you perhaps run tests you package it somehow Flake it whatever build a docker image. That's one way of doing it Now imagine the next thing you have is java You'll get to know maven the beauty of maven or and You will pack things differently some jars or wires or whatever they're using and now Imagine we have frontend as well on the stack you have javascript Wow, you would have to minimize it add shims concatenize stuff Babel other things a wonderful life and then if it's really really front and then the webpack Gulp grants. Oh, that is so much fun. I have a perfect illustration for this I took from the jess congress in munich It says there are milk products that last longer than some javascript frameworks That totally reflects the situation of the front end these days So, yeah, your stack will look perhaps like that and just share it with the ops ops will be so happy to see it They either jump out of the window or throw you out of the window So get prepared get your team with you so that it won't be you But of course, of course, you can always say it's better than a monolith an old ugly monolith this Nasty genga tower standing out there ready to fail We're going to fix this with microservices and we were way more elegant There is a surprise waiting for you because if you If you migrate from here to microservices 90% of cases you will get many little genga towers instead of one big Own with each own stack on with each own box and with own hacks to keep it somehow stable and alive, you know now imagine that There are just we have micro teams, right? Imagine there are just one or two people working on each genga tower It's a good thing. You cannot get fired because they depend on you heavily right now. So They will probably not just not fire you though the smart boss would probably vaccinate you against everything in this world because like their business depends on every Every little piece of this and you're basically irreplaceable I love it as a result. What I call it is an epic zoo of technology Imagine the front end is and react statistics is an r and then you have some admin side php legacy You have accounting made in java. You have to Maintain that whole thing somehow and these components they they bark some components meow and another components roar and Sorry, I forgot the fish fish is important, but it just rides logs so can keep silent now Imagine how can you possibly manage this whole thing? How can you keep this component sane and well for for keeping it sane? We're going to the next chapter Even deeper even more concrete We are going to repeat some wisdom of bigger companies that did it in the past already Imagine the new team member comes and says I don't care. It's all in python. I want my new fancy microservice in java script What you have to do Be aware not to offend him or her Should be polite and you just use the wisdom of netflix netflix had this issue They had the whole thing done in java it worked perfectly until they hit certain architectural limits So they had to split into microservices Which were java microservices and then they grew grew grew and eventually they had to deal with other languages as well So they developed a very very smart strategies that worked for them at the beginning They said of course at netflix You're very welcome to pick any language you like you you have a total freedom in doing that But just just in case just in case that language you'll pick will be java You'll get the failover. You'll get the discovery. You'll get logs. You'll get monitoring all for free automatically But if it's not java well, you're very welcome to implement it in your language of choice And at this point most of the developers said maybe java is not that bad option What I wanted to say is that at the first stage All right, let's say you have to split your monolith into many components But that doesn't mean you need to have that zoo right away You can still stick with one or maybe two platforms that will be well tested That will have a well defined process on the upside of building things on testing things and on deploying things So it costs a lot especially on the in the beginning to integrate every new process In your architecture. So if you can avoid it's just avoid it at the beginning and you can pick just one Reliable well-known language such as java and work with that Kidding kidding no worries So let's say we take python We need still to decide on the exact tools that we are using in python to write microservices And here i'm just going to quickly jump over the most popular framework and choices that we have Let's start with jango. That's not a joke. I mean if you're them good in jango, you can write the microservice in jango I know people who did that It feels to me like going grocery shopping on a truck Nonetheless Nonetheless, yes, you can do that still because jango is still like this very very flexible plugin architecture Which you can customize where you can throw useless things out and you have wonderful pieces like an admin panel They're Automatically but the problem is that if you take jango by default it has so many components. So if you think of that Actually each component could be a separate microservice So it doesn't really make much sense to use jango as it's shipped to you. It only makes sense if you cut Things you don't need out Then it's fine and speed wise. I mean we don't have to discuss jango. We all know its advantages and disadvantages But it is still doable So here many would suggest flask flask Everyone loves flask flask is so minimalistic It fits so well the whole idea the whole concept of microservices because it does just one little thing and it does it right I like flask. I mean flask is It's okay. It's not the most fast web framework, but it's a comfy one What I like in flask is the asynchronous networking and this is particularly useful for microservices because when you have a chain of requests From one microservice to another and they are blocking and one thing fails and the whole chain is blocked and does not serve More requests because potentially I don't know it gets out of the workers in the pool. That's bad. It's not cool So I strongly prefer and suggest you to use the synchronous tools so that you are not blocked If one of the components is blocked and you don't have a limited amount of slots of the parallel requests That you can process at one point of time. So there is a solution. There is quart Quart is a replica of flask. It's just a synchronous Copy of flask, let's say I didn't check it out in production. I just know it exists I played with it. It has a very same api as flask does. I think they are Developing it further now But it's a good option nonetheless as I said, I didn't check it in production flask. I did so if you were a flask person And I know flask is a Much better framework because it has a cooler logo and it's minimalistic It's like fiat 500 minimal thing does all you need looks fancy for those who do not need to compensate, you know with a big scale Totally love flask now. I'm just going to quickly talk about other options there. There is hug. Who knows hug Hug is great Hug is based on falcon. Falcon they say by friv by the Benchmarks that falcon is the fastest web framework in python by amounts of requests per second it can handle falcon winds It's synchronous, but it's very fast And then hug is a sort of abstraction over that simplifies building api. It's like jung arrest for falcon Makes developing restful endpoints very easy Generates you the common line interface right away because remember your microservice is not necessarily have to communicate over It could be that it's very useful if you have something like amazon cli Available right away. You create a normal restful endpoints. You have a console interface Automatically that's pretty handy. I used hug in production does a great job. Then there is tornado tornado is 15 year old now as far as I remember it's super stable. It's a synchronous. It's well tested well documented it's like more The older version of async.io, let's say That supports python 2 as well which for some cases because usually when you come to the project that Wants an architecture upgrade the chances are that they have a lot of components written in python 2 So with tornado you can have a fancy synchronous communication right away since 2.6 they Sort of assimilate the yield from or async await thing with just normal python generators So you can use yield You can use yield With python 2 and it just works very similar to the to the way async.io does I'm not putting async.io on slides because that is so obvious like that's a default thing You would go with a uhtp and async.io That's now the default way to go now the project grows and Thanks to python. Of course at some stage we hit the point as netflix did Where we cannot resist getting new languages and new stacks anymore So what can we do then? Then We should use the container patterns and there are books published There are articles published and also many many videos on youtube about that I'll describe three most popular patterns that are used and that I checked personally Let's start with a sidecar sidecar is a sort of a extra container That you put next to your main container that would enhance it To be more precise here at how it looks like imagine that you have a main container, which is a web server And your sidecar container is a log saving Application why would you need to separate it? You might ask you can have your main Application and something like a sentry agent that collects your logs or kibana or whatever the the trick is that Every web server even super old one and super legacy one can write logs to the hard drive And not every old framework has sentry integration So we go a bit another way and we say let's just assume Every app can write logs to the hard drive And we take another container that would pick logs from the hard drive and send it to whatever cloud aggregation system This way we separate totally the responsibilities of these two we say that the web server should just do the web related job and save logs as it can and it's a job of the Sidecar container to pick these logs ups process do the whole routing find the right cluster to submit it This sort of work This is the first pattern that you can use and the second one is ambassador It's a very similar concept sort of a big important fella that you have to speak to instead of speaking to some even bigger entity This is especially useful when you are dealing with some remote resources like a database or a redis cluster Ambassador will hide it for you Ambassador resides on your local host it runs Next to your main container and you only communicate with the ambassador It simplifies a lot because in your super huge application you perhaps have a super huge redis cluster that Every component that speaks to has to maintain you either need the ip or dns And if it's failing then you need a new route So this is all job of the ambassador and the ambassador is always on local hosts the same port So if anything fails, it's a job of an ambassador to get the new route find or maybe do some graceful degradation in this case or Caching things as well can be done on the ambassador level. So this is also a pretty common pattern very useful when you're writing big Big distributed systems It is particularly useful because as as I already mentioned when the new language comes in It does not affect the main container. You can have a main container in java and you can have your ambassador in any other language They communicate of a standard protocol standard ports. They're totally running independently Now finally the adapter the adapter looks like a like an ambassador, but the reverse ambassador If the ambassador pattern represents an application your main application with a simplified view of the outside world Outside cluster then the adapter does the reverse It simplifies the view for the outside world of your application to give you an example Imagine the the monitoring case like perhaps we want to have a health check On our main application and different frameworks different applications serve from different protocols They have different ways to check if the app is alive or it's down Some would provide you a url like slash health or whatever. Some would write in a file in unix That would say file is there then i'm up And some would just not do anything you would have to do ps auks and check if the Process is running in the unix machine So we can use adapter to help the outside centralized Monitoring system to get the status of every component Because we will ship this monitoring adapter next to our main application and the adapter will know How to check if the main application is dead or alive It would know if it should check a file or check a url And so then the externals the external part the Monitoring system would just use the same way to ask The adapter if the system is running or not this way you can also scale framework independent And you can keep the same monitoring system on a very very very diverse zoo good Now quickly going to the functionality after we know the patterns As we already discussed microservice usually has some business logic like the main thing that it usually does either serving some Crude operations on the database or doing some monitoring or something else And there are always communication functions, which are repetitive across many different projects It makes sense now to take this communication functions and put up in a sidecar The patterns are already described to you because they are very repetitive. They do this failover and discovery and Graceful degradation thing it looks much more elegant like that, but we still have one problem now. We offloaded all of the functionality of Network functions to the sidecar. However, Nonetheless, how should different? Application find each other you packed it into sidecar But what exactly is the sidecar doing? How is the network communication and discovery happening? You need to communicate with not only with a database your components Perhaps also need to communicate with each other and how exactly can this be done elegantly You can of course put ambassador for each service, but you know it will quickly get insane You can you can break things and lose your job. So, uh, you need a better way of Establishing communications not just with a big redis cluster, but also with the components Talking to each other like small dots And here we can finally introduce the concept of a service mesh. It's a new step. It's uh, yeah, it's a bit centralized It's a centralized component that keeps Keeps track of which service is doing what's and available where and so the idea is it's this That the communication between our little Our little services. They do not communicate only over Service mesh. They just use service mesh to find Where are other components located? They still communicate directly. So this is like dns and steroids basically you can say I can just use dns at the beginning, but dns depends on caching and In such an architecture you cannot afford caching too much because if a system fails you need a very very quick Replacement for a failed system and that's exactly what service mesh does. So you can think of it just as a Better dns system and to be more precise what exactly it's doing. I have a list here like you normally have lots Tasks like routing access control. It's another big thing that I'm not discussing right now But when you communicate within microservices, you need to decide on the common way how if you encrypt the data How exactly How do you decide if one service is allowed to ask another service of a particular operation? Then there is always a context of are you asking One microservice to another on behalf of user or you just need some meta information all of these little things You have to do repetitively in every new microservice. You bring in into your architecture And not to do that you offload this repetitive task to the sidecar And the service mesh is just maintaining the list something like the access control list routing list task list and so on good Finally, I want also to say a few words about the failover People say graceful degradation and short circuiting. There are many fancy ways of saying How would you elegantly tell your users? I'm down It's way more easy to bring the system down when you have many little services because it's a chain If one element of a chain fails, I mean if instead of showing error 500 you are showing a message I'm sorry. We're currently not working It's more elegant, but it's not good So instead of instead of doing putting effort in this short circuiting and failover I would strongly suggest you to put your effort in better logging and better tracing There is an open trace standard that was developed especially for this case when you have a big system and it's Hard to it's hard to debug it. It's hard to trace where exactly did the request fail Instead of putting your effort into excuses like I'm sorry. We couldn't do it Trace it log it make a have a system The even when we have here sponsors. What was the data doc people like they do it I use data doc for that I'm not advertising them But also new relic does it invest your time there build a reliable system where you can easily trace every request And every time you can say if something fails it was fault of this component because without this You will pretty certainly eventually lose your job Thank you for attention and I wish you a very nice zoo of your microservices Thank you in turn for this refreshing talk and we now have time for questions So we have microphones on the side or I can also move Okay So in the meantime While they get inspired I wanted to go back to your recommendation on basically the frameworks and There has been quite some articles about Why and talks about why frameworks are can also constrain you a lot Especially when you want to iterate over your products or Let's say add new features because they basically Fix the way the business logic is defined So Would you have also some recommendations there on maybe just smaller libraries or how you how you would approach that if Basically be writing is not an option Yes, of course. Well one trend is Things like lambda functions on a w s Many people do it many people do it successfully there is basically no framework That's just a function as long as it's simple it works. However If you are not so much afraid of leaving that project and you think that there should be another person replacing you eventually Frameworks are like like a common patterns that other developer knows If I leave a project and they tell you it's just the the jango rest end points and you can find right away It's a post handler. It's a get handler and you can just Continue working right away if you do a customized system Obviously, it's harder for new people to get into it on another hand. There is an overkill of using a framework Especially in microservices because even tornado. It's quite minimalistic. Nonetheless It has also as jango does many batteries included like csref protection and so on which you normally would not need in microservices So you're right. It's always a trade-off when you pick a framework You make things simpler for yourself and also for people who will be working on the project after you If you do it yourself, you usually can tailor it better. You maybe can do it even more efficient But there will be difficulty in maintaining it when it grows and then don't forget the testing you would perhaps test Things a bit differently when you write microservices because it's not so much about unit tests In microservices as about integration and acceptance tests You need to check the whole chain not just one thing and here testing frameworks that act predictably Is sometimes just simpler than testing custom code that Is not so predictable for others not for you for others that test their components speak into your components So, yeah, but there is no Black and white answer. Thank you. Thank you You said about error propagation Yes at the end of the presentation as for a follower said I'll discuss in his blog post about microservices. The reason is For this error propagation and synchronous communication And the problem is that when you add microservices to your system probability of going down actually increases because we with with Each microservice that can go down So the solution is asynchronous communication between microservices it and it may be http or anything else But did you try it to discuss it? What what what can you tell about about switching to asynchronous communication? If you mean just switching on it's from Synchronous http to asynchronous http. It will not change much. The protocol is the same it's just what I meant at least in my part was that the Framework the the web server behaves differently instead of blocking one worker doing one request You have the IO loop that would handle that request as soon as it's not hitting any IO boundaries And if you hit IO boundaries, which you always do in microservices because you're always waiting for either a database or cash Or another microservice It can serve other requests as well. That is what they meant with the asynchronous communication I do not see the effect of this on the error tracing Because the protocol is the same it's still http on as a client you should in the best case not even notice that it's asynchronous or it's synchronous Regarding the error tracing. Yeah, and again, I didn't mean error tracing on the client level like the http error codes like error 500 or whatever else as I meant error tracing inside the application because you Your request is passing through so many systems It's hard to To match by time for instance That's a typical first step that developers do when something fails They try to find logs on that point of time But it is going so fast that you perhaps will have hard time figuring out because it's exactly same time in many many services Many many things could have happened at the same timestamp So it's harder to do this chaotic time-based debugging and you need more centralized tool that collects all logs And that somehow is marking every request with some headers Which are then passed around from one microservice to another so that you can trace it from the entry point to the exit point What exactly happened where that is what I meant with the error handling? Sure, that's correct but I was asking more about dealing with Unavailable services for example, what do you do when you issue a request to a microservice that should be there But it's not there It's maybe it's not there for like 10 seconds and the question is Will your system be failing for this 10 seconds or will your system be resilent and be able to recover Of course, this sorry to interrupt. We will have to stop here now So I would suggest you to continue that during the the small break we have so let's fix and then once more please and then we have a break