 I'm Flores, I've been writing Python for quite a while and currently I'm working on Koby.io and it's a monitoring platform and we've kind of designed the back end to be microservices in using ZerumQ as well to communicate etc and this talk is a little bit based on that experience that we have from that and I think it's a very good way of making a resilient back end I think. Obviously, when we started this about a year ago, we've learned along the way, we didn't do everything perfect either, software is a living thing and that kind of works like that. It's a nice way and often microservices tend to nowadays use HTTP with JSON and I think it's nice to think about other ways of doing this. So just a kind of an overview of what I will be covering talking about today is I'll kind of have it in two halves. The first one is sort of I'll talk a bit about message sparsing and I think how we kind of came from building monolithic applications towards microservices and why that actually to me at least makes sense and then on the second part I'm going to discuss a little bit more about ZerumQ why it is a very and it kind of ties in a bit why it's a very nice way of using for communication. I'll then give a very quick whirlwind tour of some not really of very much details about ZerumQ but just some pointers and tips of what to look out for and then also I'll sort of discuss some patterns that we used that I think are useful for designing to use in your microservices when building this communication. There's not that much time so I don't really go in an awful lot of background detail so I assume you have sort of a high level of knowledge of why HTTP and the rest of the show works and you sort of have a vague idea of what ZerumQ is at least so you know it's sort of this socket library that sends messages around but doesn't have a central message broker so it's a process to directly talk to each other. So I think message passing is actually a quite interesting I think it is an essential kind of part when we write like large when we write large code bases etc things usually end up being in separate things kind of compartmentalizing you know modules packages etc and sometimes like in monolithic applications you end up having threading or something or multiprocessing and that sort of thing but things usually fall into into sort of smaller tasks and a common way especially like when going to threading models etc is to because the operating system the way the operating system kind of started implementing this and when we were writing big monolithic applications those were the tools we had available so a lot of shared memory are sort of happening but this observation from from Joe Armstrong is like well you know that is not how we work in the real world either it's not really if I want to if you want to communicate an idea we talk to each other we don't like open each other's brain and poke around in there directly so message passing is kind of very a very natural way of letting different tasks and different components work together and it's also it results if you've employed out in resilience if you employ that more then you actually end up in quite building resilient systems so as we go sort of so and yeah so so Joe Armstrong together with these other people kind of realized isn't there we're back in the in the late 80s I think they they were asked to to write some software as for their work and one of the and you know as any good software engineer they went well yeah that would be we'll definitely write that software but they ended up kind of writing a language first instead of actually the software but what they did is because they they designed the language around the smaller components and they they actually embraced this like let's let's make lots of little tasks and let's make them communicate with each other so everything is and everything the structure they set up was if you can decompose your programs into all the smaller tasks and make them talk talk to each other the task itself actually become quite small and a lot simpler to work on and they did the communication they did via message passing in in this they created they made an actor implementation essentially and they just became little little loops of like wait for a message then if you get a message do something and then maybe send some other messages and then when you don't just wait for the next message in person the next message and this resulted in a very in a very robust kind of language that was that was really good for for building big and scalable systems and the other important thing they did actually in there in when they were creating Erlang was you have having all these these hundreds and thousands of little tasks actually that they have if you have to sort of manage this because if you have an error you know that they may die and they took so this attitude of let it die the sort of one of the motors they have and to to to work around that is that they basically they they have these special supervisor kind of supervision tasks that look after look after the other ones and if one of the tasks size they they just restarted in particular ways etc and they to to build like the massive big applications they they end up building these supervision trees like with supervisors supervising other supervisors and then maybe a combination of actual real tasks and supervisors and that that has proven to be like a very good way of of writing software to basically try and decompose it as far and as you can and then communicate to you so in a way I kind of think of Erlang as kind of original kind of nano services way of writing programs but in the meantime like since since the 80s the the infrastructure around us has been improving as well so while you know on a big mainframe it was easier to deploy one with a big application kind of thing then you know it became easier to kind of have your own servers for your own project and to be kind of went through service oriented architectures a little bit in between you know it was already smaller still quite big to deploy but then the infrastructure around us kept improving as well we um we started getting uh cloud cloud services for for spinning up and new virtual machines etc you just call an api call you get a new virtual machine so so things kept scaling down and then and this basically all because the infrastructure around to manage these things kind of became into place so so packaging and distributing application uh to to your infrastructure just became easier and it allowed you to is allowed us to shrink um our individual components um and and this basically like with containers we we actually went we then managed to go from sort of a whole yeah a single process per per container kind of mapping um and and and that means basically you can you can now you just have in in your own application you just because all of that work is being offloaded to the tooling around around you your application can just become smaller and smaller and that's really nice so and i'm a i'm a big um i think community is a very uh like the organization systems that we got like um kubernetes is really nice kind of takes that further because um it takes that uh that that property of the supervision trees that erlang has um and and it means that our microservices don't really have to worry anymore about logging or about dying if they if they have an error you just let it die and kubernetes will sort of ensure that it gets restarted and you get the right number of your of your containers keep running um and that that allows us to shrink sort of our process and and we end up it ends up being easier to write our applications as as whole um whole processes or containers um and and the the other benefit of that is while erlang is kind of this this nano services and maybe this i don't know um you could think of it as like the ultimate goal but the other benefit is with containers we completely isolate it so we don't all have to write the same programming language we can write in in in in python you know a language that gives us um looping syntax rather than something like bizarre from the 80s that doesn't do looping syntaxes um uh so that's kind of where where where uh this allows us to get to so zero mq then uh zero mq is is a very uh it it's um it's a kind of a socket library uh so so it's very well supported on many platforms it's really stable well tested um and and but but it's um it's essentially a socket library but it gives nice properties so you you can send messages to each other and you don't have a central broker which is really nice to build that kind of distributed architectures uh and kind of one of the main benefits that i kind of um uh compared to like htp with rest request is um is you get out of it with the problem with uh if you do this htp request is you actually have a very strict pattern you do request you get a response and if you end up building lots of microservice that do need to do this um what you're calling another microservice and then you have to go and wait for the response and if the mic and and if you call services several layers deep etc you still have this very direct call graph of dependencies that all need to be there uh and and if you if you're not careful you actually have like the same thing as a as a monolith because it's essentially function calls but instead of doing it in process you now have an extra network layer that can go and fail as well which you know now let's do fail um so and and just like you can try and improve things with adding timeouts etc but um it doesn't you still have a quite restricted um uh way of organizing things so that's why i think like serum q brings kind of new possibilities in uh serum q itself it's like it's a very quick kind of um what i think is good points of serum q is kind of it keeps persistent connection so it can do really low overhead communication uh it's very good at like group grouping packages if it needs to but still keeping the the latency very low um uh it and it gives you reliable so it delivers whole messages and it gives them always in order etc it transparently reconnects as well so you don't have to worry about oh the service isn't there i'm going to have to back off a little bit and try and re-correct in three seconds and that may mean five seconds and all those sort of things so i kind of like to think of it as um like like python dev is like that one of the same as they have is like oh we read knooth so we as python programmers don't have to so they um do that and serum q is a bit like that you know they they do do the horrible nasty networking stuff talking with kernels and all the sorts of things that all the stuff that can go wrong what and and we just get this nice messaging uh syntax on top um and the other benefit is language language agnostic so you can just come in you don't care what what your mic service is written but obviously that's the same as an htp um the last thing is kind of it's got a very nice user level user level syntax so it kind of does things asynchronously so you just say i want to deliver this i want to send this message and it just returns straight away and it takes it away and then it goes and has to deal with all the kernel drivers and the buffering and the network etc and you don't have to you don't have to worry about that you just receive and send atomic messages and everything else happens sort of in the background and it's it's much easier to write this uh saying that there is um there is some uh you know not everything is uh great uh it it is a bit lower level so htp because serum q is the transport so htp is a little bit lower level you have to actually design your own um protocol uh which which you don't have to do with htp with htp you only have to worry about the the payload so you know jason uh is a common format you can still transfer jason around in this or you can use something else we tend to use message pack um but but the fact that you your communication pattern of when do i send the message when do i receive a message uh it is now your problem so you have a little bit more work to do um and the other thing is uh and and in those communication in when the protocols one of the good tricks is kind of to always uh serum q messages are kind of divide open frames and to always kind of start with a version frame so you can easily change your protocol just by by selecting one the first version kind of frame um the other kind of a little bit difficult thing sometimes in serum q is that you have no knowledge about connection so you don't know if your peer is actually connected because um on the previous slide i said you know it handles this automatic reconnection for you in the background uh it also means you don't know the other side if your peer is actually live you're just sending things um and that can be sometimes a little bit difficult so but that depends on your protocol if you if you design your protocol kind of right way around um then you don't let you really have to worry about it sometimes you end up having to do hard beating to actually know your peers there and that is not very nice at all um so it's nice to try and avoid that um so now i'm gonna uh i'm gonna go really quickly through these slides because i don't think there is enough time to go in as much depth as i would like but um just just some pointers about uh you know things to look out for for when um when when when using serum q is um when there is always queues associated so if you have if you have to um to to peers that talk to each other you create a socket on each side there's no central broker but you do get queues and they're just in memory and i decide so they're associated with the socket and they don't tend to really explain in uh in a nice uh consolidated way where um when those queues get created when when they get destroyed and that's sort of important because if there's messages in your queues when they get destroyed so in general they sort of get created when you when you're receiving connections they get created when the peer first connects to you when you're making outgoing connections they sort of get created when you're um uh when you make the connection even if there is no connection yet but it it's something to watch out for when the queues get created and get destroyed um the other thing about queues is that it gives you some sort of back pressure so if the queue is full depending on the socket type that he um using serum q which i'll get to in a second um when the queue is full either it will block or it will just silently drop your messages um that's okay your protocol just can just take that into account and that sort of gives you back pressure so if you've got too many messages you can you can decide to stop um if the other peer isn't like consuming them fast enough he can you can kind of block um uh in in python's bindings you kind of you probably want to reconfigure that to not actually really block your application and sometimes it doesn't really matter but even if it doesn't really matter if you actually really block then you have no idea because you just stuck somewhere in a c stack in a c a c call and not in in python world so it's much nicer to just let it raise an exception then you can log saying hey my queues are full um and then you can still decide to block or whatever but at least you're in python land the other thing to look out for if you're starting is like every single tutorial pretty much starts with this request reply pattern um it's not a good pattern uh not not just because it's not what i said about htp but also because there's weird internal states i'm not going to go into that much detail but they finally they realized as well and they're actually replacing this as well um by some client serviting um but that's not yet out but basically you can skip the past the request reply and dealer and router sockets are the ones that they're the they're the workhorse of officer mq they're what you want to use um very quickly so uh the the main difference between dealer and router so you can connect them to to too many pairs so you can talk to many ones the main difference is that the router can actually address a single of its pairs the downside of the the flip side of that is that it needs to wait to receive a message because only once it received the message it has the uh the the address of the pair to send send the message to uh the dealer is often used as the other side and in in that case um uh the only thing i would like to say for dealer is like if you only connect to one pair then you know who you're gonna send things to so that's often kind of used to connect a dealer to a single router socket um that's kind of a common way of doing that um pubsup is the other uh pattern that you probably want to use a lot um and it's basically just one way communication so essentially you you you're broadcasting information to lots of subscribers uh but it is a fire hose so if if if uh the the sending side will never kind of be be bothered by clients not being able to keep up um but it does mean when clients lose new clients can't keep up uh they they want because the nature of CMQ you won't really know that the client won't detect that it has missed information that it has missed data so when you design protocols for that you you want to uh you usually want to put in a counter to just put in a counter so that increments every time a message gets sent and then the the client at least can detect that hey i've missed something um and and well then it can try and decide what to do about having missed something um so um yeah so uh this is a call the service button it's kind of when this is a common kind of micro kind of kind of microservice thing that we um we've been saying I guess um and this is sort of when you have a yeah when you have a microservice that wants to manage a resource of some reason of some sort I find it really nice to to um kind of think of it try and think of it as a state and seeing it and actually write the states that this resource can be in and then draw draw the connections you know the state transitions that you can make and once you have that that state diagram you know you can you can think of it you can it simplifies things so and then you can start thinking about what how you know what messages do I want to send to this to to change from those states and then when you're when you're designing your your um when you're designing that that protocol uh the the nice thing the the thing you want to look out for is you you don't really want to send you don't want to design your protocol so that you send messages saying hey can can you take this action for me please because then you go back to the sort of request reply kind of style of things can you take this action for me no sorry I was in the wrong state or or or yeah sure it was successful um it's it's much nicer to to kind of make your message your message that your protocol that you make uh to make it kind of level based so instead you just say like hey can you get this resource for me in this state and depending on on the resource you're managing you may not even be bothered about waiting for the reply you can just go go and send that message and and walk away from it um obviously to get to get this the current state of the of the resource you will have to wait for a reply um so so you still have sometimes you do have to do reply um but that's kind of a a a nice way of making making your service is kind of more more resilient against um against all the the weird things you can get failure um the other one that I would um uh like like to point out is as uh basically use pubs up pubs up is great um and that's kind of you know consumers it really decouples like if if you can try and think of your of your problem that there's two services want to talk if you can somehow even if you only have like one one one consumer for for this publisher if you can think of it as a as a pubs up problem it becomes it becomes a lot more decoupled um and and it will become more resilient um and it also it also simplifies your over architecture because if you make everything like a peer-to-peer connection where you actually have to know about the other sides um you end up often with this um that uh that star architecture of like all dependencies relying on each other and and like this making using pubs up is really simplifies that that sort of that situation so so that's if you can at all get away with using pubs up it's usually a much nicer um a very good choice um and then lastly um I'd like to um mention this um this sort of because when I went going back to um the beginning where I said well uh you're uh the infrastructure kind of takes place so it takes over and that's why we can write these smaller smaller applications uh every um and when I was talking about actors as well every kind of service ends up doing the same thing it's just waiting for messages and then when it gets a message you start to do something so this is something that we um we kind of started uh because the main loop of every application was kind of same so it's always the same boilerplate so we started abstracting this away and we came up with this um stream abstraction where you where you get uh where you can basically just use a for loop um the for loop is the is the interesting part here that you want to look at so you get you just process messages off your for loop and then you do something with them and if there is no message to the for loop the iterator of the for loop will just block um termination and then you know becomes becomes very simple you just have this um stream dot sent term you hook that up to to to seek term or something like that and when when when you want to terminate your application send send a signal um it just comes basically in as a message but it's kind of hidden from you and the for loop just exits and and and exits cleanly the the only other thing that extra does um that this stream kind of this is is it uses um uh here it's demonstrated with a context manager and the only reason for that is that it also needs a context manager is that you also have um if you get an exception in in your message handling code or something uh it allows the the stream to sort of clean up itself because it has some 0 mq sockets to open itself internally but it's kind of it's a very nice abstraction that we kind of handled and and it's basically the core of it pretty much every microsoft is that we have um it's uh it's publicly available um currently in this act your z kit package which is a little bit old act is kind of just it's our own personal kind of boilerplate removal kind of tool it might not be applicable does that i really would like to move the z kit package outside of it because that's that's more of a a generic um usable thing um but yeah it's it's a very it's a very nice way of of right of removing the boilerplate in in in your services um and that's kind of the the this is kind of a repeat of like my my my the main thing really you don't have to like 0 mq if you don't like it um but if you if you think of your your architecture of if you think of it as passing messages rather than as a request apply um you re you you start to think of of your architecture different and and you end up building more resilient architectures um and yeah there's you know look outside of the box of htp there's you don't have to like 0 mq but there are other options um other than htp so yeah thank you very much that was uh i think we have time for one question okay thank you um that that was awesome um i just have a question regarding one slide where you said like kind of don't use htp so what's your take on something like you know uh rest api or you know web sockets uh so rest is usually very request-response based so that comes with what i you know think is what yeah watch out i mean it is it is useful sometimes um web sockets kind of definitely work around this because you get you get a stream and then you have to implement your imperfections on top of that stream so web sockets are a nice way of of of communicating as well maybe we can fit in one more okay give a big hand to uh flutis