 Okay, this is mic one check check mic one Mike, can you hear Mike one? Okay, we're checking the lab Microphone on I don't know what channel we're on but how's it sound? I'm walking around the room a bit You might hear the music too because the next speaker said he didn't have any computer audio Testing one two three. Oh look, it's working Okay, I tend to ramble on a bit So I'm going to try to start on time in hopes of actually finishing somewhere close to on time Welcome to our developer track I'm your host I've also got the first talk as I'm filling in for another speaker. So I am sorry But we had a last-minute dropout. So I didn't think it was fair to bring anyone else in on the presentation This is working better. Yeah, you guys can definitely hear this better, right? All right, we shall start so welcome again to the developer track and Might as well dive into our first talk for the day, which is my talk It's called rules for the road for distributed compute And as I talked about in the abstract distributed computing is Increasingly pervasive almost every computer system now ends up being a distributed system what with the proliferation of SAS providers and containers and Network computing and just the internet itself So just to start this off as a fun exercise and encourage you all to engage and really do feel free to interrupt me with Questions throughout the presentation. I tend to work better in that kind of a context anyway But to start things off with a bit of audience participation I've got a quiz for people and anyone from Ticketmaster here and not allowed to answer The question is What was or when was Ticketmaster's first distributed system? Basically how long have we been doing distributing computing at Ticketmaster any guesses Yeah, the back there 1987 good guess you're off, but it's a good guess anyone else Get warmer get warmer 82 over here way in the back No, so actually that's that's pretty close that that in technically that's right depending on how you look at it Actually our first distributed system was January 23rd 1977 it was for UNMs Johnson's gym a show there by The Electric Light Orchestra That's that's how far back it goes And technically speaking actually this system was deployed in 76 earlier in the year But this was the first time that someone could buy tickets at multiple different locations and at every single location The same tickets were always available at the same time. So that was actually a real distributed system So we've been doing this stuff for a while and it was actually a very basic and simple distributed structure at that time was basically a bunch of Terminals that we're talking to a mainframe But it was none the less a shipping system Things have changed a little bit since then. This is what Ticketmaster looks like today We have a lot of virtual machines virtual servers physical servers lots of stuff in AWS lots of Kubernetes systems over 300 different products over Well, it's actually 887 I think is a little bit low But that's how many components software components we counted those are distinct components all of which are serving different functions So our distributed computing environment is now significantly more complex than it was then And this leads to our problem case The unhappy developer They're staring off into the darkness and it feels like they can do nothing right they can get nothing done and Everything is breaking around them and this is this is the problem that we want to solve And this is what our goal is And I gotta tell you this picture I searched on the creative column commons for happy developer to see what would come up and This was the closest thing I found to a realistic representation of a happy developer because we all know They're not smiling and laughing because software is frustrating, right? But here's someone who's productive. She's getting stuff done. She's typing away She's getting work done and she's feel like she's accomplishing something you can see it in her face She's very purposeful and focused if we could just get to that level that would be considered a win in my book so We looked at the various problems we have and we came up with some High-level systemic goals that we wanted to work towards that would get us to this promised land of the happy developer Working in a complex distributed environment So first before we go into what these specific Targets are I thought we should talk about the various enemies that you might encounter once you have a reasonably Complex distribute system and by complex we all know a distribute system like once there's five or more moving components It's already complex, right? You don't need 15,000 virtual servers before things start to get ugly really at two It starts to get ugly, but five is when it gets complex so The first problem the sad tale of uncoordinated caching This is where you know, you've got to distribute the system things aren't performing. Well, how do you solve your problems? Oh, I know I'll just add another cache and then you add another Cache and then you add another cache and all of these different caches are all they're all caching calls out to other services And they all have their own made-up in validation rules for when that cache expires This creates interesting circumstances where it's almost impossible to glean the truth. And in fact, it's not there is no one truth For example, you might have one experience that shows the state of the world Hey, you've got tickets for sale still for this event on your website But your mobile application when you look at it says nope sold out It's because they're each of the different channels are calling different services who have different cache expires and neither one of them Actually is telling the truth about the system. It's actually a third state. That's somewhere behind all those layers of caching And if you do it properly if you screw this up really right and I've never I've never actually seen it done But you can create a state where the system never goes back to the original source It's just one service calling another Service that still has a cache record from before calling another service that is pulling in that cache record, etc Etc. You can avoid actually ever taxing your databases. It really runs fast But it doesn't actually accomplish anything And and by the way every time you screw something up now you invalidate all those caches all your systems That's your repair process right you invalidate the caches everything works great again except nothing performs now And so now you're offline because you can't handle all the traffic you get at your doorstep So the other enemy we have is inscrutable state if you can distribute your State in your system across enough microservices you can make it impossible to know What is actually going on in your system at any point in time? You'll go and look in one place and you'll find one record and then you'll have to call it to another service to find the Other piece of that record and then you have to call it to yet another service to find another piece of record And finally now you've figured out that oh, it's Chris Smith And I know his age and I know his address But by the way we've already changed the address on Chris Smith and it's totally different values somewhere else in the system So this is incredibly frustrating for people who are trying to diagnose what's going on in the system And it also really makes innovation very difficult because you spend most of your time Trying to figure out how to access all the different components in the system in order to make anything happen And then we have state wars. Oh, this this is the best part If you can get multi master replication going and you don't know what you're doing You can easily create circumstances where multiple masters all disagree about what the truth is So I my name might be Chris Smith. It might be Christopher Smith I might have you know a two at the end of the email address Just getting just not really trying to make a joke that is true Or I might not and some poor guy in Chicago might get all my emails because that's what the system thinks is actually my email address This can get very difficult when you're doing transactional work And you want to ensure that you have properly recorded transactions everywhere if the system thinks two different people actually bought the product It's not gonna go well for you and they're cascading failure failures This is actually my personal favorite this one This is if you build your systems correctly you can make it so that the smallest most minute failure can take out the whole system and Then and then you can go home and rest right everybody can not know instead You're up all night your missing meals literally This is or how I learned to lose weight during business disruptions because you can't do anything else But try to fix the system. No, you we you have a really nasty problem distributed system People always think of well distributed systems are much more reliable And that's true if you design them correctly, but let's face it if you have one computer Then the only thing that can go wrong is that one computer crashing if you have thousands of computers Then you have a thousand different possible failure points Just on like strictly from an electrical problem Let's let's not forget all of the network protocol problems you could have All of the problems in coding logic and inconsistencies and state between the different systems But just from trying to get power to keep those machines on you have a thousand more ways to fail Okay, so that's that's the other one that is terrible And yes, the the question of trying to figure out why things are slow So you know that a particular service is responding slowly, but you have no idea why it looks like everything It's calling is taking an inordinate amount of time, but you don't know where that's coming from You start chasing it down those things are calling another thing those things are calling another thing It's spread across your entire network and when you trace it all down to the bottom It turns out that all of those different services you're talking to We're all calling out to other services which are in turn calling to this one poor service that got Smacked with a thousand requests just to answer your one question, right? And don't worry we can solve this we'll just put caching in front of it, right? Then we're back to oh shoot problem one all over again And this is the insanity that that goes on in your head as you're getting your system increasingly more and more complicated so we we were experiencing pretty significant reality of this at a pretty extreme level to the point where we sort of said wait we need to take a step back and reflect on how we're going to get out of here and part of that thought process was a reality reflection of reality that it's not enough to have a Target for where you want to get to and how you want everything to be perfect in your utopic Distribute computing environment But you need to have a plan for rolling it out and a process for rolling out that is practical That gets your results reasonably quickly and that can be employed Incrementally because obviously we're not going to change the software and 15,000 virtual servers overnight, right? This will be a journey that We don't expect to finish for years so I'm going to talk about the various pieces that we thought would be very key without requiring a rewrite of all these systems and And I'm going to present them in the order that we thought would make the most sense for rolling out And we've been moving through this process slowly, but surely over the last year and a half So the first one is distributed tracing How many here people here know what distributed tracing is or think they know? All right, I should definitely clarify then for everyone else So I don't lose y'all so you can think of distributed tracing as a smart form of logging in some sense or a smart form of performance Metrics the key problem that you have if you do logging with distributed systems is you get logs that are going Into a file somewhere from all these different systems And yes, it's very clear what's happening in each individual system But you can't tell what happened when a customer clicked on a button because though that generated logs in like 20 different machines And you don't know how to correlate them all together what distributed tracing Does is it provides a framework to tag? That information about what's going on your system with a context that is tied to that initial button click so that each Step of the way there's a new context created that's tied back to that original context and the system can trace The movement and the execution of code across all the distributed components of the system and pull it together for you So you can see everything that happened because of that one click and the nice the other nice part about it is you get Timings of all the different steps in between so you can see that oh my goodness We're losing like 50 milliseconds here And we were expecting to lose one millisecond because all we were doing was talking to another process there this idea and this approach was popularized by Google putting out a white paper called it was dapper was the the white paper it was about But it's increasingly become so popular in the industry that there is now an open standard for it called open tracing And that's what we've standardized on internally as our go-forward way to do tracing And the nice part about it is you can plug in lots of different systems into it. We're currently using Yeager which was out of the folks at uber which does a great job of Letting you query in introspect all the interactions between all your services But there's also a number of commercial providers out there Some of them are actually in the vendor pavilion if you go to talk to them and they're talking about solutions for tracing Now you'll know what it's about But the idea of this is that it gives us an Ability to have the insight that we used to have when you were dealing with a non-distributed system with a single computer And you were stepping through things with a debugger or looking at log lines in the log file And you could see the entire flow through the system easily this allows you to still have that even though the systems are distributed It also provides you with very good guidelines for when it comes time to refactor because one of the most Depressing challenges in a complex distributed system is when you want to replace one component You obviously need to make sure that none of your dependency none of the people who depend on you are going to be impacted by how you change it Well, that raises the first painful and obvious question who all is using me And there's there's the list of people that you intended to use you and you always have that in your mind You know the services that should be using you but in a more complex environment with lots of teams trying to execute very quickly The reality is if there's three people that you know that are supposed to be using you there's ten other people who figured out that was really useful to take advantage of the service you were using and You're gonna break them without them even being notified and that's that creates a very difficult environment where nobody can trust Anybody to provide services so once you have tracing though you can query the tracing engine You can literally see here's everybody who talked to your system over the last month, right? So that's step one and this is actually relatively easy to deploy because a number of web frameworks and application server frameworks actually already have Built-in modules and middleware that will hook you into distributed tracing systems So you can actually like plug this into your Apache server or into your spring environment and you Instantly get at least telemetry for everything going in and out of the framework So that's good. That's a nice start step two generic query dispatch doing composable queries so One of the things that naturally comes out of distributed systems is you end up building lots of APIs so that the different components in the system can talk to each other in clean and Established ways and you want that abstraction so that you can then drop in a new version of the service without Disrupting all the people who are calling you you just have this contract that you know you have to fulfill and if you fulfill it Everybody's happy the only problem is most of these APIs end up being technical debt They're not that APIs are a bad thing. They're actually a great thing But the important thing to realize is every time you are building out an API You're actually making it more difficult and for your organization to be agile to make changes because APIs have intrinsic assumptions about them about how you're doing business each day and That's great. If that assumption turns out to be true It actually is fantastic if it turns out to be true That's why we build APIs to make things simpler for those cases of things that we know we're gonna do every single day But what you tend to see happen in organizations is API is actually get built most for when you're building something new and you know what when you're building something new You still don't know how it's really gonna work with your business Whatever you thought was a good idea when you first did it six months later is now tech debt because you need to change how it's going to work So simple solution don't have API's just have a data model, right? And use a generic query engine to navigate that data model We kind of selected GraphQL for doing it on the front-end side for all the front-end clients And it's great because we can all you have to do is know how to navigate the data model and you can effectively make up your own Prease whenever you want your own API's for your use case, and it's very optimized for your particular use case There's a smart query engine that someone spent a lot of time in trying to figure out the most Optimal way to pull the data from these services for your particular use case Which is a lot better than trying to handcraft an API Especially if you've got lots of different potential customers and trying to guess all their different use cases and build a flexible enough API for it By the way, we've done this We actually have internally a number of API's that we built that are very flexible that do serve a very broad set of consumers And you know what they all look like? Query languages they look like sequel they look like GraphQL. They're already there We just built our own handcrafted version So it's much easier to use something off the shelf and then you and you don't slow yourself down Similarly on the back end. We've been working towards using Kafka streams and stream sets and more recently ksql Provide a standard querying layer for any any data that's moving in flight through streams Which we'll get to more than a little bit. So the one catch with all this is that in a true distributed system You know, you're gonna have schema changes and for a query language to make any sense of whatever is going on You're gonna need a consistent schema So you also need to deal with the challenge of schema versioning when someone changes their part of the data model You don't want to disrupt everybody else in the system. So we've Employed a few different technologies in order to help us with that probably the most obvious one is Schema registry for that Kafka has and and Avro the serialization framework that sort of automatically handles changes in schema versioning We also picked a number of database tools and technologies that are Very comfortable with the notion of a schema evolving in distributed fashion because that's the tough part, right? Schema doesn't change instantly You roll it out in one place and then it slowly percolates through the system and you have to handle all that transition without breaking anything and Then the the other catch about this as well You know, that's nice to have like a generic query engine and I'm creating data But like that may not always perform well the data may not be Organized and set up in a way that will be efficient for my particular use case Especially if I'm doing something totally new that no one anticipated before which is the goal here, right to make everything as agile as possible and the way that we address that is because What we'll do is we'll rematerialize the data for a particular app to get its performance. So we might have Customers organized by I don't know something crazy like last name, right? But now we got people logging in with an email address We need to look up on that But we'll just create another materialized view of all of our customers that is indexed on the email address So that we have very efficient look up for that case. That's a toy example, obviously But you can imagine the more complicated use cases So next piece composable functions. Oh, this is so important and and and so counterintuitive for a lot of people, especially if they're working in Working with microservices a lot and distribute computing The most reusable piece of code you can have is a simple composable function That's more useful than a microservice. That's more useful than a restful endpoint that answers a query It's it's it's the best way to package up your business logic It's better than a framework for for executing on your business logic and the simple reason is this It's the smallest piece of work. It's the least amount of work, right? You have left all a lot of other decisions for someone else to work with And to give you an example of how this can go sideways Let's say that you Have a e-commerce site like like we do and you want to be able to compute taxes first when someone is about to Buy it by your product You could have a function for that that you could plug in anywhere you want your system that computes taxes, right? It has all the logic for it and just it's a library You link it into your code you execute it or you could have a microservice because that would be you know The typical way to do it have a microservice out there that's deployed you query it you tell it Here's what I got tell me what the taxes are But if you were to design it the way things send it working you might go Hey, there might be a lot of stuff that I'm asking you to calculate the taxes on that be a lot to send in the request so why don't I just give you a reference to the shopping cart and then you'll go to the shopping cart and Query it for what it has in it its contents and then you'll calculate the tax and then you'll hand that back to the system Now you've got this interesting game going on that makes perfect sense in the one use case of someone checking out Right you put stuff in a shopping cart you then calculate the taxes for it you show them the page you move on that's great What if the the opposite is true though? What if you're like trying to take? all of the products that you have and all of the people who are looking for potentially might be looking for Your buy your products because you're doing a marketing campaign and you're thinking that everybody in the system is going to be very Price sensitive so you want to figure out in advance how much you're going to charge them for any particular ticket? Well now you need to build a shopping cart for every person for every product that you have That's probably going to a database somewhere because they don't want to lose the state of your shopping cart in the middle of a checkout Flow and then you're going to send out a million queries out to this one poor little restful service that's probably running on one machine because it doesn't have to do much work and It's going to have to answer these million queries as fast as it possibly can so that your system can now make a decision about Who do I send the next email to? Right, so if instead that function was pulled out of a microservice and just a library that you could link to you could Ask that question as you're scanning through the data in your in doing your marketing campaign trying to build up your marketing campaign So yeah, this is a weird idea if you just make your function As a library and make it available everybody you actually get something more useful I do you want to emphasize though the importance of composable The composable function basically the golden rule of a composable function is that for the same inputs It will always produce the same outputs That means you can't be a function that goes out and checks for the database somewhere It can't be something that has an internal counter in it that does some kind of odd thing You need to reorganize your code so that your business logic is always represented in an input Therefore this output kind of state and of course that makes a lot of sense right in a distributed computing Environment the toughest problem is managing state right if you can separate that logic from your business logic you get so much more Flexibility now your business logic can change Without having to figure out how to move data around let that problem be solved separately for the particular use case involved Yes bulkheads and circuit breakers So We were talking about frailty before right Most obvious thing to do in a distributed system to ensure that Everything goes down is assume that nothing will and write your code accordingly right, but you know reality is hey might lose Might lose a connection to our database Okay, well we could solve that by reconnecting as fast as we possibly can to the database that probably go well Right, and that'll work out great until the reason why you lost the connection database as the database is being overwhelmed By people trying to connect to it So you get into a pretty nasty mess So instead you want to have something that's more practical and you want to think through the process of what actually should I do? When I have a failure in the system should I rate limit how much I'm trying to connect to the database and just accept The fact that everything is going to slow down should I Continue to function without that particular system right maybe there's a way that you can operate like Classic example is you might have a Commerce site that is doing recommendations for additional products that someone might want to buy When that system goes down when the recommendation system goes down That's no reason not to render the page that shows someone what they already bought It just means you can't make recommendations for additional things for them to buy The odds are they're going to want to buy the thing that they already got in their shopping cart Why would you take out that whole system and not let them buy just because you can't make good recommendations for additional products for them? Right, so you need to plan your product around the possibility that these various services might go off Do and and have strategies for recovering when they do come back online. That's the other one That's always important is the oh that reconnect problems really bad. I solved it. I won't reconnect Well now if there's even a slight network glitch for a moment Your system goes offline until someone reboots it because it won't try to reconnect to the database ever There's some really great frameworks in this space one of my favorite ones is called resilience for Jay That lets you sort of declaratively talk about How you speak to each of your endpoints and what the strategy is that you want to employ to deal with failure And that's really the best way To tackle it because writing the code yourself means you'll tend to introduce a lot of bugs It's much simpler to just to clarify say hey, I want to have a retry mechanism By the way, I know it needs exponential back off And by and by the way, I should never be hitting it harder than like, you know 10,000 requests a second because that'll definitely kill it. It won't be ready for that And so you can just declare that and it will manage that process for you in a way that doesn't create new bugs The name of the Library was called resilience for Jay So there's a bunch of other libraries like that for and for a wide variety of languages The thing I like about resilience for Jay is it tends to be pretty lightweight a lot of Java frameworks Have a tendency to get very heavy resilience for Jay is pretty lightweight and focus on just trying to give you resilience And giving you a way to employ different strategies for tackling And Then here's here's the one that actually in a lot of ways I'd like to move to the front of the line but from a practical standpoint I knew I had to roll out after some of these other things are happening and delivering some important wins for us And this is my favorite topic because I a lot of my work is in data science And it's stream all the things and and it literally means exactly what I'm talking about every time There is a mutation of any kind in your system You want to publish that data out into a stream? We've been using Kafka as the tool for getting that done And the rationale here and in the cut the parts of Kafka that are really important for this is that it's a Pub sub model Which means that the consumers of the data are completely disconnected from the publishers the data That means you're publishing data But nothing that the downstream consumers do can break you in fact It's not even connected to you so even if someone does something stupid and deploys like 10 million new Servers that all try to consume the data that you're publishing that might cause some stress for Kafka Although it's very resilient to that case, but your system that's publishing the information they're getting to sees no additional load You push the data out once doesn't matter if nobody's listening to it or everybody's listening to it It's the same workload for you your system doesn't get stressed out when other people make changes But the but the key thing is and this is really the opposite of a lot of the design precepts that are Commonly found in microservices where the idea was every microservice should have its own database its own data store And that should be isolated and no one else should be able to access that data store You always have to go through the microservice and in principle There's a lot of good reasons for doing that But the downside is you start breaking up your data into a little tiny piece of spread throughout your system And there's no way to consolidate at all and get an understanding of what the heck's going on, right? But if you publish the data out, so it's available for other people now you have a lot more flexibility in your system Why tomorrow want to combine the information from three different? Systems to make something new that no one anticipated before It's very easy to just connect the three different streams that those systems are producing Join together the information and then execute on it Okay, instead of having to call out to these different services possibly through interfaces that never anticipated how I wanted to use the data And and and make sense of it all without knocking any of them over So you can see another common thing here is we're we're we're moving the data away from the business logic We're moving the state away from the business logic and so you can Decouple those two problems the distributed system and your business logic And Finally, and this is the hardest one that we've only started touching on just a little bit Is anyone familiar with the call me maybe website for distributing computing? Yeah, it's some great great blog posts if you're interested in distributing a lot. I highly recommend reading those blog posts But yeah, there's there's this Need we have these composable functions now right so we can connect them all together and rearrange business logic and do whatever we want But we also need our data structures to be composable so that we no longer have to worry about what happens when there is You know some server wakes up that went to sleep or went offline or crashed And it's got a bunch of transactions that it had processed like three days ago That finally get pushed back into the main system and you know if you've done if you've got a transaction processing system I'm sure your transaction records will still have full integrity You'll know that there were ten new records or ten new purchases that happened three days ago You'll get that right, but you don't want your metrics on your data warehouse to suddenly show that you got ten new sales three days later or to suddenly Freak out about the numbers that it had from three days ago You want everything to to just arrive at the consistency that it should have been in the first place Even though the data arrived out of order Or potentially even in reverse store So this is where this notion of conflict free replicated data types come from and And the win with them in the conflict free part is very important It means that you can have multiple copies of your data throughout your system without having inconsistencies in what the truth is it's always possible to figure out what the truth should be it just depends on Whether you've got all the data or not what truth you see but there is always a true an accurate truth at a given point in time and these are There's a whole bunch of different strategies for this So this is actually from that Jepsen.io is the site where where this information about CRDT's is up There's a diagram from there that sort of talks about the taxonomy of different approaches to achieving CRDT's depending on What outcomes you're looking for for your particular system and believe me? You got a lot of choices here because you're trying to achieve consistency Which means you have to have different levels of flexibility, but probably the simplest Model for CRDT to think about is one. That's actually commonly deployed in in a couple of eventual consistency databases Which is simple rule of last writer wins, right? So whoever has the newest timestamp, that's always the truth or the last thing that happened but it's important to think about what that means in a in a more sophisticated kind of context where if I have a record that shows up with an older timestamp That is actually just updating two fields Right and the most recent update was updating two fields as well But they're two different fields. I should be able to combine those two operations, right? And it's a very easy way to solve it Which is you look at the timestamps on each individual field for each time they get updated And you just say the most recent one always wins and then it creates this interesting Context where when you look at it, it actually looks like all of your rights for atomic They either happened or they didn't in every context in reality under the covers They didn't it was just all comparing timestamps and everything just happened to work out beautifully But this ensures that you can avoid having a completely Dispart system when you're dealing with that multi master replication problem that I was talking about There's always a way to resolve those multiple masters. You don't need a lock anywhere You don't need some arbiter about the truth You've got a set of rules in place that ensure that your systems never go down and at the same time Because you don't have a master anymore if there is a break in the network or someone Destroys an entire data center because I don't know what was the thing that happened to Delta They were testing EPS took out a whole data center That's that's not a problem for you you lose that data center for the moment All your other data centers know how to function with the data that they have and when the data from that data center comes back Online whatever you can recover It just magically blends into your reality without you having to write special case logic for handling that scenario and that's I Don't know about you guys, but I've built a lot of systems with a like single master where I have to deal with about 20 different failure cases and writing all of those branches and code and Testing them and making sure that they all work correctly is Way way harder than just having a simple eventual consistency model that ensures There's no conflict and I get to a truth at the end of the tunnel I'm down with Mongo. He says yeah up with Cassandra, right up with react down with Mongo. Absolutely Mongo's the framework that teaches you the importance of CRDT's So so the fun part about that though actually is is realistically if you build your own CRDT's Right, you can actually use systems like Mongo and not have a problem when there is a failure Because you've got eventual consistency in your own data models The only problem is you had to do a little bit of work to do that It's nice to use a framework that has that built in but you can realize that you can use this stuff with like an Oracle database and it will make handling replication lag so much easier Okay So there you go. That's the last and the hardest one And and all I can say now is like you know safe driving Enjoy the distributed computing route. You've now got a system that Will be resilient to failure that you can make changes and understand the consequences of them before you roll them out and Measure them after they're rolled out you can make decisions about How you want to reorganize your business without being Frustrated by the existing structures that are limiting how you want to want to go about your business and you're into a free and open distributed world So that's my talk if anyone has any questions. I'm happy to answer them now Thank you very much. I Think the dog had a question, but we're gonna skip over him, but you had a question over there. Yeah J E P S E N dot IO. I believe yeah Yes, so the question is You know, this seemed like a pretty Carefully thought out process for rolling it out Rolling out these kinds of changes in an existing environment But if you were building out a new environment, would you do would you roll things out in this order? Was it did I get that right? Yeah, so um, no actually yeah, I would do it differently if it was a completely from fresh environment I would actually approach it differently so I Would I would definitely first of all you would try to start with all these things honestly like if you're doing it from scratch It's a lot easier to just do all the right things So I would say that distributed tracing if I knew I was building a distributed system from this get go I would make sure that's there before I even start doing any work because it just makes all the work after that Possible actually, I'm not even gonna say easier. I'm gonna say possible because it's it's that bad when you don't have it The other thing that I would be tempted to do is the streaming part. Okay, right from the get go It can seem unnecessary when you build the first component, but as soon as you have a separate component That's in a different, you know different distributed context Things start to magically become easier when you work on a stream model and there's other benefits that come from the streaming model as well it allows you to Precompute things in advance so that at the time of request you do the least amount of work to resolve the request Which allows you to handle scale a lot better So there's other side benefits that come from it that make me want to make sure that I'm doing that earlier on the process and then and then Although bulkheads and circuit breakers are super important and a really really good idea The funny part is you can actually get to those later because when you're first building your first set of components You don't have enough components that egregious failure in the one component. It's gonna take down a whole system It's it's gonna be Terrible, but it's not gonna be the worst thing right your failure rates not gonna be high But once you start having you know 10 or 20 components now your SLA is even if each component has a 99% SLA You start doing the math on that and it's like now your uptime is like 89% Right and that's not good So you can wait on those until a little bit later and that might help you get out the door a little bit faster and then the and certainly CRDT's I would try very hard to put those in right from the get-go They are so much so they're hard to get your head around if you've never worked with them before Takes a little bit of time It's helps to have frameworks and libraries that do it for you But if you can put that in from the get-go it reframes how you think about Architecting your data model and your systems in a pretty significant way And so it's good if you can do that in the get-go instead of having to like sort of refactor all of that infrastructure at a later point Does that answer your question? Yes? So the question was you know are there cases where graphql What what are the cases where graphql makes sense does it make sense in the back end? and and also if I heard you correctly like a Lot of API's seem like they make pretty pretty decent sense by themselves And they're really independent and you don't see the benefit of gluing them all together through like graphql or query engine So first of all, I don't highly recommend it for back end a lot of back-end stuff It's possible to do and it works But there is a lot of overhead in translating everything into JSON and going through all the resolvers That you might might not be ideal a lot of back-end systems It's really important to be extremely efficient And that can be problematic And also a lot of the goodies that are in graphql are really designed to help you when you're going over a WAN as opposed to a nor traditional back-end environment where it's a LAN, okay? And so yeah, I wouldn't say that it's for everything on the other hand when you're building all those API's that are doing different things you tend to find that a whole bunch of them are like crud operations and And guess what? You know if you just build a data model use use something like graphql and really it's not about graphql specifically, it's about the idea of using a generic query engine You suddenly find that you're not Having to build a whole bunch of API's that are really just boilerplate. They're really not adding any value But they are creating risk right they're all ways that you can screw things up and also ways You can make it more difficult to change how you do things so Classic example is you might build a retrieve service that returns back this nice big object, right? That has all the information about I don't know some order that was put through your system, right? But there might be a use case where all you really want is to get the total amount of money spent from all the orders in The system and if you use that API to get that information back now You're pulling back a huge amount of data creating a huge amount of load on both systems Just to get what should be a very simple query answer and by the way sequel is seriously a legitimate solution for that There are sequel engines that can translate onto data models in multiple different systems You just create that way we chose graphql in particular because It fit well with our strategy with react on a lot of our front-end pieces And it was very it's very beautiful in that use case in that use case. I would really go strongly for that instead of Yeah, it's really great for that and and the nice the other nice thing about graphql is it's not purely a query based model It also has subscriptions in it so you can do the push based model So it can fit in well with that streaming part of the equation where you can say, okay Now I want to know the moment that a new product comes out in the marketplace that I know to do a whole bunch of Things in reaction to that but yeah, no, that's that's why we're not using it all in the back end We have things like ksql and in in stream sets and a few other tools that allows us allows it to generically work through the data You had a question back there so the question is how do you approach the Question of functional testing in a complex environment where you've got all these different pieces moving together So the short answer on that is very carefully No, so so if you take that approach of composable functions it actually helps a lot because you can test composable functions very easily independently and Also in aggregate together because that's the whole point that can be composed together And so you can test what happens when you put them all together in a chain that works very nicely Where it gets complicated is composable functions the whole thing is you don't have the state piece of the picture involved, right? And in reality all these systems there is state that needs to be managed and that makes testing more difficult, right? so The most important thing is to have a good framework for generating a state actually in the system consistently. I'm a big fan of Essentially using a declarative model to populate your system So you'll say something like I want to create ten orders that look like this with three different Users attached to them and I want special case where one of those orders we detected fraud, right? and then have the test framework generate essentially random data that is In compliance with that declarative statement, but otherwise completely unpredictable and and exercising all the corner cases That could exist your data you shove that into the system And then you look to see when I create the system now does it reflect the state I would expect it to be in one all of those things have just happened And it's really important to build a robust Integration testing framework that works in that fashion instead of the more traditional model of individual imperative Integration tests that are like just one particular functional piece of the system. They just don't catch enough of the problems Does that make sense? Did I do it by person-by-person basis? Is that what you said? Right. What happens when one service depends on ten others? So in an ideal world and if you're doing it the right way those ten other services should be Able to be spun up with basically a single button push right in a test environment And so the right model is actually yes when you're building your tests it should spin up those ten different services that you're talking to in your test environment and Run the test against the real services not some mock Because most of your integration challenges come from making bad assumptions about how those ten services behave and Getting a surprising outcome in the real world And so you want to test against how those systems actually behave And then you shut them down after the test of right, right? So that's the real idea. That's the ideal world We don't live in that ideal world And so what we end up doing is we have Qa environments and pre-prod environments where we have some guarantees about the Approximateness of those environments to production reality And those services are available for anyone to do testing against at any time I will say that we do have challenges sometimes with Multiple people independently wanting to run tests and plot pull out different pieces And that's where it's so much better if you can just spin them up on an as-needed basis for your dependencies Makes sense Sorry, there was a question over here. Yes Yes schema versioning. Yes So there's a couple of other so the question was what else to be used to manage schema versioning and the evolution of schemas So it's a good question. We're actually in the process of reviewing the tooling that we use for that and enhancing it We've been using a Couple of in-house tools along with some open-source solutions to get us to a good place there We're using a product called a secan ckan that Is a sort of data dictionary of sorts that provides us with insight about all the schemas that are deployed in our environment although We're still getting all of them integrated into it. So not all of them, but in principle a bunch of them And the idea is when a deploy happens it actually Part of the deploy process is to publish a message to a special Kafka topic that tells the data dictionary Here's the new version of the schema and push that Into a central registry where we can capture not just the fact that there's a new version But that there was a transition and what the previous version was you can go back through that history and manage it there's a couple of other tricks that we try to employ on the on the data science side like Representing everything as a feature vector so that we're sort of new fields just show up and it's not a problem but But there's a lot of sophisticated tools out there in the marketplace That do about half of the problem and the challenge is cobbling them all together To make something that makes sense in aggregate and I haven't seen a lot of good products that actually Handle all of it all at once seed you have to invest a bit of time and effort into it So case in point Cassandra which we're using in a couple of cases it actually handles schema versioning very well in distributed context You can make a version change online it ripples through all the clusters all the data centers that you have Cassandra deployed in a very reasonable fashion that doesn't break anything during the transition In between that's great. You've solved the problem for Cassandra But that's absolutely no help for anybody who was using some kind of tool to that was expecting a specific data structure in Cassandra at any particular point And say it was like pulling out all the records from a table and putting them into a spreadsheet somewhere, right? The tool is gonna be like wait You've now got another column that I didn't even know existed a few moments ago And could quite well break if it wasn't built to handle the reality of schema versioning But the biggest trick is tack on a schema version on every payload that you have moving around Because at least that gives you the possibility later on of figuring out that there's a different version and how to resolve it That's basically what the schema registry does right? It just tags everything with a schema version and then keeps a history of all the possible permutations Yes, another question there. Heather's smiling at me there at the back so We've gone through a number of different iterations on this Been around for 40 years and we've we've Attempted the attention organizations to try to impose governance and then recognize once you impose too much governance That you're creating too much friction and that no one can move forward very efficiently So then you compensate and go the other way and now you have Jungle of different technologies out there that you got on tangle and then you compensate for that by going back to the Crazy levels of governance and so forth and so forth So we've been through a few iterations of that I would say right now. We're probably closer to the jungle than we are to the tight governance model but we are taking an approach now that's fairly pragmatic of sort of recommending and guiding engineering teams as they go to solve a particular problem and encouraging them to sort of consult with Set centers of excellence that that can guide them on making the right decisions In the right places and one of the key things that we've been trying to do is ensure that there's someone who's the title that that we use as an architect involved in that conversation because There can be this dangerous tendency to think that you know what kind of problem you have Go to the expert at solving that kind of problem and then they solve it that kind of way and it's actually the wrong problem It's that they'll joke about like you know Don't go to a surgeon for asking for treatment about something you have unless you already know it requires surgery to handle it because the Surgeon will always tell you you need surgery for this, right? So actually I've seen this problem in a lot of different companies with Dba teams Dba teams are always the experts about databases and data People Developers love them because they can go to them with almost any problem and if it involves data magically it gets solved for them The problem is is that not all data problems in distributed environment are database problems some of them are distributed computing problems and if you go to a database team to address the distributed computing problem you Actually end up with a solution that doesn't really work because you haven't addressed the real problem the database will work great But the actual application will not And so that's where we try to have these Architecture experts who can help guide you maybe not to how to solve the problem But what kind of problem you have and who would be the expert to talk to to actually figure out how to solve that particular problem Does that make sense? Any other questions like we're done. All right, cool. Thank you all very much come back for the next talk we've got The next talks on zero MQ and we've got a graphical talk later if you want to get a little bit more feedback on that Thank you very much. Oh Okay, there we go. We have Welcome you can be my guinea pig while I make sure everything works Is it too loud when I when I got it turned on I suddenly go whoa the voice of God is coming down All right, good. It'll probably quiet down a little bit when there are people in here But I would hate to blow to blow people out especially if I you know get excited and Talking louder. All right, so that means we're basically ready to go. Oh Yeah, there we go So if you're inclined to play the home game you've got tons actually have more time than anybody else But it's entirely up to you. It should be self-contained without it. So yes Excellent is my pointer visible. Maybe is that visible to you if I use that? Okay All right. Well, then maybe I will I think I'm ready to go Looks to be working well. In fact, I now have the voice of God This thing is kind of loud. I think it'll be all right once people get in here, but I turned I was like So It looks like all is well though, okay, great. Thank you. What's that? Well, they might have to work at it It's you notice I'm gonna actually do it in there or at least I'll take it off at the last minute It doesn't it's not great to give a presentation in a coat But man the speaker's room is at least this cold and here I was hacking all morning polishing things and And I needed hot chocolate. Welcome to the North Pole I feel compelled to not give a talk in a jacket, but the rest of you should probably find your park is because it's quite I Guess the machine. I guess the heart the electronics will run cool But so I appreciate if you make it through the entire talk. I will appreciate your amazing dedication or you'll frozen to the chair I don't know which Yeah, I think it would be good to take the edge off that If it's any consolation I spent all morning polishing thing in the speaker's room, which is easily as cold So we have the speakers on ice today If you didn't notice there is a home game if you want to play along the slides that I'll be giving because I just pushed so I promise they're actually up to date and the example code is in the repo there and And for bonus points in the home game Tell me what my t-shirt is actually about There's a puzzle But afterwards Because I'm gonna try to try to stay on time. I think we have about two minutes Okay, great. Thank you although If we get much lower they'll freeze to their chairs and can't leave in the middle even if they're bored and I mean You know, it's it's win-win, right? for me But I have to give a more animated talk so I stay functional. All right. Well, it looks like basically it's time to start The there is a home game if you just came in there's a URL there if you feel like Cloning the repo and playing along feel free the slides are there to people I took me a while to realize that the best thing Do with the slides is to stick them in the repo so that you can pull them rather than me pushing them. So All right, let me start by time right here. Okay, so we're gonna talk about zero in queue Basically, what we're gonna do is we're just going to connect all the things. Okay So first of all the name is rather confusing It seems like it really should be a message sir a message queue or something Because after all the name is just like rabbit in queue clearly. It must be related. Yeah, it's not really I mean, I suppose you could use it to write such a thing, but it's not in any way a server Nor is it even a centralized message broker Even though I'm Maddox is also, you know heavily Or was heavily involved with a MQP and It's not actually for existing protocols. So you don't use zero MQ to say write your HTTP server You can bridge messages, but that's not really I mean, that's not what it's designed to do Okay, so what is it now? For those of you that are young enough that you grew up with social media I have tried to distill the description down quickly enough to match the modern attention span There we go Now for the rest of us There are a number of definitions floating around a reusable messaging layer is actually pretty concise It is asynchronous unless you choose for it not to be And it can look like a concurrency framework. That's kind of a tall order for it for something that's not totally specialized I'll play with that for a minute Now the documentation part of the reason for this talk Or at least what kind of informs my thinking is there's a guide that seems to be the Bible the guide is a pretty good Tutorial it's very thorough. It's a terrible reference Okay, because it's written as a tutorial and the author writes in a very friendly style And so you can't find them when you're looking you find them as you get to them also the example code is pretty Variable I honestly kind of despise the C++ the way they wrote it, but what I want to do then I Want to basically illustrate just how flexible and easy this is to use and especially get over a couple of conceptual Humps that I think would have made the guide a lot easier for me to read if I had had them I'm going to mainly focus on the socket types because that seems to be kind of the core of your thinking and maybe we'll look just briefly at the threading performance since they actually talk you can use it just for pure threads and Basically, it's built the structure. It's built around three key objects. Okay, so you have messages you have contact you have a context that you'll usually only have one and You have things they call sockets which look like Berkeley sockets a bit They've met the interface looks like it and that's only to confuse you it's sort of vaguely true if you squint but they're much different much higher level and It's so don't expect the semantics to necessarily be the same. Okay now We're gonna start with a simple warm-up exercise What we're gonna do is we're just going to have two threads talking over What amounts to a channel something like you'd have in go or rust? Basically because it's just really really really simple and they're actually gonna be synchronous just because again I'm really only illustrating how you get the library up and running So there's no need to actually write asynchronous code So first of all if you're playing the home game In the repo I should have it on the top of every slide for people who come in late. It never occurred to me That would have been a great idea Basically, there's a very stripped-down code in echo pair dot cc Now the funny thing about this is or at least when you look at it All you really need to know is the main function is going to function as a client meaning it initiates requests Okay, and it creates a server thread to respond to those requests. Okay. Now. How does this work? Well, this is the way everything works in zero MQ. You're always gonna start off. You're gonna create a single a context It's almost always going to be a single one for an it for each process And it's the only thread safe thing in the library You're not supposed to need thread safety for anything else because if you're passing them between threads You're doing it wrong, but the context has to be shared. So it is thread safe and re-entrant and all that stuff Basically after you have a context you use that to create sockets There are at least 11 different kinds of sockets and you have to specify a transport and What you share is as I said not the sockets you share the endpoints and then each thread connects to the endpoint It needs to talk anywhere else. So this is what this looks like in code. Can you see my pointer back there? Is this useful? You can see it. All right. Good. Okay. So again, this is echo pair dot c CC so everything they night in the C plus plus interface they put everything nicely in the Zmq Name space. So here's a context. I tell it how many IO threads. I want normally one They say you probably one per gigabyte per second of data. So one will be fine for us now You actually don't need any for this program because we're not using the network and so in fact it's direct, but Normally this is almost always be one. Okay, so then what do I do? Well, I create a socket with the context and I have to tell it what type it is So this case, it's a pair which is the least used socket, but it's actually the easiest to understand initially I need a an endpoint and the funny thing is even though this has nothing to do with the network How I specify the endpoint looks like a URL Why because it has a single unified interface in proc just means in process. It means it's inter thread only Okay, so we have to bind. Okay. That should feel familiar And then I create a server thread here You I'm not going to show all the code in anything because all I want to show you is the parts that matter And I'm passing it the context and the endpoint it should connect it. Okay now Here's the funny thing Semantically main is going to be the clients can initiate requests. Why did it bind? Okay, so this is the first place you realize. Oh, these really aren't Berkeley sockets They're only trying to confuse me First of all it actually doesn't matter near as much if you like using sockets This matters to you a great deal because you can't connect to something that's not there But in zero MQ you can okay because they just in the background remember we gave it an IO thread So one of the things it does is in the background if it just automatically reconnects if it's disconnected If it's not there yet, it'll keep looking and in the meantime just queue up your your messages So you can do almost everything in either order and no one cares Which is really really nice when you have a whole bunch of processes that are coming and going for any number of reasons And that's where we start to see the ease of use So the usual recommendation is the usual rule is it kind of looks like sockets But it's more of a of a principle than a binding rule You bind for things that stay in one place that have well known Sockets well know or well known ports well known addresses and you connect if you're the sort of if you're a client or something That can come and go The only difference is in proc is different in proc bind has to happen first and that's because It bind creates the message queues that they that they talk with okay, so pair sockets are also kind of special They're not like the other sockets in the system. They're they're They're almost like Berkeley sockets You can only have one on each end meaning they act like a Berkeley socket Yeah, they have some handy properties that make them basically a channel basically a message queue And really their only real purpose is to do exactly what we're doing here Okay, so what did we do in the child thread? Well, obviously we pass the we get a contacts We create our own socket we bind to the same end point Another rule in zero and cute not all sockets can connect to all other sockets The types have to be compatible pair sockets only match other pair sockets. That's as I said, they're kind of unusual So here is what the the thread looks like. Okay, so here we got we received the context I hope it's obvious if it's not obvious to you why it's a reference. You're probably not a C++ programmer and it's okay Here's the so what do we do? We just create a socket we connect and we're good to go. Okay looks kind of familiar There's a just I had to do a little slight Synchronization since I wanted to time this I didn't have to otherwise it'll wait until everything's up, but I Send a blank frame so that the in other words the the supposed server actually initiates a Frame so that the server knows okay now start your clock and start going okay So basically all we're going to do is we're going to sit here. We're going to create a message We're going to run a bunch send a bunch of messages Actually, sorry, this is this this is the the server through it's going to receive messages going to send it back And that's all and I'm letting it block. This is synchronous Which means you know it has its limitations But again, I really just want you to see how we how everything works every program is going to do basically the same thing Okay So here's what the server is doing obviously it sends first it sends, you know something important, right? And then it receives the answers Okay, so the real points are you've just seen the basic usage you see it's very simple It almost looks like sockets, but if anything it will just work where Berkeley sockets would punish you if you didn't do it right the by the way the s receive an s send functions are convenience functions. I just there's a Z helpers file in the guide You can download all the example code for the guide And it's also a nice way to see how to sort of make it easier to use the message objects aren't as nice as just using strengths and You use you often don't need them so now performance now This is a quite an old laptop, but still 80k message is a second not not super fast Okay, so just for fun and this is the only timing I'm going to do I wanted to see how it would compare with doing this with something that specialized for inter thread communication Okay, so first of all I wrote basically the world's most bare bones Q protected with a mutex lock and I got almost double the speed Okay, so all right. It was kind of slow But then if you do it right and you use somebody's Well-written cue I got about 25 times faster. Okay, so you might say well wait a minute This is supposed to you. I thought you liked this library. Okay. Well first of all I the timing is Probably only vaguely close but a factor of 25 should stand out from the noise and actually all the implementations there They're almost the same code. They're nearly the same length except writing a cue. So why would you use your own cue? Well, basically I set you up while I was just showing you how it works. I set you up That's probably that's a secondary job for zero MQ and it's probably the hardest one for them to do very well because Everything fits together. So everything has the same interface. Everything just works So I was really playing to its weak weaknesses. Okay, it's not actually intended for just using it It's not MPi and open MP are safe. We will not be running this on our supercomputers simulating the weather, right? Okay, but just think about that code for a minute. What if we wanted to have multiple clients? What if we wanted multiple workers? What if we wanted them running on different machines? Different processes. What if we wanted to do some pub sub now all of those would suddenly bloat the code up Enormously, okay, and would be not fun and it wouldn't reconnect when everything goes down Whereas as you'll see it's actually quite easy to do here. Okay So that's the problem with specialty tools and that's really what zero MQ is really designed to do Okay, to avoid all the code to do da da da da da da da. All right So let's look at something that's kind of real. Okay, so this is going to be a very simple load balancing server This is mostly in the guide. Although I've done some different things with it, but What I really want to show you so it's in load balancing server dot CC it is just what it says on the tin, okay? and What do I want? I want to have any number of clients. I want any number of work workers And I want load balancing that will actually accommodate slow and fast Clients and workers so I can't do round robin, right? Okay, so The clients are actually even simpler because they don't need to send ready because I'm not profiling it and and if you look at The example code I kind of print some things out so you can tell it's actually working Which means that the threads are slowed down to the speed of The console right so we obviously don't care about that yet Why am I okay? So the clients are all going to connect why because they're the ephemal pieces the server is the has the well-known the well-known address The only real difference is now the clients are using what's called a request socket Rather than a pair socket. Okay, so here's what it looks like remember. This is always the same context socket type Connect. Oh and those of you that are not c++ programmers. This is pronounced forever That is my prefer you can do while one if you really want to but that stands out and you go What in the heck is that and then you realize oh, I've got a I've got a potentially infinite loop here I hope that's what I wanted. All right, so what do I do? It's pretty simple really I can send things I can receive things and Let's see. Oh That's right. I actually gave you a Python example. So that's another thing, right? It's very easy to do things in different languages because the bindings all work similarly So let's do the same thing in Python and this is in load balancing client.py. Okay, I'm going to create a context The syntax is different, but it's exactly the same thing now. I'm going to create a request socket I'm going to connect it the endpoint is all the same and then since I can't do forever It's while true. I'm going to send something I'm going to receive something the only tricky part here is that you do need to send binary data, okay? Because it expects binary data and not utf8 Okay, so what about request sockets, okay? They're the simplest way to do To do excuse me to do request reply They're synchronous So you actually have to call them there like a state machine that flops back and forth, right? Request has to go first Then it has to receive something before it can send again and so on That makes them kind of fast and simple to use you don't have to do things Synchronously, but if you want to these things are very simple, okay? The server however isn't going to care So the server will be is going to be asynchronous and it will happily support asynchronous clients I just didn't write one for you And they use a different socket they use something in this case for a request the equivalent is something called a dealer socket Which has extra bells and whistles and comes with a five thousand five year or fifty thousand mile warranty Unlike the request socket, okay? So almost every socket type in zero and q comes in pairs Why are they in pairs because each one of them implements a specific pattern and the patterns are asymmetric Except for pair so ironically pairs the only one that doesn't come in pairs, right? so if you want a Request reply pattern, then you've got rec and rep for synchronous Work and dealer and router for sort of the full Monty, okay? You've got pub sub for pub sub you've got x pub and x sub which you only use if you're relaying really And you've got for you've got push-and-pull sockets for making pipelines right data splits out gather it in So these are really all the baked-in patterns Okay, and you'll see in a minute. You can do odd things with them, which is part of why they're useful But they're specialized right so these are bidirectional they obviously have to be and Router for example knows how to get messages back to the origin pub sub push-pull and obviously unit Unilateral or sorry unidirectional Pub sub has some kind has to have some kind of routing logic push-pull does not so again What you're going to do with zero IQ is you're going to think through the patterns and figure out all right Which Lego bricks do I assemble for what I actually want to do? Okay, and we'll see a little bit of that Okay, so what's the server going to do? Well the server can't use a reply socket because it better not be synchronous And it wants to connect to a lot of clients So it has a router socket, okay, and the great and now one thing that yes, it's coming up I'll show you well. This is where we start to see why this is a whole lot easier than doing this the hard way Okay, first of all, I don't have to select and pull even though. I'm going to connect to multiple endpoints Okay, I don't have to do except this just happens somewhere on a magically I Don't have to do different checks for things that don't have a file descriptor it just works Okay, so how does that look well alright, so of course I create a socket with a context I give it a type. What am I doing here? I'm just bind bind bind bind. I can just keep doing this I can bind to whatever I like Which is again, it's not really Berkeley sockets. It's just much nicer. Okay, and I'm binding to an in proc Endpoint so there isn't any There isn't any file descriptor just to pull on but it doesn't matter It's going to act as though there is it's all just going to work I this is IPC the way they implement IPC as this is a actually a unique socket and practice So the guide says it doesn't work a Unix domain socket So the guide says it doesn't work on Windows, but it's I believe Windows 10 might now have Unix doing does so this may well work But the guide says no because they wrote it back in the Windows whatever days single digits of some kind and then of course This is the real thing. I'm going to bind it to local host, but you can bind it to whatever you like Okay, so how do we route things? And this is where you start to see kind of the logic of using this for less trivial problems. Okay So, you know, we're doing all right. I'm okay So basically what a router socket does is it wraps whatever you send it? Okay, so data comes in the review send a request data goes over it hits the The router socket on the other end and it wraps it in an envelope and the envelope knows the identity of who sent it Okay, so what you receive out of it is Not the original message, but the original message in an envelope that says this is this is where the reply is going and When you send outgoing messages on a router socket, it expects that envelope to be there And it will strip the envelope off and send it to the right one Because from it's secretly keeping a thread pool So it knows it knows where they went probably by which of the sockets in the pool came from but it will magically get there Okay, but you have to make sure if you do anything odd as we are going to do You will have to make sure that you give it the envelope. Okay, so This is where to know how we do envelopes. We have to know something about Zero MQ messages. All you really need to know is that they're Sequences of frames. I'm not crazy about that language because frames can mean lots of other things but what they mean is It is a sequence of chunks of binary data. They're just they're just binary strings okay, and That is all zero MQ has to know about messages, but it's enough to implement Envelope letter because all you have to do is just prepend extra frames with the routing data And they often like to use empty frames as delimiters, and I think they're like virtually free I think there is zero bite in the stream to be honest with you and Then you can manipulate the frames and make it do do what you want You could send it to a different a different origin if you were to put a different ID there because it is Transparent to you when you're working at this level. So this is how the routing works, right? Here's a request socket in a client. It has a client ID associated with it It sends a request when it hits the router socket what comes out is actually two more frames There's the client ID, and there's an empty frame delimiter, and then here's the original request So when it goes back, we have to make sure that this stuff is there Okay, and then here's a reply hits the router this gets stripped off the reply goes back to the same client ID Okay, so if you have that in your mind, you're halfway to kind of seeing what it feels like to work with this, okay? So now that was really easy. Okay, so how are we going to solve the load balancing problem? That's a little harder So what you would expect from what I've said if you've been listening you'd expect that the server is Going to send the requests onto the workers with a dealer socket, right? Because if a router is the equivalent of reply But with you know extra added features then the dealer must be the request and that's true But the problem is the dealer will just sit there and round rob it And that's not really what we want if we have workers that are you know sitting on another machine across the network and workers That are in threads so what we actually want the workers to do is to tell us every time they're ready Okay, now what's the problem with that? The problem with that is that means the workers have to send first okay So they can't use reply sockets because reply sockets can't send until they've gotten a request can they? So in reality the way to think about this is the workers are actually going to be making requests to the server for a job and their reply is Going to actually be Sorry, they're gonna make a request the reply that comes to them is going to be a new job so What the worker regard what is sort of operationally functionally a reply to the worker is the client's request? Okay, see how the language is getting tricky. This is why I said I wanted to get you over some humps The guide will make more sense if you already know this can happen, okay? We don't need an extra ready message because when it sends The the data is so what happens it said hey, I need some data come it gets it gets that So it got a reply from the server. So what can it do next it has to send another request so its request Functionally is going to take the reply and that's also going to tell the server now. I'm ready to go Okay, I do have a diagram coming which will make this a little clearer But the point is that the workers even though you would think that they are replying semantically They are going to use a request socket So we've inverted the sockets meaning in a sense and I'm gonna just say something about that There's a lesson for this is the socket types are named you would think they're semantic patterns Oh, I have a I need to send a reply. I will use a reply socket. That's not the case. They're actually What I want to say Where do I say this the point is that there's a difference between functionally and semantically I don't like this language I don't think it's that clear, but I couldn't find a better one what you do is you actually just think to yourself Functionally, what am I doing? Am I sending a message first? I'm a request I don't care what it means to some other part of the program if I'm sending if I'm only responding I'm a I'm a reply and so on so again the thing I want to keep clear is To use these patterns even though the type and the names kind of make you think that this is the semantic meaning It's really just the pattern of their behavior and you really just pick the behavior you want whether or not it means to you What it sounds like, okay? so The implementation is slightly hairy so in a moment We're going to see the diagram, but I'm just going to say what the diagram says so that you'll hopefully the diagram will make sense quicker So the server has to use a router again because remember the workers have a request socket So I need to reply to them plus I need to know who it is. That's ready. Don't I so actually router is perfect So what am I going to do? Well, I'm going to get the ID of the worker that just told me he's ready so I can store that I'm going to put it in a queue, okay and What I can do is Is is once I have something in the queue a worker that says he's ready When I get a request from the client, then I'll just pop it out of the queue and I'll stick the worker ID Remember I can wrap it up. I can send it to whoever I want Okay, and the router socket will strip that envelope off and then it will send it to that guy It doesn't matter that he's told me earlier because the router socket just going to do what router socket going to do, right? Okay And of course I'm going to keep the original client ID because I'm going to persist that because eventually I've got to get it back Okay, now the other thing is now. I do have two sockets. I have a front end the back end socket Okay, so even though I don't have to pull for you know a hundred clients on one socket I do have to pull for the two sockets, okay, and you can do that you can still pull You just don't need to pull on nearly as many things because so many of them can just attach on one Zero MQ socket, right? Okay, so here we go. This is the dreaded the dreaded data flow diagram So what's happening here? I get a request right what comes out of the of the router socket as we saw from the front end is there's a request But it's wrapped in this envelope with client ID and the delimiter So what I'm going to do is I'm going to pull a worker ID a ready worker ID out of my queue And I'm going to put another envelope so I'm wrapped put you know you guys all know the envelope Letter pattern and that's what's that is what's so great They gave you just enough structure to the message to do the most all the most important thing Everything else you use your own serialization libraries and anything you like this is all that they needed and that's Kind of the ethos of this library give you just enough to reuse without too much Okay, so now I've got something that has two envelopes and that's great because the router is good The back-end router is now going to strip off the first one. So I'm back to this aren't I but I'm back to this going to the Correct worker, okay, what's the worker going to do? Well, we still need to know what this is So he's just going to replace the request with the reply Send it back, right? So this gets to the router of course What happens well the router now wraps it up again because he always does so now we know who it was that just sent us the Reply, okay, and that's great because first of all I need the outer envelope to stick back in the queue So I know that he's ready for another job, right and then the rest of it is exactly what I need See how I have I now have the original envelope here But with the reply in it and now that the router the the other side the front-end router will now route that correctly Okay, so it's a little bit confusing But the fact is that's tremendously powerful because it's a whole lot easier than writing the routing code for yourself, isn't it? Okay, so how does that work? Well, so here's what a worker looks like that. This is the easier part What's he gonna do? Okay, this receive multi part here I did this is I just use the Python code for this one This is so you get all the frames at once instead of sucking it off frame by frame Because that's really easy. All I have to do is replace The final frame right I'm just gonna assign it here in this in the list or the yeah Python calls them list there are raised but they call them list and then I'm just gonna send it right back Because I'm done so the workers again are really simple and that's good because you only have to write the server once Right, but you may be writing a lot of clients and a lot of workers So the the complexity is in the right place just naturally. Okay, so what's the lesson here? Well, the point is Choose the right functional behaviors. We don't care what they say semantically we want to know what the behaviors are right and It's easy to just slice and dice mix and match shuffle together deal apart all of the message frames however, you like it and Just keep use frames for the pieces you need to slice and dice and everything is really simple. Okay now that is Is this time right? Okay. Yeah, we're all right. Okay now First of all, I do have some time here. So Let me go back to the big confusing diagram as I'm doing well on time Do we have any questions about the or any on anything really but but this is where I suspect questions may come up Is this making sense? Yeah Okay, so the question is is this something this pattern something I invented. It's not something I invented It is that I adapted something out of the guide because I thought this was an extremely nice way to illustrate sort of How you actually use the tools in practice. So there is a load balancing server there that I have kind of adapted But I think it's quite representative of Kind of the ethos in practice You don't worry about what the message frames look like you pick the sockets for the right behavior And then you just slice and dice the message frames until they do what you want If you're just relaying they even get you could it's trivially easy to pass them back and forth They even give you a function for that so you can write Proxies in a couple of lines, but normally you'll probably do something like this Anything else? Okay So here's our worker Okay now When I was doing this so believe it or not So the goal of this talk was to illustrate how easy and quick it is to get your messages where you need them to go And I got this fairly well polished up Believe it or not last night and I said you know what I need I want to show how quickly we can prototype things because that one was a little hairy So I actually kitted this up That quickly and that's kind of my point. It was just trivially for me to say Ah, let's just prototype something that has some interesting behavior. That's really easy. So Let's add a log server to this thing Okay Now what do I want? Okay? Well, first of all, I really just want to show you the other two major suck kinds of sockets Okay, so we have push Paul and pub sub How quickly we can just throw messages wherever we want them So there is and I did this in Python because we're prototyping. Why isn't that what slow languages are for? I'm only saying that to tease the Python programmers. Okay So I like Python, but yes, it's a slow language. I'm sorry So the code there's a log server and there's a log watcher So what we're gonna do is we're we're going to actually make it so the log server just routes the log messages and anybody You know anywhere you want you need to listen in you can do that and listen in on the ones you want Okay, now I will say this I only put the logging in on the Python worker and client and the reason was that if I log From the the threads which are also can run there at the same time It slows my C++ threads to the speed of Python and I just found that unbearable But that's a real I mean that that's not the fault of Zero impute that's the actual problem threads go much faster than the network, right? So you're going to have to figure out how to solve that and any in the library can't help you with that It could just put the messages where you want to go You have to figure out how to get them how to get threads to get their logging done without being unduly slow Okay, but nonetheless other than that and just my odd Dislike of seeing my C++ chug along at the speed of Python So how do we want the routing to do? Well, it's really simple But what I wanted to show you is the fact that why would I use why do I have different pairs of? Sockets here. Okay. Well what push and pull are good at is Sending out lots of chunks of data out to be run by you know many Many workers in parallel bringing and collating them back together. That's what they do Okay, so they're unidirectional you can't send anything back the other way because that's not what that pattern does So that's why I'm using this okay You might say well wouldn't pub sub be nice and the answer is no Because then every single pub sub is also unidirectional every single worker would be a publisher and The log server would have to subscribe to them, which is Pizarre, I mean you could do it but didn't make sense to me So instead the log server just sits there it binds to a pull socket and then all of the All of the anything that logs just connects to a push connects with a push socket to the same endpoint again We see that I've got there. There's there's probably a socket For every connection in this pool, but I don't care because the pool is hidden nicely away in the context and I never have to do another pull or Anything just for that reason, but on the back end I actually do want pub sub and why is that because I want anybody to be able to Listen, I don't want to have to know or care what they're doing with it I want them to be able to just pick the messages they want and If they're not there You know, I still want the logger to run so that is pub sub isn't it okay? so Let's take a look at how this looks in Python. So well, okay, so this is in the client now that I wrote in Python So he creates a second socket. It's a push socket, right? So what am I gonna do? Well This is the tricky. This is one little bit another example of why frames are really really nice Oh, I did what I didn't actually mention is that Even if you're not using the envelope wrapper stuff, it is incredibly nice to have sock reliable In-order sockets that are framed for you Okay, if you've ever sat there and had to do your own framing and you go wait. Do I have enough data? No I'm in the middle of a frame Zero MQ frames or messages are all or nothing Okay, so if it can't deliver every Byte of every frame in the message you don't get it which if you think about it is what you want in Every if that's not what you want then Go grovel with TCP and suffer right okay? So that's what's gonna happen here, but I want to send two frames. Why do I want to send two frames? Okay, this is again this little bit where Python will do bad things to you if you don't send binary Why do I want that because the way that subscriptions are done is they're just pattern matched To see whether the message has your pattern as a prefix And that's actually kind of kind of brainless, but it's actually really nice for one thing a client Could subscribe to client Python And now he only gets messages that have that Channel name or he could subscribe to things that say client and he could also get Client dash CPP so you get a little you can actually kind of create a hierarchy of Channels if you want to which is really nice now the other thing you don't have to use two frames This could actually just be the beginning of this message. I don't want to do that because then I actually have to Disambiguate that I have frames so it's only going to pattern match on the first frame So now I'm guaranteed that it does not matter if this happens to begin with this because the pattern match will happen Exactly on one frame it that frame is entirely devoted to the pattern and that's it So it's actually really nice. Okay, and what's beautiful about it is that it's Does everything you wanted and the implementation is so simple and it means that you can just if you can your language can mess with strings Then you can do anything you want with it Okay, what does it look like in the watcher now? I didn't actually put the code for the log server because it's trivial for this simple log It's just grabbing them off of a Pull socket and it's broadcasting them right So what's a little more interesting is slightly more interesting is what the log watcher is doing So he's got to create a sub a sub socket now This is the this is a great way to waste 30 minutes of your life The first time you do this forget to call set sock opt Once again, we're do we're making it look like Berkeley sockets even though it really really really isn't so yes Everything we do every option is done with set sock opt And we have to subscribe and this is just a string. I'm using as the prefix right now The guide says and I believe I probably did it What you always do the first time you do this is you create a subscription socket You go ahead and you start receiving with it But you forgot to subscribe which means you're subscribed to nothing and then you wonder Where are my messages going and the answer is they're going right where they should straight to the bit bucket because you didn't subscribe to them On the other hand because it's just a prefix you can subscribe to the empty string channel And you get everything so it's exactly what you want Okay, so what am I gonna do? I'm just gonna sit here if I'm the log watcher all I have to do is is subscribe And then I just sit here. I just receive stuff and then I do what I want with it But I can have log watchers all over the place asking for different things And it just works and nicely the just if you're wondering The message is although you set the prefix you set the channel subscription in the subscriber That invisibly goes over the network to the publisher socket, which then doesn't actually send it to you So you don't have to have all the network traffic of having every subscriber get every message and then throwing them away So which is kind of nice. Okay? So What's the lesson here? Okay? Well, the lesson is that I prototyped that at the last minute just because I could and I actually thought it after I did it. I thought, you know, that was actually a better illustration Then it would have been if I had planned out, you know something a little more involved because That's what part of what I want from something like this there are many more Sophisticated ways to use this. Okay, there are ways to to work very hard on reliable Messaging and a lot of other things But the nice thing is to start by just getting the data to flow From all your parts so that all of your buddies can actually start writing clients and workers Publishers Subscribers and then later you can go back in and do that, but they're not twiddling their thumbs saying Where's the messaging layer? Right? Okay There was one other thing that was a little bit obnoxious and I want to mention it because so it turns out every socket I started building a table. I didn't give it to you because it wasn't finished But I started building a table of what are all the differences between all the socket types Because as a tutorial they get scattered all over for when you needed them in the tutorial Most of the sockets will block if their message queue is full So let's talk for just one second about these things so these sockets the guide describes them But he starts off by saying we took Berkeley sockets and exposed them to radiation from you know waste from a Russian nuclear power plant and then we gave them superpowers and they were bitten by a radioactive spider and the pros is a little Unsettling you if you're a techie you're going that's neat. What do they do? Okay, so all of the sockets have All of them will even have one message you most of them have a queue and in going an outgoing queue some of them Don't need it, but the point is that You can control the size of the queue with Get sock off But you will care about this once you start wanting to make things reliable because as I said my what will happen What happens when that queue is full the socket types have different behaviors most of them will block That's why I said if I send logging messages from my threads They slow down to the speed of Python. Of course. I was being mean they slowed down to the speed of the network Python is faster than the network. So I was just being mean just because I can But the point is a couple of them will just silently throw things away the de the router socket will throw things away Because it's a server. It's supposed to stay up The one disappointment I have with that is I actually want to be able to change this behavior with set sock up And I was going to show you how to do that and then I'm going like wait I don't see that option turns out doesn't seem to exist So if you don't want that behavior you might have to implement some flow control But as I said reliability will cost you some effort, but it will still be easier than doing it in raw sockets, right? Okay, but it's worth knowing that's why they slowed down They filled up their message queue and then they're sitting there blocked waiting for somebody to pull some of those log messages out So that they can continue on right okay, and as I said you need something fancier, but The important thing here is my point. That's why I called the talk connect all the things If you run the example code there are workers running in threads over in proc You can change it, but the Python client by default is going over local host It's going up through the through the the network stack the The Python worker is actually going through a unit socket and none of this you don't know you don't care It just works now Again for the social media generation who have already tuned me out because there were way too many words here I do want to summarize things in a more concise fashion I love you I just like to tease people who are addicted to social media and with that I'm going to end. Oh, actually the last slide is this I've not done this before but since I put the slides in the repo it occurred to me They would be nice to just have it all licensed you can do pretty much whatever you want So I'm sorry for putting legalese in the slides, but it is there That's the Debian guideline make sure the documentation is under the same license as the codes you can rip things out I don't think you're gonna want to rip things out of my slides, but it's whatever you like. So with that questions Be absolutely So save doesn't find to repeat questions. So so if you make a network-centric set of Processes that want to be monitored by the existing commercial monitoring Process software that uses sockets to attach it to If you have zero MQ can a regular socket attached to you said it's not a Berkeley socket But what happens when a regular socket client attaches to a zero MQ? The zero MQ endpoint What will happen is it will make no sense unless you have implemented the zero MQ protocol So that's why I said you don't use it primary. You don't really use it for existing protocols That's not its purpose. So the point is zero MQ has a well-defined wire protocol and You can certainly implement that But the what you would I think do in this case if you want to get first of all you can use any number of I mean So it clearly uses Berkeley sockets underneath each one of those Zero MQ sockets represents a whole pool of Berkeley sockets and it's so it's pulling on them night and day You just don't have to deal with it You can certainly also have your own Berkeley sockets You don't have to it doesn't require you to do all your networking that way if you're Interfacing with something else if you want to pass data between them, then you write a little bridge Which may not be very hard if you if they already gave you an SDK for whatever it is or I mean in other words If you can already just throw data on to a regular socket then all you have to do is You know use a zero MQ socket to pull message pull a zero Q messages from wherever you like in your zero MQ messaging Layer and then stuff that string onto the onto the socket So it won't help you, but it will also not hinder you in any way. Does that make sense? No, you can't do that because it will because no matter what you do You can send a raw a string, but it will send it using its own protocol For example, every string is counted so their account their accounts in that stream You would have to know where they are so you'd have to actually understand their protocol and you can do it But honestly, I would guess in almost every case it would be easier to write a bridge data streaming systems like It was kinesis and Kafka have this abstraction about atomically increasing numbers and ways for clients to Express that they process a certain message in a system and they come disconnect and reconnect and then Continue to process a certain point in a stream Okay, if you want to implement something like that in zero MQ is that an appropriate is zero MQ an appropriate technology for this kind of System where you want to be able to continue processing from a given point if you have some kind of outage in a worker So what you're saying is that the message is persist essentially It's a process goes down the message is persisted on disk somewhere or in something persistent storage And then you can bring it back. Yeah There's something persisted but the notion is that there's an ID that determines the last thing you process So when your thing comes back up, it knows where to start again Okay, so the answer is zero MQ isn't Going to try to help you with that in that it tries to do one thing Well, okay, it does a lot of things, but it all is related to having an in-memory message layer Okay, so it's not going to persist anything for you If you're doing but if you have to do enough of the other things that it does well It's probably it might be easier to write that that it is to do all the reconnection and all of the the other things it does But if that's not true, then it might not give you enough because it's not going to persist for you Yeah, okay, so What are the major advantages of zero MQ over a simple queue service as your service boss? Rabbit MQ IBM MQ Or most other message queues and can you configure it for FIFO? for oh FIFO, okay, so the well the advantage is it actually is in a different space. Okay It's a lower level tool and if you want for example It doesn't have it doesn't have any built-in notion of having a server Whereas like rabbit MQ is there's a server there with a persistent queue right so The equivalent to that is if you wanted that And you in zero rabbit and Q didn't do it for you you would write it Okay, in other words, it's a tool for writing those servers Okay, so what it's trying to do is it's trying to automate the really low-level code that almost everything that that touches a Berkeley socket already does right I mean if you want to be it's a pain to have to oh I my song somebody went down. I've got a reconnect Oh, I need to frame these things even though I'm going over TCP because in their infinite wisdom nobody gave us a Frame well, they did but no one implements it They didn't give us a nice framed reliable service, right? So what it's doing is it's trying to give you enough building blocks It's it's a it's a set of Legos the difference between a set of Legos and a nice metal toy car, right? It's not meant to give you a toy car. It's meant to say do you want a toy car or did you want a dump truck? Or did you want a spaceship? We have the tools for you if that does that kind of does that answer the question? Yeah, but the MQ suggests that it's a message queue not just the blocks of a message queue So that's that's kind of the nature of my question And the second part was is it easily configurable for first and first out last and first out like a sqs is Well, okay, so so don't let me forget the second one So yes, that's why I kind of I had the little what it isn't thing because with that MQ on there You unavoidably think that it's a competitor to rabbit just just don't forget about the zero at the right? Yeah, it was a hint that there's zero. Yeah, the zero is a hint that it's something different and the zero originally meant It's a zero zero late zero copy and I think now they they have lots of zeros because some of them get really excited about these things But none of them are here is a turnkey server Started up in a knit and then call it right as far as FIFO I'm not sure what you what you're saying But every one of those sockets will have one or both of the input and the output queued and that is first in first out Right, it's basically Sucking in network data off of normally TCP and then it's framing it for you So when you ask, hey, do I have a message it even if it's all there, but one bite you don't have to deal with that So the sockets are inherently FIFO. Okay, there's no out-of-band facility that I know of so Every message that you get will be in order. Is that what you were asking or would you ask to something more more sophisticated? Yeah, that's basically what I was asked about So there it's inherently that way then so my understanding was that the creator of zero MQ has moved on to nano message and Is there a reason this this talk is not nano message More industrial or there is a reason first of all if I recall right not only did you move on to nano message? But then he just moved on entirely and someone else forked it and now has it was either nano message to I think it now Might have another name. Okay, so there have been some forks and I looked at this because as With everybody who forks they claim to have improved something here's the problem most of the forks that have happened are now defunct Okay, and those that did happen they may be working now But I don't know how many years they've been working, but it's a lot less zero MQ doesn't seem to be going anywhere And also it's better documented. I mean like I said, I really want a really nice reference as well as the guide A lot of the other ones. I mean literally one of the one of them that look quite good The answer was mostly read the source code You know the idea because you know what zero and Q is now read the source code for what I did differently I'm gonna like Yeah So the reason is I didn't trust the other ones yet if I were deeply enough into it then I could make a decision But it might I this isn't going anywhere as far as I can tell that's that's the entire answer really If you go back two slides back There was a push and pull and then there is a Pub and sub right. How are the messages sent across this push and pull pub? How are they sent across? Yeah, do you mean how like like how how are they ordered? Oh, yeah, so what's basically happened is that the the pull socket is going to use I think it uses some kind of fair. So some kind of fair queuing algorithm So that it won't let for example a very fast pusher, you know entirely dominate the stream So it's distributing for you. I don't think that that's configurable So if you really needed something specialized That might be a problem for you It I didn't see anything that it wouldn't do it didn't do that I that I would I knew I'd want but it's going to sort of it's going to try to keep them all working without letting One dominate if that is that does that answer the question? Yeah, sort of okay I mean there is no buffering in any of these places, right? No, no there is there's the The push socket suppose the server goes down. Well, then he's filling up his outgoing queue, right? Well, there's buffering here because this is another socket so it's got an incoming buffer if your server is not pulling them off They're fast enough. He's got his own his own queue to fill up So that most of the most of the steps in this all have buffering on them Okay, and what it will do is it will push it as far as it can go and then it will start filling up the queue This is kind of It'll start filling up the queue that the way where the blockage is if that makes sense and While it doesn't persist them on disk as long as the program continues to run It'll sit there until it figure till it's able to deliver it if it can't So it tries to be robust, but it doesn't persist on itself Okay, can I assume that if I need a high throughput or low latency use case applications Zero on queues is a one possible. Okay, so they claim so I did not as you saw my my Benchmarking was Extremely minimal at best. I really just wanted to know how it competed with something that I knew was fast They claim that it is in fact high performance And that is likely to be true Certainly for the network things that it's really designed to do I would however Benchmark that especially if you already know what your goals are I would just write something and just stuff messages through it and see if it actually is doing what you want Because I don't have any numbers for it. They I can tell you what they claim But they also say that they use that the sockets were bitten by radioactive spider, so I I Just wonder if I can answer the question before about the difference between 0mq and rabid mq. Oh, yeah Okay, me. I don't don't ever think about them the same thing. I believe a rabid mq and a lot of them use AMqp as their actual protocol. Oh, it does. I would more relate that to what this Zero is doing. It's really just a library for using a network socket which provides some cool functionality and a cool pattern Otherwise, it really doesn't do much. I would also In my understanding I could be wrong, but in my understanding generally you don't want 0mq to do any type of queuing It's not its job. It's there to provide a buffer just as a Maybe a network queue on a switch. We do it buffers a little bit just so we can get traffic flowing but if you're queuing you've probably got a problem something backing up and then You should actually have a program like Kafka or something else in between there which would be handling your temporary storage of messages or something like that No, I agree with that. I mean when I say it says queuing that's equivalent That's roughly like saying a Berkeley socket does have a buffer inside there So that you're not hitting the network on every call to receive, right? But as you say if you if you're intentionally queuing Well, now I'm the voice of God again because I adjusted it Then it's not that's not what it's meant to do you can't get at the queues in this You can't do anything to them except set their their maximum size So if you need rabbit mq, then this is not doing that It could have been written to use this as its transport layer But that's yeah, I mean I totally agree. They're not they're not filling the same space at all And you made another point that I like oh if rabbit mq is using a Q&P it's I won't say much about that because I don't know much about it But I will say this I Maddox the company who backs this was I think they developed a mqp And eventually abandoned it because they said this leads to too much. It's too heavyweight We don't want to be required to have a central server So you don't have to you can write a central server with this But there are plenty of tools to make it completely distributed So in their mind it does compete with a mqp, but it's a very very different animal Okay, and According to them faster Hi, I have a question saying that I saw you use you Get multi I mean receive multi parts or something similar. Yeah in your code and send multi part and modify Using an index number. I'm not saying is there abstraction layer available for like Packing up like each message is each message or for example, and in another case for example, I'm using a C program and a Python program using unit Unix domain socket to Connect between them when this because the speed variance is really very huge So what would happen like like for them if I only receive something in Python. I just receive like six parts or nine parts I'm not sure I understand the question. Are you asking is this about primarily how to use the interface? How do you get off the frames? You said you yeah, but what if like the message pops up and it can't Is it possible that when I use it? I just get more frames than I want. No Never because the frames are always part of a message and you get complete messages messages are all or nothing So if the sender sends you nine frames, you will either get nine complete frames or nothing Yeah, but I mean like for example the minimal frame for an envelope is three, right? You could make your own envelopes you don't have to use the delimiter They do and they have reasons for doing it. You wouldn't technically I mean you're not required Do you use the frames any way you like if that is answering something? Yeah, but is it possible like using the way you demonstrate it that I will receive like two messages at the same time No, no, it's always again That's that is one of the better the best guarantees of it especially if you've done your own framing on TCP and you wrote the same code a bunch of times or you've said screw it and used a Serialization library it they always come in a bundle and It's not all or nothing on the frame level. It's all or nothing on the message level. So you will always get Okay, well, let me let me qualify that that's the way zero and Q works now the interface like the C++ interface You can pull the frames off one at a time And I don't like that and if I'd had a little more time I would have actually written the equivalent of the Python interface for C++ it'd be very easy and The nice thing about the Python one is you can just say give me all the frames doesn't matter How many are there you'll get the next message, right? That makes sense Yeah, can you discuss options for transport security for transport security? What I can mostly tell you is that they do have a security system now built in I have not tried to use it They did mention the person who wrote I think some a litter wrote some elliptic curve code for them So it may be well done. Okay, but I have not tried to use it. So I can't tell you much on the other hand You know there is in my experience Much of the time people do the all-purpose Thing and just send whatever they like over a SSH SSL tunnel So you always have that option, but it does supposedly now have a built-in security system Might look at salt actually uses your MQ for all of its command dispatch And it they those guys work really hard on getting some transport security On top of it and take a look at what they've done there They've had it audited pretty extensively as you can imagine no one likes having your infrastructure orchestration layer be a compromise so Last I looked was like two years ago or something and somebody had actually forked The zero MQ and created a sassel implementation for authentication, etc It does seem to attract forks so We're at the end of the hour. So I'm gonna End the questions even though this is going great if you guys want to bug Dustin after it's over that that's fine Absolutely, but let's thank him. Thanks. Very nice. Do you want? Thanks a lot. What's that? Yeah, yeah, yeah, I didn't know what you were pointing at because I was not thinking very well. I'm just running on coffee Hello, everyone. Welcome to the developer track At this point this afternoon. We're gonna be talking about lost in user land with will here and be presenting We're gonna get started real quick here. I'll let him take off and then we'll do Q&A at the end I'll just sit off in the corner there, but when Q&A comes Tell me to grab point in my direction to grab my all right well without any further ado Let's all find out about the joys of core utils Excellent. Thank you. My name is will Rose Kranz and thank you all for coming to find out how to get lost in user land So my name is will Rose Kranz and I work on the edgecast content delivery network at Verizon digital media Basically that is a very large network where if you have bits and you have clients we will send the bits to the clients We have over a hundred points of presence around the world. We serve terabits of traffic Zillions of connections a number that is always growing and not worth remembering exactly We serve lots and lots of sites that you're familiar with and lots and lots of video sites cat and gifts mainly and then All together it adds up to a non-trivial percentage of the total traffic on the internet so we are very concerned with performance and that's sort of a key part of my job and Basically the summary of all of that is cat gifts We do also serve edge cats net in addition to edge cast CDN net if you ever need like an Edge cat supply it will give you a random cat gift every time you go there So more specifically I am tech lead on the HTTP platform team We are responsible for building and releasing our core HTTP server software, which is called Sailfish Every time we release it. There's a really involved process of canary deployment where we're constantly looking into any possible performance regressions things like that so performance is a key part of my job and part of everybody else's job there and Generally speaking we have this very large distributed system lots of nodes around the world Running Linux so Linux performance is something I'm really interested in and we also have a software life-cycling community internally There's lots of different communities that people have started that are kind of separate from the management hierarchy One is the board game community some of them are more or less serious This one is called the long march forward and I created it with another colleague named Marcus Hildum And we do things like courting Coordinating OS upgrades on all of the systems that we have deployed in production And we are also really excited about end-of-lifing any sort of system or any code anytime somebody deletes a substantial feature or chunk of code or Shuts down systems that are no longer in use we award them with what we call the Darwin award Because we realized that management is always really excited about pushing This is the new thing that we're going to build and celebrating we built this thing And there's almost never a moment to really celebrate We destroyed this thing that has caused massive amount of maintenance overhead over the years and wasn't really that profitable and didn't help us that much So we kind of took an initiative to celebrate the thing that I think a lot of engineers are actually more excited about sometimes because the thing that has Been dragging them down and that thorn in the side that they've had for years And almost never gets kind of celebrated and the third bullet point is that I have very boring hobbies So I decided to write my own Unix user land because the Unix user land is this piece of legacy software that has been more or less Consistent in its user interface since the 1970s and I'm really interested in performance and I Have this theory that every large corporation has a directory that new guys are told not to run LS in I used to work in visual effects and we had internal pipeline utilities that would dump lots and lots of frames of animation Into a particular directory Previously I've worked in places where everything was on a really slow NFS server And it wasn't that many things but because it was coming over network It was really slow to run LS in and because I work at a content delivery network We do storage as a service for other people who decide how they want to organize their data So we can't be blamed for that directory and I can actually admit to it So I was really fascinated in why is LS so slow in these big directories that have millions of files in them And is there any low-hanging fruit in terms of performance? Is there something I can learn from kind of trying to implement my own LS and some other related utilities? So efficiency was something I was curious about How close could I get could I approach the efficiency of the LS and GNU core Utils? Can I maybe even exceed it? C++ is a language. I'm pretty comfortable with it's reasonably low level it's kind of like see which my next kernel is written in new core Utils is written in and All of that kind of dates to the late 80s or early 90s before C++ was as common as it is now So if you imagine a next-generation GNU project that had been started in the late 90s instead of the late 80s It probably would have been in C++. So I did that since it's native code pretty similar to regular core Utils I'm very lazy So I only worked on the easiest utilities to implement because as just kind of a hobby project on the weekends If you can write one in like a day, it gives you instant gratification, which makes it a really fun project So I created W Utils My name is will but I named it worst Utils because that helped me justify Not trying to match every feature in GNU And then I started doing a bunch of research on the history of some of these utilities and sort of fell down a rabbit hole that I found interesting so We have this concept of a Unix user land I need to define Unix and I need to define user land in order to explain what that is I'm using a very expansive definition of Unix in this context that includes things like Linux and Minix that don't share code with classic Unix Reasonable people will disagree with that definition of Unix But it turns out there is a lot of legacy even though the Linux kernel does not directly share source code with classic Unix The current version of Linux even if you build the Linux kernel 5 that just came out on a brand new system today There is legacy that dates back to really old Unix in terms of design influence and a lot of the supporting utilities user land Simple utilities are part of it mostly for historical reasons some simple languages like awkward part of it But languages like Python and Ruby are not because they're newer and they're bigger and much more complex you don't go to like awkpie to awkpip install modules for awk so The user land is fairly constrained and simple and then you have all of this other complex stuff like GUI's big languages the init system which obviously init systems are so Simple and boring that nobody would ever bother to like reinvent a new init system That's more complicated and significantly different different than the legacy 1970s version. I assume That's the best joke of this whole talk and it's all downhill from here And In it the code runs in user space It's not kernel mode, but it's part of the OS because a user user isn't directly interfacing it with it Things like cp and LS are the user experience or the user interface of a classic non GUI Unix system So that's what I was focused on but like I said, I'm super lazy and I wanted instant gratification So I just focused on easy stuff. I didn't write languages I didn't write a C compiler. There's people who are way better at that. So I wrote stuff like cat Which shouldn't be interesting, but it kind of turns out that it is So I started with false and true Which in C are numerically in that order rather than true and false because in C false is zero and true is one in shell True was zero and false is one The shell and Unix of course were created basically the exact same time But the shell predates C So they didn't agree with each other for various reasons that made sense to people at the time And this is a piece of legacy that is completely insane and we just accept I just really wanted to point it out because we get used to some of these quirks that a new person approaching the system Would be like but those who literally the exact opposite false is true and true is false and it's a very post-modern approach to truthiness So anytime that you want to implement your own version of utility check the documentation some of it's good some of it's terrible Can you true is documented as do nothing successfully? And then return a status code indicating success which is zero and then false is do nothing unsuccessfully pretty straightforward. So let's Evaluate the really early implementation of true This is the complete implementation of true They said that I shouldn't have a lot of code on the slides, but I think you can all grasp this It was a shell script that was zero bytes and because it had no commands in it None of those commands could fail and therefore it returned success which is true Which you really have to admire the simplicity you had to spin up the shell interpreter in order to do it So it's a little ram that was probably unnecessary But as an elegant solution you have to admire it then lawyers got involved. So by 1984 This is the implementation of true. This is the complete implementation of true circa 1984 It is all copyright text and some source control notes Fun fact, this is less than 512 bytes of copyright notice So on a 512 byte block file system, which was in use at the time and the file systems were still mostly implemented in like assembly So they didn't have really complicated systems for merging I notes together and stuff They got this metadata completely for free. It took no extra space. So on some level This was also a sort of admirable engineering accomplishment to include all this Duplicated data between all the scripts, but in some of them you just got it for free because it was still less than 512 bytes But new in the 80s they looked at classic Unix and they were like we can do better We can be more efficient spinning up an interpreter is insane So the GNU true the current true dates back to 1999 if you check it and get hub it's been through a surprising number of iterations and This is a 31 kilobyte binary, which is about a hundred times the size of the shell script that had 300 bytes and If you note in the comment True will occasionally return exit failure in the case where rights fail So GNU totally unprecedented new feature in true. It was a true that sometimes returns false Because they added dash dash help and if for some reason it failed to write the help output It would return false and and I was really impressed by this sort of new feature this amazing new approach So in in my version of true and false I was like I can take this to the next level and make a false that sometimes returns true Because if they get to take credit for failing at printing help for a utility that has no features Then I get to take credit for successfully printing the help for a utility that has no features So if you do dash dash help or dash dash version It will print out the help text and then I think rightfully claim success So having learned from you know that history I I Was inspired to create even more features and then down at the bottom There's if exit failure is true return exit failure So bin false makes a point of returning true Because see and she'll don't agree on the definitions and Then if you download the slide deck, I've included a bunch of links to things There's links to the original source and things like that that you can check out on your own time So a couple of other utilities obviously true and false pretty minimal I decided to write my own cat and I looked briefly at doing my own DD. I was really interested in the History there. I didn't actually release a DD with this talk because it turns out DD doesn't do anything like I thought I did So cat of course can catnate files and print it to standard output There is a famous page called useless use of cat where if you cat a file and then pipe it to something and only use one file One of your colleagues will during code review tell you you're an idiot because cat is only for concatenating multiple files Of course, they could have just made cat a shell built-in and then it wouldn't really matter and The less than greater than syntax for redirecting a single file wouldn't be more efficient But in 40 years nobody has ever bothered And then DD a lot of people learn this as the disk duplicator when I was first learning Linux and Unix That's what the utility was explained to me if you download an ISO image off the internet for like a CD ROM You can copy that onto a disk Take an image of a like a USB flash drive put it into a file things like that But if you look at the man page for DD it's convert and copy a file And in the man page for DD aside from the name DD convert is the very first word And that's an important nuance that I think a lot of people don't really notice So if you look at the whole man page Copy file converting and formatting according to rules and most of us don't really use DD for that If you scroll down in the man page the conversion is ascii and ebsidic and As it turns out the name DD does not stand for disk duplicator or anything like that the original name comes from JCL in IBM language on mainframes and it's for data definitions in JCL because the IBM mainframe guys who were collaborating with the Unix guys needed to trade tapes and stuff sometimes and in order to Be compatible with an IBM system you needed to use DD to convert the text between different character sets and then write out an ebsidic tape for your IBM using colleagues and Obviously we don't use a lot of ebsidic it still exists. It's probably not super important day-to-day to anybody in this room The thing I want to flag about this is that there's this legacy utility that had a very specific use case in the 1970s That nobody intended to use for backing up USB flash drives because USB wouldn't exist for decades That we still use because it turned out to be kind of useful for something. It wasn't intended to And so there's this enormous amount of legacy stuff with character set conversion in this utility that we never used for character set conversion that is still maintained and could still go wrong and stuff and we're going to revisit how deep a rabbit hole of Weird legacy backwards compatibility we can find in the modern stack So I just sort of wanted to flag that as part of what is going to be the theme cat Doesn't just concatenate files whatever it claims It also has a bunch of conversion and processing stuff the most interesting thing that I found in the man page when I Actually sat down and read the man page for cat if that it'll do line numbers Which most people Never bothered to read the man page for cat so they forget that it has all of these extra flags But that guy in your code review who says cat is only for concatenating multiple files Doesn't use it for line numbers, which is a totally documented valid feature And you should give them a little guff for that during your next code review I All right I also found article on history of cat that goes into a lot more details and then the useless use of cat file my cat an Original cat written in assembly because all of the original utilities were written in assembly before the C compiler was stable And they had to do ports So the next utility that I wanted to work on was LS pretty simple one of the inspirations for why does this take so long? Getting a little more complex than cat. I didn't Focus on implementing every last feature of LS I wanted to start with just listing files and then the ability to maybe do a long listing where you have to make it actually Stat the file to see features about it rather than just looking at the name I didn't implement all the sort features and stuff like that LS is Thousands of lines long when you actually pick it apart and it's very complicated and doing all of it is surprisingly difficult but All right, so the W utils philosophy is to just by start by doing the dumbest simplest thing and then see what happens because like I said very lazy so I'm using C++ you can use boost or in C++ 17 There's library from boost for file system access that is part of the core language now Almost they change some of the spellings on things just to make it difficult But it looks like this in either boost or C++ 17 Give me a directory iterator for a path then iterate over the directory and I want to print it out It's pretty straightforward this is For the default invocation of LS basically equivalent to thousands of lines of C in the new core utils version One thing I will flag C out has a really bad reputation for performance But I didn't bother to use printf turns out the main reason that see out has a really bad reputation for performance is because it has Legacy interlocks for compatibility with printfs because they each hold their own separate buffers if you disable the Backwards compatibility with C so that printf and see out don't intermingle their output by accident Turning off that locking makes the performance just as good as printing out in C. It's fine Again, there's this legacy that is there by default that if you throw it away then everything just suddenly starts working faster And it's great So next step benchmark it figure out like how much slower is this how? Because like obviously going through boost there's this abstraction layer on top of posix LS is talking directly to low-level system APIs So there's going to be some cost using this higher-level API So timing a directory that has a million files in it just to see how long it takes to run LS new LS is Invoked as Regular LS without LS dash dash color equals auto that slows it down a lot because it has to stat every file So if you want, you know turbo charge your LS without having to build your own just an alias LS in the shell And it goes a lot faster that took four seconds and then mine using this complicated abstraction layer that's in C++ object oriented simpler API, but much bigger 1.4 seconds it was much much faster was almost four times as fast by doing the dumbest simplest thing So I think well I was disappointing. I Thought I was gonna have to learn a lot about LS in order to beat it But that was when I sort of started to learn that a lot of this legacy stuff is not optimal on modern systems and if you just Start by doing The simplest dumbest thing in a modern system that is designed for modern systems It just goes faster pretty much for free and you get these negative cost abstraction layers that yeah They're big and they're a bunch of code and it's a pain to link boost your application You have to add include path, but the benefits versus writing 5,000 lines of C from using this abstraction layer are that You wind up with a lot less code you have a lot less maintenance cost You have a lot fewer places for bugs. Oh, and it just happens to go four times as fast for free. So great So next step make it a little more complicated I want to add color output to my LS. This seems like you know at that point I'm in the big leagues for LS implementations And there's a couple of ways to do this. You can blast out antsy characters. There's these control codes You can do you can check if is a TTY standard out because you only want to do it if it's interactive not after piping to something But like that won't work on Windows You can check the term environment variable Which will say which terminal you're using to make some judgments about how to do color and whether or not to do color And then like there's an official library the term cap library and utility called t-put that uses it You can use that and then the requirements that I wanted to do was to do color in a correct portable way My false works fine on Windows. There's not a lot of complexity there So if I can make a totally portable thing that is using just core language features and C++ Then ideally it would be a portable thing that'll work on any OS even if I use something new besides Linux some other Implementation of POSIX so blasting out antsy the core problem with this is little bobby tables You're doing in band signaling a Terminal intermixes the actual content with the color stuff and that's kind of grody Using something like HTML. It's separated out into like CSS where you can make judgments about colors That's not just in band. There is a standard. Unfortunately. There are several standards Another xkcd reference and the way that color handling is defined it changes between all the revisions of the standard because as Systems evolved since 1979. I think the first antsy color standard was Systems became able to display more than eight colors, so they had to make some changes And even if there is a standard a lot of old terminals had color standards that were common enough to be de facto Standards that aren't compatible with the antsy standards Windows support Windows 10 theoretically does support antsy colors But only in like the latest revision of windows and if the terminal is running in a certain way and like not PowerShell Which does colors differently? That's not great terminal emulator support even on Linux is super inconsistent and You can leave the wrong colors because there's this stateful thing where you print out a control code saying become red and the text printed from that point on is red and then your application ends and the terminal doesn't know that your application ended because it just sees a stream of incoming text and it was never told return to the default color so then Everything printed after that just remains red because there is this implicit state it is only stored in the terminal or terminal emulator in the modern world and You cannot query that state to find out what the color is supposed to be or what it has been changed to at least not in a consistent way so Maybe blast out antsy if you check is a tty you Can avoid breaking some things and then like tty's really don't exist on windows only kind of sort of pty's kind of do and then What is a tty? It turns out that is a tty. It's a teletype Checking the term variable This is something a lot of applications do and it's why so many command line utilities break weirdly when you use a different terminal emulator And some like oh, it doesn't give you color out. It gives me color output on my system Oh, are you using a slightly different revision? Oh, you set it to X term to 56 color instead of X term color Linux rather than X term things like that There is also a separate color term variable separate from the term variable that you can use to decide if you want to do color so You cannot make rational judgments about how to output color based on those variables it is Not merely a very difficult thing It is impossible partly because modern terminals like console or GNOME terminal that you're likely to be using Just claim to be X term because everybody just gave up on using newer versions of the term variable because it didn't work anyway So why bother? continuing with it I've been writing this in C++ like for file system access there was this convenient library boost that did file system stuff And I could write like two lines of code. It worked great Is there a great library for C++ that we'll just sort of handle this and who cares how it works? No No, not at all There's some that claim to there's some that kind of seem to under some circumstances One of the ones that I kind of like the API for that I found on github that Shall rename remain blame this blameless Use term cap, but you only looked in one directory, but not all the directories. So it understood X term dash color as your term environment variable But if your environment variable was set to X term or X term dash 256 color it would just fail horribly and think it wasn't allowed to do color output Despite the fact that X term 256 color Loves color output even more than X term color because there's this legacy of files being divided in different directories And if you try and roll your own thing to Parse all of it out you're gonna screw it up because it's massively complicated So you use term cap It's not portable. It's not something that really exists on Windows, but it exists on Linux. It's pretty stable. It's mature You can use the utility called t-put that uses this term cap library It's a demonstration of the t-put set f2 that makes green text set f5 That makes magenta text and then set f3 according to the documentation outputs yellow text That's not yellow though. So what is documented has very little relationship to what actually happens Here's a Wikipedia guide to color keys in certain ANSI APIs and what they tend to come out as with certain things so it's a Real crap shoot like what color you're actually going to output when you try and output a particular color because This table goes off the screen further That's not even a full screenshot and that just says Ubuntu is one of the columns There's a lot of terminal emulators on Ubuntu and I don't even know which one they're claiming that is for so I went to this term cap database That's the thing was what's that? Thousands of files referencing different terminals That you will never use that you've never seen that you will never encounter none of your users will ever sshd your machines using I found Almost 3,000 files in user share term info But remember how I said there was that library that failed when you set it to X term That big path with thousands of files isn't big enough to cover X term which is a reasonably common version because everything claims to be an X term even if it's not tons of legacy So one of the things that you can do using the information from this term cap database You can use stty which is a related thing for setting your teletype configuration settings You can put GNOME terminal or console into uppercase only mode as a legacy compatibility mode to terminals that did not support lowercase So that works fine color doesn't work yet Legacy hardware that you won't use and I sort of fell down a rabbit hole trying to understand how far does this legacy rabbit hole go? This is a photo of Buffalo Bill who became famous during the 1800s for his Wild West show Okay, don't worry. There is not an entry in the term cap database that goes back to the 1800s But it goes back pretty far It goes back to 1963 Now those of you who are Unix historians will note that zero Started in 1970 you can only represent 1963 with a signed time value because this is from before Unix existed and obviously decades before Linux existed But when Unix was first being developed The guys who were working on it got the crappy old terminals that were hand-me-downs from the more important work So they got the very first ASCII terminals that only supported uppercase Consequently like Unix v6 boots by default into only uppercase mode and in gnome terminal You can still flip it to uppercase only mode for compatibility with a 1963 terminal If you set term equals TTY 33 that still works the term cap database will look up the valid control codes for using it And if you had this terminal connected to a machine and SSH to any Linux box in the cloud that you provisioned five minutes ago It will still work and have valid control codes for talking to this TTY model 33 the ASCII standard originally created in 1963 did not get uppercase and lowercase differentiated until 1968 so This was the beta version of ASCII that we still have backwards compatible support for and this is why color is hard So the 1968 version of ASCII it got adopted as Standards get lots of names. It's got a CCIT CCIT version number and things like that one of the names for the spec is ITA 5 That was when they added the lowercase to flip your gnome terminal set STTYL case and Just to point this out If you run LS you get this mojibake gibberish stuff with characters it doesn't recognize That's not a problem with uppercase versus lowercase That's a problem with color if you unalias LS so it's not trying to output color the uppercase output works fine So in 2019 we have perfect support for uppercase only terminals But outputting color will put the gibberish that's all misaligned and has weird characters in it So that's neat and a hundred percent not useful to anybody So I mentioned that it was ITA 5 is the adopted version of ASCII there were ITA versions 1 through 4 that came before ASCII ASCII is a 7-bit code Which causes all sorts of problems with needing to like you you encode to go through Not 8-bit clean systems with old protocols It is a 7-bit code because the code that preceded it the old versions were 5-bit codes and That terminal there is a World War 2 era system using ITA 2 Which is a precursor to ASCII that evolved in a very linear line to ASCII It is made by the teletype corporation the same as that TTY 33 that we maintain backwards compatibility for in Linux to this day It ran at 45.5 BOD Which is impressively fast for for World War 2 given that you know a home user would still be using 300 BOD modem in the 80s and I found a video on YouTube where somebody had made a USB adapter to go from USB to a World War 2 era TTY 19 Because the serial protocol is so consistent between these really early systems the serial ports used for modern computers Were directly inspired by these clocked serial protocols that used a Single-wire for data exactly the same as RS232 This dates back to before we really talked about these things in binary terms so instead of one you have mark as a signal because You didn't plug it into a computer and you weren't going to do math with it yet because nobody had a math a computer that worked with the terminal interactively and With passive components you don't need a microcontroller or anything to do the character set conversion If you set your Linux box to uppercase only mode you can build a very dumb adapter Does not require a processor does not require any CPU power to convert between the character sets and still use this as a terminal on a modern system This predates the information age this predates stored program computing this predates Shannon's limits for signal processing It still works fine. I've seen it happen on YouTube ITA one is the ultimate original version of this ITA code series of character sets that we can trace ASCII back to with some compatibility carriage return in line feed only originated in 1901 but The code was originally developed in 1870 by man named Monsieur Baudot. He's French And his name is where we get the word Baud from because he invented this clocked serial communication almost exactly to RS232 that we would use today and the number of bits per second that it would use Was something that he invented prior systems didn't have that kind of consistent clock the way we would use today on a serial communication system Fun facts ITA one had a bell symbol. So if you've ever wondered gee, why does ASCII have letter ding? It's because on the original international telegraph alphabet They had a ding that you could ring a bell so a guy could come into the room to see a message get printed on a paper tape By one of the really early typing systems. It also has a null character You'll note the keyboard the keys are helpfully labeled in the super logical order four five one two three Because you use the three keys there with your right hand two keys with your left hand and then you just had to cord binary you had to memorize the binary code for everything and just know the combinations and Null was if you didn't have any keys pressed The original interpretation null was I'm not sending anything if you didn't have any keys pressed It wouldn't send any high voltage signals down the line and it would just be waiting at low voltage That's not something that is a useful representation in ASCII But the null character in ASCII is still all zeros so that you can send nothing down a serial line and Just leave it as low voltage That's where this comes from And because you could input totally arbitrary binary sequences on the original version of the 1870s keyboard While the character set didn't officially define carriage return or new line or anything like that You could enter the codes from the later 1901 revision of the spec on the 1870 keyboard So if you connected this through an adapter that did voltage conversion and character set conversion You could enter a command like LS and then enter the code for new line And this would still work with a modern System if you set TTY 33 as your term variable while you were SSHing So Yeah, I just took you back to 1870 for What you can get to work with term cap more or less with some passive electronics But not needing to write any drivers use a microcontroller or do any really exotic things because of the backwards compatibility support that's still there and This picture of Buffalo Bill that I found on Wikipedia is undated It's fairly clear so it's plausible that it was from the 20th century and this picture of Buffalo Bill is actually Decades newer than the oldest thing that you could use as a terminal on a modern Linux box This is why my LS implementation does not support color It's impossible Even if you use LS with weird modes like I showed you just get weird gibberish in your terminal There is no good way to do it. We have failed as a people With centuries to work on the problem literally it goes back to the 1800s all right, so That is my whirlwind tour of LS one of the other utilities that I wanted to work on was CP All right, you want to copy a file? That's a pretty simple thing to do so Standard file system is the API that I use for my LS implementation Works great. I tested that there is Method file system copy underscore file Works great totally follows the WU Tills philosophy of do the simplest thing because this is just Weekend hobby project for fun But I already told you just use standard file system or boost file system to do things with file system If you're gonna do it so repeating that for CP is like it's not fun. It's cheating so There is a lower-level interface that is only Linux specific and it's a lot more interesting called send file which The C++ standard file system method and send file which is a Linux specific API invert the order of source and dest Because in all the years that we've had units We've never agreed which order of the source and the destination go in when you're copying files depending on what you're doing So that's fun Also, just kind of a random fact one of the things that you send is the size of the file to that function The size of the file is an off T type rather than a size T It's not an offset So everything is backwards and all the types are wrong But it goes fast and it's an interesting thing and it gives me an excuse to kind of talk about some storage stuff So if you time my really simple versions that are each one line long With send file you have to open the file yourself and then send file handles to the send file API With the standard file system API you just send file names. So it's a little simpler. There's a couple extra lines in the Send file version that are trimmed from the slide Second and a half to copy a gigabyte file with new CP and then Half or two thirds 0.8 seconds using the really simple dumb ways that are not thousands of lines long And fun work. I noticed during the testing new CP takes a lot longer if the file already exists So the less important the work is the longer it takes because you've already copied the file And you're doing it over and over while you're testing your thing to prepare for a talk It just starts slowing down if you don't delete it yourself between the runs That I don't 100% understand why but new CP has a lot of different modes for compatibility with different scenarios And it will do things in extensive you have sparse files and things and the existence of the file that you're copying over It's not just the amount of time to remove the file and then copy over it It over thinks it and makes bad choices So how does storage work what's going on with this copy? You've seen that I Am in the software life cycling community where I have an interest in legacy. I have an interest in performance I'm going to give you a complete history of storage that is a hundred percent as in-depth as my history of terminals There is rust that gets dizzy and not This is my complete history of storage devices and as a first-order approximation when you're trying to write performant code This is 99.99% of everything that you need to know hard drives there is a Magnetic domain it goes in a circle and the iron oxide gets dizzy with tapes There's iron oxide on a tape and it goes in a circle on a different axis and seeking to your data takes a lot longer Historically there was drum memory which rather than being a disc that went in a circle. It was a cylinder And it went in a circle and the rest got dizzy and it takes a long time for it to come around to the other side Of the circle because it's a mechanical device everything else SSDs that use NAND flash nor flash 3d cross-point battery-backed RAM Basically all the same thing from a programming perspective because there is not a physical device going in a circle that you have to wait For it to come back around because if you're anything like me, you are unlucky And that means that the data that you want is at the wrong point in the circle And you're going to have to wait a very long time for it to come around When it's in chips It's fine So what does that mean for performance terms if you're trying to actually take away something useful other than buffalo bill pictures from this talk Well for dizzy rust Find busy work in your software Regardless of whether it's tape or disc. They're different by orders of magnitude. It's fundamentally the same thing when you issue a read You're telling the colonel go do this and then if you wait for it doing nothing You're going to have to wait for a little piece of rust Work its way around and maybe multiple times if there's a bunch of indirection in the file system and things like that and if you Dispatch that into some sort of a background process You can find some other work to do like if you're looking for copying multiple files You can background the read of the first file and go looking for the second file things like that depending on the exact application What busy work you can find? While you're doing an asynchronous read will vary But generally speaking if your rest gets dizzy find busy work because you're gonna have to wait for it And use big buffers use lots of lots of memory. The memory is faster than the disk so if you can dedicate some RAM to things like page cache, it's gonna work a lot better and Copying between buffers is a lot faster than copying out to disk So even if you're wasting a bunch of RAM and you're wasting a bunch of CPU cycles copying between buffers in RAM Like who cares if you can avoid some disk because you're ordering things to be more efficient totally worth it For other avoid busy work get to it You do not have to wait for a piece of rust to go around a circle and complete a rotation So while yes, there is latency on SSDs It's really interesting how much there is depends on the controller depends on the connection Whether it's SATA or SAS or NVMe things like that all super important when you're getting super into the weeds But generally speaking it's going to be so fast that if you are spinning up a background thread just for a single read job The cost of spinning up a new thread is going to exceed the value of the interlocks and waiting for it and doing this and doing that Oh, are you done it might done because it will be done And then you will just have all of this overhead is spinning up a thread creating a New buffer and coordinating locking between all of it So just do it and then crucially avoid copies and that explains why send file is so much faster than GNU CPU Because send file is not implemented in the C standard library. It has actually implemented in the kernel The way GNU CPU works You say kernel Please give me the data in this file and the kernel reads the file Ideally it's already in page cache. Maybe it has to go to the disk You don't actually care, but it gets into page cache. It's copied from page cache Into your address space. You go great, and then you probably copy it Into another buffer in your address space, and then you go kernel Here's that thing you gave me you can have it back, but put it in a different file now So you've got three copies that it goes through where you roundtrip it from page cache to your application To a destination buffer back to kernel On the other hand with send file because you're saying kernel I don't want any of this you deal with it The kernel copies from one place in the page cache to another place in the page cache, and then it gets flushed out to disk There's only one copy. That's a third as many copies. I'm bad at math, but a third is less as a result it goes a lot faster and whenever you're dealing with non-mechanical storage you always want to avoid copies because While your SSD is much slower than your memory bus you ideally want to use your memory bus for other things than just a bunch of Unnecessary copies particularly if you're using numus systems, which you may not be on your development workstation or laptop But secretly you will be on your machine in the cloud when you ship code to run somewhere in Amazon They're big systems with multiple CPUs So you may wind up having a copy data between physical nodes in a numus system And if you just do anything to avoid those copies then they're free because anything you don't do Doesn't take any time to execute. Yeah, that's It's got bored. I talked too long All right So takeaways you've learned a lot about Buffalo Bill He lived until 1917 Which it turns out is long enough that he was aware of the first tank battles in 1916 during World War one and People drove drove Model T forwards to his funeral And he makes an interesting metaphor It was kind of silly to put a picture of him in the presentation because he has nothing to do with this But he is a guy that we conceptually imagined being purely in this period of the Old West as if he just stopped existing when the 20th century happened and he actually lived well into World War one he was aware of Aerial combat with fighter planes happening in the European theater. He was aware of tank battles He was aware of all of this stuff that we don't think of somebody from the Wild West in the 1800s knowing about and That serves as a good metaphor for the fact that backwards compatibility is a nightmare of Lovecraftian knot that takes you back to the Stone Age if you keep following the thread and will haunt you forever because literally in 1870 there were still parts of the United States that Had Indian tribes that were unconquered and they were literally Stone Age people because they Didn't have large infrastructure for spelting iron or making bronze So when I took you back to the 1870s in parts of California, I literally technically took you back to the Stone Age So in band signaling is always wrong I always want to emphasize that it is the reason for sequel injection It is the reason that color is hard in terminals It is the reason that there are exploits in SCP protocol Unfortunately the bullet point right after that is that in band signaling is inevitable because we need to maintain Backwards compatibility with these legacy systems, and if we have a signal we will use it So one of the takeaways is if you are going to be doing in band signaling that could potentially be Mixed in with your actual data Be thoughtful about how you use your metadata Because like the metaphor of Buffalo Bill it will be around way longer than you think and like DD It will be used for things you did not expect and you will make someone's life terrible In band signaling is inevitable The cheapest code to maintain is also the fastest code to run is also not code Anytime that you can delete some complicated legacy systems from complicated legacy infrastructure Delete a bunch of code from your application. That can no longer make your test suite fail That can no longer get stuck waiting for IO. That can no longer be the cause of a vulnerability because you're doing in band signaling On and on whenever you can use a really simple solution that will always be better Don't be afraid of code that runs on a computer For a lot of this stuff writing C++ even though that's theoretically a super low-level language. It's very scary You can use super high-level API's that aren't that bad to use in practice and you can have very compact code If you're not chasing hyper specific API's and super specific Things and trying to handle every edge case in C++ a lot of times just running code on a computer is pretty easy and pretty efficient and then Yeah, just when in doubt prefer simple solutions All right, so that is my talk that is my tour through the weird history of some things that I learned while I was implementing my own user land and You've been exposed to more legacy than you ever needed or wanted to so are there any questions So on your journey, do you ever look at on your wild wild journey? Do you ever look at busy box? Yes, I'm aware of it and it's super cool on embedded systems. I didn't specifically Dig into the history of it or benchmark my stuff against it, but yeah New core utils is definitely not the only available implementation of any of this user land stuff busy box works great probably Better than you knew in some cases because it is more recent as well How do you disable the see out printf the legacy support and why isn't that a default? Ah? Because legacy when C++ was first becoming popular everybody had legacy code bases in C and If you started using this cool new C++ library, but you've got gibberish It would theoretically result in sort of like if you had two threads outputting to the same pipe You would get intermingled output from the see out buffer and the printf buffer So nobody would have adopted C++ at the time if they had done that and Now we're stuck with it probably forever. Maybe how do you disable it? There is a link in the slide deck if you Got a link to the slide deck on my website If you go to willrosegrants.com it's up there In I clicked through a couple of slides that had just links On the slides and one of them is linked to that and you can see me use it in the WTills repository Which is also a link from my website It's like two lines of codes. It's actually really simple and just Don't do this and then it doesn't it goes faster. It's great So So DD copies files and CP copies files too. Why do we need to? Because DD converts and formats files and incidentally happens to copy them The reason that DD is really still popular is that so IBM mainframes Mainly used fixed record sizes for all of their data formats so DD has a bunch of settings for reading a certain number bytes and block sizes and Offsets and things for doing all of its manipulations with its character set conversion Assuming that you need to convert to only uppercase ebsidic That's an interesting late legacy that dates back to IBM's use of punch cards Where you would allocate a certain number of rows on the punch card and then when they move to electronic systems They sort of adopted that for their default data format with fixed record lengths as a result When you were trying to interface with hardware you could say read from this device file because in Unix everything is a file and read exactly one megabyte and that Ability to set alignments block sizes Buffer sizes things like that doesn't exist in CP could Nobody ever added it because we had DD Long after we stopped using it for converting between uppercase and lowercase and it was useful You were able to write programs and more efficient C++ libraries Was it as a pop don't those libraries don't some of those libraries also have a lot of legacy code. Yeah Yeah, yeah So something like the boost file system library boost itself is massive It's a ton of different libraries. Some of the libraries in boost haven't been touched since C++ 11 and if you try and use like boost graph library You're gonna party like it's 1999 man. It's great fun. Any other questions? Okay So this is a crazy project If no one's told you directly and you mentioned that you reduced the scope to be able to get enough here to show us So are you done do you have more plans? What's next? Not done. It's a fun thing to tinker with now and then At some point I would very much like to write my own shell or something like that It's a big project to try and tackle all What makes up user land, but you know if you pick at it one by one on a weekend Over time. I'm sure I will wind up adding more to it. I Probably won't bother to write a C compiler because The GCC guys have been working on that for decades and they're still not done There was a question back here somewhere No, I guess we're done. Well, thank you very much. Will that was very entertaining and Thank you all And if you stick around for half an hour, you'll get the next round of talks, which I believe is about fast string processing with HTTP Hello. Hi everyone Okay, I think we have a quorum here. So we're gonna get started Welcome everyone to the developer track here at scale This afternoon. We have dummies talking to us dummies talking to us about Working in serverless and GraphQL to build reliable and scalable apps I'll take Q&A at the end to bring around the mic so everybody gets a chance to be heard But beyond that, let's take it away. Tell me Let me just set up my mic. Can can everyone hear me? All right. Cool Great to be here. This is my first scale And I was looking at the schedule today and I saw I saw that there's something about game night I'm going to be here at every scale now. That's that's game night All right, so I'm gonna talk a little bit about building applications that are kind of resilient and scalable out of the box Exploiting GraphQL and serverless as a way to kind of architect such applications easily It's a fairly simple idea. It's just kind of a neat way of putting things together Before I get started just to quickly sample the audience. How many of us are front-end developers? How many of us are back-end developers? Not the application developers, but more on the DevOps and the sysadmin side of things. All right, you're gonna have a rough time, but So all right, so I'll start with a quick intro. Okay, wait, no wait, I forgot to ask How many of you have heard of GraphQL? All right, how many of you have used GraphQL already? Cool. All right. This would be a good intro and how many of you have used serverless functions of the service? All right, cool. Awesome. Okay, so quick intro about myself. I am Tanmay I am I created a open-source project called Hasura and we created a company around that a while back which is Hasura.io, but a fairly young open-source project, but fairly rapidly growing and and and we work in this kind of GraphQL and quote-unquote serverless space. So to kind of motivate the the problem that we have in a way is if you if you think about putting together an application on the Putting together a full application, right? The traditional architecture that you might have Which is probably what you won't have at scale But what you might have initially when you start doing things and when you move from like monolith to services is that you have a front-end app and this talks to an API and The API kind of talks to multiple other services, right? So the API for example talks to a database or the API talks to a payment service or You know, I'm talking about like a food delivery app It talks to maybe a restaurant service to To figure out if the restaurant is open or can take delivery can take orders or not And then maybe to a delivery service that has complicated business logic for figuring out how we should actually assign the delivery person And do the delivery, right? so so this is kind of like this is what you might have and and and This will start becoming a problem very quickly and you will start missing or at least I would start missing the monolith As soon as because you're kind of creating more services. So obviously you need to start worrying about It's it's not really scalable It's not really as scalable as you thought it would be right You have to worry about scaling each of these services independently or the scalability of your API server Actually depends on the scalability of these other services. So it's not like you've got API being scalable for free If you think about it being resilient That also became more of a challenge than when you just had a monolith you now are dependent on every single network link Let's say for example, the restaurant microservice fails. Do you roll back the payment? Do you you can't you can't you can't undo you can undo a database transaction what but you can't undo an API call, right? And and and so kind of building in that resilience after you have this setup When you realize that there are arbitrary network failures is is a bit of a pain, right? And also microservice is a little bit 2015 now. So so maybe we can maybe we can do something new, right? so so one of the first things when people start to think about Making things more scalable or more resilient is To kind of async as many things as you can, right? so that at least failures that happen are not synchronous and and You you when you do this that the expression on the on the things face is exactly the expression that you end up with because Asyncing all the things is not is not fun But if you async stuff, this is kind of what things might look like, right? You have an app the app talks to an API Now the API kind of talks to I'm calling it a database, but it's basically your state, right? And does some operations on the state Which atomically kind of creates events that are captured and then these events get delivered to services And then the services asynchronously process stuff and maybe write back to a stateful layer, right? So they don't again don't have to write back to the same database but state, right? So so they kind of events get triggered things happen you write back to state, right? So and so instead of kind of synchronously talking to multiple services You asynchronously talk to other services, right? And this is the kind of this is the first step towards setting up an async architecture And this is nice because because you have a persistent kind of event system and you're persisting your state you're getting resiliency kind of You're building in resiliency into your system because if something fails if there's a network transient failure or a Hard failure then you basically have as long as you've captured events you can You can theoretically kind of redo things again, right? and and and that's kind of the first step so In in that light and there is there is there are still problems with this and we'll come back to that later But in that light if we think about If you think about serverless or when I say serverless I I mean functions as a service I don't mean the CNCF definition of serverless, which is a little bit complicated But the the simpler definition of serverless, which is just the functions of service piece is And let's take a look at what that is for those of you have not used serverless or functions of service You write a small function typically in a language like no JS This is where most of the serverless action is and so you write like a small function this function When it runs it responds with a JSON that contains hello world, right? That's it. I just write the function I don't do anything else and then You have a cloud vendor or a serverless vendor or your on-prem vendor. You have some tooling you use that tool to say Deploy this function call it hello world and trigger this function every time there's an HTTP call So attach it to an HTTP endpoint And so as soon as you run that command what you get is you get an HTTP endpoint Every time you call that HTTP endpoint an event will be triggered by your vendor the or the cloud vendor Which will then kind of trigger this serverless function. It'll provision the resources to kind of create this function And and and run this function and then take the response and then send it back to you, right? So as soon as you do this you can curl that endpoint and you get a response back, right? This is this is the serverless experience this is scalable Quote unquote because you don't have to deal And and you don't actually get to deal with anything you don't you don't have to worry about You don't have to worry about what happens when there are a thousand invocations of the function because when there are a thousand invocations of the function There are a thousand instances of these functions that run, right? But this kind of scalability that you get where you don't have to do any of the underlying operations of kind of maintaining an API Server because it's just a function or the freedom that you get when you don't have underlying VMs to manage for those servers for those servers or containers to manage And when you don't have to worry about this you kind of think that It's freedom where you just get to write code and you just get to deploy but this is This is not really This is not really freedom. This is it's it's weird because it's You're forcing the developer that's writing code to not do things that they would have done when they were writing a Traditional API server, right? So you can't do global variables You can't think about what happens when there are con-con requests to the same function. There is no you you cannot There is no state. There is no mechanism for you to actually capture state across invocations of the same function And so when you take that out of the function, obviously that function is scalable So it's not it's not it's not magical. It's not even special It's just the fact that there's a little piece of code that runs and each piece of code runs independently And that's it. But in a sense, it's freeing if you have an architecture around it Because you kind of get to scale for free, which means that you're not really thinking about I guess the font sizes might be a little bit small But you get to scale for free because you don't actually have to worry about Managing infrastructure. You don't have to worry about monitoring infrastructure to figure out how to scale stuff And and and as a developer you get almost end-to-end ownership of Of the entire life cycle of your business logic, right? You write the code you deploy the code the code will work the code will scale forever if something If something happens, I own the code. I change the code. I redeploy it. I still get complete ownership And and and because there is no ops things are happening for free I'm using a lot of air quotes because all of this comes at a cost But this mental model is weird, right? So it's kind of like the way I think of the way I think of serverless functions is that it's kind of like Nano services the reason why we move from a monolith to microservice For me the way I think about this is that it's not just about It's with monoliths monolith scale Facebook and hence nobody can argue with that right but But it's it's that ownership doesn't scale right and that's the reason why you chunk out into so into having different microservices that the Management or the the the ownership that you have of this entire code base across a large orb will not scale And so it's easier to think of ownership when you break it up into services, right? and if you Perhaps take that line of thought to an extreme you you get serverless functions where it's complete like it's hyper ownership And this developer doesn't need to talk to anybody else And just does their own work inside a little block of code Manages and manages it end to end entirely, right? So And So you sort of get freedom and you sort of get scalability, right? And for the purpose of this talk, we'll call this being scalable We'll come back to this we'll come back to more aspects later Resiliency around this is also very interesting because of the model the model that a serverless function has is that When the serverless function is running, it's not it's not something that is running Already that is kind of auto scaling up and down the thing doesn't exist And it is called into existence when there's an event, right? the the idea of it being event driven is is built into is built into the paradigm, right and I Mean for better or worse and and what that allows us to do is is think about Making our serverless functions resilient fairly easily because if you think about it each function Think of each function as just being ephemeral because it is And and it doesn't have any knowledge of state, right? And if it doesn't have any knowledge of state, you have to trigger them when certain things happen So you trigger them on events. So technically if you can replay events or you can retry them or Whenever failures happen or whenever there's a back-off or whatever you want to do you should be able to build in At least a certain amount of resiliency Whenever there are transient failures, right? It comes at the cost of you having to think about your serverless function being idempotent Which means that if I try it multiple times things should still work Which is a problem, but which is kind of sort of what you already which you have to do when you have an event-driven system anyway, right? and and and the tooling is getting better to not have to worry about this so much but But in a sense because each function by itself can't too much and each function is triggered on an event if you can persist events and replay events you should be able to move towards it being resilient and And and a combination of it being resilient and it being scalable by the cloud vendor So as a developer now, I don't have to care about the ops of anything, right? Technically. It should just always keep working I and I can ignore all kinds of transient failures and so and so so when you think about So when you think about writing your code in serverless functions, you're getting some sort of resiliency and scalability out of the box, right? You're not you're not explicitly thinking about it You're not explicitly adding it to your application after you've built the application So let's take a food ordering app as an example, right? If this is our state This is our model that is there somewhere in the state, right? It's not it's not necessarily a table that has multiple columns It's it's the abstract data model that we have in our state, right? And so this order this order object has is created by a particular user has a validation state Once the order is validated. It has a payment state once it's paid for, you know It's approved you check whether the restaurant is open or whatever or maybe you check that before doesn't matter And then you check if you want to assign a delivery agent to it and you assign to the agent, right? So this this order object is going through various states and if you think about and if you think about Doing all of this business logic with serverless functions The the way you might want to think about it is you think about your entire business logic is state machine, right? So you have a new order and where all of these kind of validation state paid state approval state Delivery person state is all false Then you run a serverless function That does the order validation and so the state changes to validate it equal to true and maybe some other stuff again apologies for the font size, but If you can just make out that the blue is changing to red That's kind of one state change right and similarly you can kind of keep going to state changes So the next serverless function comes in does your payment stuff or validates that the payment is done does the approval stuff? Run some machine learning algorithm to figure out who the right delivery person for for you is based on your rating Based on your entire life history And then and then kind of assigns a delivery person to you based on their entire life history And then and then you kind of move to this final state where the order is being Is being processed and is actually being sent to you, right? So so so that's kind of So so if you have your state and you have a mechanism to trigger serverless functions whenever there are state changes And your serverless functions will drive those state changes, right? So that's kind of one way to think about running your business logic with serverless functions For your for your for your application logic, right? The the cost of setting up the system is that you get to write code Inside serverless functions easily, but the cost is that you need to build a generic event system that captures events Well that delivers events right and if you have this generic event system Either you might have it or you might use a tool like how sort of what? But you might also but but you can set it up yourself You can do a bunch of things on the cloud vendors are doing this tooling right and and so if you have like a good eventing system Which is generic and a good event capture system, which is generic or which works for your use case You should be able to start moving stuff to serverless functions fairly easily, right? There is There is another cost to this Which we'll come to but at a high level just to kind of make this clear for to kind of just draw Put the two architecture side by side you had the kind of earlier architecture where there were service level dependencies Between your API service and other microservices and now you've taken the back portion of it We still not talking about the front-end app yet and that's where the problem is But you've taken that kind of back-end portion of it and converted to an event-driven system where things or your API's or your services Are triggered asynchronously whenever events happen, right? The problem is that whenever stuff happens on the back end Right. How does the front-end? How does a front-end app know? What is what has happened right and this is historically always been the problem with doing stuff asynchronously And you can replace front-end app with any client, right? The it's it's great It's great to say that we're making things asynchronous But it's very painful for the end user or the developer that's building the experience for the end user to consume Asynchronous information because finally on my application I need to see the state changes that happen as well or maybe some Representation or abstraction of the state changes that need to happen on my app, right? And so when stuff is happening asynchronously on the back end How do I how do I get that info on the app, right? Because the app is designed to make an API call the API call does ten things and I get a response And then I just show that response, right? But now I did an action that forked up a bunch of things When am I gonna get the API response? How am I gonna show that final thing on the app, right? and it's it's painful and Which kind of leads us into Talking a little bit about graphql. So so that that final piece. We're going to we're going to try to see We're gonna see how graphql is going to be helpful. So before we do that. We need to understand take a little bit of a introduction into what graphql is right, so Back in the day from my point of view We were all using rest API's right and so this is maybe what a rest API look like forget everything I said about the back end and now let's come to the app and the API ignore what the API does So let's say I'm building a user profile page on my app if I'm building a user profile page on my app and I make an API call to fetch the user info Let's say I fetch the ID and the name so I do slash API slash user ID equals one I get a json response that says ID is one name is Elmo. I render this on my on my page. It's great But then tomorrow, I realize that we want to make the user profile page a little bit richer. So let's add Let's add more information here I also want to show the last the last known address for this user on their profile page So then I make another API call To fetch slash address user ID equals one So I'm making two API calls and then I get that information and I show it on my app And this is not ideal because in this in this toy example I was making two API calls, but in a in a more serious example You might be making five or six API calls, right? and and As a room full of back-end developers Y'all are not very empathetic to front-end developers Right because this might sound like it's not a problem, but this is a huge problem for a front-end developer because writing API calls and and the last portion of the front-end app that you want to work on is the bit that does the API calling Right because that's just always the worst you make an API call the response comes something happens on the UI You make ten API calls like ten things happen on the UI independently It's just it's just the world's most painful experience to build when you have to make multiple API calls, right? And I don't know if you have this experience But it's also super irritating when you load up a page and it loads up really quickly But then there's an API call that happens in the background you click on this button But the button shifted and it's replaced by another button, which is which is a destructive action, right? And every time that happens to me, I just get so I get I feel like punching my screen And so what you do is that as a front-end developer you go talk to your back-end developer Hopefully your friends and you say can you can you give me one API call? Right, don't do two API calls Just give me one API call and give me all of the info in that API call and then the back-end developers like really You already have it. I mean what has called two functions. I don't want to do this work I have other more important work to do but then like you know You you kind of convince the back-end developer the back-end developer goes on leave for a few weeks comes back whatever Maybe a month later. You have one API endpoint that gives you all of this information, right? And and you're happy because now you're just making one API call for your entire page And this is great till you realize that Till you realize that you want to fetch different amounts of data For a mobile site and different amounts of data for the desktop site on desktop I want to show you your user profile. I want to show you the last 10 addresses I want to show you the last 10 orders one on the mobile site I just want to show you the user profile one order Maybe the last five orders in just one address and then as soon as you do that You again have to go to your back-end developer and say Yeah, I'm making this API call and I'm fetching like 500 kb of data every time Which is fine when you're on this great Wi-Fi connection But it's terrible on mobile and just speeds are slow. Can you add another parameter called like fields? And I'll give you a comma-separated list of the like the actual fields that I want So I'll do like I'll say user info id equals one and Feels equals id comma name comma address comma whatever right and then you kind of come up with this syntax yourself And then you decide how to pass the commas and then you're not allowed to have commas in your names Which you don't anyway probably But but whatever right like it's a thing. It's the thing that you've now built with you and the app back-end developer And and and now you've done this right which is just a nightmare Process so what the good people at Facebook did is they said let's not do any of this Let's reinvent Everything and call it graphql and what we'll do with graphql is we'll not have multiple API endpoints We'll just have one API endpoint and this will be a post API endpoint And it's typically a slash graphql endpoint and we'll post queries to it. There's no get post put patch delete I mean and you I've not seen too much too much put patch delete anywhere I've only seen get and post mostly, but so kudos to you if you're using put patch delete, but But we but you do away with all of those methods you just have one method which is post you post the query to it And in this query you describe the shape of the data that you want Right so instead of kind of hacking around with the query params and coming up with your own syntax to represent What slice of data you want or what joins you want to do across various API models? You you just kind of make this query where if you can see on the left in this gif if I query for user ID The json just contains ID one if I query for user ID and name I get user ID and name So this language on the left is called graphql right and if you look at graphql It's basically like json with the values removed and the quotes removed right It's kind of it's the it's the skeleton of the response that you want And this is why front-end developers love it because it's just so easy to see What you know what you want to do and what this query means and the back-end developers hate it because now they have to You know build a graphql parser and whatnot Because you need to pass this query. This is a new language So you need to pass this query figure out what to do, but there's great tooling around it So you don't really have an excuse, but Or you can use also again, but now this is okay But so so you can do so you can kind of build your own graphql server and your graphql server will pass this graphql Query it will do the same thing that you did always with the API call But the contract is that your server will now return a response that has the same shape as What was requested by the user right and this is graphql or this is one core part of graphql that makes graphql awesome, right and Get rest calls are replaced by this notion of graphql queries and post put delete patch Rest calls are replaced by this notion of what is called a mutation right and they both look fairly similar So so that's what graphql does right? But graphql also does something else and that's the bit that's super interesting to us Let's say on the back end. I have this order object, which is changing Hopefully you can notice that like there are two colors that come in and this is a table So this is an order object table and I have and I have the status of this order is changing because Asynchronously something goes and updates this order table this value this order row right As an example right so there is an order that was created after sometime the payment suddenly just turns to true Because something happened asynchronously or the fact that it's dispatched suddenly turns to true because it was actually dispatched as an order, right? So this object mutated on the back Asynchronously now when this happened what I want to do on the front end is build an experience that shows you on your app When you have the order page open that shows you payment tick Delivery tick right even if I close the page and come back to it, you know, wherever I left off You whatever happened in the middle. I just kind of does it shouldn't matter and I should be able to kind of show you the latest state The way I need to write so I'm consuming this information asynchronously from the back end And whenever this changes, I should be able to build the UI for doing that. So before graph QL What you would have done with a typical API is the first option was that you would have done polling The only thing worse than making multiple API calls for front-end developer is to pull for the same API called multiple times That's possibly the only thing that's that's worse than just doing multiple So this is a fairly straightforward idea I will just keep refetching the order object every x seconds and I will pretend that this is real time Because I'm how's how's the end user ever going to know, right? And so doesn't matter and surprisingly it actually works fairly well in most cases It just is bad because a it destroys your soul a little bit every time you set up polling on the front-end app And you set up a cleaner because you can't keep polling even when the person navigates away from that page, right? otherwise you'd be pulling for everything and it also feels bad because On on mobile you're eating up a little bit of extra network You're eating up extra battery every time you do polling, right? So that's one the second option is web sockets and the reason why web sockets is not fun Is because you move straight from like yuck the nightmare There's this there's no middle ground because web sockets is just the world Is very is very bad And it's it's it's been a part of the HTTP spec now for a while, but it's it's incredible how The lack of documentation of the spec and how browsers deal with web sockets is astounding Like the the way browsers deal with web socket connections on cores requests is very different Browsers will not send custom HTTP headers Overcores no matter what you do, but it will work if you're on an Android app It won't work if you're running it in a browser There's all kinds of crazy things that have that's happening and you don't want to deal with that socket, right? so What graphql says is that you know what we'll come up with a spec for doing this real-time business as As a part of the graphql spec itself. We had queries for fetching data We had mutations for mutating data and we'll have subscriptions for watching data, right? Or for watching on events or listening to events It's almost as if like there was a new rest verb called watch and that's called a graphql subscription, right? And so with graphql If you remember that order object Subscribing to it is as simple as let me load that. Yeah, so if you can see the bit on the right here You open up a subscription and you say I want to subscribe to an order where the ID is 57 and The shape of the event should contain the payment info the latest payment info the latest dispatched info Every time this changes Send me send this event to me or send this if every time there's an event So either you can have a live query type thing or you can have an event type thing whatever you prefer but Every time this changes send it to me as a front-end developer. This is all that I do I Don't I don't deal with web sockets underneath it. It uses web sockets because graphql clients use web sockets underneath it And if you're building a graphql server, you still have to write a web socket server So best of luck for that, but but at least as a front-end developer. I don't have to do anything To consume real-time information I just have to write a graphql query and replace the query keyword with a subscription and I'm done, right? I don't I don't have to do anything else Which is which is amazing because now I can kind of consume information. That's happening asynchronously fairly easily, right? So if you kind of put this idea together with what we thought about when we did like event-driven serverless Your app now talks to a graphql API The instead of a rest API the graphql API is Closely linked to an eventing system and has synchronous logic whatever logic you need to be that you think is synchronous kind of goes into inside This API it talks to your state your stateful layer And and that's your graphql query that you're running from the app, right when it talks to your stateful layer Eventing happens and that triggers serverless functions to to kind of do stuff And so you're doing your you're processing stuff in this case payment and restaurant and and delivery and whatnot and And then whenever that happens Your app can open up a graphql subscription to say I've created an order Now I subscribe to this order and then you chill Right and then whatever happens whenever it happens However many failures happen whether it happens in the next few milliseconds whether it happens the next day. It doesn't matter, right? Everything just keeps working That's kind of what the system looks like what you can do is The cost of the system is as is fairly apparent is that you have to build a real-time graphql server You have to build an eventing mechanism And and then you have to convince your back-end developers to move to serverless best of luck with all of the three but But the the kind of the other way of looking at this and and the reason why I'm talking about this also here Is that's kind of what we've been building is that orange box can be replaced by an open-source component called hasura Which is which is what we build which gives you a real-time graphql API and an eventing system on top of Postgres Which kind of helps you set up this kind of architecture out of the box, right? And and in fact this pattern is something that we're learning from have seen from our users And that's kind of where the inspiration of the stock up comes from as well. So what I'm going to do next is do a Do two demos one demo for you to kind of look at what an application That is built like this looks like and the second demo time permits a demo of actually setting this up with with hasura as well so Kind of a little bit of a prayer to the demo gods now for all of this to work if you guys have phones If you all have phones, please choose to take out your phone then scan this QR code or head to this URL And I'm keeping my fingers crossed So I'm just keeping this it's called for those of you want to type it in it's bit.ly slash three-factor hyphen app And if you can scan the QR code you can scan the code. All right, so Let me open this up. So this is kind of what you should What do you should land up on? And this is the food delivery app that we've been talking about right, so I'm going to place an order as Tg 17x right and So now what happened is so this is a react app The react app is speaking to a GraphQL API to fetch your past orders. I don't have any past orders I'm going to place a new order Then it fetches like the menu items and I order a doughnut and haggis So it's the best Congress a new thing and and and then I subscribe to the order and The order validation is a AWS Lambda. It happens asynchronously Now it's blocked in the user. The payment is a synchronous function because I need to talk to the stripe API does Stripe payment gets processed updates the database and says payment is done I get that kind of tick asynchronously then the next Lambda kicks in actually it's on Google Cloud Functions And then the next thing kicks in which is a delivery assignment thing, which is again on Google Cloud Functions, right? and As we can see on this chart that I have also built is That you will see these state changes Happen as you folks are placing orders, right? So the red line represents the total number of orders placed the brown line below that so that that let me The line below that represents people who paid for their order or the orders that have gotten validated Once the order gets validated the green lines represent people paying for their orders You can notice that people don't like paying for their orders, which is why it's substantially lagging But as soon as people pay for their orders that state changes triggers the next serverless function Which goes and does the state change for Assigning or approving whether the restaurant is open and whatnot or can't take your order and then triggers the delivery function, right? So so that's kind of how how it works when I don't do this demo to you I pay nothing for this business logic to be running And then when I do do this demo to you You can do I mean you can we can have like thousands of orders get placed at the same time and I don't have to worry about it, right? I Don't have to worry about scaling it right in a sense in a sense still a toy demo the the other portion that I'm going to show you is Kind of simulating what it looks like if this was if we had a failure, right? So if you can see the screen I am going and placing an order for Dosa and idly which are two South Indian food items delicious and then as soon as I did that I placed a thousand orders in one Shot so I have a UI button that places a thousand orders and then after placing the thousand orders these The validation start kicking in right so more and more of these orders are getting validated And now to kind of simulate failure what what I'm going to do is once these orders get validated I'm going to hit pay for all of these orders and And so I paid for all thousand of them in one shot So you noticed a full state transition where all of them got paid And then I go to AWS and I throttle and I throttle the function And and this is kind of like a network failure, right the LAN cable got disconnected in the data center And so you notice that like 500 or whatever got processed and then it just flattened out, right? And this is this is not a failure of the delivery assignment function. It's a failure of the Of the order validation of the restaurant approval function, which is the third step not the fourth step So third step is failed and so we're kind of flattened out on the third step And we're panicking and I'm calling up like various people and now they're like looking at cables They're still looking at cables right and now they kind of figure out that there's a network failure So let's reconnect those cables, which is me going to AWS and unthrottling it And saying unlimited whatever don't throttle this anymore. And so as soon as I kind of remove that throttle And AWS is back these continue from where they left off, right? And and and this was Having this experience for Everybody in the organization including my application developers. I didn't have to I didn't have to think about it More than I usually would just just by building in this style I I just wrote my application the first time in this weird three-factor asynchronous way And and whether there were failures or not or whether there were thousands of orders or not things keep working. So that is Not my order history. Sorry viewing history, right? Cool. All right. So so that's kind of what that's kind of what An application that's kind of built of this built like this looks like I have a little bit more time so I'm going to spend the next five minutes and show you a live demo of how Hasura itself works And and kind of fits into this helps you build in this pattern. So let me Pray to the gods again because this is a Live demo from scratch. All right, so so has to rise kind of a simple Docker service what we do It's an it's an open source engine. What we do is we You you take the Docker container you point it to a Postgres database and it generates it generates your real-time GraphQL API for you and And then it does eventing as well, right? So I'm going to deploy this on Heroku I'm sure all of you are familiar with Heroku But as soon as I go here, let me call this the scale 17 X demo, right? So what I'm doing right now when I click on this deploy is that I'm asking Heroku to provision the Hasura container for us and A free Postgres database Heroku does this for free bless their hearts and Kind of allows us to to run this container and to run a free instance of Postgres It takes about it takes a few seconds to deploy. It's a tiny Docker image about 20 MB Docker image and that's kind of what The UI looks like and I'll I'll I'll go into and we'll kind of look at a few GraphQL queries to understand it as well so that's your GraphQL app endpoint so mobile and web apps that you're building will connect to that endpoint will make post requests to this endpoint And we'll try out a few GraphQL queries here, right? I'm going to go ahead and create a table in my database just for trying things out So I create a profile table that has an ID and a name and Creating some random names here and so I do standard database stuff I'm just creating the aspect ratio is a little bit screwed up, but I'm just created table and I've gotten to rows inside it and what housework gives us is a GraphQL API so I can query a profile you know ID name Auto-complete my way through it and I get that json response, right? and So what we're doing internally is we're a compiler so we compile GraphQL into SQL queries We add right access control rules and whatnot. I'm not using any access control right now So any of you can go and destroy my entire database don't do that but But but this is kind of what a GraphQL query looks like and to kind of show you a few more examples of what we can do with GraphQL what I'm going to do is I'm going to just import an existing database so that we have a We have slightly more models to play around with so that's my database URL and a All right, so that's me connecting to Postgres directly and that's the table that I have in Postgres I've done this via the UI so the tables already here. I also have a Database dump. I'm just going to import that into my database, right? So So now this is kind of creating all of those tables and now it's starting to import data, right? So as this happens if I go back to The house for UI It'll tell us that there are these tables. These are not exposed over GraphQL. So I'm going to expose them over GraphQL There are foreign keys. These foreign keys are GraphQL are hints of GraphQL relationships And so I'll say yes convert all of them into relationships and now as soon as I do that what I can start doing is query albums, this is a music database Let me show you what the schema looks like Chinook that that's the So that's the schema that I have here. It's a music database. I have albums albums have artists albums have tracks And and as soon as I import that here, I can do albums on my API ID title And I can find the artist and the name, right? And I'm getting that response here, right? So whatever 300 elements or so In the in the shape that I wanted to write and I can This is the way you'd make queries to like say that I only want. Let's say the first 10 and You'll only get 10 responses, right? So so that's kind of what GraphQL looks like Underneath it just to kind of make this make sense for you. This is a post-end point. I'm sending this as a String to the server the server takes the string and and response with this The experience as a front-end developer is amazing because if I make an error, let's say I misspell something and I call it Whatever TLE right I get a little red underline if you can see and then if I hover over it Says can't query feel TLE. Did you mean title? Yes, I did mean title There's AI in this in this API experience, right? So so I'm I'm I'm the as a front-end developer I like using GraphQL, right? What I can also do is query for the GraphQL API in this case is made by Hasura But you can have your own as well. So what I'm going to do now is query for the music tracks I'm going to make an aggregate query to fetch the count, right? So if you notice you'll see that every time I run the query, you're seeing that the count is changing, right? And the count changes because well, I'm importing data, right? And now what I can do is I can convert this to a subscription And you'll see that this data just keeps changing, right? So as a front-end developer, I don't care I don't care. There is something changing on the back end. Can I make a subscription to it? Yes, I can I will subscribe to it. It works. I'm done, right? And so now if anything changes on the back end, I'm kind of getting that information real time on the app And I can kind of deal with it, right? So so this is kind of the GraphQL portion of it The other portion of it, which is the eventing side of things is We allow you to create event triggers. And so this means that you can create a trigger called You know, whatever. Let's say echo Every time on the profile table, there's an insert We'll go call this webhook, which is httpbin.org slash post Right, so now if I go in and insert something, so this gives us an excuse to try out a mutation So let's do a GraphQL mutation to insert data. This is like a post So I'm going to insert profile I'm going to insert objects name Whatever asdf returning ID, right? So I inserted a new profile. I'm going to insert a bunch more Right, I inserted these Every time there was an insert, I had an event trigger attached to it Which means that I can go in and these events get delivered They did not get delivered Presumably because I made an error in my url endpoint But and and I can now set up retry logic on these to ensure that they get delivered multiple times Or I can respect a back off header so that you can implement your own kind of exponential back off But that's the payload that gets sent That this is the data that was inserted. This was the person who inserted it and and the response says that Whatever this not found because I had a bad url, but But you get the idea, right? So so this is kind of what this looks like and I Yeah Let me just convert this to a full screen again, right? So so that's kind of what we do But but this is just one implementation of doing this with Postgres If you have your kind of own setup and you have a more complicated data data layer setup You'd probably have to piece these things together yourself But but this architecture and this idea of using real-time graphql as an api of having eventing on the state which is Atomic so that you're capturing stuff reliably which is reliable so that you're delivering stuff reliably And then having that trigger async serverless functions I call this the three-factor pattern because it reminds you of 12 factor And it sounds simpler, but it's not simpler. It's much worse And so and and that's kind of that's kind of it, but Feel free to please do reach out to me if you have any questions and check the architecture pattern out at threefactor.app Check hasura at hasura.io. We're a young project. It's open source. Please try it out. You know, give us feedback I also have funky stickers. So please come and collect some funky stickers If if you all are here over the next one or two days and you guys are doing or exploring stuff with graphql and serverless Let's I would love to talk about that. I'd love to learn from you From your implementation and what kind of problems you're running into especially with serverless How you're dealing with state how you're dealing with contracts Are you putting multiple serverless functions in the same repo? I do you have multiple repos for each of them. What your local dev CICD stuff looks like Would love to talk to you about that. So Feel free to reach out, but that's that's it from my side and we can do questions Yeah Let me put you the mic sir wind here Thank you. That was great. Just two questions here. So with graphql some Folks that I know that implemented it they ran into performance problems when it was under high load I'm wondering if you've seen anything with that and then my second question I haven't implemented much serverless stuff, but Is it possible as a best practice? Is it common to implement a microservice? With serverless that does that even make sense? Yeah. Yeah, um I'll I'll I'll take the graphql part first. So so graphql and performance is a big problem because uh So there are there are a few ways to alleviate it But what happens with graphql frequently is that the way you implement a graphql server Is if you were implementing a rest server, you would have URLs that are mapped to functions, right? Like a typical rails app or a jango app or a flask app. You have a url that maps to a function in a graphql server the The graph of the query. So if I do query albums artists album has a function an artist has a function So whenever I make that query the album function is called the artist function is called data is kind of aggregated Put together and you get that response. Um unfortunately in in, um In in bad implementations, what will end up happening is that you'll end up making n plus one queries So let's say for example, I'm fetching albums and artists. There are 10 albums So what you'll do is you'll make one request to fetch the 10 albums for each of the 10 albums You'll make one request to fetch the artist Right, so you're making 11 requests to your database or to other services to aggregate this data into a json api. This sucks Um, this is horrible, right? Um, and and this is why And this is painful to implement and there are ways to get around this and there are ways that you can't get around this But the way that we get around this for example is that when you make this query You can see that it's super fast. Although it feels like it's on a server And you can make like a thousand of these requests per second in 50 megs of ram and we do this because we compile this to a single SQL query Um, and so like however large your graph kill queries we compile that to one One SQL query we inject the right access control rule so that you're looking at your data as an application user And that kind of solves a lot of the performance problems that you get Um, there are other kinds of performance problems with graphql that you will also see with the rest api That are not unique to graphql, but those you'd solve in the same way that you were doing on rest You do some server side caching instead of querying from the database you query from redis, whatever, right? Um, the the second portion of that question about serverless is Yes, um, it's an emerging pattern to think of your microservice And deploy your microservice with serverless to think of serverless as just a new container Right that that's like like we had we had VMs. We had containers now We have serverless, which is this new thing and we'll just package our entire microservice and deploy it as a serverless function And then it'll hit the main function and then it'll delegate to one of the routes inside the microservice Yeah, that works. I don't think there's I mean, I see some people doing that. Um, there's also a very systematic way of doing that There's something called zappa Yes, yes, but but but with docker you can still reason about so for example Let's say you want to create a request counter You can have a counter variable That you keep as a global variable that gets incremented in memory, right? Because you have you have the notion of Something persisting across invocations So in serverless, you don't have that So if your microservice is written in a way that it does not depend on the fact that it is running continuously even when there are no requests Nothing's happy. It's you you don't rely on the fact that it's running Like you don't have this notion of a global variable that you can keep or a temporary file system that you can save to And so if your microservice is written in that sense in in that style Sure, it's kind of like serverless and and there are tools emerging like zappa that help you convert a flask microservice Into a bunch of serverless functions automatically but But but serverless is not really like the functions approach is a slightly different mental model. It's like if you if you've programmed with haskell, it's like It's like you only ever have a pure function You there is you You can't do anything you there is no global variable. You can't refer to a global variable. You can't do anything It's just that one function. Everything has to happen inside that function, right? Um, and so so so that's the change that's required But you might be able to get away with wrapping your entire microservice into serverless function You first Yeah No, the problem the problem is observables will help you handle the event once you get the event That is that is how observable observables will help you. So graphql client for angular js uses Uses observables, right? But the problem here is what is the mechanism to capture events on the client? That's the problem Observables help you once you have the event. How will you get the event? How will it return how will it return stuff like imagine that order page I closed my order page I opened it again Like how will I how will it the server push data to me? That's what that's yeah So here when I created the order after I created the order I got multiple events, right? I got orders validated done orders paid done something else has happened done So so that mechanism is is the problem Yeah, yeah, and a common pattern for this is called back end for front end bff So you build a rest api that aggregates over other rest apis. Yeah, but like I mean Yeah, we mute. Yeah, like soundcloud had this bff approach But you're still building those apis with graphql the idea is that the the graphql developer doesn't Handle multiple routes. I build one graphql server and then you as a front end developer can query whatever you want I don't care and in production. We'll whitelist it in production We won't allow you to query whatever you want in development. You can query for whatever you want then we whitelist it so that Um, it's not a problem Yeah, so we have a serverless see if you have one is uh With using serverless and everything's its own function. How do you manage the? repositories for those functions and maintain it without having to have the likes etc and another one would be What have you done for concerns of like connection pooling where it's a normal thing for api servers, but serverless It's gonna have to connect every time. Yeah. Yeah. Um, great questions Uh, we have two blog posts out that talk about exactly these both of these problems. Um, I prefer the the way we solve the structuring code organization approach is I'm a big fan of mono repos And uh, and so we just have we just put all of the functions In the right kind of folders in the mono repo and then we have ci cd automation that picks up the right functions from the right places Deploys to wherever Like aws or google cloud or wherever it needs to be deployed Maybe at a different endpoint or whatever right sets up the environment variables goes updates things Uh, so that works fine. Uh, but but I I think the answer to that question really is How are you thinking about the ownership of your serverless functions the repo ownership and the business logic ownership should Should be closely linked, right? So if you think it's this case where you want to have one repo per function go for it But if you feel like no, it's I I feel like this team is managing a bunch of these serverless functions together That's fine as well for connection pooling. We did a few very interesting experiments. Um We ran aws lambda, uh, which would then connect to the which would connect to an rds instance directly Without going through a connection pool, it'll just like open up a new connection every single time This is the worst possible thing to do, but it was just to see what happens, right? Um, and so very quickly it errors out like you start making 100 requests you make 100 invocations It will fail. So the way aws encourages you to solve this problem is by creating a a connection pool variable Uh, that is outside the scope of the function and and and what ends up happening is because the because the cloud vendor is Is running those containers Is recycling those containers They actually keep that that they keep that variable Um in memory So the connection maintains across invocations of your serverless function, but you don't have any control over it So maybe you get connection pooling. Maybe you don't So the way we've solved this problem is We very frequently for rds deploy a pg bouncer Um, and then the and it just runs on an ec2 and the pg bouncer just connects to rds I don't know why rds doesn't give you a good pg bouncer by default like I think digital oceans managed database offering has a Has an inbuilt like connection pooling system But as soon as you add pg bouncer you can do we we we tried out 5000 invocations in a second on Uh on lambda which we're connecting through pg bouncer. No errors Like complete zero error rate, uh aurora. You can't connect to aurora directly aurora is the serverless thing It doesn't work. It errors out faster than rds postgres does It's it's better to just have pg bouncer. So yeah database, right simple structure This is exactly I have it but small things changes Instead of a microservices the microservices is built on grp services. You're right instead of being a cdp The api gateway i'm keeping it as grpc gateway So now I would like to use your hasura framework. Got it. How do I plug in these are options to plug in On the grpc gateway right as a frontend Um, as so your frontend is making grpc calls right now? No, okay. My frontend is going to call the rest api calls. Your frontend calls the rest api call And then your rest api gateway does grpc, right? No, no, no the grpc gateway Supports the rest api calls. Oh got it. Got it. Okay. So it's like a Proxy got it. Got it. Got it. Yeah, sure. Um, so we Yeah, so so there are there are two three ways to do this So on the frontend for example, like you do graphql the graphql query string is sent over http The graphql query string can also be sent over grpc But grpc is not a very common frontend to gateway thing But let's say you want to keep that as rest and after the gateway you want to do grpc stuff the we don't We don't do grpc yet But I think if you I think if you raise an issue on the repo we should be able to add grpc support fairly easily You can yeah, you would in if you have to use it the way you currently are using it without replacing a component You would have to keep hasura in front of grpc Uh, and yeah, you'd have to keep it in front of grpc But the ideal way of doing it is for hasura to support grpc Correct, which is what you should not do and which is why you what you should do is open an issue and say support grpc And we'll support grp. That's the that's the right way to do it Thank you, uh, how do you deal with the authorization right? Um So we took inspiration from postgres's low-level security um postgres rls, uh, we re-implemented Yeah, we re-implemented postgres rls at the api layer So you can give declarative configurations on top of your models To specify what access people have so for example, I can go to the profile table and say permissions uh user can only access data where the id is equal to Some session user id right i'm doing all of this for the ui But you can do this declaratively as well like whatever i do through ui is just a is a declarative setup as well And then you kind of say whether you want access to these columns you want to limit Um, you want to rate limit like the number of responses that you get Whatever size limit the responses that you get. Um, so that's kind of one way of doing it. Um But this is basically what the api layer does is that whenever you get a header which contains an authorization header or a cookie um Like our system resolves that and converts that into session variables like user id and role And then the permission rules Uh refer to these variables So the sequel that we create instead of when you query for query profile id name We convert it to select id comma name from from profile where id equal to cookie dot user id So that's the that's the compiled sequel. Uh, it works Any other questions? All right. Well, oh, sorry. We got one more I always think they're all done There's one left Did you have any problems with Start times for the serverless like For serverless? Yes. Um, yes It's uh, so so yes and no so if you have consistent load Like you all you right. Let's say you have a website and you're just always getting Uh, 10 api calls every instant Then this is not a problem because the cloud vendors keep it warm for you But if you don't have sustained traffic and it's spiky Then what ends up happening is that it takes time for the first it it can take They don't even give you a guarantee the cloud vendor tells you it will take Maybe 200 milliseconds, but it can actually be anything. There is no sla on how much time it takes to start Uh, in our case, this is not a problem because it's async anyway The serverless function is running asynchronously anyway, right like something the the api layer the graphql api layer is not serverless The graphql api layer is a container which is one to an autoscale not zero to an autoscale Um, so so that is running always and the event that triggers serverless That can the serverless function can start whenever it wants. It doesn't matter because whenever it asynchronously happens Uh, I will get the update on my ui because it's asynchronous. I don't actually care about there's no hard limit on The user experience is not made better or worse. Does that make sense? There is no gateway here, right? I call the server that first request ends. I create the order That request is stopped. It's ended. I create the order. I get order ID Done now. I open up a web socket to say subscribe order ID So that first request is ended within like 10 milliseconds It's create order ID. I instantly get the response back and now it's stopped. Does that make sense? We're at the hour so i'm gonna cut it off there and let our speaker get away. Thank you very much Uh, okay presentation and for answering all our questions. Thank you. Thank you See you folks at game night Hello everyone, welcome to the developer track. We'll get started here with our next talk We've got alexander here talking about fast htp string processing Without further ado, I'll let him get started and get you guys into it and we'll talk We'll have questions at the end. I'll come around and answer your questions if you raise your hand. Thank you Okay, thank you for introduction Good evening. See huckers So let me start from introduction who are my what we do and so on so we develop custom software in High performance network in area more than 10 years and usually we Uh, the area includes security applications like the application for those VPNs Whatever usually we develop the stuff in the linux kernel and provide some Performs optimization of existing solutions and so on one example of Such project which we proud of is positive Technologies application firewall. The project was mentioned by gardener magic quadrant in 2015 in visual nerve And also we develop tempest f double u this project which I will be talking more about this presentation And this application delivery controller open source, of course, so it's embedded into linux cpip stack And if we talk about application delivery controller, then usually this Aggregation of hp accelerator and firewall which provides you efficient load balancing Web cache Whatever you need from hp accelerator Plus additional filtration of d-dose attacks Web securities and so on In the case of tempest tempesta, it's since it's built into linux cpip stack. It's very fast. It's comparable with kernel bypass Approaches and at the same time it's very flexible. You can manage it Just like you manage Apache nginx Whatever So if we compare tempest f double u with nginx, we typically see Two to three times better performance and if we compare it with sr star an example of dptk based Web server, we show very similar numbers for For cpu cores just begin of the graph We started to work on tempest f w when we started To work on positive technologies application firewall that time we had to develop web application firewall this Kind of application firewall which works on hp layer and provides you web security filters out as much as possible web Attacks attacks also It protects from Attacks from on application layer and usually such applications are built on top of usual Hp proxies h a proxy nginx, whatever and that time we realized that usual Hpx Proxies aren't good for Fit creation logic. There could be perfect for Application logic for example to deliver content to manage it to mangle it or whatever you need But not for security aspects And that time we started tempest f w W inspired from net filter. So net filter is a kernel firewall working in cp ip stack So we just made a continuation of cp ip stack to htp and now we Have tempest f w. I want to make a disclaimer that I will talk A lot about nginx during my presentation and the reason Why I choose only exactly nginx is that nginx is very simple and everybody knows it So it's just a good reference for Presentation during the presentation. I will concentrate on hpm processing especially crucial for high performance hpm processing and security aspects I will Consider two main questions. The first one is Hp state machine efficient state machine and Hp strings processing and You might Wondering why do we talk about hp strings if we have hp2 after all hp 1.1 is about strings And hp2 is mostly about binary data However, if I would start from the last reason actually The state of the art that hp2 was Adopted in most hp servers in addition to existing hp 1.1. It means that we have a hp 1.1 server and we have a special decoder for hp2 and when we decode hp2 then we pass the Decoded data into standard hp 1.1 Parse this just how how the most server behave. The second thing is that we have Dynamic tables compression in hp2. However, they are limited in size and some of the headers Can easily outsize Dynamic tables the example of the headers cookie uli and Referee and We might note that this Hp fields are very crucial for security So let's have a look at dummy hp float against the nginx Access log switched off. We see hp parser as Hot spot in the profile and I want to mention that in this exactly Profile we see that the profile is flat. It means that if we optimize the bottlenecks then we want to get dramatic performance improvement in this case So, uh, how usual hp Parsers work typically we have a loop over ingress data and we have Couple of switch statement the first one is for states the second one for Input characters. So we have a auxiliary variable state. We have an ingress Character b we start from the loop and first statement. We pass to current state and current Character, so we typically made a jump And probably we get the instruction cache miss here next we go to the end of the loop and we repeat the Next cycle with new state and here we go. We just We made made the spin, but we actually Just moved down for the next state So ideally we should Get what we have on the Right, but actually we are just spinning all over the loop. That's not good Uh, if we look at the request line parsing in engine x, this is a snippet from real code and There are a couple of notes about how engine x works the first of all it works with copied network i o It means that all the data of the requests are in memory and if we process Get method then it means that the method is fully in realm We don't need to bother with various data chunks and Now engine x uses the knowledge it spins over the data character by character calculates the length of current token then after that Runs the switch on the length of that token This we now we're going to see what gcc optimizes for such state machines. So when we have a Uh switch statement we have a true Uh possibilities how the statement is organized. The first one is on the left Is that we have a enum of sequential numbers from 0 to 26 And in this case we instead of scanning Case statements in the inside the switch one by one. Uh, we can run a look up Table. So as a middle of assembly snippet, we have a look up table. It's just a sequential array of pointers and With the first jump we just Jump by address of From the look up table. So with two memory Deferences we get to the required state the second Possibility for switch statement is uh not Consequent states. So in this enum we have various paths States and now we cannot organize look up table because all the Offsets and the tables are very different and the Hydres index in the table is quite too large So we fall back to binary search and gcc compiles your Switch statement into binary search over the states Next let's have a look at nginx Parse typically it's about nine kilobytes of size It's like Bit more than three times less than cpu layer one Cash size and it In total it implements only a bit more than 80 states nginx path is very simple This because it needs to do only basic Technization and do not care about strict hp validation but For security reasons we need to know What the headers are we need to understand which headers are passed to backend servers and so on so We need to provide much more sophisticated HP parses to carefully analyze what the headers are so in tempester currently we have more than 500 states and Timpester uses zero copying ior it means that all the Functions about ior aren't Aren't so important in profile. So Large and Accurate hp parses becomes the even more Strong bottleneck of the system And the bad thing about Zero copy ior is not Is that currently we need to bother with data chunks we can get HP methods separated in a couple of chunks so we cannot compute the size of the Talking so We started to do the same as regular does we just use direct Direct labels and jumps plus two switch statements. So the corner The the main point of this Technique is state a definition. We have a case statement and we have a direct label and now we Execute a switch statement only the first entrance to the function And the bad thing about this Technique is that now if in engine x case we have a four or while loop and we have We have a program counter. We have a data counter inside the Loop header Now we need to move all the logic inside the move marker and Now the state machine executes the code on each state. So typically such state machines generate much larger code. However, just here on optimization purpose does Something similar. So if we look at the assembly code of state machine with 20 or 30 states We might see that about five or six states. We will have the same logic. I mean increment of data pointers to check the bounds and so on Next thing which we did is to use gc extension labels as values and now we remember labels address in parser And now we can remove switch statement at all. So we don't execute any lookup tables or binary search and now we use the auxiliary variable only once when we start to pass HP request. If we have several data chunks of the request then we still use the auxiliary variable only once Let's have a look at the benchmark. I just want to mention that in the benchmark we usually set Task to a particular CPU. We choose the smallest numbers because we need to remove all the system overheads and so on So in states in state machine with very small number of states The approaches are almost indistinguishable. However, with a lot of states we see dramatic performance improvement two or more times If we will look at branch misses in such state machine, we see that While switch statement state machine is two times less than the go-to-driven machine. It still has much more branch misses and it's Very very interesting that While code of state machine is almost 1000 bytes of code is much larger than layer one CPU instruction cache. We still see twice less number of instruction cache misses. This is because We do less jumps and we decrease probability of cache miss even in larger state machine The next point which we observe it in the state machine is that if you Program your state machine then you place your most probable states and the first states at the top of the machine And you expect that the less probable states you can place at the bottom of state machine And they will be executed later or maybe not prefetched to the cache at all However, gc doesn't have such knowledge about the state's probability and it Free to move states around the code and we observe that the less probable states Became at the top of the function while our the most crucial states Move it to the end of the function. That's not good. So if we use compiler barrier, we can force gc to keep the states as way they are and just Addition of the compiler barrier to state definition improves performance for four persons Next we consider stronger optimization compiler optimization and surprisingly at stronger optimization where we see lower performance and If we look more carefully at the what was the Optimization switches just we find that some of them Actually makes the performance worse and if we carefully choose the right switches, then we can get the more performance the green the green switches are auto-vectalization Options you can by default Switch tone on a higher layer of optimization and unfortunately not all of them Can be done and all the code for example in our hp state machine auto vectorization doesn't Do anything at all. So they are simply no optimization On the compilation phase this bit outdated gc tutorial actually Gc can optimize and vectorize much more sophisticated loops than this depicted on the slide But in our case it just doesn't work at all Next thing Intimpesta and nginx we do the same trick So if we have Some request methods Same get of two four characters Then instead of matching the characters one by one we just Compile We just Make the integer pointer a compare two integers. However, since we have no Have no idea which data came to the hp parser and we have no idea about its alignment We can hit CPU pinarty on misaligned word reading and this another Mica by benchmark and now we see that misaligned integer access Can worse performance in two times And it's simply it can be simply fixed if we just made First check with the address aligned to four bytes if yes, then we Treated as integer pointer. If not that we read characters by characters very straightforward However, if we try to employ the optimization technique in real life parser, we might Find that Performance results are becoming even worse. What's the problem? In the benchmark To measure the speed of which Memory read carefully we had to keep Benchmark code in one compilation unit and the function which Defends memory in another compilation unit. So now compiler cannot Cannot make the tricks about reading the memory and have to execute exactly that code Which it doesn't doesn't know that doesn't have knowledge about However, in real life parser, we don't need to keep such lightweight function as parity and just inline them into the parser and now the Gcc have knowledge about optimization of the matching routings and In that case we Can probably generate just different code that's not code. There's nothing special about the code. There are not more Significantly more branch misses. There are no more cache misses The code from a profile point of view just looks About just just as the another version. The thing is that assembly code generated by gc is Much more different and gc is just confused by more complex Statements in if branches and generates less optimal code. You can find more discussion about the topic in our issue on github So we end up with state machines and now we come to hps rings and Let's first observe why HP strings are important in hp parsers First of all, this is a usual URI. This is my Almost my URI from booking com when I tried to book a hotel in pasadena And if you check a lot of options on booking com, you easily get the URI of almost one kilobyte And on the right we have a state machine of URI parsing from nginx And surprisingly we see that This very long switch statement parsing character by character URI and we just saw that the case Switch based Automatons aren't so quick So if you have a slow logic from one side and you have a very efficient data to Make To give a hard time for the logic. So why don't not to use the data in your DDoS? DDoS attack so this If if you want to DDoS is not wise to use a simple index request instead of a simple request. You can just use large URI to To make The parser work hard and the large URI also efficiently load your backend logic. So you can just efficiently Nail the target server by DDoS with more complex URIs and the second thing is that in TPS we Search for injection attacks in URIs. So the second Example is injection in URIs. It's also not so long but still Also not so small URI which we can must carefully analyze and to Be sure that The problem is really important. We can just put a lot of more or less real-life requests to the Server again this access lock is switched off and we can easily see that both of the nginx Parse loaders are on the top and in this hp flute we Use very complex requests and now the profile isn't flat. So we have a very clear bottleneck in the hp processing What we can do with hp strings? First of all hp strings are very different In comparison with common c-strings. They do not end with Zero byte instead they end with various Dillimeters it could be one byte delimiters two byte delimiters and even one delimiter CRLF can be two byte or it can be one byte left only and second Think that we must carefully check A load alphabets in LFC a load strings for security purpose And the standard stale spn function provided by glipc Does very poor job on scanning the accepted Characters and the second interesting function, which is good to optimize is stale comparison And in in just in many places in modern hp passes We see special cases when you need to compare ingress string with some statically defined String in the parser. So the second string should not be converted in case And some hp passes we observed that Switch statements are used just as we just saw for nginx gliparser And we know that switch driving fine state machines aren't fast at all Um, let's make a little study about what the people do in this area. So Uh, nginx provides the most straightforward logic just character by character Processing it does very careful analyzing what which characters are lowered and not and pico parser library hpm From h to all seven Can Eat by 16 characters at once and otherwise them for against a load alphabets Cloudflare made extension for pico hp parser. So now they can eat by 32 Characters at once and They used avx true extension for that This example high psem starai Instruction works. It might be hard to read The functions on the slide This In tree hgc in tree index. So I select with both the most important pieces of the code. So first of all, we place our allowed alphabet in static Static memory we define it as cons. We provide alignment For the most efficient reference to the memory and the bad thing about PCI PCM see a starai instructions that It can treat by eight Ranges or 16 characters at once And if you have a range for example as the second range And in the range we have only one character. Then we have to spend the whole range to Declare only one character range So that's not comfortable. That doesn't suit all the alphabets in rfc So Cloudflare used a more simple approach. It's not so Precise as pico parser approach, but it's very straightforward. We just Check two bounds of the Allowed variable and also add a tab as one more Allowed character. The code is straightforward Simple and fast. This is a comparison of the two parsers. We see that cloudflare version Two times faster on large data, but Still somewhat slower on smaller data. This is because AVX Instruction set is more heavyweight than PCM see a starai I used the upper limit as one 0.25 Of kilobytes because this is the highest This is the largest Data chunk which can come to our zero-copy Processor this upper limit for is a net frame So if we compare the The string processors with stlspn, we see that stlspn is incredibly slow. So it's not good to use it And now we compare with our processor Our string matching is faster than Than both the Than both the algorithms on large and small strings and has very Very minor increase of performance in With the growing of the increased data. So The algorithm works in several stages. So the first stage is a function prologue when we process the small strings So we do exactly the same as nginx. We define the static table uri for load characters and we run very simple dummy Switch statement over four characters not more the limit for the Four characters is exactly the last statement When we have massive branching So if we add more characters, then we will slow down the last statement. And also we use static branch prediction likely statement to make the Court as Close to the entry point of the function as possible. This is because small strings are more sensitive to branch mis predictions The main Main loop body is This one we have free course of basically the same logic And we since avx Instruction set works with 32 bytes. We actually start from 128 bytes of processing The thing is Inside the function we execute the same code in parallel in four data streams This this allows the processor to use the data quickly and Execute code in parallel So after the main loop we have a Function epilogue this just very close to prologue. Now we since we Finish with 16 byte strings. We have to process a String with tails less than 16 bytes Just the same technique And here we go. This is the main Main description of the algorithm. We use a skid table representation, which depicts it on the Figure this representation uses eight columns 16 Rows and we start from loading Static table What we did here in the your IBM Table we We describe a ski Lines A ski table line by line So the first line is encoded as b8 We We read the ask here first ask here ask here low in In reverse order. So we start from p little p big At sign and zero and the third Row is Small q r big b big and two. So it's essentially f h Um Number so this how we encode the ask him lines And now we execute shuffle of the ask him Constant table in accordance to Ingress data. So if we have data s pr Then on the first position we Uh have Characters from the first line of a ski tables. So we good. We leave the Bitmask on the same position zero's position However arm is from third line of a ski tables and We need to move uh or shuffle the bit mask for field for field A ski line to the second position where our our character Is placed. So essentially now we just arrange a ski Lines and we say that uh at the first position of this link We have a character from the first line of the ski tables as the second position We have a character from the third Line of a ski tables. It's that's straightforward Next we want to do with the same with a ski table columns and arrange them in accordance with Ingress data to get the A ski columns. We need to do a bit more operation. We need to shift Characters for form bits left and split them. However in Our evx and sc extension. We have a shift only for two bytes. So we need to add Additional and instruction to clear the Move it Less significant bits. So by the end and Left shift we have from our PRR symbols. We have 0707 and it means that In first and the second Position we have a characters from the very last column of a ski tables. This is seven and we May do the shuffle and place the This column notification Define to the first and second places Now we the things is trivial We just perform and or over the two vectors and define which characters are lowered and which not Next we count zeros and return the how many bars Good for us the good thing about this approach is that um Since we used dynamic In previous slides we used statically define a tables of encoded Characters which allow it for our ski tables and now we can dynamically define In the table which characters are lowered for our ui which characters are lowered in our user agent or any different hp header and the resulting algorithm with dynamic Tables instead of static tables will work with very close speed So these are two examples of attacks. The first one is from black hat talk. We The injection uses At sign character is probably not the most Most common character in your eyes So if you don't need to handle the characters in your application, you can just Define this range for your eye and exclude the character the same you can do for user agent I was surprised that it's It's handy to launch attack in even user agent Could be static header, but nevertheless there are attacks in such headers and Dollar dollar sin is isn't a common character in user agent. So we can exclude it as well So we can define dynamic tables which characters are lowered and not So the last one is string comparison. This is very straightforward and very simple. We just Do Two things Three things the first thing is we convert Case of the characters in only one string. So in all the characters we compare In first vector we compare each character with a and that Constance and if the character is in from from the range then we convert the case We use a couple of tricks from hackers delight book to Reduce number of comparison in first two lines and also we had to subtract H zero from both the sides to Move from sign it sign it come from unsigned comparison to sign it So, however, the idea of the things are very very simple. Since we once we Determined which characters must be converted to different case. We just execute and operation with cast case Conversion constants and next we just do all with our original string. We get the string with Converted case and now we can compare it with the second string also not so much operations very simple And runs quickly It was mentioning that lips here also use I think this 2.18 or 2.19 version of lips here. I didn't check the later versions But that version of lips here Uses avx instruction set but In avx 2 we can get more speed and the results are on the slide The last topic so we end up with the parsers and the last topic which I want to discuss is FPU context switch the thing is that When we use Single instruction multiple data instructions in your programs or just compile arbitrary CIP program with highest level of optimization and compiler uses auto vectorization Then you use Registers from FPU unit. So the registers for single instruction multiple data lives in a different unit and The thing is that when you switch from user space to kernel and back from kernel to user space You don't spend time on saving and restoring the Single instruction multiple data registers. However, if you switch from one user space program to another you have to save and restore single instruction multiple data registers so Basically, kernel doesn't use Single instruction multiple data. However, if you need to Use the extension like crypto or some other Kernel code you can explicitly Save and restore State of FPU this is done by these two calls And since tempest works in Linux HP IP stack we care about the stuff and to Reduce the effect of context switch. We save Single instruction multiple data register when software queue begins and HP IP stack starts working and we restore the state when we Finish our job and let the user space to run further and just to Think just to see how how the Single instruction multiple data concept switches are built. There's another micro benchmark. It shows dramatic performance problems. However, as we just saw for Programmer alignment This Micro benchmarks aren't Do not show the problem for Full program. So if we are very bad at micro benchmark, it doesn't mean that you see the huge Penalties on context switch for your user space program. However, being in kernel is good that we Can eliminate the problem of FPU context switches As much as possible So I'm basically done this useful links. The first two links is my blog post about the topics which I discussed in this presentation Relatively old It's also good to read Cloudflare work and hackers delight. It's always good and useful So, uh, thank you Questions and we'll be Happy to see you on our github. So thank you Do we have any questions? Yeah, just one quick question. I don't really I haven't really understood you say that For this finish that machine Gc can sometimes we order most popular most important values with least important And you said that you eliminate this problem with just the memory barrier, but why memory barrier is just enough here? Um, one moment. You're about this slide, right? Yeah, um, look at the left we have Basically, this is a marker language. So so we have a state and the next each The state well Considered that we have a states processing Get header post header and the last state to process unlock Methods character by character. So when we enter the state we Since get and post request are the most frequent in Real-life traffic. We want them to have as Closer to entry point of the function as possible That's that's clear. However gcc rearranges If we use direct labels gc is free to rearrange the code as possible and Compiler barrier is Exactly the fin cut which prevents Compiler from rearranging code across the barrier So if we have a barrier we say to the compiler's precities that please leave the code or Both the barrier at the both the barrier and which is after the barrier It must be leave after the barrier a compiler is free to rearrange the code above the barrier But not cause Yeah, I actually actually I always have been using the barrier in different semantics But this one which we figured it out for state machines. It's very very interesting Just a question for me actually a memory barrier. There's a little bit costly. Is it is it possible to Use profile guide and optimizations to get gcc to recognize that the get and the post should be first sort of cheat This is not a memory barrier this compiler barrier Maybe I didn't get your question So you're right. It's a very different thing. It's not that expensive. I see a good point Yeah, the question Well, cool. Well, thank you very much. That was quite an awesome. Thank you guys. Thank you for coming That was great. Thank you guys very much And this ends the developer track for today Yeah, of course