 Yeah, thanks for the introduction Happy to get a full house today. So yeah, I'm Thomas I work at local stack where I mostly drive the engineering and product development efforts and I was also involved in a lot of Re-architecting of the core code base when we move this from a project to a real company So local stack is many things. We like to say it's a platform for cloud application development and testing But at the core that makes all of this possible is an open-source AWS emulator. That's On github that you can find So maybe just a quick show of hands like who is working professionally with AWS Okay, pretty pretty big crowd So then local stack is maybe for you because local stack is also a drop-in replacement for AWS that runs on your local machine So you can grab your terraform file or whatever it is you're using You can sorry this a bit small the code will be bigger later It's been a local stack. It runs in a Docker container and then you can terraform apply And you can start pointing all of your other AWS tooling to local stack So we're like just running AWS C like commands against this infrastructure that we deployed which is a bucket and a queue and event notification So it seems the integrates with a lot of the AWS tooling and That was a simple example But if you go to the developer hub you can find like big application stacks serverless stacks with 15 services Big range of services that we support So how do we how do we do it? Like it's pretty outrageous idea, you know, we go and rebuild AWS. It's like this huge System and there are a few strategies to have we're like one is we use Python and Python is a great language for for a system like local stack because we bring in so many third-party System tools that we need to integrate And maybe not everyone from the team would agree. It's like okay dynamic typing global state pollution blah, blah, blah But in their heart, they know it's it's the right choice So if you go to the Docker documentation and you work with the SDK like this is the amount of go code You need to just run programmatically Docker run and we need to do this a lot and this is the Python code So, of course, I'm gonna choose fight like I need all my extra time to read the AWS documentation to figure out how this piece of service works But in seriousness though, so Python allows us to reduce complexity. It just allows us to get things done really really quickly Reducing complexity also is is like a big topic. So everything runs locally which allows us to do make a lot of assumptions So AWS, you know, it's a huge distributed system. It runs on huge data centers We don't have that we run in your docker container and everything's local So there are much fewer distributed systems challenges involved. There are some and We can build some of the core services in like really simply so SQS for example, which is managed queuing service It's a pure Python implementation. It has 2000 lines of code very simple model You know, there's an API that creates a new SQS queue and that's it We leverage a lot of the open source AWS ecosystem. Luckily, AWS is giving us more and more We standardize all of the repeatable parts. So service emulators and I'll talk about this which is basically our implementations of the AWS services We've started standardizing be very systematic about development testing and maintenance of the services and We really embrace plugability and I'll get into the details of what that means afterwards So I want to spend some time talking about service emulators because that's the bulk of our work and what the team spends most of the time on and I want to first start off talking about how it used to be. So before we Found the company and we actually got a team involved. This is how things look like So when you point your AWS tooling to local stack, let's say you have your AWS CLI You say Kinesis put record Kinesis is just a managed streaming service. Not very important Your client makes an HTTP request and it sends it to web server that we expose we call it the edge proxy and historically local stack was just a wrapper around third-party tools So we would bring in a third-party system in the case of Kinesis. There was a thing called Kinesi light. I think it's still around it's a Emulation for Kinesis someone just went and reimplemented Kinesis so we can sort of take that and unify it under our API This is again, this is how things used to be and then we would have just a proxy listener around that and Now the the code begins because this is like interesting I think to understand from a code perspective what you had to do is there was a simple interface It's called forward request and you got the HTTP request parameters. That's it That's what you got and then we were able to sort of instrument the request before we sent them to the back end so for instance, we would have a feature that Would allow us to configure a certain amount or a fraction of requests being returned returning an error so chaos testing for instance very useful for that and That we could layer that on top of the third-party back ends So that was very useful and we had a bunch of those and now problems Started appearing. We had more and more services. We needed to start it We needed to integrate the services together. So the services typically don't live in isolation They work with each other and in the case of SQS for example, there's a lot of code don't worry about too much about it But down here, you know at the at the delete queue action We actually had to go to SNS Which is the AWS publish subscribe system and actually unregister the queue after we deleted it and you can see actually Okay, now we need to parse the request. That's parsing. AWS request is non-trivial We needed to you know We had this big if else blocks with the different actions to do different things and it just grew and grew and grew and No, we we had a big uptake in contributors. So this is the this the contributor graph in the open source repository we had increasing number of code commits and when we started in Summer of 21 with the company. We had six engineers and now we're almost 30 So we needed a way to sort of systematize all of this So how it looks now We leverage again as I said a lot of open source tools from AWS. They make it easy for us sometimes one thing that's We use a lot are the specifications from the services. I don't know if you know this but all of the service specifications are open source They are described in a language that they came up with so it's called Smithy and you can see for instance the Amazon SQS Specification here with your operations and the input shapes for the operations and so forth. So essentially AWS is just a very complicated RPC system and We use a photo with photo core, which is the AWS SDK for Python And they actually have concepts now that we can use so they have a class called service model and operation model Which allows us to just parse this and reason over it So we can just load the SQS service and then programmatically iterate over all of the operations So that's very useful because now we can go and write code generators So we have just this method basically looks like this. It's generate code. We load the service We generate we basically expand the the object graph from the service model We generate all of the types and then we generate the API and then we use this awesome Python tooling for Formatting code out of lake eyesort and black and we literally just have a bunch of print statements into this output buffer And then we format it with these tools and outcomes like nice code. So this this is what it looks like So you can go and scaffold lambda for instance It generates you can see the the types in the preamble You can go and search for the for the API. So you have a class here, which are all of the API stops and you can search for the create function and see all of the parameters for instance Okay, and this this we can now use for several things. So one of the things we do for maintenance is we get weekly pull requests that Give us the updates of the API set have occurred in the last week So we we regenerate all of the APIs and we just do a pull request and this is how we keep track of what's going on We can't jump on all of the requests or sorry all of the new API methods obviously we have a small team But at least this gives us a way to kind of keep track of what's going on and stay on top of things So in total AWS has currently like 350 service specs With in total almost 14,000 API operations and that all amounts to 80 megs of Jason specs We actually like Jason specification is a big bulk of just the container image Which is kind of silly, but we need it and in local seg we currently have 93 services implemented and and growing so you can see that there's just a lot of code necessary to be maintained And I'll go into a bit of the strategies that we used to do that so with the With this vast, you know number of services that and all of the code generation That's nice But if you go and try to generate easy to for instance, which is the one of the oldest services and the infrastructure is code You wait a while and then you get a file with almost 24,000 lines of code And 600 API operations and then you look at this and you're like, okay, what now? So we kind of need to choose what to work on we do a lot of things reactively So when people ask for certain operations, they need for their stack we go and implement them and we kind of choose the ones We we know from from talking to people and looking at usage analytics What we need to focus on we pick those and the rest we do we catalog and do reactively But like this system is now much cleaner than before because we have like a structure We have these API stops that someone can go and implement and the emulate the code oftentimes is just pure Python code But very often also we need to bring in third-party systems Which is also why I can't go into the details of how the services work because there are 93 of them and they all look different But they're unified under this sort of API So services sometimes either use back-end services use third-party services like in the case of kinesi light that I that I explained earlier Where we really use the third-party tool to emulate the service or oftentimes? It's just bringing up open-source Systems like that is what AWS does all the time they take your open-source project and then they sell it as a service So elastic cash is just ready as a service. So we just spin up our redis instance for you and We have our own package manager abstraction for this We actually load a lot of those systems lazily because the image would be huge if we package all of these Third-party services from 93 services into the Docker image So we do a lot of that lazily and Python gives us a way of easily sort of maintaining our own packages And maybe looking into one concrete example to give you an idea is a lambda is a very interesting Example but AWS generally is a lot of container technologies So think of ECS EKS those are all like container technologies So we try and use a lot of what AWS gives us here and in the case of lambda For instance when you go and you know Lambda invoke with the AWS CLI and gets routed through local stack to a lambda provider And it does some housekeeping of the functions that are in there And it actually uses our internal S3 provider to store your code, which is what? AWS actually does it works in the same way And now we use the Docker sockets to instantiate new containers The containers are based on the lambda base images that AWS provides So we used to use a very old kind of unmaintained set of community images that were great for a while But then they got abandoned and now it's it. We're much more close Closely coupled to AWS, which is good because this gives us what we call the parity, which is you know behaving like AWS And we use the runtime interface emulator Which basically just takes an HTTP request and then makes a lambda invocation out of that So that is actually a pretty tricky piece of code that they just give us which is nice So there's a lot of you know complexity involved in some of these services and you can kind of think about you know Like think of 93 different diagrams at this that look completely different So maybe what's more interesting to focus on also for the Python community is sort of the This this API wrapper around it and how it works So I want to spend some time talking about the web application framework that we built for local stack We serve what we call the gateway which was previously this edge proxy Which is the thing that receives your request. It's just a web web application server And we serve that through hypercorn hypercorn is an async IO HTTP server that supports HTTP to which frankly pretty hard to find the pure Python implementation of an HTTP to server That does what we want But because our framework is built on Verkzeug, which is the library that also fuels flask Which is all synchronous code. We need like an ASCII whiskey bridge. So we had these conflicting Requirements that we solve just by building our own bridge. I think Django also has something like this because they have a similar problem And then we have our own that's when our framework begins we have a thing called the handler chain so a handler chain is just a set of request and response handlers and It's an implementation of the chain of responsibility pattern So if you've ever used like Java X server API, for instance Then this may seem familiar to you because that's essentially an implementation of this So we call our handler chain with the request context the request context just holds the Verkzeug request that comes in from from the web server and all of the additional metadata for the sort of to know which service the request should go to and The handler chain again holds all of these different requests and response handlers and they enrich the context as the request goes through this handler chain The the request context gets enriched with all of these with all of the metadata for the AWS request So this is a this turned out to be really really useful and I'll show you in a second why So this is one of the one of the handler implementations. It's this what we call the service name parser So based on the request that's coming in we need to route it to the correct service and turns out this non-trivial So this determinator we have service name I think has almost thousand lines of code if you're interested in how that works the author sits there And then we just get the you know We load the service model now that we know which service it is and we attach it to the context And then we pass it to the next handler which is you know could be the service request parser For instance, so it only does things when something's in the context And then we know which parser to pick and then we can parse the request and attach that to the request context So this is very similar to whiskey middlewares only that it's It's it's more sort of in Imperative rather than the way that the middlewares work, which is kind of wrapping things like an onion So I'm not going to go through this But this is sort of an illustration of the entire handler chain of what's going on so you can see it's pretty involved But if you just focus like on the on the left hand side like even just taking a glance at this and Looking at the list of request handlers You can kind of just by reading it kind of understand what's going on So, you know this push request context par service name And fourth course and things like that So this gives us a way to kind of have one place where we put all of the logic so this you're looking at the guts of local stack here basically, okay, and One thing that we had to do a challenge that we had to solve is migrating all of these old Service implementations that we're using this old proxy listener interface to our new handler chain framework And the adapter pattern is Like the perfect tool for this and I want to just briefly explain how we did this So adapter is just you wrap something make it look like something else and in this case We when we started the migration we started playing around with this new framework with the handler chain Like we did still didn't have the the internals quite figured out, but we were able to serve these new service implementations through our old framework by just making this handler chain look like a proxy listener You remember the interface from before it's just forward request. So it's the same interface. We had to do a bit of sort of Emulation of this new handler chain within that method and then like as the the migration progressed It took around a year. I think to get all of the services migrated There was a sort of a tipping point where we had more new services than than the old ones and Then we put this new web framework in place that I just presented And then we had some legacy services that we need to serve through this old through the new abstraction So now we have like the adapter in the other direction So we we serve this old proxy listener interface through the new handler interface I'm not going to go into the code But it's just to give you an idea that you can you know migrating frameworks in this way with like the Simple adapter pattern is just it was great. So is the it is it was very very useful pattern okay, so one thing that I said in the beginning is we embrace a Plugability and you can you could already see some of that You know you can we have this pluggable handle chain where you can just inject handlers and kind of change the flow of the Of the requests as it comes in But also a big problem that we had in the beginning was Importing you don't worry too much about the the code is just a bunch of import statements Which was quite frankly terrible because we we had to import all of The service code and we just had this one sort of registry where we just import the code necessary to start the service and Because transitively it loaded Quite some generated coded just this import statement took about five seconds So starting local stack was pretty slow in the beginning There's obviously no way to discover unknown packages and I'll get into that in a second and You know all of the other obvious problems with this So one thing to understand how local stack is distributed and how it's structured We have local stack core which is the open source tool Which contains like all of the guts of local stack that are presenting and a good a fair amount of services are open source lambda for instance open source and Then we have like the stuff we make money with is is in a Close-source repository and we also have an extension system where you can modify local stack and plug into specific APIs For example this handler chain you can extend So we had and and we need to separate this into different Python modules So we bring everything together and there was a problem of just discoverability We had to discover the code in the different distributions Sorry Yeah, this is what happens There we go, so we built the system. It's called plugs It's basically a dynamic code loading framework also open source and it solved the problem that other plug-in systems have Which is discoverability of entry points. So entry points, maybe a quick show of hands. Who knows about Python entry points? Okay, very few people so it's basically a way for your Python distribution to expose Code to other distributions and a lot of systems use this you may not know about it, but setup tools for instance actually uses Uses entry points itself to advertise additional commands in your setup line so we can we have a bunch of code that's Structured as plugins. So I'm this is a this is an example of how you can use plugs Plugs provides different interfaces to define plugins So you can just have a class called plug-in that inherits from this base plug-in class and it works in the same way But this abstraction allows you to do things like function plugins, which we use a lot in in local stack This is just an illustration, but you could expose Configurator functions for instance as plugins So a plug-in just has a namespace and a name in this particular example We use the function name as the plug-in name and now You have the all of the ingredients that you need for an entry point Which is the left-hand side is that you have the namespace in the top and then the left-hand side is the name and the right-hand side points to some piece of code and Big problem of the things like plug-E and and other plug-in frameworks is that you need to either they they're based on conventions So the frameworks know where the code lives in PyTest for instance You've probably written like a conf test pie or something This is just a convention and PyTest knows where to look up your plugins there But we wanted to give developers freedom how to organize their code and where to put it So we couldn't really rely on conventions, but we also couldn't have this big registry anymore of you know maintaining entry points So what plucks does is it discovers these entry points from your code? So there's a build step involved now, which is the drawback But you can just write your plug-in Regenerate the entry points, which is done automatically in our build chain And then you we distribute the entry points from that and then at runtime You just have this plug-in manager class You set the namespace that you want to get the plugins from and then you just load them So in this case, you know these configurator plugins they Configure a runtime and then you pass that runtime to the configurator functions And you can do all sorts of stuff with this and we use this extensively in fact our whole extension system is another layer around this and it allows you to In to build third-party Python distributions put them on pi pi and install it into the container and it's That's a little bit involved, but if you're interested in that come talk to me afterwards It's a it's a great way to sort of put additional functionality into local stack Okay, I said like plug-ability We need to pull in a lot of third-party tools one thing that I wanted to talk about when just running out of time is Monkey patching so actually Baltimore was also here in the audience gave a talk about local stack last year At Euro Python it was more oriented around how to use local stack But there was a piece that a bit of the talk goes into runtime cold patching why we need that We again we bring in a lot of third-party tools and what makes Python so nice. This is something I appreciate very much It's just openness of the software. It's so easy to look into third-party software And also to patch it like no need for bytecode instrumentation or something You just you know replace attributes and then you're good to go and we do that a lot There are some problems with this obviously and again, you can find a little bit more details in this talk Okay, so much for plug-ability. So local stack is a pluggable system in that it's it and you can plug into local stack itself But also we plug into other systems a lot and Python helps us do that So the last thing I want to talk about is parity. I mentioned parity in the beginning It's it's what we say when we mean that local stack behaves like AWS it needs to because it's it's supposed to be a drop-in replacement And there are certain strategies we have in place to maintain parity One is this you know auto generating the APIs and getting full requests and things like that But this is just specification based parity is all about behavior based Matching so this is such an important concept in fact that we have a blog post just about this So you can find that in our engineering blog But just like a brief excerpt So a thing that we built around pie test and we if you go to the local stack test code base You can see almost every feature of pie test being used there It's such an amazing framework and it does so much for us And and we build a traditional tooling around this. So here you can see what we call a snapshot test a snapshot test is it it runs against a real cloud provider and then it collects the responses from that cloud provider and puts it into a JSON file so we do some like these transformers you see in the beginning They're responsible for replacing things like auto-generated IDs and and dates and things like that from the actual response because we can't match those where when we randomly generate ideas like Ideas we can't match anything. So we have a way to replace those But when for example generated IDs are referenced multiple times in the response We check that so that's important And you can see a bunch of fixtures being used here So the lambda client for instance is just the AWS client the both the core client create lambda is a factory fixture So it creates lambda functions and then tears them down at the end of the test. So we use that a lot So this is a great framework to have It's basically just contract testing and Then when you run this against local stack, you get a nice report That things are missing or things are things don't match And again, we can't jump on everything but this system gives us at least a way to write a test We know runs against AWS And then we can sort of pick and choose what we want to implement. So in most cases We don't need to fully support the entire API operation We can just we get away with with implementing like 80% of it and then you your use cases enabled and Now now that we have all of these tools We can reason over the API. We can generate, you know, lists of services and API operations. We have this parity testing We have the recording of the responses. We can go and generate Coverage pages. So if you go to the docs, you can actually very fine-grained. See, you know, this is the this is the lambda coverage overview each API operation whether it has a test or not Whether it's AWS validated the test So whether we know that it behaves in the same way and then you can even click on on particular API method create function, for instance, and see which tests covers this particular API operation like this is this is a completely different type of code coverage Which is much more high-level. So it's basically API coverage And we're sort of exploring this idea of how to communicate to users the coverage How well tested local stack is other than, you know, that the classic code coverage metrics Okay, so coming to conclusions some takeaways open source software is just fantastic I mean the the community gives us so much and we try to give back to the community Maintainers of a bunch of open source projects. We also contribute back to open source projects And just openness also of the Python ecosystem and also the language just enables us really really really really well The dynamic code loading is a big part of local stack And and if you have this problem of modularization and multiple distributions, and you need this you need to pull this in Go check out plugs. I think you can really solve some of the problems you may be having Those are some obvious things as well like pick your battles We have to be very careful and deliberate about where we put in our effort You remember the EC2 file with the 600 API methods. We can't implement it all But we can focus on 80% and again Python just helps us get these 80% done really really quickly Consider the patents when migrating frameworks some of the Patents that are well established in software engineering. They a lot of them come from object-oriented programming But they can be a real Facilitator for these kind of lift and shift migrations that we were doing with all of the service providers Have code introspection in place So be able to understand your code base and reason over its generate reports and things like that immensely useful and What I learned is Scaling your team also means scaling the code base. We spend a lot of time thinking, okay We need to scale this team now We have six people now and then we need to be like this many people at the end of the year figuring out where the bottlenecks are in the code base and and proactively addressing that with framework migrations is is just as important as just the functionality of the product and With that one last thank you to the community and the contributors to local stack We have I think by now over of almost 500 contributors So also if you're interested in contributing, please go find us on github and thanks so much for coming