 Thank you, everyone, for coming along to my little talk this afternoon. I know it's Sunday afternoon, last talk of the conference, and people are fading a little bit. Fortunately, this is not a deep dive, so you don't have to think too hard. My name is Hamish. Unfortunately, I haven't been able to be at the con this weekend of a good friend's wedding on, but it is my honor to come along and talk today. I am a recovering engineer, now product manager at Coordinates, who's here at Coordinates? We do geospatial stuff primarily, and we've got a web app, which hopefully some people have used. A lot of Python and Django is kind of our deal. So I'm going to talk about flags. A bit of a backstory into a little project that I'm going to talk about is sort of had a genesis at Gather conference in 2015, which is a cool sort of conference that ran for several years, which sadly is no longer in existence. Back then, we had this flag referendum going. There was lots of talk about all this flag design stuff, and there were people doing talks about flag design, and it was really interesting, and I thought that was cool, and everyone was submitting their homemade flag designs, like laser kiwi and all that kind of stuff. I'm no vexillologist, but I have Python. So I thought it would be fun to have a look at where it would be possible to actually generate a flag. And so I did, and I came up with this thing called that would basically take a genome and it would generate a flag design. And this is an example. This is one of my favorite ones that came out of it. The first part is a color palette, and the second part is the actual designs that go on there, so we've got an English cross with a diagonal cross on top of it with another cross on top of it, and that's all encoded in that URL. We're in that sort of bit of text, rather. And that's pretty neat, and you could basically generate all these random things and it would generate these flag designs. I'm going to try and actually do it in front of you, which is always risky. So is that super tiny? No, no one can see that. So basically you could go like, if he calls flag.parse, create random genome, flag.show. It's a randomly generated flag. Let's create another one. Hey, that's the same, because I didn't actually call it stunning. Which is, you know, honestly, half, better than half the submissions that we've done. And I also made it that you could evolve the designs, so if we have a look at that one again, it's pretty crazy. But of course, if f2 equals f.evolve, f2.show, that wasn't very dramatic. Okay, so the idea was that you could take a design and you could evolve it and you could create a new design. I had big ambitions of playing with a genetic algorithm that would then, you know, try and fit it to some sort of objective measure of what a good flag was. And of course, that was impossibly difficult. So I sort of forgot about it. And then a friend of mine who was running an art festival said, hey, would you like to do something? Again, I'm no designer. I'm an artist. But I can do these flag things. So I thought, hey, it would be really fun to make something that was online and interactive. And they thought that was really cool because they had lots of artists making really crazy things, but not a lot of sort of digital stuff. And so I said that I would put something together. And so I thought I'd put that thing online. And it being 2015 and kind of, you know, we get a bit stuck in our tools sometimes. We've been using AWS a lot. And I sort of wanted to learn a bit about it. And sort of EC2 was kind of the way to put something online. So I was like, okay, flag and Postgres, you know, that would be dead simple. No, I just need an EC2 instance. And it's going to have to have Ingenix running Gunicorn to run a Flask app that is behind a cloud front thingy. And then RDS, which is the Postgres-like AWS server. And that was pretty cool. It was a bit of a pain to actually set that up. I probably spent more time configuring all that than I did in building the original kind of flag generator thing. But it worked. And, you know, the AWS, they give you, you know, a bunch of free credits. So I could run this thing for next to free. And it looked like, hey, get me out. That was random. Of course, I didn't actually test this bit of it. Okay. Running on there so I can grab that. And here we go. So now it's running in a Flask app and it's generating random designs. And it would save the flags into a database and people could then click the button and evolve the design more. And then you could favorite the gallery. And that was all pretty cool. So I was pretty pleased with myself with that. But it was still running in EC2 and RDS. And, of course, I kind of forgot about it. And then your free credits run out. And then you got a bill for $60 a month. Which I hear is a common story with these things. It's kind of how they suck you in, I guess. But that's kind of expensive to actually do that. It's just sitting there. It's almost never being used. And it's just spinning in the cloud and wasting me money. So this year the same friend was organizing another art festival and said, you know, could we do that again? And I thought, yeah, that'd be cool. I can add some new things. I can make it animated. It's going to have stars and circles. Not stars and circles. But I don't really want to do it in EC2 again. So I had to look at if there was another way. And basically, because all I really need is somewhere to host some static content and some way to generate these flags on demand, I didn't actually need all that stuff that was running all the time. And people were talking about serverless stuff. So I thought, hey, maybe that is an ideal way to start learning about it. So what are the pieces that we can use to do all this? The first one is Lambda, right? Who's actually heard of AWS Lambda? Okay, lots of people. So AWS Lambda basically lets you plug in a bunch of code and something will happen and it will return some sort of result, which sounds kind of trivial, but the fact is it's running in AWS's infrastructure. You can call it a million times and it will just happen kind of magically. It also can be triggered by a whole bunch of stuff. So you can trigger it on an API request. You can trigger it on all sorts of events that happen in AWS. So you kind of look at the list of triggers and it's kind of daunting the number of things that you could sort of plug in from one end of it. And then it's got full access to the internet, but it's also got access to all your internal resources. It's got things like Bodo sort of built in as well. So you've got all the power of that particular library to talk to all your AWS stuff. The pricing is interesting. You pay per megabyte memory per second of execution time. And you kind of got to say how much memory the thing is going to use. So at the lowest tier it's 120 megabytes of memory, and you're paying for each second and 100 millisecond intervals. But the pricing is insane. It's at that tier 3.2 million free seconds, which I think is more than a month of just execution time free. And then beyond that, it's 0.000000000000. 0, 0, 0, 0, 0, 0, 0, some numbers, 0, 2, 8, 100 milliseconds. So it's very cheap to actually do all the trivial API stuff that I'm trying to do is essentially free, because if I actually manage to hit 3 million seconds of up time, then I've become sort of an insanely famous artist. And paying the subsequent bill of infinitely small number of cents per millisecond, it would just be irrelevant. So I can actually try and do that. So I'm in ABS Lambda. This is what it looks like. So that's the console. You basically just go to Create Function. We can author from scratch. It supports a whole bunch of runtimes, but hey, 3.6. Not that one, this one. You have to give it a roll, which gives you basic execution, which is cool. Test function for fun. Right, and it creates this Lambda function. You can add a trigger. So you can say how it is actually executed. But just for now, we can actually go and create our function. We've got this coded thing here, which is kind of crazy. So we can just sort of save that. You can test it. You can give it all sorts of test data as well. So you can set up sort of like, I guess it's a bit like a unit test for your extra lambdas. You can actually run the test. We can see the result. We can see how much memory was used. We can see the build duration, all that kind of handy stuff. It seems a bit kind of 1995. I'm editing code in a little thing online. It's a bit like uploading zip files to like FTP servers or something at this kind of level. But the fact is I've just created a function that runs in ABS. So that was super easy. If you are doing things more complex, then you can build your virtual environment, and then you can just basically upload the whole thing. It's got a whole bunch of stuff available already, like Boto I mentioned before, and requests. But anything else you want to add, you can package in. And there's also a bunch of really handy, there's a GitHub repo you should look up. I think it's called Lambda Functions. It's basically a whole bunch of pre-built stuff that you can bring into your project and upload it as well. So we've got a Lambda Function, right? And now we need to have some way to talk to it over the internet. And so I can't remember when Lambda first came out. It was a little bit limited. It had a bunch of functions, but didn't have a lot of ways to actually interface with the functions. And then API Gateway came along. API Gateway gives you a whole way of setting up endpoints that are publicly accessible, which let you talk to not only Lambda, but a whole bunch of other services that are available to. But it's cool though, because you can set up different resources, different models within API Gateway. Authorizers, so if you want people to be forced to use their sort of two-factor, authenticated AWS account to talk to the API request, you can set that up. If you want it to be publicly available to anyone and anyone, then you can. Different methods. You can set up different deployments. It's got monitoring baked into CloudWatch too. So you can see everything that's happening within AWS's own monitoring infrastructure. Even beyond that, it's got all the business logic that you kind of want around a lot of these things as well, so you can create usage plans. You can generate API keys. You can set rate limits. So you can have a particular set of API keys around a usage plan that have very low rate limits, then you can have a different set of API keys, which have much higher rate limits, for example. What's really cool is you can generate documentation from your API, or you can import APIs to find in Swagger, for example. And again, the pricing is pretty cheap. $3.5 per million, that's US dollars per million API requests. I should expand on that as well. With API Gateway and Lambda, it's per requests, and then the transfer fees as well. So if you're pushing data in and out of US$3, then you're going to pay those transfer costs. Usually, if things are in the same data center, same region, a lot of stuff is free as well. That's pretty cool. So if we go back to our AWS console, then we can actually create an API that talks to that Lambda that we created. So I can go create API, test API. So you can see import from Swagger and all that kind of stuff. We can create a different kinds of methods and resources. So here we can create a get method. So at the root of our API, there's a single get method that we can do. It's going to call a Lambda function. And we are in API Southeast 2, which is a fancy name for Sydney. You just plug in the name of your function, which was test function for fun. Save. OK. It kind of adds the necessary role permissions as well along the way. And here we can kind of see a bit more of the detail of what's going on. When a request comes in, it is then forwarded to the Lambda function. It is processed. The result is then passed out again. And along the way, you can do things like transforming requests and adding headers and adding other tests along the way. So there's an awful lot of power there that I've only really had a little sort of play with. But fundamentally, there's all the usual kind of things you might want to do with an API. You can set up. So here we've talked about gateway responses so if you want to set up all your 500 pages and 400 pages and you can do that all here, you can set up your authorizes, different stages. So at the moment, we've got an API that is going to call our single Lambda function and return something. At this point, we need to deploy the API. So we're going to deploy it to a new stage, which we will call test. And if we open that, it will. So I've just created an API, like it was that easy. It's a trivia API for sure. But you can think of how any little like, you get an idea for a slack bot that you want to create. And then like, ah, but then I need to have a thing that runs a bot server that is going to handle the incoming web hooks and that kind of stuff. Well, now you can probably do this all in the console, just typing things in, hacking around in a couple of hours just for fun. And that's really cool. I really like the way that you can now have a little idea. You can spin it up and test it out almost trivially. And the cool thing about this as well is so you've got a whole bunch of, you can now start adding different types of stages. So you can have like a development stage and your production stage. And you can switch, you can deploy different things. And yeah, it's neat. So we talked about usage plans. So you've got things like usage plans that you can create for different APIs, throttling burst rates as well. So if you're infrastructure, this thing scales more as infinitely, right? So if your infrastructure that your API request is going to call is going to be destroyed by a sudden influx of 1,000 requests, then you can make sure that doesn't happen as well. Quoters, all that kind of stuff. You can also throw them by custom domains as well. So by default, let me have a look at this stage. So that's a pretty crazy URL, but you can put your own custom domains in front of that as well pretty easily, which is neat. Any questions so far? Richard? Yeah. Yeah, so the question is, what do you do when you want to save state? Yeah, you're going to have to connect to a database or any other appropriate service for that. Yeah, RDS is the Postgres black one. So you can connect to that. Or I'll talk about what I did for the flags one in a second. But yeah, that's true, right? All you're getting is compute time. That's what you get. But you've got effectively, infinitely scalable, incredibly cheap compute time. So the other thing about lambdas is when you actually configure them, you are going back to lambda. So you have to say how long you think it's going to run. So you have to give it a maximum timeout, and that maxes out at three minutes. So that's a three minutes or five minutes, one of those. Three or five minutes, I can't remember which. And you've got to say how much memory you expect it to run. If you hit the memory cap, then it's going to get killed. If you add the timeout, it's going to get killed. So that's where things get a little bit different from running your own stuff in EC2 machines, for example. So of course, the last piece of this particular puzzle is S3, in case you're not aware. S3 is Amazon's simple storage service. Big database of objects, where you pay for what you store in access, and it's very cheap. But the other thing that some people might not be aware of is you can also use it as a static web post. So if you give your bucket name, like example.com, call it example.com, and then you point your example.com domain at the S3 path, then it will just start serving content as if it was a static web post, which is a really cool feature. But it also interesting, dots into bucket names causes other people all sorts of pain, if you've ever had to talk to buckets via the API before. So Foflux 2017 uses all of those things. And you can go visit it, so Foflux.nz. So basically what you're looking at is a really cool animated thing. And it is hitting an API, which will then generate the random genomes and send you back the image. So it's generating the flag. It's saving it to S3, and then it's redirecting you to the S3, or returning the S3 URL so you can then display it. It's got stars and circles. Didn't work out that great in the end, but I kept them. And in this case, what it's doing is it's taking the genomes and evolving and giving you images. So this has been running for a while now. It's had a little bit of use, I guess. I'm storing maybe 10,000 different flags. All of these cost me like $0.20 a month. So that's pretty cool. That's really handy. Now the way that I actually got that on there, Zappa. So I think someone mentioned a lightning talk I was told. So you want to see before I was in the ADS console and I was sort of typing in code and then manually setting things up. If you've used ADS in anger, you'll know that everything is behind an API. You can manage everything about ADS via an API, which is what is awesome. It means that you can do all that kind of stuff that I was doing point and clicky. You can do via APIs. And there's a whole bunch of frameworks out there now that are built around creating the kind of apps that I was talking about and letting you build your application like you would build any other. But then you push it to the appropriate serverless services to actually execute it. I haven't looked at many of them. I've only really looked at Zappa and even then not in a lot of depth. And the reason I looked at Zappa was because it seems to be very geared towards Flask and Django deployments, which I was using Flask and so that seemed to make sense. But it's cool. It will build and package your project. It will deploy it into a Lambda function and to an API gateway, configures everything for you. And then it gives you handy CLI for actually deploying and rolling back. So if I go back to here, I'll just close that one. And we go. So at the moment, I've got this hooked up to, so this is the Zappa instance for 5Flags.nz. So I can do things like Zappa Tail. And that will basically talk to CloudWatch and get my logs and give me the most recent activity from the logs there. It's kind of like a really slow way of just tailing your logs normally. So it's not ideal. But there's a whole bunch of combined options for filtering and the kind of stuff. It's also slow. There we go. So you can see all the stuff that you would normally see. Well, you can filter your different log types. You're going to get a lot of internal lambda execution kind of noise in there as well. So you want to filter it down to the things you're actually interested in. You can do Zappa deploy to initially get your stuff out, Zappa upgrade and your stage. So the update dev. So Zappa update dev. So if you make changes, it will then find all the changes, repact them up, deploy it out, and just make it available. So Zappa is really cool. It's, I don't do that on my mobile connection, actually. Yeah, so you can deploy a particular stage. You can update a stage. You can, oh, sure, roll it back to previous stages. Take your logs. It's also got some handy extra features, so scheduling. So you can set up basically scheduled lambda functions really easily with Zappa, which is cool. So you think about all the things you might use salary for, for example. You could look at whether you want to actually use that for lambda functions instead. You know, you might have a whole machine right dedicated just to running salary workers. Do you actually need that spinning all the time? Also, I mean, some of the stuff we do, it is bursty. You know, you might get a whole bunch of requests, feeds a bunch of events into salary. And we've had issues in the past where, you know, a large spike in activity can actually lead to real big problems in salary queues. We just wouldn't have that problem with lambda. Of course, unscheduled. So that's pretty neat. Some downsides with Zappa in particular, and I'm pretty sure this is just from what I've had to look around, this seems to be reasonably common. Ease of use is awesome, but you're really papering over a lot of the complexity. And I feel like what you want to do is understand the complexity in a little bit of detail first before relying on one of these frameworks, particularly right to the last point. The default IAM roles that I sort of asked you to set up are really liberal. And it's so easy with AWS to slowly creep your policy to the point where you've got some lambda function role over here that turns out has global read access to history over here, and that credential leaks, and everything falls over. So I feel like there is some danger there and not necessarily understanding how AWS kind of works, how IAM works, all those kind of things. IAM is the user management kind of portion. With Zappa, it kind of cheats. So instead of setting up all the different routes as API endpoints with an API gateway and then a whole bunch of lambda functions that might handle the different Flask methods that you've set up, it kind of just pack it as the whole Flask gap into a single method, into a single lambda function, rather, and it gives you a single API gateway endpoint and just kind of lets those components, particularly the lambda function, kind of handle everything. So I kind of feel it's cheating a little way. So if we have a look again at all my lambda functions, I've got one lambda function for that for Flags app, even though there's five or six different endpoints under the hood. And in API gateway, again, there's sort of five or six different endpoints that you might actually hit. But what it has is just a single proxy thing, which basically takes every request and passes it right after the Flask. So it's kind of cheating in a way, but it works. And it's really great for hacking things together quickly. So I'll talk really quickly. I think we've got 10 minutes left. Is that right? So that was kind of a fun project, right? But one really serious use case that at coordinates we used lambda in particular for recently was a migration project we had going on this year. So coordinates, we had three kind of on premise, sorry, three instances of our whole stack. We had two local kind of on premise type installations, and then we had our own version running on AWS. And we had to merge the three into one. And we had shared identifiers between all three projects. We needed to be able to merge everything while it was running. So there was essentially going to be minimal downtime. We're talking about an evening of downtime. And the particular thing that I was looking at was the analytics system that we had. We had six years of analytics history. Right now we pump in 50 million events a month. All the shared identifiers in that history had to be rewritten to merge all our history into this single project. And it had to happen with no downtime. So one thing that I find this kind of good for is those kind of ETL kind of tasks, the extract transform on load kind of tasks. So in our case, we were using Keen.io. Does anyone use Keen.io for analytics stuff? Yeah. It's really cool for just running usage and kind of thing where you get a million different events happening in your system. And you want to see how many of those events summed up by some dimension based on the user. And then it's kind of like a purpose-built database of event collection that lets you spit out analytics data very quickly. So we had an enormous amount of history of that. And they have this handy feature where they can stream raw event data into S3 buckets for free, which is really neat. So you pump in all your event data. And every five minutes, they push it back out again into an S3 bucket. What's handy is that Lambda can be triggered by S3 puts and posts. So you can trigger Lambdas on all sorts of different events in Navies infrastructure. But S3 puts is a really interesting one. So basically you can tell it to anytime S3 sees a new object, run a Lambda function. And that Lambda function can do anything. So in this case, what we set it up to do was every time one of these buckets of data came in, extract the data, find all the identifiers, change the identifiers to what we know that they need to be now, do some other maintenance along the way as well. So we're like, oh, these things were originally designed sort of five years ago. And of course, all those people are idiots. The same people still work here, but now we're a lot smarter. So we can fix some things we've always wanted to fix. And then we can push that data into a new Keen project. So what we had was three live running projects that were then streaming all of our analytics data into a new project, but now with non-conflicting identifiers. So that's pretty cool. And what it meant was that come migration day, we could basically point old project, you point the whole system with all the merged data at the new merged analytics system and things just keep working. You know, it's the same data, but all the identifiers in the database have changed, all the identifiers in the analytics system have changed. Users completely aren't aware that anything actually happened. The other part of that was backfill. So, you know, we had half a billion historic events that we had to pull out of the existing projects and do the same transform and load process on. So there's a couple of things we can do there. We can massively paralyze it via Lambda as well. Some of those things we can aggregate. So we're like, well, actually, let's have some special Lambda functions that will actually run queries on our analytics system and take, you know, a day's worth of queries which might be like a million events and then spit out an aggregate of the dimensions we actually care about and so you might get a thousand events on the other side. To actually call that, we can then, we plugged a API gateway in front of it so that we can say, well, I want to migrate this bit of data or this timeframe, hit a button and it will asynchronously go and call it. With API gateway, you can do things synchronously or asynchronously. You can do, so the event, the example that we did just in the console before is like a synchronous one. It's calling the Lambda, it's returning the result of the Lambda. You can also do asynchronous calls where it'll immediately return, yeah, yeah, I've triggered the thing. Now it's up to you to actually make sure it actually happened, which is handy because you know, we're doing things that might take, you know, half a minute and we want to be able to run a whole bunch of them and monitor their progress. Yeah, so we did it, we massively parallelized it and then we got an email from concerned engineers at Keen saying, hey, that's great that you've done the migration but you're damaging our infrastructure because it turns out that actually running thousands of these things at the same time is not being a good citizen. You see this graph through of like, you know, the pink line is us. The other lines are sort of the more normal background activity that was going on at the same time. So basically we backed it off and we said, okay, we can actually run this and it'll take a couple of weeks to actually process all our data but we can do it backwards in time and again, no one's really the wiser that we've made this major transformation under the hood. So just last of all, just some gotchas learning some thoughts from my sort of relatively limited experience for all this. Big gotcha, async lambdas will sometimes fail for mysterious reasons and the policy is they get reached quite twice and then discarded. So you need to be able to, this is the kind of fun thing you find, you know, sort of 10,000 event, you know, 10,000 executions into your migration process. Oh, there's some extra data. Where did the starter come from? Oh crap, I need to delete it all and start again. Because what happened is it actually did the ETL process and then I suspect what's happened is the system that actually checks that it's run correctly has fallen over and then it goes, okay, I'm gonna try to get in 10 minutes and it's 10 minutes as well. So you can't look at after five minutes, you're like, after five minutes, you're like, oh, it looks good. But then you come back in a quarter of an hour and all your data's wrong. Sorry, I threw out a stick. So you can look at dead-let accused for SQS and SNS for unexecuted things but you do need to handle double execution properly. Latency is a problem. So that faux flags up, I think latency can be about a second sometimes. You're not gonna get the same with that approach. The ECT approach is gonna have better performance. There's a thing called Lambda at Edge now which I think might help a lot. There might be a whole bunch of configuration you do to improve it but generally it seems latency is not great. Of course, if you worry about lock-in, then if your business is built on this, then it's gonna be pretty hard to switch. That said, it is just Python functions talking to gateway routes. So a lot of these frameworks, some of them are designed to be able to cooperate with Azure and all that kind of stuff too. And last of all, if you're an API-only business, you wanna be careful because your barriers to entry are really low. So you kinda see this with the likes of Keen.io and other sort of API-based companies where I think a lot of them are racing to the value add on top of it. That because you can come in and just build your Slack bots or your analytics platform or whatever without having to deal with Opsbird and that you traditionally would have had to around the same sort of high volume services. All right, so that's me. Any questions? So just give me a second while I try to come up with it. For the production use of Lambda you were talking about, are you using Zappa to manage those, or? Sorry, sorry. For the ETL and the analytics thing, are you using Zappa to manage those, or? Are you using what? Are you using Zappa to manage those? Oh, Zappa, no, no. So no, we did that before, we did that before, they came before the flags thing. I guess also because it's a business use, a little less inclined to like use the thing that has a lot of IAM or permissive IAM role to do something that important. That said, if I was gonna do it again, I would definitely look at actually the proper framework for doing it rather than you build your function and then you zip it up and then you upload it and then you test it, it's very like a Z95 where you're sort of uploading things by FTP. It feels very similar. I do think it's gonna be a little while before the very big list of frameworks is rationalized to the two or three really good ones. Yeah, yeah, I think so. Yeah, I mean, you do have limits, right? So you gotta look at that execute. It depends on the kind of data that you're capturing. Yeah, I mean, this will give you the back end stuff. Yeah, you'll need to talk to RDS or something similar, yeah. So the Lambda function, the API response that you're creating has a funny URL. How do I make it pretty? You can plug your own URLs in front of it too, so your own domains. There's stuff in there for doing that and you can also put certificates in there too. So you can make it sort of SSL protected under your domain relatively straightforward. Yeah. Sorry, we've run out of time. Thank you, Hamish.