 Okay, so I'm gonna get started with the intro So Hopefully all of the materials been distributed to you There's a little bit of an issue it seems with Windows being able to read the USB sticks So Cody's gonna work on reformatting a couple of the sticks for you So hopefully you won't get too far behind on this if you if you do don't worry Just follow as long as best as you can All of the workshop material and the software you need is on that stick So if you if you want to take it home and play with it later, you're welcome to don't worry so Here we go. So today we're gonna look at building applications on 0vm and Swift and just kind of what that looks like So the objective is to learn how to build a complete web application using just Python Swift and 0vm No other components. That's it. So So We're just gonna talk briefly about these components and we'll just get right into the hacking. So 0vm it's single process isolation. It's based on the chromium native client It's lightweight and embeddable so we can we can put this very easily inside a data store like Swift It is not Docker And it's not a replacement for traditional VMs or containers. It's kind of a different beast Um, it's a it's a very secure container. So the the security is twofold When we start a process in 0vm, we can we can statically validate that it doesn't do bad things And this is not something we created. This is this is something that we have to credit to knackle knackle does this out of the box Um, we have taken a little bit of a step further. We've we've reduced the the syscall API from about 50 functions in knackle To six. So we've really locked it down And that means that not only is it a very small attack surface, but I mean you've just got six syscall functions It's very reasonable to to audit these and test them thoroughly So you have some some guarantees about security Uh, we'll also have to be upfront with you. What are the limitations? What what is this thing really? So language support is python 2 7 c and c plus plus Um, currently no c c extensions for python unless you want to go and do that work yourself Um, it turns out porting is really hard um So but so anything you want to run on this it needs to be cross-compiled to knackle Um, I just want to show you. I don't know if you can see these really well Um, this is kind of what what a simplified version of what happens when 0vm starts up So, um, it's given a manifest the manifest defines what resources it has access to And that's all it's going to get Um, the manifest says okay run this program And that's the the nexie or native client executable It's going to go validate that and so it runs the static validation And if the validation fails it will stop and throw an error Um, if the validation is okay, it will continue And it's going to initialize channels, which are the io abstraction. So, um, if if in my manifest I've defined that I have access to these three files I'm going to create a channel to give 0vm access to those files I'm going to create those file descriptors and I'm going to hold on to them Um, then I'm going to allocate memory for the for the untrusted program So that's the that's the code that that you are writing And then I'm going to pass execution off to the untrusted code and let it go nuts So opening a file uh to your code you actually don't open files if you if you do an open call Um, it it looks like you would expect but it's not actually opening a file All it's doing is just going and getting a file descriptor that's already open Um, it's just totally transparent if that file doesn't exist Then it will just save you know file not found basically Uh, reading a file once you have that file descriptor You can read and write from that file as you wish There are constraints in the manifest about how much you can read and write from from into that file So as long as you don't exceed those limits, uh, you're good. You just keep going So, um, let's talk about Swift. This is this is the open stack summit. I'm I'm sure you've heard of Swift before So, um, but just to remind you a few facts about that. It's a massively scalable object storage You can store millions of files petabytes of data and in a cluster you can have thousands of nodes One of the really interesting things about Swift is that you can extend its functionality through middleware and that's what we've done This is and I apologize if there are any Swift developers in the room But this is this is our this is the Swift architecture. I'm going to be running with today Um So in Swift, you have uh, basically a layer of proxy nodes and a layer of storage nodes And they all communicate with each other to do various things Related to storage So keep keep that picture in your head Um going back to middleware Um, what we've done with zero cloud, which is a middleware application that we've written for Swift We enabled data local computation So, uh, we do that by by basically intercepting an HTTP request In the in the Swift pipeline and looking for specific things in it special headers That will go and run computation if it just looks like a normal Swift HTTP requests It'll just behave exactly as you expect if you're getting a file It'll just get you a file if you're posting a file. It'll just post the file. That's it But if you add an additional headers until 0vm to do things then it will go off and run these computations for you It's it also sets up so when you actually do these computations it sets up kind of like a CGI style environment for you So it has these environment variables that you would expect in in a typical CGI environment You can build your own HTTP APIs on top of this Um, there there's also plumbing to do parallel map reduce computations over large groups of files Um other interesting thing Um You could debate about whether this is good or bad, but it's interesting. I'll say that All you need is Swift you you so you can you can deploy a complete web application onto Swift with this middleware You don't need on any other data store. You don't need any other web servers. These are the only components you need And we'll actually do that today in the workshop So with uh With Swift and zero vm and zero cloud. Uh, this is what your architecture looks like you you have your Swift cluster And then you have zero vm running on all of those nodes Um, I just want to put this up here just if you want to go see how this is configured It's quite simple to to install it. Um, the middleware is just python code So you python setup dot py install on all of your nodes Then you have to alter a couple of the container proxy and object node configuration files And set a few parameters. It's it's quite simple to set up And we have documentation on this As far as executing code it's uh, it's as simple as This short curl command So I talked about some of these um, some of these special headers So you notice the uh the url there example.com slash v1 It just looks like the the location of Swift. That's like your your your, uh Your accounts like home directory or home location in Swift. Um Without the headers like zero x zero vm execute, this would just do a normal Swift post But in this case, we're posting a a python script. We're we're telling the middleware Hey, this is this is we want to execute this on zero vm We're telling it. This is the type. It's a python script And here's the data go run it for me and it'll just go run it And it will return the standard output So this is the very very basic example of how code is run, but we can get a whole lot more complicated And it's going to get really hairy later on Um, I just want to make a quick note about the project says it's I would say it's a usable beta There are lots of bugs to find There's plenty to optimize um The project is sponsored by rack space, but the code is Apache 2 so Anybody can use it anybody can contribute anybody can fork so and without further ado Let's hack so um Agla take it away. Thank you Lars So for those of you that came in late, please get a usb stick and a little cheat sheet All of the materials are online So you don't really need a usb stick to get this going but We are simplifying this for you So you don't have to go install vagrant and download all the All of the things that come with setting up vagrant and all of that And since we don't really want to bring down the conference wi-fi If you do not have a usb stick raise your hand and someone will bring it to you. All right. Can someone Lars? all right I think there is a an issue with windows machines. Uh, kodi right now is Working on making it work on How long Minute, okay So hopefully uh, hopefully you can catch up as soon as kodi is done But if you do have a machine with um Where the usb keys work basically what we have on the usb key Are some handouts One of the the same handout that is printed over there We also have a an ova file that you have to import into virtual box or fusion Which whichever you prefer it should work in both. We have some virtual box installers if you don't have it installed We we have party for windows people because you do you'll want to copy paste and Virtual box doesn't really allow that very nicely the tutorial This is the same material that is available online on uh zero vm dot read the docs So We we print it for you to the usb key just in case there were any connectivity issues The same material is on a in a pdf format, but it when you copy paste from pdf it doesn't Come out so nicely, especially with the code there is a little um How to import it and how how to get going so I we assume that most of you are Familiar with like using basic computer stuff and also how to use virtual box and it would basically just in virtual box you would go import the appliance and Select it. You should not have to change any settings. Just continue click click import and it will import it for you after you you import it You will want to ssh to it by typing ssh minus p And uh vagrant at local host The password is very secure. It's vagrant. I don't think we wrote it that anywhere Like I just realized that right before the workshop that I don't think we actually say anywhere What the password is except on the documentation online? So Password is vagrant and the user is vagrant as well after ussh into Your virtual machine You need to run a couple set up set up commands so What you have is dev stack running on your usb key and the dev stack doesn't really like being restarted too much So you will want to go and rejoin Once you rejoin You will need to get out of the screen by typing control da Control ad it's their way and after that simply run setup lab Script and it will set everything up and everything should be Up and running and to validate that you have a valid Environment you should see something like this Yes all right, so I will not keep checking where everybody is because all of the material all of the steps are available on the usb key As well as online if you do get stuck on something, please raise your hand and we get One of the red shirts to help you I Will send the away team and But everything should be there. We're not we're not doing anything super magical here on the stage that you can do on your own laptop all right, so Yes, so once you ssh into The virtual box. There are some directories there. There is a solutions directory Lars will go over a much more complicated example there and That has lots of code So he will magically type in all of that coding there and the application will just happen and run really fast And you'll be able to to do the same I will copy and do actual copy paste typing thing because live coding is prone for typos and So so yeah, like everything is in here all all of the setup script. You do have a little Can type There is Swift running you should be able to check. There is nothing too fancy right now there There's a couple things uploaded to make things work just in case we don't have internet. Yes Uh-oh Yes, if you're on windows, you'll have to wait a second or Yes, all uh, so all of these commands are available in Where it says hands on with zero VM tutorial PDF NHTML so in the tutorial It is Section three There is a if you were to set up the vagrant box that we have already set up on the usb key for you You would come here and follow these steps and it would work. Hopefully the same way And after it's installed, we start at section 3.3 So once again pdf doesn't really copy paste very well So you may want to go to the html version that's on usb key or it's a Zero vm dot read the dogs dot org And it's going to be the same thing over there Yes All right sending the red team So i'm going to give you all a couple minutes to catch up so i don't lose you too much but All of the The things hopefully are there and running So just a quick check how many of you were able to type zpm off and actually get some uh swift credentials back Okay, so What zpm off does it basically spit Spits back out the swift credentials that were in the source file that you sourced and uh, this is just to validate that. Yes this Everything is working as it should and You can proceed with writing code and running things Yes, david okay, so the documents are slightly outdated and uh So once again, if you just want to follow the directions online, they are on zero vm dot read the dogs dot org And uh, hopefully most of you are but at zpm off raise your hand if you are all right Yes, and it is uh the same Documentation we printed to the usb keys So both in html format and pdf all right, so after we have The zpm off works we need to set some environment variables and um This is just to simplify passing uh doing curl commands So we don't have to Retype these wonderful long tokens and urls Uh, the first token is the off token and the second one that we need to export is the url force swift And uh, everyone probably has different ones at least the token parts. So do not type mine and Things will not work I haven't tried that, but i'm pretty sure that's going to happen. All right, so uh now that we have our uh Zpm off working swift is running. We got the credentials back. We set the environment variables Let's run some code and it's really as hard as creating a file and The the one thing that is really to miss you do want to copy this first line File python python and Now that you have a python file with your source code Simply execute it by running curl command and Hopefully this works and it says hello from zero vm. So this is your hello world of zero vm and uh It's really that simple. So once again all we had in the example Where four lines of code really could have probably cut it down to Three lines for the basic simplicity, but we do want to show you that there's actual things happening so this is hello world of zero vm and There's nothing complicated about it now. Obviously Most of us probably want to do a bit more so Let's try something harder And i'm creating a a small python script and another small python script This is my main that imports the other script Yes I'm sorry so all of these steps are in On zero vm. Three the docs the docs.org No, and I think there is some There's like a cheat sheet type things that we kind of explain how you can run basic things We are not replicating everything that is online because once again things change and There would be a lot of uh Trees that we had to kill to print everything out So but you should be able to uh, the first example was uh section 3.2 post a python script That that's all I did. I just was following this these directions And to get the first one running and right now I am working on posting a zero vm image and To post a zero vm image. I will need uh a little bit more A few more steps I have two source codes. Yes Yes All right, let me All right, David gets 20 seconds. Everyone else gets couple minutes What is the status of windows? Yes Okay. Oh, you got it working. Awesome How did you fix it? Yes Yes, we we just uh used curl command to send it uh in the documentation Oh, I'm so sorry. So yes, we're doing execution by using curl just a simple command line client and uh You can also invoke it through python I skipped that part, but uh it just is The same thing through code How many of you have run the first example successfully? How many of you would like a little more time? All right, oh Yes Maybe Uh, yes, I believe that is but We're starting simple All right, so I know a few of you have Uh successfully completed the first example Uh, what we did so far was we made sure that we can use uh Swift and zero vm together by typing zpm os and just verifies that things are All working properly we exported the tokens so we don't have to retype them all the time We wrote a little sample python script to that says basically hello world or hello zero vm world And uh, we executed it by using curl command You can also execute the same command by calling the python script Now we're going to use a slight uh, we'll do a slightly more complex example and I think so far I had created the two source files My math dot pi and main dot pi with the source code in here That you can copy paste if you want to follow along We also need to create a system dot map file That is basically like a manifest for The code we want to execute So I'll try to not Okay, I guess I already done this before So I create The file that basically says what it is. I want to run it and where I think it's on. Yeah, there it is. Okay So just wanted to talk about this real quick and and give you opportunity to to catch up So the um, uh, let's look at the top. Okay. There's just name. This is just metadata. This isn't terribly important. Um Execution what we're saying here is that we want to Execute a very specific Executable and this this funny file colon slash slash syntax tells us that this is kind of a a well known program That's installed on zero cloud. So zero cloud has a registry of of the programs that are available In this case the pretty much the only one available is python. So it says we need this image So wherever you run this that image needs to be there um, and then We are saying run that and then Start with this script. So it's just like saying python space main dot pi enter That's that's pretty pretty much what this what this is saying You can add additional arguments. So if you if you're writing a program that that accepts Positional arguments you can you can add them here and you can you can parse them in your application The devices When zero vm starts up We We have a lot of things that we need to be available in our file system in the python standard library So we need not only the python executable, but all the standard library like the uh, json module Uh, the random module things like that All that's included and by having by by specifying this device We're saying mount this into my in-memory file system. So take that well known image there that's at the top and Make that available in my file system. So my my program can do things with it So that means that allows you to import python code And then the last thing here is we have an io device. So we have standard output So it means you can print something and then zero vm and zero cloud will Figure out what to do with that output and pipe it back to you. So in the case of Running curl if you print anything out to standard output it will just get sent back to you as the response of the curl request Thank you lars All right now that we have All the files created we need to create a tar file with all of our information I'm sorry All right now that we have the tar file Uh, let's execute it by using curl command again. And this is all it is It's simple curl. You can do a true code if you don't want to type all of this fancy content stuff in and Wow, look at that. It actually ran and uh, it did amazing computation And uh, you know, hopefully it's correct. Someone can double check I i'm not entirely sure I can condone the use of swift as a giant calculator though You can doesn't mean you should All right, so obviously this is the also a very trivial example But what we did is we have two python source code Source files we we had to make a uh json map file and we also had to create a tar file. All right, so Let's move on. Yes This thing takes a second to come on. There we go. So I want to add one other quick detail. Um, agla Can you show the main dot pi file? Yes So notice there's an import statement here. So the one of them is this custom module We created my math. So this should give you kind of a clue to get started on creating your own libraries and and packaging up applications that consist of multiple files as Most interesting applications do right? Um, so the reason why this works is that the Root or slash and the file system is is in the in the python path. So anything in that location can be imported and Um, there's some magic happening behind the scenes, but when you send this tar ball And you include All of these files in the tar ball. So there's the main dot pi and the my math dot pi All of those will become available inside the zero of m instance in the file system. So these imports will just work So if you if you package up the tar ball with a different structure So if you put things inside folders Your import statements would look a little bit different or you might have to amend your your python path variable To or your your sys dot path to to be given access to these modules Thank you Lars. Yes So the question was do I have to send this tar ball every time the answer is no This is useful if you want to just do a one-off execution just one time What we're going to see in the next example actually We're going to upload a similar tar ball into swift And then we're just going to send a small job description to it And say go execute that tar ball of code So you can do that as well All right, so now that we use a very nice segue. Yeah Now that we use the swift as a calculator, we'll use swift as a little excel storage place thing Since i'm going to have a few more files i'm going to create a directory and you can call it whatever you like It's not important I'm going to create another main Another file with code And in this case We're going to get to it, but we're importing CSV and we have A little bit more code Uh, I know I typed very fast here. So you should do the same just copy paste and um So I have created A source file and i'm going to create a tar ball with this source file Now i'm going to create a swift container Where i'm going to upload a set tar ball file And uh now we need a little bit of data So i'm going to create two data files That have some very, um Confidential information about people's names emails and their balances so You can put whatever Things you like in there, but If you want This is just to show how things are going to work All right, so now i'm going to upload these two data files that I have created to the same container and Hopefully it should be there. They hopefully it doesn't take up all of the space in my swift cluster I am going to create another json file that says Where all of my things are so in here we have quite a bit more You see we have the data files. We have the tar file and we have We're pointing to main So we'll first we're going to run it Run our uh code with the first data file And to run it Let's use the call command again so after um Calculating all of the information now we see that Uh bob has a negative balance and it's kind of sad. So let let's um Let's change it to To use the different file All right, so now if we're rerun the same code, hopefully we'll get a different result There there we go. So Hopefully a few of you were able to follow if not you can see how simple it is to actually run code Yes Good question Do you want me to show the json file? Yes, please All right, so the format of this file is the the same as that boot system map file thing you saw Ignore the name of that for now. Don't worry So We we see some similar things here, right? So we say okay, we need python we're going main.py is our entry point for execution and then Um, we've got a couple of devices that we're going to be given access to Um, again, we have python and standard output just so we can print things um, the new things here are um So input is the data that we're operating on so Um in the in the presentations before we talked about data local I'll get up here and I can maybe point. Yeah, so we talked about uh data local computation. So This right here this input Um This kind of gives a clue to zero cloud as to where exactly we want to run this in the swift cluster So we say, okay Give give me this file and we use the the tilde to act kind of like a home directory That means like my account use my account as the root of this of this path And say look in the container example and find a file called data 2.csv And then when I submit my request Zero cloud will go figure out where it is In the swift cluster and send it to one of the machines that has a copy of that data So if you're running with replication like three replicates I think it just picks one there's it's no real smart algorithm It just picks one and sends it there. So then it will be executed Right on that. So it won't be pulling that file over the network It's just going to be opening as a file on on the on that local machine, right? So with a small amount of data that doesn't really matter too much But if you have a huge file that matters, right? Sorry say that again No, it doesn't actually download it from the swift container. So we we put it up in swift We send our code to swift And then we give it commands remotely but everything happens inside of swift Which one No, that's not so that's just a description of the job. It's just a short text file that says Go run this application on this data Can you look at the json file again real quick There's one more thing okay, so The last thing here is is our application which contains our source code files. We need this to be available so This is kind of like our This is our complete bundled application that includes all of our code including main.py. So The the image is like the the application image. This is this is a special name. These aren't arbitrary names These are actually Very very special symbols to zero cloud So it will it will mount this tarball and it will make all those files available in memory in the file system And then it will go and execute those. So if you omitted this And just ran that You would get an error because it says well, I can't find main.py I don't actually know what the error would be, but that's essentially what's going to happen You couldn't find that so you need this to provide that code there Yes, david Yeah Yeah, it's just a string. So you can say main.py and then any arbitrary positional arguments Yeah, I think you're going to have that in your next example, right? We might you can also set environment variables So there's uh, we'll see that in some later examples. So in this in this in this exec Dictionary there. Whoops. Okay You can uh, thank you. It's down here now So you can supply I think it's called env so you can supply an env key And then which what whose value is another dictionary and you can just set environment variables Yes, um No, you can have multiples um, so so There are some limitations there. Uh, actually each node can only have really One input can you scroll down a little bit to devices? So this guy right here This has to be either a single file Or a glob pattern if it's a glob pattern What we're going to do is so if that if that glob evaluates to like 10 files We're going to start 10 instances one for each of those files. So really you only get one One input device here um That kind of seems like a Like a silly limitation, right? Like you can only read one file. What the hell right? But but there's actually a good reason for this so Because the the computations need to be located somewhere in the swift cluster Let's say you're processing 10 video files that are 700 megs megs apiece, right? If you've got 10 inputs, how do you decide where to locate your your computation? Right, you have to pick one and then and then you you don't really benefit from So you're local to one and you're remote to nine others So you're not really benefiting from this this data local computation. So you kind of have to Decompose your problem in a slightly different way Yes No So the the the question was how do you control the access to the data that's uploaded in swift? Swift does that so so in we kind of in the pipeline of of of Swift when you send a request to swift There's there's a pipeline and it goes through different filters. We kind of sit in the middle our middle where sits in the middle But there are a couple of things that happen before and after so if you Once you get to the the the zero cloud layer where you say, okay I want to I want to execute on this file if the user that is submitting this request Doesn't have authorization doesn't have access to this file Um, it's going to fail Because because the the zero cloud middle wire is very polite and it asks swift. Hey This user wants to run this are they authorized to do so? And if they're not it just fails So swift does all of it Yeah, cool. Did it really? Okay. That's good Yes, so you had two zero vm uh instances running. Yeah, and then uh, yes, so they're Um I don't know if it's yeah, there's a way you can do like like a second stage of computation. So you do like so If if that's like the the map phase So like like doing mapping a function to all of your all of your data You can add another layer the reduce phase to collect them all and do something else with it Um, I don't know if we have anything in the material On actually we do later later on in the The uh, the the big scary part of the workshop. There's a there's a map reduce Component of that so show you what that looks like If we don't get to that If you go to the example application tutorial snake bin Here I'll let you do it Oh Sorry, okay In part three there's a search function that does a very trivial map reduce search across all the files in the in a particular container So you can kind of see how to do it there And you can use that as a template for any kind of generic map reduce operation, right? All right, so I think we're ready for this part. We had three very simple examples and uh Lars will show you that you can actually do some really interesting stuff With us too not just hello zero vm Yep, okay How's oh, we're actually on live read the docs cool. Okay, so the wi-fi is good here All right, where's my terminal? All right So Low res awesome, okay all right, so First just just a quick check How is everybody with those examples raise your hands did anybody get through one of them? At least one. Okay. What about two? All three Okay, that's At least half. I think all right okay, um so Instead of trying to describe what we're trying to build i'm just going to show you Um, does anyone played with the the go language? Going you have a few of you, okay So go has this cool little playground where you can go to a website play.golling.org and you can type in any code And you can run it and it will just run on some server somewhere and return the output to you and of course you can run like Obviously they're letting you do anything you want right so that they lock it down somehow um So you can basically write any code you want here They do limit some things like they you know you can't like read and write files because there's That doesn't really make sense You can share these things so if you you know create some some cool snippet of go code you can just Share it with somebody So what we're going to do is we're going to build a um A uh We're going to build a clone of it except with python and we're going to do it all on zero zero cloud and swift It's not going to be as fast and probably um, it's not going to be as cool on the ui side because um I'm not a designer so all right So this is what I mentioned the the search part that's in part three where you can do the the map reduce So the first thing um, we're going to create something that looks sort of like pace bin um, it's just going to provide a very simple Um html javascript ui where you can type in code you can click save and it'll give you a little shortened url where you can access it later And uh, and then of course you can you can uh retrieve these scripts later you can Um, also upload them using curl. So you just post a script and get a script So the bulk of this is is going to be like boilerplate setting up part one So part one's the biggest part part two and part three are much smaller So here we go. So you have your development environment set up. Uh, let's just create a project directory And you can you can follow along in the docs here. I think that'll probably be easiest. Um, I'm just going to stick to the script So I'll also point out that all of the files the boilerplate files instead of having to copy and paste them are in Slash home slash vagrant slash solutions And they're labeled, uh, nothing part one and or sorry nothing part two and part three Yeah, so for example So like for example, uh on the first section, this is going to be the index.html for part one Okay so, um The first thing you do is set up a couple of containers. So We have these swift commands here that will do that for you Um, one of these containers is going to be empty the whole time and essentially All the reason why we're creating this is just to give us Um an endpoint and we're going to so if you send any kind of request to to anything after that container So if you do snake bin dash api slash foo bar, whatever we will we will Capture that that that you or I and do something interesting with it Uh snake bin app is actually where our app will be hosted And then snake bin store is where all of the pastes are going to go So that's just kind of how we've we've separated things. So I'm going to do that. So swift list Swift post snake bin api We're going to create app as well And create store And then we'll just list them to make sure they're there for sanity check Okay So up until now we've been using this this boot system app or job json thing To to define The execution behavior for applications Because this is going to get a little more complex. We're going to abstract a bit away from that We're going to use some tools to help us so We're going to use zpm the Zero van package manager to create us a template for for this application. So it's going to it's going to help us out a lot here So in my working directory in home snake bin, I'm going to do zpm new It created a yaml file. I'm going to look at that It's got a lot of comments in here um We're we're not going to be modifying a lot of this So in execution I think one thing is missing here. I think we actually do need a name So I'm going to call that snake bin We already have some nice defaults here. We have python 2 7 and we have standard out We're going to do something special with standard out. What is it? We're going to set content type. This is going to do some magic for us. We'll see that later But this is very important to do so Standard out before has just been a way to communicate back to the client Um by setting this we're we're saying that we're kind of going to overload the behavior a little bit It can communicate to the client But if you if you write out very special things to this device You can also spawn new jobs from within a job So your code can go off and like a chain call another application for example So you can create any arbitrary pipeline of of of computation Okay, and we're going to give it An input file And the path Swift It's in my uh my account Now this is actually a container name So this this is this is interesting. You can actually read from a container And when you when you read a container, it's made available to you as a SQLite database and you can actually query information about that container We're not going to be writing to it, but we're going to be reading from it So let's see that later on I I thought this was pretty cool when I first saw this It's probably incredibly evil, but I don't care Okay, the help section has some not very helpful defaults. So we're going to edit those We're just going to change arcs to an empty list and remove that I think that looks good And then in bundling so before we were creating a tar ball manually Um bundling is going to help us help us do that So for one it's going to generate that that uh that json file And pack it inside the tar ball in the right place And uh it's also going to to pack all of the all of the Well any of the files that we specify here. So in this case, it's some some python source code and uh an html file So do snake bin dot pi Get file dot pi. I have almost all of this memorized Save file dot pi Next dot html Okay, I think that looks good Here's the final result. So if you if you're really lazy, you can just go to the final result and copy and paste the whole thing That's totally fine Yes, sir So if you want to have your own custom third party python code made available Sure Yeah, uh Yes, uh what you can do is you can actually specify directories here So let's say If you're If your project is called, I don't know project jim Whatever if your code is in there These are the the paths here are relative to the the the directory of the yaml file that we're working with So if you just put that code there In a in a directory, it will just recursively grab and pack everything for you You'll probably need to hack your sys dot path To to a lot to import from that but Actually, if uh if project jim is a proper python module with an under under in it Then it you should be able to just import project jim Yeah, I don't think there's anything currently that will make that super super easy for you. That's something I want to do though It's it's it's generic so that you can do like you can do things with c and not just python I would like to create some templates that are a little more opinionated for python. So that's that's that's a very good idea All right, how's everybody doing just a quick check? Is everybody following along or you guys hanging back? Okay All right, cool Okay, so I mentioned before we can kind of build our own http apis so Let's let's think about what that's going to look like I've kind of used the the syntax that that github uses on their apis spec because it looks really simple and very intuitive to me So I've just kind of copied the way that they define things so We say here is if we submit a get to just the snakeman api container Uh, we'll just get the html form for uploading a script. So just like if you go to payspin.com You'll just get like a basic html form, right? We're expecting something like that oops That can go away If we post to the same url, we will upload a script And then of course when we when we upload a script, we're going to get the short url back And if we subsequently get that url, we will get that script If you're in a browser or something that looks like a browser You will get the rendered html page with the contents populated just like payspin If you look like to some other generic client, you'll just get the raw contents of that paste without the html boilerplate So here's the code. This is where it's going to get a little hairy. So First I want to create a snakeman.py file. This is going to contain most of the code for the app So I mentioned this This device here this standard out with a content type message http Um, we want to create a little helper function to to to use this device to our advantage to do special things so We're going to create this http response function here Let's plop that in So this is nicely parameterized so you can you can add in extra headers if you want you can you can change the content type You can specify any http code that you want Um, the thing I really want to draw your attention to though is this right here so If you just send in some text here This will be written to the standard out device and return to the client If you write a very very special http response just using plain text http You can actually spawn new new jobs from this spawn new computation We're using whoops We're using sys so I want to import that Okay, the next thing so we've been creating these these job json files manually That's kind of tedious to just create dictionaries and python code and dump it out So, uh, I've made up a silly little class here to kind of abstract that a bit It was kind of important to show you the format so you know what this thing is doing and it doesn't just look like magic So we're going to add that This makes use of the the json module So we need to import that as well Okay, now this is the interesting stuff This is where I mentioned we can we can actually query a container directly So if you remember We defined our input file as the container. So when this thing starts execution That container will be made available on this file. And so we can just read it like a a sqlite database Um, so this this is a way to check for duplicate content. So Here we want to generate a a little short name For the content, you'll notice that we're using random here But we're we're explicitly setting a seed because Randomness isn't truly random inside 0vm if you want to have really random behavior You need to have a random seed generated outside somewhere your entropy needs to come from somewhere else 0vm will not give it to you And then here we have a couple of functions for handling Post requests and get requests and I apologize. This this code is super gnarly I can do much better, but Bear with me So I'm going to be lazy and I'm just going to copy and paste the whole thing and I hope you do too Can also find it in the solutions folder. Yeah, you can just copy it right into your working directory. Whoops Okay So here's here's our our execution point. Um So here we have a what looks like one of the standard cgi environment variables. We can look at the request method Uh, we we switch on the request method and then Uh, we in order to behave nicely we we return an error code if somebody tries to do like a Head or a put or something. We tell them. Sorry. You can't do that All right, and I think it's time to Bundle deploy and test Um in that code we okay. We do need a few more imports And you probably I just scrolled over it quick the the way we're getting the seed for the the the random short URL We're actually just hashing the contents of the script that's uploaded And using that as our seed that's pretty much it Okay, there's the complete content. Let's go past that All right now we need to populate get file Oh boy, then we need save file Okay, so these get file and save file scripts These will be handlers for the various types of requests. So the the gets and posts Uh, how these actually get orchestrated is buried here in The main file in snake bin So let's look at the look at the get request We do some kind of ugly parsing of the of the The the request path If it's just a plain get as we define our api We're just going to read the index html file which is packed into our tarball. It's available in the current directory And we can just oops. There we go. We can just stream it out May I go for it? Um, so do we have any windows users left in the room? I fix the keys Most of them anyways, so here we just return the page itself If there is actually a a file specified after that, so if it's snake bin api slash abc 123 whatever it is We query the container to see if that exists This is the snake bin store container where all the pastes are stored And then we're going to generate a job And we're going to kick that job off So what this is going to do is this is going to say okay go and execute get file Give it the path of the file that was requested So that file which is the short url is this thing. So it's going to be snake bin store slash file name And then we're going to kick it off here. We're explicitly setting the content type application json And we're setting the zero VM execute header. So we basically We're ending we're ending this session and we're sending a message back to the middleware And the middleware will intercept this and go do a net new execution the reason why we have to do this is because um The question asked earlier was can I attach to multiple files? And I explained that well if you want to do everything local to the data It doesn't really make sense to do that. So because right now we're attached to the container file There's literally a sequelite file that we're attaching to we need to send out a new job to go attach to that script wherever it is in the swift cluster and then Do something on that in this case send that content back to the user Sorry say that again so Yep, so that happens here with this this this http resp call So by simply writing out to this device That content will get read and parsed by the middleware and they'll say ah, this looks very special I need to go create a job from this Otherwise if you just write something else out if you write any other http or just any text or even binary Uh, it will just get it will it will end The job and just return to the client Otherwise it can keep going on and on and on until it hits a time out Yes, david um There yes, so between these jobs. There is some information that's passed from one to the other Uh I don't actually know that part of the code super well, so I couldn't tell you which ones But all I can tell you is that it does happen So each time you submit a request it will spawn a new instance For specifically for that request You will not have any long running demons or anything like that. So Yeah, okay, so Because we're uh, we're awesome and we do things live Let's uh, oh, did I miss something? I think I missed something. Oh, I totally missed something from get file I'm going to be lazy and copy from solutions Get file Oh, it must be just get file to get file The same thing for save file. All right um Here's our html and javascript Our ui I'll just add that in Okay, so we're ready to bundle deploy and test So I will zpn bundle We've created this dot zap file. It's really just a glorified tar ball And I'm going to deploy it. So you remember we have these containers. I'm going to deploy it to Snakebin app with just this file name Okay, so now we've deployed it So we see a couple things in there. We see the tar ball itself and we also see that Zpm has extracted the boot slash system map Uh out from there. That's just sort of a convenience for zero cloud So it doesn't have to go into the tar ball and read it out every time it gets a request to run this application Okay, so let's see. I need to see what my environment variables are So we can make these commands, uh nice and easy So if I echo my os storage url I want to take this part after the v1 This is like my my account id basically And I want to set this to os storage token Okay It's the scroll. Yeah All right, I'm just going to copy this and paste this in the terminal and I'll talk through it a little bit Okay, so we need to do some very special things so in in order to To indicate to Swift and the middleware that we need to Do special things when requests come through for these containers We need to set some metadata. So we're basically saying for for anything that's interacting with these three containers That's api app and store Um, everything should be handled by This application here So this is just essentially just configuration for the deployment of your application We'll do it for app as well and store So here we that's why we set the The variable so we don't have to type it Okay And the last thing is permission for anonymous execution. So as it is we can execute this But we would have to provide an off token for each request Obviously if you want to run something like the go line playground or a paste bin which can be used anonymously for anyone We want to we want to specify that So there's some other zero cloud specific headers They use a similar syntax to the To the Swift acl syntax the read and write acls So this is just a fancy way of saying allow anonymous execution on all these containers Okay, and we're done. We can test it So, uh, let's go command line first. Let's upload a script. I have 10 minutes. Okay, cool At least we'll get through the first part and then you'll see the The wonder All right, I'm going to create an example file really simple and then I'm going to Upload it using this curl command Something was not happy Live demos What am I missing? I'm missing something. We did not get anything back. What did we not get anything back? I did did I not put anything in the file Put something in the file Oh Do you think it's uh, not Wrapping correctly. Whoops Oh Okay Thank you Well, maybe it's putting a new line in there or something. Let me go smaller font Sorry It did not return. Why didn't it not return? Live debugging. All right Woo I wonder if I missed Uh, probably missed a step. Let's check There we go. Okay. I think my my zap YAML file was a little bit messed up. I probably just skipped something in there Okay, we're good. So here we have our file I'm going to curl it to Return the contents and because it's all running in vm. It's kind of slow. Whoo. All right Nice. I want to show you guys execution. I think we have time Let's do it. Yeah, we have five minutes We have how many five five minutes. Okay. I can do it in five Okay I'll get file.py and then spslisions snakebin part 2.py snakebin Pi what else do I need? I'm going to redeploy Let's see if we can get at the first try this time. I'm jealous. All right Let's run. There we go. Nice It's very impressive that you got this running super fast the whole snakebin thing and you have no idea how many times I've run through this in the past month Well, don't tell them that. Oh, sorry, you didn't you didn't hear that I need one of those little men in black devices, you know to erase everyone's memory I just want to show you one of the quick things. So I've saved this to this short URL We can actually If we curl it, we will get the script itself If we append execute to the end of the URL, it will actually execute it. Nice Thank you. Thank you Question I'm only able to write that file. We didn't turn off auth. We just Allowed for very specific things So so so what we're saying is it's sort of like setuid We're basically saying okay, we're allowing users to execute our application But we're within the application itself. We're very tightly controlling what's happening You look perplexed Maybe I'm not understanding the question So the the owner of the application, which is me with with these credentials that I've set up while developing this I have access to those things and by by setting all of these By all of by setting all of these these metadata attributes on the containers. What I'm doing is I'm allowing I'm allowing anonymous users to Execute this application within a very specific set of constraints It doesn't mean I'm allowing people to run any application on my containers. It just means they can run this specific application on my containers Containers technically throughout the the lifecycle of of an execution it will touch all three of these containers So it needs it needs to be allowed access to those containers either read or write access I have that same reaction when I saw this feature land It's it's it's it's kind of black magic. It's very new, but it works Thank you. Any any other questions? Okay If you want to chat more about it, we can we can go out into the hallway and hang out I believe we're actually the last thing in this room. So are we the last thing in the room in this room to do that Okay, please take a shirt if you haven't already or certainly welcome to okay Thank you very much. Thanks for coming