 Welcome to the last session in this room for the day. So I'd like to welcome Graham Dumpleton to the room. Thank you very much. Okay, thank you. As I said, the talk is about warp drive. This is a personal project of mine, which I've sort of been working on for a year and a half to two years and haven't said much out there about it. The goal of this project is to make web application deployment easier. So the big question then is why would I do it? Isn't web application easy, deployment easy enough now? Yes, no? So let's take an example, Django. If you're using any sort of application development, hopefully everyone here is using virtual environment. We won't shame those people who aren't. So we should first off, we'll create a virtual environment and we're gonna activate it. So we're actually now operating in the inversion environment. So if we now go and do pip install of all our packages, they'll get installed into that virtual environment instead of the system packages. And our next step is we'll try and run up, manage, run server. Now, the assumption here I've got is I've already got my app in code in place, right? That's all been done before. So I'm just gonna go through the steps here. And so that works, that brings up our application, but you'll note there's a warning there. We missed one step. So there's another thing we have to do. We have to initialize the database for applications. So we need to do a Python managed py migrate and that will go and create all our database tables in our data. Now, another thing we have to do, which if you were to run that up now, yes, we'll define, you'll get your data and you get it better, you won't have to log in. So another thing you have to do is create a super user. Otherwise we can't actually log into the admin interface to Django to do anything. And finally, we can run the server again. And in this case, we actually wanted it to be accessible outside our box. So we did need to also tweak the configuration of how we start up that server so that it would listen on all interfaces and be accessible outside the box. And we get an app. So there's a whole bunch of steps there which we had to do. Now, it looks simple, but it's not. It's always still a step you have to do and you have to remember. And if you're an experienced Django person at second nature, right, you can do it with your eyes closed. But new people have to learn all this stuff and have to go for all the docs and work out what all these steps are. Now, what happens if we want to deploy this to a real web server? We were using run server there. That is the inbuilt Django development server, which you're supposed to only use for development. It's sort of by default, single process, single threaded, which means it's sort of not going to handle high number of requests, there's issues around ability to serve static files and all those sorts of things like that. So we're going to go and use a real web server. So let's say you look at modWiskey. Apache is notorious for being hard to configure. Yes? Yes. So here's our little bit of configuration that we need for running modWiskey and setting it up with our application. So let's go through a couple of things. Well, first up here, we need to tell you about where our static files are, which our CSS, any other GIFs and images and so on. We're going to tell Apache that it's actually allowed to serve up those files. Apache has a security model whereby default, it doesn't actually allow you to serve up things from arbitrary directories. That's part of the security model. So you need to tell it that we're allowed to actually serve up those files from that directory. And we're using modWiskey. So what we're going to do, if you're using modWiskey, hopefully you're all using what's called embedded, not, get it wrong there, all using what's called demonMode. The default of modWiskey is a thing called embeddedMode and you shouldn't use that. That's a bad default, but that's the history of it. So you should be using this thing called demonMode. So we have to go and set that up. It just sets up things so that your web application is running in a separate set of processes and we can figure out how many processes and the threads we need, various time-outs, where our virtual environment is located, where our application is located, and finally we're going to set up our actual application. So this is telling us we want to run at the top of the website, our Django app, that is the whisky.py script. We're going to run it in this demon process we set up here and we're going to force this into what's called a main interpreter context. And again, if you're doing this sort of stuff and you don't know what I mean by application group because global, maybe you should ask me later and why you should be using it. And finally at the bottom, we also need to tell it that Apache can serve up our whisky.py file. Otherwise nothing's going to work. It's got lots and lots of complexity in all of this. Now, because we set up static files, we also need to configure Django a bit more. We have to go in there and tell Django where to put our static files. Because normally when you run the Django run server, it's just going to internally merge together all the different locations of static files and essentially serve them up from all the places that exist within your source code. So we need to tell it where we want to put those static files. But we also then need to run the collect static command for Django managed top pie to tell it to take all those static files from all these different locations in one. So what do we learn? It's not so simple, is it? There's all these different steps to do. So initialization sets, we need to create the virtual environment, we need to activate it. We have build steps, we need to install all of our required Python packages. Now, I only mentioned installing Django there, but that's not going to be enough. Usually you're going to have lots of other packages in there, be it Psycho, PG2 for Postgres database, you might have decided not to use Django's own template system and want JINJA 2, or you might need XML libraries and a whole bunch of other things. So you also need to install those and normally you're going to have a requirements.txt file and I just installed one packages but you might have to hold a whole bunch of those to install. And we also need to collect together the static files so that we can tell a real web server like Apache where they are so we can host them up. Then we have deployment steps. We need to initialize or migrate the database data. In this case, we only showed the initial setup of it, but the same migrate command when Django is used later to do database or migrations as well if we make changes. And we need to configure and start that whizky server. So why should you need to know about all those details though? That's the way the complexity comes up in all this and why people have so much trouble is that yes, there are instructions but if you're doing something slightly outside of what the instructions follows for that narrow path that's the tutorial or the hello world, then that's when things get messy and complicated. So what if all those building deployment steps were managed for you and you didn't even need to know? So this is what warp drives about. It's about hiding all that complexity. It's the magic glue that can make this possible in an easy way is the aim. And importantly, it's for local deployment on your own box during development as well as production, okay? So generally you think about, okay, I just off on your development side where you might be happy with run server. But then you go and deploy to production system and everything's done differently. That's where you're using your real server. And your environment is effectively totally different to your development environment. And that's where a lot of problems come in because your code may work in Django's development server but they're not work in your production because of the way the server works is very different. So let's say we can bring this together. Let's say we can look at how we can do this so it works the same way in your local environment as well as production environments. So warp drive, let's assume we've got warp drive installed. Let's go through now how we would use warp drive to do what we did before. So we're initially gonna go warp drive project Django and what this is gonna do is create a virtual environment for you for that project if it hasn't already been created and it's gonna automatically activate you into that environment. We now want to actually do all the build steps that we need to do. So we've installed Django, we had to do our collect static and things like that. So we're just gonna go warp drive build. Importantly here I'm doing warp drive build. I don't need to know what those individual commands were. Great for a user because you say warp drive build, warp drive build, warp by build. You don't need to do all the details. And if you wanna start the server, we just go warp drive start. Now if you look very closely there, start modWizgi. So I didn't configure Apache there, did I? That's actually started up a version of Apache for you with modWizgi in it and configured it all for you and you didn't need to know anything. It's to do that, it sucked the information out of the Django setting file so it knows where the static files are and can automatically put them in place and sets up all the configuration and things. So it's all automatic and that's it. So that's just going. Now this example I use Django. Warp drive does more than that though. So it will detect a manage.py file and say yes I'm using Django and I'll do all the special stuff that's required for Django. But if you've just got a flask app and you've got a wizgi.py file in there along with your requirements.txt, same steps. Warp drive build, warp drive start. Warp drive build again, we'll just see the requirements.txt file and install all the packages. So these are wizgi.py file, it'll start up modWizgi.express with that wizgi.py file. And so I can do that as well. If you are using tornado and you don't have a wizgi.py file because it's an async app but then all those guys are probably next door on the other talk. You've got an app.py file and you're starting up your own web server there and that could be tornado or it could be twisted even if you're using that. Warp drive will detect that as well. In that case it will go Python app.py and run it for you. It assumes that you will then use the correct port and away you go and it's working. And finally, what if you're using Jupyter Notebook? Jupyter Notebook, you have to run ipython... Notebook, is it? And it'll start up your web application. So if you put that in an app.sh file, Warp drive will go great. You've got an app.sh file, I'll run that instead. And again, you just need to make sure it's running on the right port. It's all for you, all that are for you. So they are just make all this simpler. Now, I've used modWizgi as an example but and I'm the author of modWizgi if you didn't know that. So you might think, ah, look, he's biased. I'm just gonna make it use modWizgi. But I have made it to default for good reasons. I think it's still good option compared to the other ones even though a lot of people are deserting me in going to youWizgi now. But I do have the ability in there if you don't like modWizgi for some reason, if you like religiously don't like what I've done. Then you can say, ah, I'm gonna use gooeycorn or youWizgi or waitress. If you're in that situation of running Django or if you run a Wizgi app, then you can make that decision and override it. The same thing, it'll create all those default confings in there and default configuration has started up. Now, one of the things that we're using because we're using Apache there, it was automatic and be able to be configured to use Apache for hosting static files. YouWizgi has a similar command line option you can use to have it host the static files for you. So in those two, I'll go and use those. But in the case of gooeycorn and waitress, neither of those have an inbuilt ability to handle static files. So what happens? Usually you go off and serve them up with a separate web server like Ingenix or you use White Noise, which is a Wizgi middleware for hosting static files. Now, you could go off and set up Django yourself to have that in there. But what Wolf Drive will do is it'll go, okay, you're using gooeycorn and waitress. I know where these static files are. I'm automatically gonna wrap your Wizgi application with White Noise to host those static files up. So you don't even need to bother about that. Okay, I'll worry about that for you. And in all those cases, I did a manual step back there of putting static route to say where the static file should be. If you don't do that, I will actually go and set it for you as well so that when you do collect static, it'll go and put them in a place I know where they are so I can use them later. So magic, lots of magic. Maybe too much, we'll see. Now, well, I haven't looked at initialized database yet. So initializing database, we did that Python manage.py migrate command before. And now, Wolf Drive itself won't do that for you, per se. It's not a bit of magic that it does for you. I did it at one point, but I actually determined it's actually a really bad idea. But what Wolf Drive does have is a concept of action hooks. And what they are is that you can define certain scripts which Wolf Drive will know about. And you can put in those capture information about certain steps that are involved in the overall life cycle around deployment application or running application. And one of them is setting up a database. So you can provide this set up action hook script and say in there Python managed.py migrate. And that way you've captured that information about that command inside of your source code repository. And rather than you having to manually run it, again, you've got this simple command of Wolf Drive set up and it'll provide that step. Similarly, if you want to capture the commands you need to do for migrate, again, you can have migrate in there. Actually, I should point out that one also down the bottom has super user creation. So I put two in there, migrate and super user creation. For migrate, you only want the migrate in there. So separate script called migrate and you can capture those commands in there. And you might need to do other things in there as well if you need to do other sorts of data migration outside of the database as well. So the example of database migration hook. And I wish Django did allow this to happen a bit nicer. So one of the important things about doing database migrations when you talk about in a production environment, you need to make sure your database is actually running first, okay? So I've got a little bit of Python script here, check database and I'm gonna invoke that using Python managed up shell. And this is a mess, I think they're changing it so in the next version of Django there's gonna be an easy way to run shell against a script file. So I don't have to do this mess and then I'll run migrate. So that's an example of that action hook script where I can capture these commands. So action hooks, things you can do, stuff to do with the building of the image. So things that you want to do before any of the Python packages are installed, you may want to do this if you need to install extra little libraries that you're gonna build up a custom module which requires those libraries and that might require further setting environments. And then in between these two steps all your Python packages will get installed and then you can do extra stuff after. So if you need to do something which is reliant on the packages you install. So these are all things where you can actually record all those special steps you need to do as part of it. And all of that together will be run as warp drive built. You don't need to remember all steps. Similarly with deploying, you can set up environment, last minute generation of configuration files if they're dependent on environment information to pass them from the enclosing environment you're hosting it on. And just before you run up the server you can also have things to do deploy if you need to run a background process or something like that. So that's build deploy and we've also got this life cycle hooks. And I mentioned set up and migrate. Now I've got two others there, verify and alive. And this becomes important if you, especially running in a containerized environment, if you're using Docker, you may deploy your application in that Docker container and start it up. But you actually need a way of testing that that application is ready to start handling a request before you change the router configuration to start sending requests to it. And that is something where you might just do curl against the web server but you might need to do other things like check the databases running or check that data is in place because it was needed to be there copied down or something like that. So we can capture those whether it's ready or not in this verify step. Actually, that's done wrong. I'm missing one. There is actually a separate ready one. I've left it off the slide. Verify was one actually related to checking whether the build of your application was done correctly. So this verify one is a ready one which tests whether you're ready to start accepting requests. And finally, you might want to actually run a command periodically against your application to see whether it's still alive. And that way, if you detect your application is not running any more properly, you can shut down the container and create a new one, for example. So different life cycle hooks. But one of the big problems when you talk about difference between development and production is I've mentioned environment in a bigger sense but there's actually even more than that. It's just environment variables, okay? You might, in your deployment system, it provides all these different, well, those scripts there, my scripts before there with my deploy and script, for example, that may have settled these magic environment variables and that may be partially based on what things being passed in from a hosting service. If you're in a development environment, how do you execute a command against your application development environment if you need a special environment variable set? You don't want to actually go and have to duplicate all those environment variables setting that you had set in deploy-env. Following what I'm talking about? So imagine you need to override Django setting module because that's different for development and production. You don't want to have to manually set that command line. So I've got this command warp-drive-shell and essentially it will go through some of those deploy steps of deploy-env-deploy-config. Make sure that your shell environment is all set up exactly like it would be when your application deployed. Then run commands to do stuff. Or you can do warp-drive-exec. If you don't want a shell, you just want to run a run-off command. And this is where, for example, I do that and you can get to see some of the magic environment variables that warp-drive itself is setting, which wouldn't exist in your normal shell. So there's things that warp-drive is going to set. So that's all good on your local box, but what's happened lately? Obviously containers is the big thing and Docker sort of burst on the scene a few years back at one of the PyCons in US. And it's the raging craze at the moment. It's getting a bit of competition now with Rocket and there's also the standardization going around this with the opening initiative. So hopefully you've all heard of Docker. Yes. So we want to deploy this to Docker now. What do we do? All we need to do is warp-drive-image-jango. And that's going to go off and create me a Docker image now. And I can run it. You have not had to worry about how to do that. You've not had to create a Docker file for yourself. So what's this actually doing? So I have a Docker base image, which has in it all of the Python tools, well, the Python interpreter itself has all of my Apache and warp-drive installed in there. So it's all in there in the base. So what's going to happen is that when we build that image you're effectively taking that image and we are injecting our source code for the application in there and we're going to do warp-drive-build. Exactly the same command as you did on the command line when doing it in development environment. And that way you've got the consistency, you know that things are going to be run out and built in the same way on development system as in your production environment inside this container. And you get your final container, which you can run. Now, if you did want to do this yourself, like I've got that warp-drive-image-command, but if you do want to do it yourself, then you can use that image direct. So at the moment it's called grumpy-dump-olden, warp-zero-centos-34. You could, if you're familiar with Dockerfile, that's just a Dockerfile. You just copy the source file in, you go warp-drive-build, warp-drive-start, and that's all the rest to it. It sees you've got a requirements text text file. It'll go install all the packages. It'll set up all the code and all the ways it needs to be. Now that's the manual if you're doing a Docker. When I did warp-drive-image, that's actually using a tool called source-to-image. Source-to-image is a nice thing around all of this because if to do that Docker-build yourself, you would have had to actually then create a Dockerfile somewhere, so you have to check out your source code, you have to create your Dockerfile, run Docker-build, and get all that, and then get your Docker-image up to where you need to deploy it. But imagine if you got your source code up on a Git repo somewhere and, well, you don't have it, if you wanted deploying that into an environment, it sort of becomes a bit of a mess to check it out, create a Dockerfile, Docker-build, and so on. So source-to-image, I can actually run this thing. I can run as to I build and just tell it my repo name. I tell it, what is my builder image and what the name is. It will go out and check out that source code for me, run up my builder image, index the source code, warp-drive-build, stop it, commit it, and gives me a container at the end of it. So that's what warp-drive-image is actually doing. But you can do this separately as well if you want. Now, source-to-image, if you now want to deploy that, you could take your Docker container, you've built that way manually and just run it up on Docker-service yourself, or if you've got some existing pipeline using jankers whether to build or like thing and test it and deploy it to some service, you could do that. I'm from Red Hat, obviously, this is my little salesy picture, okay? I'm gonna go away with one slide. We have a thing called OpenShift, if you may not have heard it, a lot of people might have heard of Heroku. Yes, so Heroku is a platform and service out there. It's been around for a number of years. OpenShift, this has also existed for not quite as long as Heroku. And both of these things have used containers all the way along, but Docker is long now. And what we did with OpenShift in the last year and a half has gone and rewritten it totally. It's now based on Docker and Kubernetes for scheduling. And the tie in here is that source-to-image I mentioned there is actually supported by OpenShift. So I can very, very simply go into OpenShift and I'll select my warp drive. I'll give it my git URL and I'll deploy it. And that's all I need to do, okay? Nice and simple. And you're guaranteed the way it happens is the same as what you saw me doing the Docker image manually or on my environment, because it's still just build warp drive start. Nothing different. You can also run that on the command line if you want to deploy from the command line can dig on and do with that as well. So integration possibilities and what am I up to? So my goals and all of this, I want to provide a way that makes it very easy to deploy a web app direct to a host. So that can be in development mode or even just deploying direct to a host. Using Docker or Rocket and supporting Docker at the moment to be able to deploy to a platform on the service such as Kubernetes or OpenShift or other container platforms feasibly, it could be done as well. Because if you can just bring out a Docker image, then you can just take that and run it up on Kareena from Rackspace, for example, or if you're running a Docker service up on DigitalOcean yourself, you could put up on there as well. And finally, technically I could also make this work on legacy pass environments. So I like to say legacy, I like to say legacy because they're not Dockerized yet. I have had an earlier version of this working on Heroku. So it could also be brought over to these other pass environments as well. So overall goals, I want to create a best of breed Docker image. So I've got a Docker base image here. And if you've ever done any Docker stuff, what do you do? You go off and look at what's the official Python image on Docker Hub? And then you scratch your head and work out what do I have to actually now do that to get my image application in there to build and get everything installed. And everyone's working it all out themselves, which is stupid. Everyone's reinventing the wheel yet again, not necessarily doing it the way that's giving, using best practices. So the idea is to come up with a base image which has all these best practices in there, has all the things we need in there. It has the important things like making sure you don't run as root, has all of missions in a way that sort of locks it down as much as possible in case someone does break in and they can't do strange things, all these sorts of things. So that's one aim. The next is to take that image and it's not just the Python in there, the fact you've got it in there is to embed these warp drive stuff. So you've got these build it script, wave building it up and then being able to deploy it. So as you saw in the original Docker file, it's from image, copy source, build, start. And that's everyone, that's all anyone ever needs to do. And I wanna make these scripts again, just not production and Docker but also in your local development environment if you're doing development or direct to a host and so on. And works with major hosting services. So where am I at with all this? What am I looking for? So I said I've been working on this about a year and a half to two years, progressively tweaking and so on and so on. And yeah, I'm at a funny spot right now. It's there now. I haven't decided that I'm at the point of I'm doing the final spit and polish or whether I'm just adjusting the deck chairs on the Titanic because I've given this, I've given this talk a few times before and people go, oh, that sounds really good. And I'm sort of like said, okay, I like people to give me feedback on this and no one ever contacts me and so I don't know whether to go ahead with this not because who here's got their own open source project? A few, you know how dangerous it is putting an open source project there, right? So if you do start, people start to rely on you, start to nag you, I need more Docker, I need help with this. So it's a big challenge, a big commitment and I've got a few open source projects out there and I'm sort of the boardage. I don't know which way to go on this one yet. So any encouraging feedback, that's what I'm after. Is this a good idea? Do you think you would use it? I guess it's the important thing because it's great. You can put it out there if no one uses it as well. So that's what I'm after. There's not a lot of Docker on it at the moment. I have done a few blog posts on it but I will now, after this conference and the rest of the year, starting to be actually getting the more docs in there, still doing a last bit of polishing and some of the stuff. But otherwise it works, it's being used. I've been playing around for ages and using it and just doing it. And just last that little chick tweaking. So that's my details, my email address, where are you at me on Twitter? If you wanna know, I see a little bit of, the little bit of docs that exist already is www.getwarped.org. There's a Git repo and that is my blog address down at the bottom where you can find things. Now at the end there, I did mention OpenShift. If that looks appealing to you for OpenShift, so it's an environment which you can deploy Docker containers to. These are Docker containers you bring in from outside or you can use OpenShift to build it up for you with our source to image tool. If you are interested in that, I've got some books. If you don't want paper, because you think that's ecologically unsound, then there also is an ebook one. It's available for free download. So he's not giving away for free, I'm not charging as well or anything. They're not big enough to hold a monitor up there, right? So they may not be that useful. So if you are interested in it, I'll get a few of those as well. And that's it, that is Warped Drive. So if you've got any questions, I'll gladly take them. Any feedback? Thank you, Graham. Only a couple of minutes for questions. So quickly, is there anything special that you have to do for Python 2 versus Python 3? I have base images for both. So yes, you could. I have base images for 2.7, for Debian and then 3.5, for CentOS I've actually got 3.3, 3.4 and 3.5 for various reasons. Yeah, thanks, Graham. I was wondering if I wanted to sort of grow my app out. Like I need like my database container and my Apache container and my web front end and I kind of wanted to create my own little city. It was like, if I wanted to get Warped Drive to kind of do a big deployment, how hard would it be to extend what you've got? Warped Drive is only intended to help you with deployment of your Python web app component or Python application component. For things like database, obviously you can, with those action hook scripts, do things in there to help you set up environment variables to match. So for example, if you're hosting environment, past environment rails for database locations and things like that, but they don't quite match your code of where it's expecting it. You can use those scripts to change variable names and things like that. It's helping you with that sort of thing, but it's not helping you with how to deploy any of this outside. That's where you still need a platform. So one example of that is OpenShift. So OpenShift can help you of deploy a database or the web app or all these other bits. Or if you're using Heroku, if I decide to go and add the support in this for Heroku, then again, this will just help with deploying the web app to Heroku and then you use Heroku's separate database as a service for the database bit. So it's not helping in the outside of just Python stuff. Oh yeah, I've got a question. I've got an app in production that's Django and Tornado. You can warp drive, handle both of those and the one hit. Dependent, what do you mean by that? Because generally, Tornado is using the async, right? Okay. In containerized world, you would generally deploy them as two specific containers. There is no reason why you could not run, use warp drive to run two instances on the same host. You're just telling it to use different ports. So there is a default port in there of 8,000, but you can overwrite that. And even if you're using a particular website like ModWizki, okay, I've done all this configuration for you. There's a file there. You can define all these extra options for what ModWizki express if you need to customize it. Same with Goonicorn or Waitress or Uwizki. You can customize it. So you can change the port. So you can just then create two instances of warp drive in different base directories and start them both up. You're still gonna need, the important thing is that it's not doing demonizing or anything like that. You still need supervisor D or system D to manage the actual warp drive start and what the process starts. It's not backgrounding or anything. Yeah, I think that's all the time we've got for questions. So one last round of applause. Thank you, Graham. Okay, thank you.