 Okay, so welcome to my talk, title is Pythonic Approach to Continuous Delivery. Yeah, I work for Blue Yonder. We are a company. We are building data science applications and we also operate them. My job there is, I'm the main developer of the so-called data services. So we are offering some external data like weather or public holidays and so on for all those machine learning models to enhance the data. Yeah, and as I said, so I'm the main developer. And in addition, I'm also the main operations guy. So the question is how to operate and develop at the same time and services with just a single person like me. So more or less you can see so what we came up how this could be done. So who of you have heard of continuous delivery yet just for me? Okay, so maybe there's a strong bias if you're here. Okay, so just a short outline. Yeah, first of all, I want to I will explain what continuous delivery is with some definitions and allergies. So what my understanding is maybe it's not completely in agreement with everybody else, but that's my opinion. So how does such a delivery pipeline look like? And there we really dive into some boring details. Then the biggest question. Okay, I have working Python code. So how do I start? And there we really assemble such an exemplary production lines or a delivery pipeline out of some building blocks. And this we will do in a Python way. Then you could possibly go wrong some tips and tricks from my side from my experience and maybe a short outlook what might be in the future. Some wishes and a short summary. Okay, so let's start what is continuous delivery. So I think to understand what continuous delivery is all about is to understand what's the main workflow of traditional software development. And there we have two of such so-called silos. So we have a team of developers. They are doing all those code stuff, releases, features, continuous integration, things like this. In the end there's a product which is such a package with a version. And on the other side we have operations for operating, for example, a web application and stuff like this. So what the company is really doing. And there we have terms like packaging and deploy, lifecycle, configuration, security, monitoring, all this stuff. And so the more or less traditional way of looking at things or really how to do things was we have such a wall in between. It's sometimes called the wall of confusion. And yeah, those two silos and once the developers are finished with a new feature, they just take it, throw it over the wall and say, okay, it's ops problem now. They should look how to deploy it, how to lifecycle it, how to monitor it, how to configure it, how to do security stuff. And now what's happening is there's this new development of DevOps. And this just means we tear down this wall and we say, okay, all those things, it's just one thing. So we cannot divide it with a wall and say, okay, you do this and you do that because it's just one thing. We want to deliver this web application. So this is what one could call continuous delivery because now we don't have this wall anymore. Or maybe with such another picture. So one could see continuous delivery as extending the development, so with code and versions and stuff like this into production, operations, and also extending operations into the developers workflow. And the important part is that now the development really includes the entire value stream and value stream is really just, you have an idea, you think, our customers may like this new feature and then the value stream starts from that idea until it ends up and gives value to the customer and that's the important point. There's no wall in between. We just say, okay, that's the value stream and development includes the whole value stream and it really is important that we get feedback inside these development cycles. Okay, now to emphasize this continuous delivery, so okay, we now know we have to deliver value to the customer, but continue delivery means this is the same child thing. We want to release early, but we want to release often. And the same continuous is far more often than you think and this brings with it a real explosion of complexity due to these increased demands on security, safety, failover, monitoring tests, and this is only possible via automation. So any manual workflow will just completely destroy a real continuous delivery. As we heard, it is far more often than you think. You cannot do it just a little bit faster than before. It's a new quality. You really have to do it automatically, your delivery of all your features. Okay, there's another thing. It's called the Pokayoke. Pokayoke is any mechanism in a lean manufacturing process that helps an equipment operator avoid mistakes. Its purpose is to eliminate product defects by preventing, correcting, or drawing attention to human errors as they occur, and that's maybe such an important point. So we now need to leverage automation, and automation is something different than the workflow of humans. As a human, you can detect errors by just looking at things. You feel there might be something wrong or it doesn't look correct, and that's a complete different story once you go into automation. Machines are dumb in that side. So we really have to build in mechanisms to detect failures early on, the earliest possible. So we could see all this stuff now, like such a car manufacturing, automated robots production line. And now, if we compare it to what we did so far, that traditional software development was just programming those robots. But in the end, we don't earn any money if we don't deliver cars. So that's just the new side of the thing. It's just irrelevant to program robots. We have to build cars. Okay, so now let's look at this production line. How does it look like? And once you really do it, you always feel kind of in such a jump-and-run game. So first of all, each change, that means some software developer somewhere around the world is committing something, is deployed to production, full stop. Each change goes into production. There's no more, ah, let's wait until someone who might be responsible for it then decides or whatever, or it goes through, I don't know. So you just have to keep in mind, each commit ends up in production as soon as possible. So there's some time in between, but let's say five minutes later. Five minutes later, your commit will be in production and hit the customer. Yeah, exactly. So unless it is proven to be not production ready, and that's like in this jump-and-run game. So what we do is we design challenges for such a change, for such a commit, and this is this Super Mario thing. And if we fail to detect a wrong change, which is called a bug, it will end up and catch the princess. So we don't want that. So we have to design challenges for this hero. Yeah, and maybe there's more or less a hint. So I don't think that it's possible to start a little bit and then have some other workflows manual as before. So I would recommend to really start with a lightweight, a small pipeline, but you automate the whole process. So once you have such a walking skeleton, really where you first just commit something and it ends up in production. So this is the first step you should do and not let's start in the beginning and then we keep the rest as it was 10, 20, 40 years before. So yeah, here you can see once again this jump-and-run level kind of thing. So the delivery team, so we don't call them developers anymore in contrast to operations. It's just we deliver value, so it's the delivery team. So operations and developers. They check in, maybe this is the first feedback, a merge conflict first feedback, it gets back, or then we have the traditional continuous integration which approves the correctness of the code. Maybe there we get feedback, let's say two minutes later, red, okay, fixing bug, committing again. Maybe this one goes through and then there are stages later on. So the next thing would be an automated acceptance test. We will see in a minute what is in there. Maybe some acceptance tests, yeah, and way more. So this is, let's say, the minimal production line we have to implement. This picture is stolen from this book from Chess Humble, so who has read the book? Read it, it's very good, it's full of practical tips and it's really helpful. Okay, so another picture from that book just to give you a feeling. So it hasn't to be that it's really all stages, we call them stages, those jump and run levels, that they have to be streamlined, they can also be executed in parallel. But important thing is that naturally we want to increase our confidence in that change until it hits production. So in the beginning, in the first stage, we are not that sure, we know maybe that all the unit tests passed, but is it really production ready? So maybe it's the logo red or is it green? Maybe that's not tested by unit tests, but maybe it's important because it's our company logo and so on. So going to the right means we have more confidence in our product. The same is to the right, the environments where we test these things needs to be more production-like. So we have an environment which is called production, this means maybe an EC2 instance on Amazon. So it's a good idea to try to be as production-like within the environments as possible in the early stages. So if it's too expensive, okay, then you have to somehow do trade-offs here. And of course we get faster feedback to the left. So once someone commits, first thing is this commit stage, this is the traditional continuous integration, you commit and then unit tests and so on. And there we get feedback in two minutes. This increases, I mean, the latest feedback we get is from production and that's the most dangerous and most expensive one. Okay, so now let's start. We have some kind of working Python code, our web application, name it Hello World or whatever. So what do we do? And yeah, the first step is we need a proper deployment artifact and yeah, so in the Pythonic world, of course we know, let's call it a normal Python package. I think all of you know that there is no such thing as a normal Python package. They all look different, they have different versions, dependencies and so on. So let's call it a standard Python package, but you could do anything else like proper Debian packages or some people use Docker for it, for the complete packaging, so binary package of your artifact or if you want you can do it by hand with just some tar files, it's up to you. Some constraints, it should be uniquely versioned, so that really if you know that's the number, you should know that's a revision in Git. It should somehow manage the dependency, so what do we expect our environment to be? And yeah, there's a small hint, so we built a small tool called PyScaffold and there was a talk on Monday, less known packaging features and tricks and who was there? Some people. He had something similar on his last slide, it was a cookie cutter template, so this is nothing else than just a template of how to build a standardized package, so at least for our company it's extremely important that all the packages inside the company, inside our production delivery pipelines really look the same, and the same means, for example, we see later, so it's really easy to just say pip install PyScaffold, put up my app and then you have a package which does quite some things, for example, gives you automated new versions per commit, or doc tests and swings documentation and you name it, a Git repository and stuff like this, so it's really handy. And yeah, with these two lines, we have the first thing, we have a proper package. So next thing, I think most of you know continuous integration, and yeah, in principle this means we execute our unit tests as we do it. It's, yeah, so you can use any continuous integration tool for that, build port or travels, there are quite some tools out there. Best is, of course, it's really a server running somewhere, which commit is automatically executed. That's a very important step. So really, if there is a commit, we really should know at latest, 15 minutes later, whether this commit was good or bad. Anything else would be a fail. One thing, okay, yeah, at least from my experience, it's a good idea to already start to have different stages. So really, maybe some unit tests are running really, really fast and others take, let's say, one minute. Then it's already there. Best practice to have several jobs even there so that you really have the fastest feedback you can get. So after 15 seconds, the first feedback and then the longer running tests take up to 15 minutes. So just to help you, if you're not creative enough for those challenges, for our hero, normal unit tests, we know all of them, what it is. And there, my definition would be we only test the pure code there so we don't need any environment. Environment might be some running instances of services like an S3 bucket or file system, maybe that's debatable, or database, stuff like this. Then integration tests or component tests test how the parts get stuck together and there we can use databases, maybe some small dummy databases. So that's up to you. You can do whatever you want. Then, of course, something like static code analysis might make sense. Pylin, PyTracker, things like this. I think it's a good idea to measure the coverage so that you detect if there is a huge new 20,000 lines of code with zero tests, then one might ask what's happening. Doc test is a real good thing so that you test your documentation that it's the latest. What we get out after continuous integration is really a fixed artifact, which means we have now a version and we know this version is tested with our continuous integration. For example, in this PyScaffold, PEP440 is the PEP4 versioning. Somehow the Python community decided that for continuous integration where we really need for each commit another unique version, it has to look somehow this way. So the first three numbers are really tech. So if you tag git tag something, then this is the tag. Post says we are after this tag because we are developing further. And dev15 means it's the 15th commit after the last tag. And the thing behind it, it's ignored by PEP but it's the git ref. So it's quite easy to then reference from your version. When you find a package somewhere, you can directly see to what git commit. It's quite handy. Now it's time to fill up our artifact repository and maybe I forgot to say you need an artifact repository. We use DevPy for this and I think Holger is not here. So he gave the keynote this morning. He's the core developer of this. So this seems to be the logo of the DevPy server. It's what you know. It's an on-premise or on-source solution for this pypy.org. It has some quite good features so we use it quite heavily. And I can reference to a talk by Stefan Erpp. He's also from Breyonder. It's at 12.30 in P1 and he really deep dives into DevPy. So if you're interested, it's really interesting what we do there with some indices and private indices and inheriting indices and so on. Quite interesting. Okay, now that was the fun part. So if you're a traditional software developer that's the part where you know what's happening. It's the continuous integration part. And now comes what I have felt for you as painters in configuration. Maybe you can add packaging or release dependency management. So for the next stage, this is the acceptance test. And the acceptance test, we need an automated deploy because now we triggered continuous integration and continuous integration automatically triggers our acceptance test. The acceptance test really tests the behavior of our service or web application or whatever. So it's really, you should design them in a way you now are the user in front of your web application and you really need to test how it behaves. Something like if I click on the item and type in my credit card thing here, then I should get an email saying that I have bought this thing. So it's really the full stack. So everything you want your application to behave should be tested here. And for this is clear, it has to be automatically deployed into some production-like environment, let's say Amazon or in-house somewhere on a server, like Debian or whatever. Yeah, and I think it's crucial to have really the same code which does the automated deploy for this test. It's the same that you use later on for the deploy into production. I think it is not a good idea to have a script which is called acceptance test deploy and another script which is more or less a copy is then called deploy into production because once you find a bug in the deploy, also the deployment is tested here. So we test here if we are able to deploy our web application into production so that it behaves exactly like we want. It's really hard to pin down that you also have to fix the bug without deploy into production. So that's not what we want. Yeah, as I said already, the environment has to be as close as possible and this is just my experience. So somebody asks you, what's your guesstimate of how much time is needed for our automated deploy into production and just do it, multiply by three, no matter what you thought, what you need before. And I don't want to say that you shouldn't do it. It's just that you should start very small. So pick up a really, really small thing maybe where you think this should be done in three hours and then you need one day. So don't start with, I think we should get this all running in three weeks because chances are high that your management will maybe tell you after three months, come on, what's happening here. Maybe we have to stop and that would be bad. So start small and then you get a feeling where are all those pains and what are the blocking things. It's really a strong advice. You can do whatever you want. A normal shell script is good enough. Why not? It worked years before, but configuration management tools, who knows what configuration management tools are. Okay, kind of some. Yeah, it just eases everything by orders of magnitude. So there are things like puppet, salt, chef, you name it. We use Ansible, it's the Pythonic tool. It's written in Python. You can quite easily extend it with your normal Python skills. And Pythonic also because it's really, really simple. It's lightweight. And the very important thing is it's declarative. So you don't, in shell scripts, you write do that, do that, do that. And here it's more or less like SQL. You just say, I want in the end, have this. And this really eases your deploy quite a lot. And this could be an example Ansible playbook just to give you a feeling. So what do I have to do to really automate my deploy? Okay, we want to deploy something on a host called web servers. They are defined somewhere else. Some IP addresses, things like this. Then, okay, we want our app to be installed. We use PIP for it. There is a module in Ansible. So we say the package named MyApp and maybe we created it with PySephote and it's already tested and uploaded to the DevPy. We want to install it into a virtual amp so that we are not really messing up with some already installed system dependencies. Yeah, and there we say, okay, as an index use our own DevPy because there we uploaded our package. Yeah, it's maybe not a perfect example. So really rethink once again. If you don't just type it and use it for your production, it might screw up because there's minus, minus, minus, pre. This would also install for each dependency development versions. For example, for requests and so on. So maybe you should rethink it. For example, first install all Pint requirements and then just install in the end your latest development version. And then just start your app. I did it also in a declarative way. This shell script, it's called MyAppStarted. If it would be plain as a script as you do normal, it would be called StartMyApp but then it gets broken because it's running already. So always think in declarative descriptions. Okay, so acceptance tests. As I said already, they really prove the behavior. Really be careful here. That's your money. Full stop. It's your last chance. It's really the final boss of this thing. And yeah, tools. There are really some tools like Behave, for example. They are built exactly for these kind of things. If I put an item in a checkout and press button X, then you write it really in that sentences. So even your management can write those acceptance tests and you execute these like tests. Selenium 4 if you have a GUI, things like this. But yeah, do whatever you want, but do it. And that's it. So that's the last step. So all those acceptance tests are passed. That's the last step. So you might want to have some additional non-functional things. These are things like performance measurements so that you don't have any surprises that a checkout now takes three hours. Security and maybe explorative. That might be only feasible with manual tests. So really if you have an experienced tester who really tries to screw up your thing, but that of course is then... You have to wait until the tester has time to test it. So that might, if you really have to wait for it, that would be such a blocking thing until the tester has really the time to do these manual checks. But additionally, you really want to have such a manual approval that would be such a button where someone is pushing the button. But it is not for taking over responsibility to some manager who then gets fired if there's a bug. It's more or less for things like you want to do a marketing campaign because your new feature is really, really a breaking change. So you want to coordinate somehow, maybe, or some other things. For example, if you have some legal issues, maybe you're not allowed to leak some information early on. If possible, you can do things like there are many different possibilities. Canary releases, that's something that Facebook does that maybe you can already give some beta versions to some users, maybe randomized or you select some users who can already do it. So that's up to you. You can do whatever you want. What we do is we just have one button and then, okay, now it's deployed. Another important thing is it gets quite complicated to have all those stages and this workflow, which is completely optimized, you don't have someone who says, we should not forget until next Friday to execute XY. So you need somehow coordination, past unit tests, past integration tests, past acceptance tests, no security testing, things like this. And yeah, we do it with Jenkins. There are some specialized continuous delivery tools. I know from ThoughtWorks, it's go.cd or there's IBM Urban Code. I never use them because we just have Jenkins already there, so we glue those jobs together. And as you see here, there's a job called PodDeploy Current. Good enough, this one is blue, but PodDeploy Latest is red, so there seems to be an issue with the latest deploy. It's of course bad, so it should be covered somehow early on, so these tests were bad here. Yeah, so our manual approval here is to click on the Jenkins build button and yeah, this means deploy to production. So you see already, it's not really a specialized tool for that, so build and deploy to production is maybe something else, but it works. So the question is what could possibly go wrong. It sounds quite trivial to do it, but it's not. So my advice is to really keep everything, simple stupid. Don't do any complex things, you will screw up. Always if we had an issue somewhere, it was always the first version was too complex, then we did a real dummy implementation. We said, okay, that's the simplest solution, and this is always better. You have to automate all the things, really all the things. Okay, I do it because I'm really lazy on doing things twice, this is already for me not acceptable. But there are quite some other arguments for that. You complete delivery pipeline from idea until customer is in version control. So you have, for example, predictable recovery. So you know, one minute, okay. So you know Amazon goes down, you can deploy it somewhere else in another region because you know everything is in version control, there's nothing missing. And machines are just better in such repetitive tasks like I have to reconfigure that script in order to be machines are just better, they don't do errors in repetitive tasks, and you can concentrate on the value delivery. So you can think of features which might bring value to the customer, so that's the important point. Good advice is to really maintain, read factor, block time for your delivery pipeline for your automated deploy. You have to migrate to new versions of Ansible. You have to migrate to new versions of requests and so on. So really do it. Don't just think, okay, it worked once and now we can stop. It's automated, it never is automated. We have to continuously improve the deployment process. And yeah, so the cloud thing is a really handy thing here. So if you have to work with tickets, it's quite hard to do automation. If you for a new F3 bucket would need to write a ticket, it's really not a good idea. So better have some cloud thing there where machines can automatically get all the things automated with APIs, things like this. So the future, yeah, okay, I'm ready. Okay, so we learned from Guido, he has the same opinion than me. Dependency management, packaging, all this stuff is somehow not so perfect in Python, so we really have to find ways to make it better. If really you automatically deploy to production after each commit, you really want to know what's installed there or is it now a new version this morning and we deployed once again and all of the sudden we have a new version over there. So it's not really perfect. There's the two worlds problem. We have, for example, the devian world app is installing half of your dependencies, but then comes PIP and installs another half, maybe in a virtual M with complete different versions of all the dependencies. That's not perfect. And yes, such a pythonic, let's say, really lightweight and easy to use. Continuous delivery tool is also missing. A tool which is really aware of the delivery pipeline which knows, okay, this is the acceptance test stage and we deployed version number X in that yesterday by user number, that and that. So that's still missing and many, many tools, really many tools are optimized for manual workflow which means there is the operations guy sitting on the terminal and typing in at get install Y. So Jenkins cannot type Y, so you immediately broke your deploy. You have to have a tool which types in yes for you in a non-existent terminal. Yes, that's not okay. So we have to work on it, we have to hack on it, we really need to improve the tools there. So really short summary, okay, CD is really, it rocks, it's cool. Agile, we have just feedback from the customer so we don't lose money. Automated is better than manual, collaboration is better than silos. You can do it, just start example building blocks once again, Python for packages, DevPy, Artifact repository, Jenkins for continuous integration steering of the pipelines, Python unit tests, that simple stupid for tests and Ansible for automated deploys and you really need some courage. Once you do it and commit something and it ends up in production, you really say, okay, that's something new. Okay, but that's it. Okay, we'll all have time for one short question. Okay, one short question. Have you heard of the Jenkins workflow plugin? I heard of it, but I haven't never used it. You should, because then Jenkins really becomes a continuous delivery system, not only a CI system, I struggle a lot with having those single jobs that are connected. I think you're connecting the jobs manually more or less, but the workflow plugin is really a game changer for CI and CD. I heard that I might try it, yeah. You should look into it. It's in the backlog. Thank you. Okay, if you have any questions, if you have any questions, we are at the Blue Yonder booth, so I'm over there, so you can come over and ask questions there or here.