 Welcome, really glad so many people here Maybe someone to see the Facebook talk, then you're in the wrong room. So maybe you have to switch if you're really here for that talk I'm really glad. Thank you so, yeah, it's a long title and So a brief agenda what we want to do today. So first of all we have to Settle some of the definitions. So I want to give you a brief overview of things like what is a delivery pipeline what I understand about this About dependencies, maybe some packaging and then I want to do a small walk through through the different possibilities of how to bring Python into production and I will also give some pros and cons Yeah, and in the end, I mean all of you of course want to know the one question. So what is the obvious way? so Yeah first of all, I want to give some Insights, I think what the future might look like. This is my very personal point of view, but yeah Yeah, it's it's difficult to to do predictions for the for the future. We all know this and then I want to Do a small discussion. So I want to know what you think what the one an obvious way should be Okay, so what are we talking about and why so here of a small picture? Just sketching out what a thing is what what we call delivery pipeline. So who here has heard of this word before Yeah, some of you okay, so Yeah, so it means how do we bring things we develop? In the end into production, which means we deliver something to our customers Whoever it be whatever it will be it doesn't matter and here we have some stages and I will go through these stages So there's development some building packaging testing staging Q&A and then in the end it hits production of Of course not every each of these single steps is always present in each project There are small projects. Maybe they have they don't have any Q&A, but that's more or less the the general View how such a pipeline looks like Okay, so let's start we start in the beginning also how a normal delivery looks like we start in development So we have some requirements there We all know we are most of you are developers, I think so we know what we want to have in our development stage So we want to have fast iteration cycles We want to change something and then immediately get something get some feedback on whether it's good or bad and I would say that's a hard requirement We need to be able to run our automated tests in our development environment Maybe not all of them, but at least we all noticed some unit tests things like this. So yeah Maybe a nice to have a thing is that we have some kind of an emulation Like the production system. So maybe we are able to start our flask server things like this Or maybe we have a small Postgres running like in production. So but this is already a nice to have for some projects It will just not be possible to really have a production like system on your laptop and We also have some risks here. So we all notice. Oh, what happened? It works on my machine So this is the typical problem. We all have on the developers box everything looked fine but in the end production breaks and This also happened to all of you. I think so It everything went really well, but we forget to commit the very important file. So in production It is missing. So this is also we have some kind of a dirty working directory Okay, so Then the next stage would be building and packaging So of course You would say something like a python. We don't have a compiler, but that's not true So they are extensions see a siphon extension Things like this. So in general, there will always be some kind of step where we do a compilation of things So we build our artifact What would be required here So I would say a very good best practice is we want to build this artifact just once and then use it everywhere Yeah, there should be also, of course, there should be a possibility to compile for our target system Which means mostly the production system, but also maybe for the target system. So we need some kind of So either our build environment is is really similar or at best. It's it's really equal Or at least we have to find some ways to do cross compiling or whatever And Another thing is if we want if we have an artifact and it has a version it should be unique So if someone in the company says I have version 1.0 2 it should be really unique thing There should never ever somewhere Tar file or whatever was the same version But in the one there was this one config item missing and in the other one we compiled it in this should never happen So this should be just unique and I would say that's purely required If this is not not the case then you should at least think of how you can get there in the near future Then here the nice to have is something like an artifact repository for Python Maybe there is def pi or there's artifact to re nexus Whatever it is So it means we have a server where we can put our artifacts and everyone in the company can just pull off these artifacts Which we build in this build step. Of course the risks here if we have this Artifact build it it might of course be that there are bugs. So maybe there's some misconfiguration Maybe we have some dependencies missing whatever. So this is the risk what we have here. So the next step Testing so we could argue. Maybe testing comes before building the artifact or maybe Yeah, some other ways But I would say in principle because we want to use the artifact and test this artifact. It should be after the build step So what are what is required here? it should be automated and we heard this in the talk of Matelli With the testing layer. So tests that are not running automatically We don't call it test. We call it another development step Maybe or whatever. So if it's just someone saying I clicked through it and it looked good. This is not a test This is something else, but it's not a test and the conditions in these tests they should be as near to production as as possible and I mean that's clear. So in the end we want to test if Production will work out or if it breaks. So it should be just very similar We want to have reproducible conditions there. So not that at 12 o'clock it fails But on 1 o'clock it works again just because there is some condition changing And and that's more or less the quantum mechanical thing. So maybe you know a Schrödinger's cat If we measure something if we test something we always interact with the system but these interactions these changes we we do in our Thing we want to test. They should be as minimal as possible. So for example a thing here is that you install pi test and pi test brings in a requirement Which is in the end missing in production because you didn't write it into your requirements file Because in the test it was present because pi test installed it But in production we don't have pi test installed. So it breaks these this would be a change which is Happening just because we test it Then here nice to have it should be as fast as possible just that if we commit a change We want to have as fast feedback it breaks or it doesn't break But of course there are tests that run maybe for several hours just because it's it's a matter of fact Then it should be as fast as possible And also nice to have if we if we are able to really test all the tests we have for all commits Not even just on master But maybe if we have feature branches or even short-lived branches we we test on every branch Or pull requests or whatever The risks here, of course, I mean it's it's hard to read the tests test the test environment But not production. I mean that's obviously the thing. So, yeah We know a pretty Famous Example of this the the VW diesel gate thing was exactly this So the test was measuring that the test has the right conditions, but in production It's a complete different thing. So this is exactly what is here Maybe we don't want to do it in purpose like some people in VW, but it might happen Okay, so the last step here Before we hit production is some staging or quality assurance Requirements here we have we have an automated deploy. So this is not a different We really test how we install it on our target systems and it should be really a production like system And We don't want to have any Changes for for this staging So for example with the pie test example in this stage We do not want to have pie test installed on this Q&A system. So it should be really a production like deploy a Nice to have here if we can afford it We just have a real one-to-one clone of the production system if we are able to afford it Good if not, maybe it's good enough to have some smaller servers smaller databases But in the end it should be really as production like if we have a real production like system Then we also would have the policy possibility for example to do a B test So that for example 10% of our customers already hit this new Q&A systems because that's the best test we can have so We then are pretty pretty sure it works for everyone if it works for 10% of our customers But the risk is very small. It just hits 10% of our customers and not 100 the risks here Maybe that's not so critical anymore. We have cloud. We have things like this, but 10 years ago These systems were always purely maintained. There were things like all we forgot to update the last Database migrations and it's running a very old Maria DB and it's not not really working We we don't have installed all the the dependencies there. So this of course should not happen So it should really be as may as good maintained as the real production system Because in the end it's part of this value stream what we do development and so on so there's really no no Yeah, we cannot cannot afford to to say, okay, it's just the staging system it if it's read It's it's not that bad that should not happen Okay, then we hit production So here are some hard requirements I know that some of them might not be present for everyone of you who is running a production system But I would say same thing here Maybe you should fix it as soon as possible. No compiler in production This is just a matter of fact. So we cannot afford it. It's a security nightmare. There is no compiler No internet at all. So there's no Point in reaching pi pi whatever This might be the hardest thing, but I mean we can just say this is a requirement and if we If you can afford it to have it then it's really nice. We exclude a whole bunch of security issues and Some kind of health monitoring So nice to have here that the deploy is automated in the meanwhile It's it's it's happening everywhere. Everyone is using Ansible things like this, but it was not 10 years ago Automatic monitoring maybe it's possible to have some infrastructure which automatically monitors some of the health checks some automatic self-healing which would mean okay if if One thing goes down and automatically gets rescheduled and comes up again and Maybe so this would be the really the best thing have some rolling updates updates Which means we have a new version and we don't have to schedule a downtime and bring hold bring down the whole stuff Deploy the new version and bring it up, but we can maybe start with one server then the next and the next so a zero down update upgrade and if we in this process Find some issues we can just do a rollback and we are just there where we were at the beginning of our update At the risk here, it's pretty simple your business is going down. Maybe it's even fatal So you will not recover anymore and you are out of business um Maybe that's the most extreme one, but this really is production. So we cannot afford any failure there Okay, so now I Schedule maybe some let's say good setup. Maybe it's not the best in the world, but I would say this is something which would be Seen as good. So we have our normal double double developers box. So laptop or whatever For the building packaging and testing stuff, maybe the staging. So this is This is more or less not not so clear We have some kind of continuous integration server Jenkins, whatever you name it Travis and Then in production, it would be good to have some kind of modern Clusters scheduler. So not Just some hosts which can go down and things but maybe some mesos cluster or whatever So because there we can get things like rolling updates for free because they implemented it already So we don't have to care for it. It's just implemented already Maybe a bad setup, but I think most of you already seen some some of those Setups so you just develop and build everything on your Computer and in the end production means you have some server on rec space Which is two years old and you install with some pseudo app get things you never you can't remember This happens for example for some hobby project, I think this is completely okay if it goes down It doesn't matter, but for your production system, I think this would be a bad setup Okay dependencies, so That's my definition. I made it up all shared software components that need to be present in the correct version So that the application works correctly Okay, maybe we can work with this so this means somehow software needs to be present But as a thing called dependency hell It's pretty famous because normally you screw up these kinds of things always for example You can have those conflicts and transitive decision and dependencies So a depends on b and c and c on d in different versions, so you cannot really resolve it Then there's a another issue. I'm in Rona have brought it up also in the keynote In Python, we only have application global dependencies That means if last depends on something I cannot depend on this in another version This is a this is possible in JavaScript or maybe some other languages But at the moment it's not possible in Python. So we all have to agree in the whole application on one version of a dependency Pip that's a sad story, of course. We all know it still doesn't have really a proper dependency Resolution all it does is it takes the first one which comes Which yeah fulfills the first requirement and it just takes this one, but so there is a issue number 988 It's some years old four years old Yeah, but the last comment there is that this year there's a Google summer of code project and a student gets paid to implement this in Pip so fingers crossed maybe we will have some proper dependency Resolution as far as it can get because still it's global dependencies and there are just unresolvable problems This is another thing we have this System Python dependencies for example a Debian comes normally with a pip installed who knows which version Pip is on a Jesse No one It's it's 156, so it's really really old The newest one is nine there. There were some Steps missing in between but still it's a really really old pip version So if you just want to install Something in Python on a Debian Jesse, which is not the very newest but the last stable one Or in the meanwhile, it's not the last stable one, but it's it's still fully maintained It's really hard time because you have to upgrade the system pip and this crashes Maybe everything if you do not really take care what you do there Okay, then the next thing package management. So how do we manage those? dependencies or Yeah, so in the Python world I think we all agreed in the meanwhile, there's one package manager called pip and that's it There's one package format. There's this wheel But still there is quite some I feel there's quite some confusion around there's still Set up tools use quite often. There's still this detail stuff Used somewhere we still I find always some eggs lying around in my working directory And I have no clue where they are coming from but they exist So it's really a sad story and if you look up some How should I install this package? Also in Stack Overflow, they are quite outdated sometimes So they would propose things like setup tools on things like this and yeah So it's still not really throughout the whole language environment Universe that pip and wheel is the standard and there's nothing else Also templating so here's the developer of a pie scaffold Wave your hand Hello But there's no standard one So if we if we want to set up a package a Python package It's still really pain in the ass to to really set it up. So what do I have to write in my setup pie? How do I get swings document? Documentation build automatically things like this. So there's pie scaffold or virgini or I think yesterday in the lightning talk There was another one Yeah, okay, so I have to be a bit faster It's gotten better now. We have set up config and set up tools SCM things like this So it's gotten better, but it's not really good Um Okay, maybe so, okay, there is one big problem and this is we have the whole world of system dependencies managed by things like yum or ebkut and There are the language dependencies and this is not only a problem of Python at all. It's throughout the whole IT the whole IT Thing so there is npm for Javascript as Conan for C++ things like this. So every language comes with its own package management for the language packages But they somehow interfere with the system packages and they rely on some system packages to be installed Things like this. So that it's a huge problem. I think it comes from the step ops Problem, so we had the big division between operations people and they Managed the system dependencies and then we have the developers and they care for their dependencies and their own language And still we have this chasm and it's so I see no real Movement that that we really want to break it up. It's gotten better, but there is this problem So I would say this is a fail. We have to fix it somehow, but there's nothing really inside Yeah, so as I explained already, I think so there are some historical reasons for it Why do we have dependencies at all? This is yeah, we maybe disk space was expensive. So it's better to share some shared artifact so we install Let's say she let's see just once and not 20 times for each application running. So these are the things why we have this this chasm So what is the current state? So I don't have too much time left. So we have to go through it quite fast So now I have some approaches that we can do For example, there would be the classical approach. So we start developing Then we build a normal package. So this is the classical approach So I think all of you would agree that this is more or less what you would come up So we built a normal package then we push it to get On Jenkins we built some artifacts. This means we build our wheel we test this and then yeah We release it maybe to a def pi server and then our wheel is up there or on on the upstream pi pi server And then in production we just go to our repository and grab this artifact with the with the version So this is the classical approach. So The the pro side here is this is the standard thing. Everyone understands how it works We have good tooling around we can use pip we can just use all standard tools and it's really well understood the cons maybe is We have to resolve the dependencies in production. This is might be a problem And Yeah, and and we have to build those wheels for all target systems. So if we have some sento s or some Debian then we have to build it for Debian and for sendo s Then there is this approach we somehow conserve our virtual environment There are many of those implementations. Oh, just one minute left. Okay And So for example, there's platter or packs or DH virtual and so the principle idea is we just pack the virtual and Together and ship the whole tar file and then we just extract the tar file in production so, yeah, the biggest Pros on this side are we don't have to resolve the dependencies on the target system and And The cons side here is we still don't have the system dependencies. It's just the Python dependencies So we still have to have installed all glibs C's in the right version or NumPy Alex and L whatever Then there's the OS package approach. So I think you all Can imagine so we build Debian packages for example proper Debian packages The biggest drawback here is of course that we need to We need to package all dependencies we have as Debian packages and it's a real It's a huge amount of work And then of course we all know there is Docker and this solves everything it does not What I hear Want to address is containers as package management. So if you just use Docker It's not it's not the same like if you use Docker as package management, which means that you build your Your docker file once put it to an artifact repository and then grab this image in production as your artifact If you just use Docker on your laptop and then you rebuild your image on production It's not containers as package managers and I Mean there are some there are of course some pros because you can use the same image everywhere and it doesn't matter which host OS system you have but still we have some huge problems for example We still have the division between system and language packages It doesn't go away. You have to install things in your docker image With apt-get and then switch to pip or whatever however you do it and Security is a huge issue if someone updates the The shell shock Vulnerability on the host system. It doesn't care because it's still running in your container So somehow someone needs to fix all the updates there and then there is another thing I would call it the next package manager For example, there's conda you all know it and this means okay We we we just invent a whole new thing It's not pip. It's not apt-get. We invent another panic package manager and they are taking care of Already pre-compiling all day all the dependencies so Another thing here would be maybe nix who has heard of nix Oh some of you okay It's a complete new thing and I think this is the first time This would solve this chasm between system dependencies and language dependencies So this would be a complete language agnostic approach We can use JavaScript and other packages, but it would go down Directly to the kernel. So if we then install something with nix we can Yeah, we all have Everything is installed and of course there the good thing is if we then update a System dependency It would be really in our application. So this whole Nix takes care that our application gets restarted things like this a Last thing but yeah, okay. I have to stop this would be the Google approach. You just take the package and Download the source code and put it in your source source code. The good thing there is you don't need the dependency resolution It's everything just in your source code, but of course you all know the bad things there You do it's just screw it up. Okay. So That's it. So the future Who knows Thank you. Thank you very much Thank you very much for a nice talk. We have time for one or two very quick questions So on the Linux side, there's new kinds of packaging formats are coming out which are include flat Okay, so on Linux, there's these new package formats designed for this problem Which are called flat pack and then there's another one called snap So do you have any experience with those and how would they tie into this? No, sorry, I don't don't know Thanks. It was a good talk instead of a short discussion. Do you have a brief opinion? Yeah, so of course I thought a bit about what what could be a solution to all that and so I Mean the first thing I wrote here is containers will stay so they won't go away and I think it's a real It's it's a good thing. So containers will stay we have those image formats But I think we have to break up the division between system and and language Dependency so I'm pretty sure we would need one language Agnostic so it doesn't matter Java script or C++ one package management, which can resolve dependencies And goes down deeply until the kernel and so maybe there are some So the nearest thing there would be using nicks inside a container that could be a solution Which might work out but yeah, I think it would take some years to get it through Every so yeah, I don't know but that would be maybe my my shot Okay, so with this outlook, let's thank Sebastian again for a stock