 Great, okay, I guess we can get started. Everybody please give a warm welcome to Michai. Hi everyone, thank you so much for joining me today. So I will share with you some of my personal experience regarding the virtual development environments. So let's take a look at the main key points of this presentation. So the first one is consistency, or how your development environment should be similar in the actual production environment and in development. About diversity, or how you should handle the distinct environments for distinct projects. The isolation, or how your different environments should be different, from different projects should not overlap and not break each other. So in the beginning let's see how usually people like Python developers are doing their workflow. So in order to have all of the dependencies locally you use PIP, or the OS packaging tool, installing pre-compiled packages from the operating system package repository, or even use ease install to have all of them locally. But how do you do it? Do you install everything globally, or you use Konda or Virtually Env or any other similar tools? We're perhaps on the playground. The last one actually is quite good when it comes to consistency, but otherwise it has a bit of an overhead. Okay, so now when you have finally your environment up and running, your version control in place and your features are freshly development, how do you deploy it and where? So you have definitely multiple options. You can do it on a platform as a service provider, or on infrastructure as a service easily. You can even deploy it on the old server that lives in your attic. The last two are quite similar from the deployment perspective, although you can't really rely on the trusted things from your attic. So Virtually Env's are really popular nowadays. They're accomplishing their goals perfectly fine. They're super simple. They run on top of the shared system libraries and packages, but here is a small problem because running on the shared system libraries that you cannot influence might be a problem. These underlying leaps are different in different distributions, while some of the distributions try to be on the bleeding edge. Other distributions are more conservative and concentrated on the stable packages. So even though it's possible to specify the specific version of the internal Python dependencies inside the virtual environment, frequently the dependency tree is not only bound by these internal Python dependencies, but also depends on the system-wide used leaps. That brings inconsistency between different machines. Moreover, this inconsistency might turn things upside down in production too when a given system dependency differs in some manner from the one from the development machine. So whether something breaks, there's always this notorious excuse. So imagine for a second that we can have the exact same thing in all of the environments in which application is running. Even when you write in Python, which is an interpretive language, which results are not compiled into some kind of self-contained binary. So let's containerize it. Let's pack it into something more robust than a simple virtual environment. So we actually build a kind of binary with all of its dependencies enclosed into it. Let's put it in a container. So as the Docker became an industry standard for the containerization, all of my examples will be based on the Docker technologies. When I was rehearsing this talk, at this point I got some questions like, Docker is a virtual machine, they said. And I was asked how it's different from other virtualizing technologies. In fact, virtual end, sorry. In fact, containers are not virtual machines. They are neither a lightweight virtual machine. They're just a process that runs under the same kernel. The process is just isolated and it's running in its own name space. So here we can see the same picture as before but using Docker containers. So we have the Docker engine running on top of the shared system libraries and packages. And the containers are all different. And still it's somehow similar to the virtual end situation. But the only different thing is the fact that containers are just self-sufficient. They have all of the dependencies inside them. So they are portable, thus they are more robust. So let's talk about container anatomy. Basically containers combine just the two features of the modern Linux kernel. Those are C groups and name spaces. So C groups are enable user to prioritize and limit the system resources for a specified process. Namespaces actually completely isolates an application's view over the operating system. So it includes process trees, networking, user IDs and mountain file systems. So let's see what is actually inside a container. It contains all of the system dependencies, own libraries and has limited end points for access to and from the outside world. The container is built from an image which is immutable. Docker images are self-contained as told before. They have everything needed for the execution bundled inside. They are isolated so different images don't share any parts of them. And they are immutable. What's in it basically stays in it. And of course they are portable so you can move them to any machine and they will work just fine. Speaking about containers, the container is basically just an isolated process that is disposable as well. So if you break a container, you just delete it and start from scratch from the same image. Containers are running over the image file system layers which is AOFS. So everything happens in the container is just a layer on top of the image layers. So let's take a look how the AOFS works. AOFS is a unification file system. It means that it takes multiple directories on a single in a host, stacks them on top of each other and provides a single unified view on them. To achieve this AOFS uses a union mount. That's how the images are immutable. A running container is just a layer on top of the image layers that are by the way read-only as you can see. And that's how containers are disposable so everything happens in this layer without being committed to the actual image. IsVolatile does its dispose at the time when the container is removed without altering the structure of the image. Now we've stacked multiple file system directories. It means that we can just add or override files. Does its cumulative. So how to actually delete a file without simply rewriting it with another one? To achieve this, the union file system driver just places a white-out file in the container stop layer. So the white-out file just effectively obscures the existence of the file in the read-only image layers below so the file is basically still there. It's just not visible to the user. But beware, even though you can delete files with white-out, it's worth remembering that it's still cumulative and the size of the image will only increase. So don't commit to the image on necessary files. We'll end up having huge files for no actual reason. Okay, so up to this moment, it seems that all the data is immutable. The layers in the image are read-only. So how actually to work with the real application data? A volume we will actually use for these volumes to mount this external resource to the container. A volume basically is this external thing. You can just mount a directory from the host to a directory to the container. And the volumes are not anyhow managed by the docker so they're never recycled or the docker never deletes them when you remove a container. Okay, so now that we know all the needed primitives, let's see how the images are actually built. So for that, we will need just two things. The first one will be the raw product or the source code of our application. And the next one will be the cooking recipe, which is represented by a docker file. So that's basically how super simple docker file looks like. And every line from this docker file represents a layer in the actual image that will be built. And let's go just line by line and try to get a deeper look to this. So the first line represents the base image on top of which we will build our own image. It will take this image from docker hub. It's a sort of github for docker images. Then we set an environment variable inside of our container. Then the run command just basically runs the command supplied to this command. So it will just create a directory for our code. Then we add this specified file to this directory. So afterwards you can just run pip install on it and have all these local Python dependencies inside the image. Then we simply add the code inside of this image. You may actually wonder why I didn't combine the fourth and the sixth line having like adding all the code before doing the pip install. But actually there is a reason for that. It's because docker caches the layers. So if nothing is changed, it will just reuse the layers of the image from cache. So when it changes the code, it shouldn't just reinstall doing this pip install each and every time. So now that we have the docker file, let's build an image from it and let's run it. Also we'll take a look on how to gather the logs from a running container and we'll try to sneak inside some of containers to gather even more control. So let's begin with the building the images which is defined by the first argument. Then we specify the image tag which is basically the image name and we can even append a column and add a word and a number which will define the version of this image. By default this suffix is the word latest. And then we specify this dot which tells docker to search for the docker file in this particular directory. If you want to issue the build command outside from the project directory, you should specify the full path in which the docker file is living. And now that we have the image in place, let's run it. The first attribute is a self-explanatory. Then we just map the port from the container from the running container to the port from the host where the docker daemon is running. Then we specify the image name that was passed before the build stage so we make sure that we run from this specific image. Then we just run the usual command for running a dev instance of Django. In this case, all of the application logs will go to the stdout. Now we want to run this as a daemon as it's detached. We can just specify it with this attribute which is detached. And in order to be able to identify this container easier from other containers, let's give it a name. And I was mentioning before about the volumes, so let's actually mount the code into the container so we can make the changes to the app on the fly when the dev machine. Well, this is how it's done. And well, this command works, but it's quite a mess. We'll come back to it later. So I was mentioning before gathering the logs. So when you run in detached mode, the logs are not going to the standard output or so. So we can grab the logs with the docalogs command. Minus f is basically the same, has the same effect as the minus f argument from the gunutl command, which follows the stream. And here is the name of our container. And if you need for any reason to get inside the running container, you can just execute a shell inside it. You do it with docexec. Minus at means that it will be interactive and it will allocate a pseudo TTY to be able to interact with the shell. Then it's followed by the name of the container. And then we specify just this shell that we want to run. But beware, usually the base images don't have inside of them a fancy shell, so don't try to run phish or dsh or other unless it wasn't installed to the image. Also using exec command, you can just execute any other command inside of the container. So the process is totally isolated from the other processes on the same workstation. It has all of its specific dependencies bundled inside. And these libraries and bins can be distinct from the ones that are installed system-wide on the host that Docker engine is running. And it's disposable, so whether something goes wrong, you can just stop the latest specific container and start again from the same image. But that's obviously not enough. You may wonder how about distinct external dependencies and where is the promised production-like environment? Well, here it is, we'll just need more containers. And that's how your production environment should look like. With different containers, all of them with their specific goal, within their specific scope, and working together in a wonderful synergy, in theory. In order to achieve this synergy, we'll need some kind of orchestration. The orchestration is the automated arrangement, coordination and management of complex interdependent systems and services. It should be simple. We'll consider different approaches for production and dev environments. But the simplest way is to do this with Docker Compose. And this is just how a super-simple Docker Compose file looks like. You can see that I've defined there two services. The first one is just the database using the stock Postgres image from Docker Hub. The second one is the web application which we were building before. And as you can see, it has all of the variables, the attributes that were defined before manually, defined here statically. So, and we bring up these two containers just with this command. It's just as simple. And do you remember now the awfully long command from the slide before? Now all the attributes are statically represented in the config file, so all you need is just this. And if you want to use Compose in Prod, you can, or any other environment which is testing, staging, or whatever, you can create a base YAML file that will contain all the generic configuration and just extend them for specific environments. So, let's say that this is a production Compose config. And as you can see, at the web container it's just not using the Django admin running command, but it's using Uwisgi, and it depends on Nginx as well. So, running Compose in production is not the only way of doing the orchestration and it's perhaps not the best way as well. I will not go much into details, but let's do a small overview of the industry, solutions for orchestration, for logging as well, and for secrets. So, for orchestration in production, you can use, of course, Docker Compose as described before, where you can use the Docker swarm. If you have a swarm of instances running Docker, and you want them to socialize with each other. Also, there are external tools that are good for orchestration like console or ETCD. I've also heard people being happy with ZooKeeper, though I've never tried that. So, secrets, there are multiple opinions regarding them, but obviously you shouldn't store the secrets, the passwords, or any other API keys, and so in the version control systems. With Docker, there are different ways of passing secrets to the application, from the environment variables to mounting volumes of the secrets, or any other way at the runtime. However, there are third-party tools that are designed just for this task, like Docker Vault, which is built on top of the Vault utility, and KeyWiz, they're storing the passwords safely into the host and passes them to the actual running containers at the runtime. And if you're using Kubernetes or Google Container Engine, you have all these features built in. Speaking about logging, starting with the version 1.6, we have different logging drivers, and you can easily use them. It strictly depends on your logging decisions at the system level, but as for the old-school basics, you can just use syslog and be happy. And then you have this one person in your team who uses Windows, good for them, because there is no Docker for Windows. But seriously, if you're developing on Windows or Mac, you know that you cannot run Docker natively on your machine. But now we're not forced anymore to use virtual box, VMware, Parallels, or other hypervirus or virtualization tool, because there is the new Docker for Mac and Windows, and it works blazingly fast, at least on Macs. On Macs, they use X Hive, which is a super lightweight virtualization solution for OS X with almost no overhead, and on Windows they use HyperVib. Well, summing this up, that's a nice way of building environments that behave in the same way in production and in development, so it's really easy to make self-contained packages with all the particular dependencies inside them. So even with some exotic dependencies that do not exist in the wild world. Also, having them immutable guarantees that there are no difference in time, no diverse problem related to this kind of car. And also isolating them from each other and just explicitly specifying the interfaces or allowing particular containers to communicate with other containers, minimizes the risk of unexpected conflicts, and all these characteristics improve the overall robustness of the application in relation to its environment. So that's basically it. Thank you. So as we are behind of the schedule, I will just be hanging around this entrance of this room. We have two minutes. Oh, okay. Yeah, we have time for, I think, one or two questions. Does anyone have a question? Hi, thank you. An excellent talk. Absolutely five stars. I have a question, so Docker versus Docker Machine. So a lot of people say like Docker Machine, I don't like, you know, Docker is superior. What's your opinion on that? And also what's your opinion? Like, so my use case, if you kind of develop on Windows, you're forced to use Linux kernels to run your images. Yes, but now we can, you can try this beta Docker from Mac and Windows. And it's, oh, it's not. Okay, it works for me. Anyway, yeah, if that's not working in your case, you can just use the world's cool way of having a virtual machine and running a Linux kernel and then use Docker Machine or whatever to connect to it and work with it. But as it's not working for you, you can file a bug because it's still in beta and the guys will help you and perhaps you'll uncover this problem. One more question? Going once, going twice. Okay, then please thank our speaker, Michai, again. Thanks.