 Hi everyone, thanks for being here. I'll talk about using Python LXC, which is Linux containers and Linux to create a mass VM hosting managed by Django and AngularJS We have a schedule part one is my part back and an architecture and part two is Oliver's part This is the front end in Django and AngularJS Now first part back and an architecture First about me. I am Daniel Kraft From D90 and my Twitter account is wham-dum-dum. I am doing computer since 1984 1985 sorry, and I am online since 1987 obviously not in the internet at that time Who are we we are? Creating a service for pre-configured ready to run virtual service with root for many open source web apps Think of it like a one-click installer All this hosted in Germany by the way with a hundred percent renewable energy and That's how it looks like You basically choose some templates like Django in that example choose a version of it Give it a name click on add container is this you will yes Now you have this container in red because it's off it's turned off Then turn it on then it becomes green then you click on the URL on the reachable at HTTP and There's a Django And here's how it works We have two parts in this architecture. The back end is called con Short for container management. The front end is called site This is just our names Con has two modes first mode is it could be run or the it can be run as a demon Then it is a XML RPC server Otherwise, it's an XML RPC client to its own demon So basically you start it as a server as a demon first and then you can Use it as like a console script which connects to its own XML RPC server So this is how it looks like if you don't call it as a demon Shell script with I don't know if you can read that. That's not so important It's it's more about these are this is anything you need to Manage virtual machines on a host you can I can't read it here. I have to look that you can build I'll show an example Shortly you can remove templates create containers duplicate containers start and stop them and so on So con calls itself. So it eats its own dog food because it calls its own XML RPC Methods Just as like as you'll see shortly as the site does It can be called by others like the site It contains anything needed to work with virtual machines so Very important part is con Works completely without sight that means we can Use the server part test it individually only that we can run it locally all without any user management or something This is just a virtualization level a layer The site on the other hand based on Django and AngularJS calls con Which can be many We are XML RPC. It does accounting and payment. It creates PDF invoices it manages user accounts or and the registration and so on and Also, the site works without con. This is also important to test it and to run it locally Of course when con doesn't run or isn't available. You won't see any containers now back to con We re-implemented an existing solution for repeatable bills It looks like that maybe someone Knows what that is Yeah, the docker file more or less So we essentially do the same this is because of the history of con we first Ran on docker, but docker didn't have the features we required So we added the features and had about 80% code on top of docker and just 20% managing docker and At some point we threw away the docker part and re-implemented the 20% ourselves and this is part of it So this is a very very simple language that essentially starts up a lxc container runs a command inside it like apt-get update and Closes that container again and for the next command it creates a new snapshot of this template that now Was configured with that command and runs for example apt-get upgrade dash y This results in a large template tree because What you see here is one line is a snapshot of one command you saw earlier so one of one file we Actually call it con file one of one line in this file results in one line here in the hierarchy and The longer lines are the final templates that you can use in to loop So con is using lxc for virtualization shell in the box for the web console IP tables for network accounting or Linux tools here are some rules for that if you didn't know and It's using a lot of the C group magic from the Linux kernel for accounting like the CPU act group Where the CPU act usage counts the nanoseconds per second which are used on the CPU and the same for the memory which Gives RSS active inactive memory file memory caches and so on It's using offs for storage. This is a layered file system. You can mount any number nearly any number of Your ordinary directories on top of another and they derive from each other That means if you have a file a in the lower container and file b Container sorry in the in the lower directory and a file b in the higher directory and you mount both then you see file a and file b and They do some magic with deleted files and so on. This is a very stable solution So that leads me to the failures we had yes many Let's talk about b3fs We first choose b3fs instead of ours because it's fast. It's even fast for millions of files. It works First it works very good. It has writeable snapshot That means you can at any point use any sub volume in a b3fs make a snapshot of it and write in both and They diverge It has live quota with sub volumes That means you have at any point the disk usage of this sub volume in some with all the snapshots below it and Just the diff what is the difference to where it was snapshot it from and it has an instant creation of snapshots It's like the 10th of the second But maybe you have seen that this is the Linux Ios deck or a diagram of it You can basically stack anything in Linux on top of another and Like like block devices then file system then a file system image then partition inside it and and so on Without knowing exactly what that does It's not needed. We did Use a device manager for rate 10. This is on the hard drives on the physical hard drives on top We used LLVM Logical volume manager on top. We used which IO. This is the KVM virtual disk IO layer On top we used a partition On top of that for one partition. We used x4 on top of that we used an image file which we Mounted as a loopback device and put b3fs on it. This was a test setup because the image file is Quite nice for handling and for backup. You can just turn it off copy the image file somewhere and run it again However Then the b3fs cleaner died b3fs is a lazy file system It does what it needs to and cleans up later That means the b3fs cleaner has to run and it has to clean up later and It died during its job and we lost data. That's not meant to be There's a thing in linux in the linux IO stack. That's called barriers This copied from an article on LWN in a sense a barrier For bits the writing of any blocks after the barrier until all blocks written before the barrier are committed to the media That makes sure that the journal of the file system is consistent Looks like the barrier didn't find its way through these layers. So some point in this stack Obviously after debugging it didn't work with barriers So we tried again The same basic setup we used the rate 10 we used LVM on top We used virtio with KVM because this is our default setup We didn't want to throw that away that helps us very much with backups and things like that So on top of that we use the partition and directly a b3fs and hooray We tried to crash it again. It didn't so it the b3fs looked stable from that point barrier standing Well, that's another thing about the b3fs cleaner. It produces a lot of memory fragmentation If you have never heard about memory fragmentation Yes, it exists and linux has a table of it that you will see when you see a kernel trace back in Dmask and one line of that is a page allocation failure in that case order for The order is the Potension of two of the block size in memory that couldn't be allocated. This means that a 64 kilobyte block Wasn't available of continuous memory This is pretty bad because that's not much and there's no defragmentation tool in the linux kernel This is if you have this state it will never run again Except there's a memory freeing Okay, so We threw b3fs away and used hofs Which is a bit slower, but much more stable and we are happy with it Next failure XMR PC First we use zero RPC a really excellent tool. It's pretty fast. It has a good Serialization you can basically just fire off messages. They will arrive somewhere and It's a lot faster than XML RPC But it was leaking file descriptor. We're not using G event with which we can't because we are currently bound to threats and Then we used XML RPC, but it was very nice. We were a little bit Yeah, blue eyed it's in German We used bytes for anything that was transferred like for memory usage for network graphic for disk space and so on but there's a to Potential to the 31 limit of XML RPC and And we couldn't use bytes anymore. So we had to serialize all large numbers to strings or Moved to megabytes where we're possible. So it's running for now until we hit the four gigabyte limits again And have problems with megabytes too Okay, that's it for my part. I would have a lot more failures, but times running and I'll Give over to Oliver for questions. I'll be there directly after his part Hi, I'm here working at the front end of to whoop My name is Oliver Rock I Choose Django in English is to get the web UI started pretty much fast I use Django for user accounting invoice management and as a mediator from Dennis X amount of RPC API to a JSON API I can digest with AngularJS First of all Django is using CSRF protection for a lot of use if you Activate the middleware so we have to tell AngularJS to take the token from the cookie and send it to every asynchronous request The next problem AngularJS templates will collide with your Django Django template language Because they all use the double curly braces. So we have to tell Angular to use for example curly brace and dollar sign internationalization Django uses PO files, which pretty much like soap or blown dust To have a consistent state between the Django views and the JavaScript views you can use the Django views IATN JavaScript catalog which takes the profiles and Generates a JavaScript you can include into your site and you have a function like get text to have internationalization You wouldn't use document write in AngularJS It's just for example the next We have a lot of requests depending on non-user permissions. So we have to include a permission denied exception That is delivered by Django, but a standard HTTP service by AngularJS doesn't handle a 403 so You have to create a custom Interceptor for that. So we have a factory permission denied interceptor You can handle a request request error response and response error in this case It's a response error for three. So we Set the location to a slash. It's the front page and the registration login page Another good product is the double entry book keeping bookkeeping its Django account balances You have full audit trail. You have always a debit entry and a credit entry or credit entry to debit entry So you won't lose any money. It's pretty simple. You have to define a source It's our bank account to this nation account the user the amount and the user and that is privileged to Transfer the money The most important thing is keeping the DOM On the front page of to whoop you have a list of your containers running that is updated every two or three seconds If you missed to track by an ID AngularJS will replace the whole DOM every two or three seconds So you won't be able to interact with all your containers because just in a click all down is gone and replaced by another one Yeah, it's all pretty much simple, but it's due to the fact that Django and Angular are Simple to program any questions Yeah Thank you Daniel. Thank you Oliver any questions come to the microphones and we'll ask you in turn So let's start with this chap here what kind of version of B3 FS you used for tests What version did you use for B3 FS tests? We were starting with what kernel Ubuntu 13 10 and we're testing it again on 14 or four I think it's one point something in the latest version Where the B3 FS cleaner also did these memory fragmentation things? Might be true. It was the wrong person to ask this, but so if you use an JSON For the front end why use XMLRPC in the back end? It was a Decision for development speed the XMLRPC module in Python is well tested and really complete and all you have to do is I don't know derive I think from the XMLRPC class and It makes the server automatically you have no effort with it You just define methods that can be called from outside. This is just for development simplicity So you switch away from Docker because some features were missing Dockers are still in current development and do you think they will catch up with the features you need like soon and? It wasn't just about features it was inconsistency to there were a lot of Accounting things that Docker Returned when when when calling it that were inconsistent in itself and We we had to work around that I don't exactly remember what that was and we had to do all the C groups magic I was shortly talking about ourselves and much more and we included all that and Accounting network accounting like the IP table stuff and so on completely into our own product and That was the most code of it. The the viewer Alexi virtualization isn't much I think Docker is improving on monitoring and instrumentation and another question. Do you have now? So you have Django and Angular? So the other guy and you know Django and Angular and it's your side. Does it support like progressive enhancement or? How does it? How do you handle that? Like if you go to a URL directly sets So working so I didn't get it progressive enhancement. Do you use it? What's that? So so when you load your site and then you don't expect any JavaScript to be running and the site still works or Non-client rendering no no we've got lots of time for questions. So come up to the mic I have a question if is your format container format compatible with the Docker one because if you made a so-called fork of Docker and if I want to use your Hosting infrastructure and I didn't want to you know vendor lock-in. Can I move to the Docker? Hub or something? So your question is if we have a compatible layout of directories Yes, no, we don't you could manually copy the things around and I think it would work and write some configuration files for Docker for each container, but You can't use it directly in Docker. Okay, do you do any kind of network isolation between Containers of the same customer and how do you do that? We have a network isolation all containers have private IP addresses and we only forward Configured ports for each container that you can configure yourself. So it's like a web firewall management thing But we don't support private networks right now between containers We are here to learn and we already implemented a few things we heard from people talking about downstairs, but And one and one of these things is private networking. We had a reasonable use case we heard a reasonable use case for it now because our aim isn't to orchestrate Applications together, but to have one container that contains anything you need like a Postgres and a Django and whatever But we have now a good use case where a private networking is needed and I think it'll come in a short time Hey, congratulations on a really cool project and Do you have any plans for an API at the moment? Do you have can I can I script those these containers? Not yet. Very beautiful. You can inspect what the site does with This with the browser and use that but it's still based on session cookie and and stuff like that You can of course use that but we are on making that a lot more beautiful and documented especially Yes, how long did it take to reach this point? We started in August last year. I think that's it Okay, any final questions we have got time if anyone has a final question Otherwise put your hands together and thank Daniel and Oliver for a very interesting time