 Okay, thank you, Jörg. So, yeah, as a disclaimer, you will immediately notice that I have a French accent, so this will be a signal to this morning. Okay, so I'm Pierre Chania. I'm working as an operation engineer in the SRE division of Creteo. And in this talk, I will try to share with you some insight into our deployment and operation regarding our mesos infrastructure. So, I will be quite fast because there are a lot of interesting and technical content in this talk, but I will share the slides just after and you can come by our booth if you want to have some details. So, first, a bit of context regarding our company. So, at Creteo, our business, we are operating the attack industry and our business is basically to predict the most relevant ad for the right user at the right moment. And since for the last 12 years, we've managed to do that pretty efficiently. And right now, we are able to present some interesting metrics regarding that. Like, we have 30 offices around the world. We reach every month 1.2 billion distinct users on the internet. We operate in seven different data centers around the world and we are hosting more than 20,000 physical servers in these data centers. We are also constrained because of our business. Since we are doing performance marketing, we have some services that have to respond under 10 milliseconds. And this is quite a challenge to operate such an infrastructure. So, at Creteo, like in many other companies, we are always transitioning. And from a necessary perspective, transitioning means four different things at that time. The first thing is hardware. We try in the long run to improve our TCO regarding the hardware, meaning that we see the residency more at data center level, rather than at the server level. Regarding the OS stack, we started a big switch historically from a Windows-based infrastructure to something more Linux-oriented. We're in a runtime. We've introduced a bit more diversity. We are historically based on the .NET ecosystem. And last but not least, regarding our platform, we aim to improve our ability to deliver our self-service to our end user, which are basically internal users. And what we've learned so far at doing this transition is that to get a stable and a maintainable service, you have to make its component simple and highly modular. And this is why we chose Mesos initially. So I won't detail on why Mesos is simple and modular because there are a lot of interesting talks regarding these aspects in this conference. But I could also mention that it's highly self-sufficient in the sense that you can run containers in many different ways. So just think about it. In our case, we are running internal image-less containers, meaning that we don't fully rely on the image-based isolation, but you can do it if you want. You can also use different run times. And we've learned yesterday that we will soon be able to maybe run VMs on top of Mesos, so that's really amazing. So where are we at Crutio right now? We've started a small POC almost two years ago. And right now we have 600 agents on our seven data centers. We are running more than 150 production applications and these thanks to the two generalized framework which are Mercin and Apache Aurora. And we will try in a near future to introduce a machine learning-oriented workload and to move really huge blocks from our legacy infrastructure to the new Mesos infrastructure. So what does this imply from an operational point of view? I mean, the SRE vision. Here I wanted to highlight a few points which are really important if you want to move from like we did from a small POC to something a bit more production grade. So the first thing is that you have to automate everything. In our case, we are operating a huge legacy infrastructure which is composed of more than one, sorry, 11,000 of Windows machine plus the Mesos infrastructure. So it's absolutely impossible if we don't automate everything that we actually make something work. So in our case, we are using Chef to do that as a config management tool. So okay, with config management you can deploy things, et cetera, but it's not the only thing we are doing actually. We perform also complex operation using these tools like if we want to reboot a machine, if we want to pixie reboot a machine, if we want to upgrade an OS, if we want to upgrade our Mesos cluster like we do here from 103 to 104, if I want to scale up my infrastructure, we do all of that using Chef thanks to internal development. So a second aspect is you have to configure your infrastructure defensively for predictability. So the first task for an operator to do that is to basically do a map, identify your full domains. And these are mostly related to your network topology in your different data centers. A second aspect is that you have to enforce limits for your task. I won't detail on CPU and memory because there have been some interesting blockbusters around that, but do you care about your disk and about the Unix user that makes your task running on your Mesos infrastructure? If not, you should because that's also part of the predictability of your infrastructure. And last but not least, you should perform backup and also try to restore them. You don't want like me to be in this situation on a Sunday afternoon when you're on call being called by someone in your company saying, hey, we had an incident. You have to restore everything. And you end up with this kind of errors. So please try to do backup and to restore them. And thanks to DCOS, we will have apparently amazing primitive coming for doing that automatically. A third aspect in our case was our ability to move smoothly from a service from the legacy infrastructure to the Mesos cluster. And to do that, we've introduced service discovery components using console on every single machine in our data centers so that we are able to perform smooth transition from the legacy to the new infrastructure. But with the service discovery components, you can do a lot more. We are using that also to, for example, provision load burn, et cetera. So this is clearly a key component if you want to envisage transitioning from a legacy to a new system. The fourth aspect is about observability. Don't forget that transitioning from a legacy infrastructure which is a statically partitioned purely hardware-based to something relying on containers. It's something completely new for your own user if they used to run on such old systems. Why? Because historically, there are instances where are completely static. They didn't move. They had monitoring that was maybe deep in the company but statically configured. But now, apps move continuously. Instances are reloaded because of operators doing something, app being restarted, et cetera, et cetera. So you have to get some insight from your platform. And here I wanted to highlight some very interesting components and metrics that are available in the mesos using different mesos components. So for example, a few metrics for any networking in these kios. There isn't another aspect which is important is it's tracing. For some purpose, you may want to be able to debug your app and have the lowest possible level of insight. And to do that, you can perform tracing. In mesos, you have a lot of awesome components and one of them is the Perf Isorator. Using that, you can retrieve some traces and these are awesome if you want to perform low-level investigations. In our case, we're also working to introduce LTTNG as a tracing solution. And for those of you who are able to attend to the tracing summit, there are some of our colleagues who present this integration. So the last aspect I wanted to mention is networking. Networking, in our case, was a really complex aspect. So our first intent was to provide services around networking, like automatic DNS provisioning, timeout profile, ability to manage sticky cookies, potentially automatic TLS configuration, et cetera. But we faced two categories of issues. The first one was about our load balancing stack. So we are using HAProxy, which is an awesome product. This community is bringing a lot of addition in the product, and so you are able to perform almost everything at the layer 7, with an HTTP. But the reloads were not that stable. In our case, we were either in a situation in which we introduced resets at the TCP level or purely latency spikes. So I encourage those of you who are using HAProxy to upgrade to the laser version, because this community has done a lot of interesting work, and so that these issues are now fixed. Another aspect is regaining predictability, again. We've noticed that we were quite frequently subject to the noisy neighbor issue, which is quite common when you're doing container-based systems. And so we did a lot of investigation regaining that, and in our case, we ended up implementing something around net TLS, which is something interesting from the kernel, which is managed using C-group, and which allow you to basically classify your different flows and to apply QoS rule on it. Okay, so here it was supposed to be the fun side of the talk, like a few incident we had during the last month. So first one, regaining DC outages. So it was called by someone, again on Sunday afternoon, I don't know why. Always on Sunday afternoon. July was sunny, et cetera. And, okay, someone told me, hey Pierre, so we have one DC starting to burn. So basically, every server has been shut down, and you have to help us to restart the whole infrastructure. And what I can tell you is that it was really easier to restart something mesers-based and container-based compared to our legacy. Trust me, we spent a lot of time performing that. So clearly this revealed that a mesers-based solution bring a lot to a company like our. I also wanted to mention some disaster recovery scenarios such as a Merlison application, the DC worldwide. So we had one of our users, which basically was privileged for some reason, tried to call some APIs and ended up in destroying every application on our infrastructure except his home. So this was an ACL concern, and so I encourage you to be careful about these problematics. You should either perform multi-tenancy or be really careful about isolating your different API paths. So for us, what's left to answer? So our first concern is regarding isolation. Clearly, we aim to improve our isolation regarding network and IOs aspects, disk IOs. Since we will also try to move complex and latency-driven components, we're also looking using CPU sets potentially to have more predictability on our components. There is also something that will require a lot of investment from us, and this is basically described by the graph here. You can notice that the usage versus reservation between the usage and the reservation, there is quite a gap. And so in the long run, we will probably have to invest into the notion of revocability over subscription and why not potentially bin packing? Because using bin packing, you can not only reclaim hardware, but potentially, if you are able to do it, you can also reclaim electrical power. Just imagine, if you are able to defragment your cluster by packing the task, you can envisage completely stopping some of your servers if they are doing nothing. So if you have a correct deal with your energy supplier and your data center, you can potentially save some resources. Last aspect is about maintenance primitives. As you may have understood, we are doing some complex operation with our IT automation tool, like reboot, et cetera, and we would like to anticipate more of this operation by reclaiming resources and inform the different frameworks that we will do something, and at some point, they have to move their task. Okay, so this slide was about collaboration in the company. And it was to mention that if you provide a good level of services and a good level of collaboration between SRE teams and software department in your company, you can get awesome contribution. And this is one of the good examples because one of our engineers in the software department noticed that by trying to move a dotnet-based application on top of Mesos, he observed weird behaviors regarding the sizing of his thread pools. And basically, this guy managed to introduce the support of CPU shells and CFS into the dotnet CRR, which is basically the equivalent of the JDK in the dotnet ecosystem. So clearly, collaboration inside the company gives you a lot of value, and exchanging around this aspect with your developers can bring you a lot of added value. Okay, so that's all. If you want to know more, because this was a bit technical and I was quite fast, faster than was expected, feel free to come by our booth. We can detail a lot around that. And also I wanted to mention that we are hiring, so if you want to work in an amazing department in either of Paris or in the US, feel free. Thank you very much.