 Bienvenue, encore une fois, pour les gens qui m'ont déjà vu à l'hôtel de la seconde session. Je m'appelle Julien Stroéker. Comme vous pouvez le voir, je vais faire le même drac ce matin. Les gens ne sont pas de France. J'ai originé de France, borné en Montpellier, en France, vivant en Montréal, au Canada, et travaillant pour Microsoft Seattle. J'ai beaucoup de fonds en travaillant pour Microsoft et j'aimerais dire que j'ai du plaisir avec la technologie d'open source et j'ai beaucoup de focus sur la technologie de DevOps. J'ai partie de la division du CSE, pour l'engagement du succès des clients, je pense. Nous avons juste une réorganisation, nous avons changé le nom, c'est pourquoi. Mais l'idée est d'engager avec un client qui est en train d'utiliser Azure et d'être un de ces groupes de feedbacks avec l'engagement de l'engagement, donc d'engager avec un partenaire et un client et de donner le feedback à l'engagement de l'engagement, et de l'envers de la technologie d'open source. C'est beaucoup de fun. J'ai vraiment aimé. Aujourd'hui, je vais parler de l'autoscaline mesos cluster. Donc, l'agenda pour ces 45 minutes, je pense. Un petit peu de concepts. Je vais partager le code, en fait. Il y a quelques slides et la plupart de mes presentations, la plupart de mes presentations seront live demo et code. Donc, quelques slides, et ensuite, on va changer la présentation à la demo. Donc, un petit peu de concepts. Comme je l'ai dit, parce que la plupart de mes temps à Microsoft, j'ai engagé à faire des codes avec les partenaires et les clients. Une d'entre elles m'a demandé une façon de faire l'autoscaline sur DCOS et sur Azure. Donc, je vais vous montrer le code. Donc, je vais vous montrer le code, c'est partie de ça. C'est un peu de concept. Je ne peux pas partager le code avec les clients, mais j'ai essayé de l'extraire et de jouer avec ça et de partager ça pour faire l'open source. Donc, c'est d'une POC. La idée de nous, c'est d'évoquer un cloud privé de public et d'être capable de l'autoscaline. Donc, je n'aime pas ce monde parce que l'autoscaline est magique. Et le problème de l'autoscaline, il n'y a pas de manière magique pour l'autoscaline. Parce que je suis sûr que si l'autoscaline est magique, pourquoi elle n'est pas dans le produit aujourd'hui, pourquoi elle n'est pas sur DCOS, je veux dire sur Mezos. Autoscaline, c'est un aspect juge, ce n'est pas un problème, mais c'est un aspect juge. Vous pouvez l'exprimer par CPU, GPU, RAM, Troupert, netwalking ou quoi-même. Donc, c'est un scénario custom pour chaque case. Donc, celui-ci, parce qu'on veut qu'il soit simple, 45 minutes, encore une fois, je vais juste partager mon vision sur ce code et prendre un scénario plus simple et plus facile. Oui. J'aime faire la joie, parce que je travaille sur Microsoft, j'ai dû faire un slide de marketing. Et, encore une fois, c'est le même que j'ai présenté hier-bas, juste pour vous donner un petit concept. Je ne sais pas si ici les gens sont en Azure, dans la salle. Oui, non? Wow, 1. Donc, oui. Je pense que c'est bien juste d'expliquer ceci. En Azure, nous avons un service qui s'appelle ACS. Nous sommes pour Azure Container Services. C'est juste un moyen de déployer un orchestrateur, un orchestrateur, donc un orchestrateur, Kubernetes et un DCOS. Le cool chose, c'est que, pour exemple, pour un DCOS, nous travaillons dans la salle et quand vous déployez un DCOS clusters, c'est un cluster backing, supporté par Microsoft et la salle. Donc, les meilleures practices. Et quelque chose pour l'autre provider. Nous appelons ceci ACS, les container services. Nous avons un peu de services différents. En Azure, nous avons annoncé un nouveau service, qui s'appelle AKS, pour les services à Azure Kubernetes. Mais oui, nous avons un peu de services C'est l'ACS, et nous avons aussi une source d'open source. Quand vous déploiez et utilisez l'ACS ici, c'est le Microsoft Way. Vous pouvez aller sur le portal ou en utilisant le CLI. Vous avez un peu de différents interfaces. C'est mon cluster, je veux 3 master, 5 nodes. C'est le site que je veux. C'est tout. Nous faisons tous les magiques pour vous. Mais si vous voulez une configuration custom, comme l'intégration VNet, vous voulez utiliser GPU, une version spécifique d'ACS et d'autres. Nous avons aussi une source d'open source, l'engin derrière cette magique sur le portal et sur l'ACS. Nous appelons donc l'ACS Engine. Si vous voulez un déploiement custom, vous pouvez toujours faire cela avec l'ACS Engine. Ça fait le sens ? C'est tout pour la construction d'ACS Engine. Vous avez juste à définir. Mais il n'y a que de 50 lines de JSON, vous avez juste à définir la structure que vous voulez pour déployer sur Azure. Vous avez des options de mandatory, donc je veux utiliser D.C.OS. Vous pouvez aussi spécifier la version, je veux une master. Vous pouvez spécifier un bunch de différents nodes. Et c'est tout. C'est ça. Vous avez juste passé ça pour dire que l'ACS Engine va générer et parler à l'API, derrière la scène, et ensuite déployer tout, tous les VMs, les networkings, les storages, etc. Ce n'est pas ce que je vais utiliser pour mon demo, et la chose que j'ai envie d'insister ici, j'ai envie d'insister sur le fait que j'utilise un custom label pour Mezos, donc je déploie trois différents pools d'agent, donc basically three stack of nodes, if I can say that, so three times, sorry, three times this VM, which is a standard D2v2, it's two CPUs on Azure, so two CPUs for Gease of RAM, so I want three times that one, and for that one, I want you to also include the tag, the Mezos tag, the constraint, for workload with the value stateful, we're going to understand that later, same thing for stateless, and I also have the public node somewhere, here, three public nodes. So the cool thing with that, you can change the parameters, pass that to ACS Engine and then it's going to deploy that for you. Again, think about that, 45 minutes, so now 40 minutes or 30 minutes, the simpler, easier scenario, right? So we're going to focus only on the stateless workload because we're going to, I mean, what I want first is to easily auto-scale without any storage constraint, for example. We're not going to start to play with stateful scenario, right? Here, the idea, it's okay, I want to be able to easily scale my CPU, for example, or I want easy, I want you to scale, when I need more CPUs to deploy my application, I want you to scale on that matrix, make sense? So then, when it's done, pass that to Azure and Azure doing the magic for you, like I said. At the end, it's just a bunch of VMs, master, nodes, like I said, storage. So if you're not using Azure, the same thing on AWS and GCE, basically. And one thing interesting here, you remember, I'm supposed to have three agent pool, which on Azure, we call that the VM skillset, so bunch of VMs together, let's do that. And here, I tag one with stateful, one with stateless, and we suppose I have a third one for the public agent, right? So on Azure is that thing here, I don't know if you can see, but it's as we call the VM skillset, so one for DCOS agent public, another one for SF for stateful, another one for stateless. So I just click here for the stateless one, and per default, I have three instances running on that one. That's normal, because I ask for three, right? So this is what I have on my deployment right now. Now it's came to the code. So I think I share on the slide the URL of the code of my auto-scaler. So basically, it's a Python tool, let's say that. I'm using click. I don't know if you heard about this framework. Click is just a cool Python framework to follow with you to do CLI frameworks. So this is the way that I code that. So here, when I start to code that, I say, OK, I don't want to do something just 100% compliant and related to Azure. I want to be able for people, if they want, to also use the same logic for any cloud provider, public or private, right? So the idea is I'm using a provider, so in my case, Azure, and then I deploy a bunch of nodes, so one for stateless, one for stateful, and it could be different constraints. Let's focus on stateless. And here, on that example, I have four. I want to be able automatically to scale for four, four plus two, plus two, plus two, plus two, depends on the demand. I want to scale up, scale down. Makes sense? So what I'm going to do, I say, OK, fine. It's not really complicated. I have to find a way to ask mes us how many resources I have, how many resources I want, how many I'm consuming right now, and then do the math and talk to the provider first. And I mean my class provider, and the provider is going to say, OK, because you're using Azure, this is the class that you have to use, and you're going to say to Azure, OK, scale up or scale down. And then, it comes also to constraint. You remember, on my deployment, I also specify some constraints. So I have one for workload, and workload as stateless and stateful. Why the constraint? So when I'm doing the math on term of CPU conception, I want to do that only on the stateless one. We don't want to play with the stateful right now. So on the code, this year, it works. It's really, really straightforward again. Kind of shitty code, but it works. I have a cluster class. I was talking to provider class, and that class talked to different provider. In my case, it's Azure. So when you start the CLI, you just have a switch option for provider. In my case, it's Azure. And because Azure knows that we're using Azure, and all the logic to scale up and scale down for Azure, it's here. And yeah, that's really, really straightforward. That guy's going to talk to Azure, and then when it's come back, he's going to redo the math. It's just a loop, actually. I have a timer every minute, so every 10 minutes, for example, I want to ask Mesos the conception usage, do the math again, and then again send a request to my provider. Makes sense? So let's do the demo. So first, the link here. So that's the code. Like I said, three classes, cluster provider and provider Azure. And my main file, Python file, actually. And here, I have all the options. Again, I'm using the click framework. So this is all the options to specify. So I'm not going to go deep on the code, but at least I already explained how it's supposed to work. If you want to state to have more explanation, actually, I try to describe how it works. And this is all the switch and all the options that I have. So for example, I have to specify the provider name. That one is monetary. I am specifying that I want to Azure. That's the only one that I implemented right now. If you want to implement it for AWS, GCO, whatever, you're more than welcome to do pull request. The timer, per default, is every minute. It's going to poke the API, the mesos API. But if you want to have the 10 minutes, 20 minutes latency, I can also reduce or reduce that. Here, I call that the cap. So the trigger. When I'm doing the math, so for example, if you're consuming 80% of the CPU consumed by my application, this is added trigger, the 80 percentage, that is going to scale up. So I can change that. When I'm going to reach 50%, for example, on my conception, I can trigger the scale up, or 60, or 75. For example, that's this option here. Same thing for the scale down. If you want to implement the scale down, actually. I can also have specify the limitation. You can scale, scale, scale. But maybe you don't want to scale to 100,000 nodes. So if I want to stop the tool to scale at maximum 20, and something for the minimum 3. Then for Azure, if you, I mean, only one person using Azure here, but the way to talk to the Azure API is we call that the SPN. We stand for Service Principal. Basically, just a key password, subscription ID, and so on. So for information to be able to talk to the Azure API, so trigger action like scale, add VMs, reduce VM, and so on, the location, and so on. So that's last option here. It's really specific to the provider that we seek. So if you want to implement AWS, for example, I'm pretty sure that for AWS, you also have specific options that you have to specify. So it's something that you can have, same thing for GC, and so on. So this is how it works. So I'm a cluster here, deploy. And again, if you remember, one master, three different nodes for the public agent and stateful and stateless. So for example, if I'm going on that one, that first host name, if I'm going on details, you can see here on workload. I don't know if you can see. On the back, can you see? So that one, for example, is a stateful node. So we're not supposed to touch that one here. We can if you want, but I can say let's keep that simple. Same thing for the second one. So here, the way that I deploy that on Azure, a different subnet here. So I know that that one, one, two, three, it's supposed to be the stateful one. Normally, the next one, that one's supposed to be the stateless, actually. So yeah, we have the stateless node here, and so on. So I already deploy an application, actually. That's my beautiful Hello World application here. And if I just show you the marathon file here, while I specify, again, let me know if you can see. But I deploy that one only on the stateless. So it's just a web application, right? No storage. It's not a stateful application. That's the one that I want to handle. I want to be able to scale any stateless workload. Because when I start with Mezos and DCOS, I mean, I guess it was the same thing for you. When you don't really understand that's consumption stuff, right? When you have to specify how many CPUs you need, how many memory you need. And sometimes, you deploy, deploy, and then you can see waiting. Yeah, why do you wait? I mean, why were you waiting? I have resource available, right? But yeah, when you say I want one CPU for this application, even if you're not consuming the CPU, it's going to block the CPU for this application, right? And here, I'm running three instances on my web. So again, back to my configuration here. That one, it just, in term of transition, I'm asking for one CPUs for that application. That means I'm consuming three CPUs. Again, on my nodes, on my stateless nodes, there are three times, as we call a d2v2 on Azure, which is one VM, one d2, it's a two CPUs. So I mean, I have the total that I have, it's, come on guys, six, right? So I basically consuming 50% of my CPUs here, right? So let's launch my awesome auto-scaler. So here, it's supposed to work like I described. So I specify all my parameters here. So then I'm crashing all the logs for debug reason. So it just trigger and detect that it's Azure. The timer is 10 seconds. The scale-up cap, it's 80s. That means that when we're going to 80% of construction, this is when it's going to trigger the scale-up, sensing for the scale-down. Maximum node is 20, minimum is three. The endpoint I'm going to reach, actually, that's interesting. This is where all the magic happen. Basically, I'm curling that endpoint here. If you don't know that, this is where I'm grabbing all the resources. So when I'm doing that here, on the leader mesos slash slaves, I have all the resources from the slave, right? And basically, I'm parsing this JSON file. Because that one, from that one, I can see the CPUs available, the CPU consummate and so on and so on. So back to my thing here. And then bunch of credential information. And then, every 10 seconds, it's going to poke the API and going to check the consumption. So you remember the scale-up cap, it's 80%. I'm consuming 50%, no magic here, right? So let's scale to one more here. See if it works correctly. I'm scaling to four. And we have 66, right? If I'm scaling one more, we're supposed to still be good. Oh, 83. So now, scale-up kicked. So now it's calling the provider. The provider is calling, so the provider knows that we're using Azure. So the provider is going to call the provider Azure classes and going to scale. So the way it works on Azure, you remember I talked about the VM scale set. So it's supposed to scale the correct one, which is the DCOS-Mesos SL for stateless. And we're supposed to scale, if I'm doing instances, so it's creating instances here, and it's going to scale. And the cool thing with the VM scale set is going to bootstrap the Mesos agent and everything, everything. And then after a few minutes, we're supposed to see three more VMs here with the stateless tag and so on. And then it's going to be able to deploy. So here, we're just at 80, but the ID, let's say, we scale to something like 10. It's going to deploy. We have six instances, right? So it's waiting for more. So the ID is going to scale to three other nodes. So that means I'm going to have, or two other nodes, sorry. So I'm going to have four CPUs available, four more CPUs available, and then it's going to be able to deploy my application until 10, and so on. So it's an easy way, actually, to scale. Again, stateless application, right? Just web app. And the cool thing here, it's supposed to also scale down. So when we're going to reach, like, under 20% consumption, it's going to do the same, the opposite, actually. It's going to scale down to, from six or eight, I don't remember how many we're supposed to have, to two less, and so on, and so on, right? So it's just a cheap way to save money on Cloud provider, right? And again, stateless application. Like I said, autoscaling is very complicated. I don't know if you're about that. Microsoft, we work with a company called OpenAI. It was more on Kubernetes, but just the story about that, it's we developed their own way to autoscale GPU training. So OpenAI, I'm pretty sure you're about that. So the player, the guys who played Dota 2, and a machine beats the best player in the world at that game. And yeah, everything was hosted on Azure, but the story about that, we developed their own autoscaling model based on a bunch of different scenarios, like networking, storage, and everything, everything. And I asked the generating team why we didn't, like, not open source, but include that in the product because it's a good story. It's supposed to be cool or so. Maybe gonna help or provide other customer. It was too custom for the needs and this is why we engage the conversation about that. We also have the same kind of code for Swarm and Kubernetes on open source set, actually, but again, it's still stuck to one kind of resources. On that case, it's CPUs, but maybe you want to scale for GPUs, for example. GPUs make sense also because it's really, really expensive. So maybe we want to be able to scale GPUs, launch a training, right, and then shut down of the VM. So yeah, that's just the concept and the idea of that. So let's see. I tried to talk to just to wait before the... Okay, C scaling, let's see on Azure, so it's creating, running. So I guess now it's bootstrapping like all the Mezos and DCS bits. As you may be not ties, it's the way that I did that for now, it's just a docker code. It's really ugly, running on a master, right? But it should be really easy to push that and deploy that on a cluster. And the fun thing also that I did I also create a universe package. So normally, yeah. So basically, here on the package, it's basically the same thing that is going to ask you for the service, I mean the provider name, the scale threshold and so on and so on. For now, we only have just Azure, but it's the beginning of the way to run that inside DCS. For now, it's again docker on the master, really ugly, but I'm pretty confident that it should be able to work also on the inside the cluster. Let's see what we have here. Okay, gonna wait anyway. Do you have any question in the meantime? Who is doing auto scaling right now on DCS? Okay, which way doing that? Or so custom code, I guess, or? Okay. Sorry? Okay, based on, so you're, they say they're running their own code, right? To scale, based on what? Still based on resources available on the cluster or? Yeah, it's based on, we all scale services and clusters. Okay. Dependent on services we scale on a metric that the service owner asks for and clusters, then we scale based on CPUs. Okay. Well, yeah, we choose CPU and memory and choose whatever's worse. Yeah, do you scan, okay. But it's still pretty dumb. Yeah, it's tricky, right? Because you get into the situation where you're tracking a lot more things than just CPU and memory. Oh yeah, probably. In marathon, for example, what defines whether a task is waiting or not depends on whether it can get an offer with all the things it needs. And those could be CPU, mem, GPU, but they can also be like attributes. So it could be like tags for a specific OS or tags for a specific region or something. So yeah, our autoscaler is still kind of dumb. It will handle CPU memory or other things like that where we can just scale on the worst one. Yeah, exactly, and that's also a good point. Why people use, I mean, one of the reasons we're using public cloud actually is to be able to also to scale on different region, right? And sometimes we want to scale on different region, not only on Micas Westerrop. So it's also a good point, something that you wanna watch. And again, just not CPU. If you check the code, it's only maybe 200 line of code. It's really, really simple. But again, the idea of the session because it's end of the last day, right? Just to share again, just to give some ideas. And the fun thing, I didn't fight any open source code about that. So I thought it could be fun just to at least publish something and people can start from that because in my case was the case with my customer to start from that. And they kind of did the same thing, but custom, they not use my code. I mean, that was the deal, right? But they doing that on OpenStack and Azure. So the public cloud that they are using OpenStack and they using Azure on the public one. Yes, so our code is open source. But it, like you said, it's a very specific problem to your organization. I'll support AWS. I mean, people would be free to try and use it, but whether they would get it working without our infrastructure is questionable, I guess. Yeah, yeah. Cool, thanks for sharing. So yeah, in the meantime, yeah, in the back there. In the meantime, I have a new guy here. No, that's not the correct one. Yeah, it's coming, actually. Yeah, in the back question. Wait for the mic here. And please share, guys. Like I said, it could be just if you wanna share something about the topics. That was also the idea why Mesosphere asked me to do this tour because I start and engage on that topic, but the idea is also to share and if we have multiple feedback, we can also talk with them with Mesosphere on the topics, right? So yeah, question? Yeah, my question was a relay. You are now handling a stateless stuff, but you're also doing another thing which is partitioning your cluster into two parts, actually. Yeah. So meaning that, okay, it might be interesting for if you have very, very large stateless load and you want to split your cluster into two, but sometimes it's more maybe mixed on that or do you have any ideas? Yeah, so yeah, that's a good point here. It's really, really straightforward reading one tag and just do the math on that. Actually, on the report, I think it's published, let me check. Yeah, future resource tracker. So it's another branch. So when I was motivated, I started this thing. It was kind of, it's kind of the V2 on that. So be able to track all the tags that you want to watch, actually. So just not one, but be able to say I want to watch like CPU or mem on CPU and mem CPU GPU, but also I want you to tag like, so I want to check the tags or the stateless one, but also that tag. So maybe on that way you can combine, I mean, there's just something in mind, right? But you can combine, okay, that node or VM or VM scale set or whatever is going to add a stateless tag, but also, I don't know, GPU tag or you can have custom tag, right? And do the math also on that one, if you want. Yeah, but my point is more about, in order to do that, you have to have strict conditions regarding resource allocation on the mesos cluster. For instance, no stateful node, no stateful application will install on a stateless node. Otherwise, when you decree the size of your stateless application, you are about to kill some stateful application. So basically, you are splitting your cluster into two parts. Maybe it's not a good practice, you're right here. And the thing is, it would be probably possible to explore, for instance, the tags on each deployed node to see whether an application is stateful or stateless and may I destroy this node, for instance, but in this case, you have another issue which is fragmentation. Yeah. Yeah, I got your point. You have to repack the stuff and... I got your point. And oh wait, that just confirmed the point that I said that custom, there is no magic way to autoscale, right? It's just a way, and again, the easier 45 minutes way to do that. But yeah, totally agree with you. On that code, I can add more, let's say, smart way to scale, like reading the, I don't know, the variables or whatever I want or the name. But yeah, I'm totally agree with you with that. I mean, fragmentation is one point here and maybe not the best way to do that, but again, 45 minutes talk. Are you doing autoscaling? Yeah. No, but I did similar stuff for OpenStack long time ago. Okay. With migration of VMs, it was easier because you could... Yeah, yeah, yeah. Yeah, right, yeah. Just a few remarks. Yeah. The first one is, there is the Marathon LB autoscale project, which aims to scale application on top of Marathon. To me, your project is a good extension of that one because basically, when you're blocked, because of resources proposed to scale you up on top of the message cluster, you can act at the underlying level, meaning that you can increase your cluster size. Yeah. Because you're blocked in increasing number of instance due to resource leak. Yeah, yeah, that's a good point. And also, the thing on, I mean, I know where's Azure, right? I don't know about Amazon and other cloud provider, but the thing on the VM scale set, the idea of the VM scale set on Azure is to be able to automatically autoscale, actually. So, you have this magic switch button to, okay, autoscale for me. But the problem of that, it's Azure gonna read the CPU construction of the VM by itself, you know what I mean? It's not gonna be able to read like Marathon or Mezos, right? This is why we need that tool. And yeah, to tell you agree, I also saw this project. And again, I'm not in the engineering team. That's also one reason. I'm not in the Microsoft Azure engineering team, right? My work is to engage with customers. So, unfortunately, I don't have time to invest a lot of time on that project. I would love to. But yeah, that's a very good point. Okay, and the signal remark was, our intuition so far on crypto is that you have to gather more business metrics to. Start really doing autoscaling. And if you want to push the thing really far, potentially you have to apply some machine learning stuff on top of that. Meaning that you have to understand what's the pattern and to act upon conditions. Yeah, and we know that executive people loves dashboard, right? Metrics. So yeah, that's agree. And I also like the talk this morning about, I mean, I never heard about that before you talk about the poor bin or whatever. Just bin packing. Yeah, bin packing. And that's, I mean, that's awesome. I mean, the concept is awesome. I'm pretty sure that executive people would love that. But it's, I think it's pretty tough metrics to which, right? To gather, I mean, to have the correct math on that and reach the correct. Yeah, exactly. If at some point we are able to do that for sure you have to prepare this operation carefully and also have a lot of insight through the business and your applications. Yeah, and I'm curious, which tool, I mean, or people open question, which tool are you using for the metrics? Are you using Prometheus or, I don't know which, which, yeah, Prometheus, yeah, here? Or the guys, no? Any, and do the row way. No dashboard, then. Okay. And we also ingest a lot of logs and stuff into Splunk. Okay, yes. And that provides some useful metrics as well. Okay, cool. Yeah, so that's it for me guys. If you have other thing that you wanna share or question or whatever. No. So if you wanna reach me, my Twitter, it's that one. Or I think you're gonna be able to have the slide anyway, but if you have any question. And if you are using or planning to use Azure, let me know. Again, no worries, I have nothing to say, I'm not in the sales part. But if you have nice technical challenge on Azure, let me know. I'm not willing to come and help you with awesome people from Microsoft. Because what we want and what we like is challenges, actually. So that's just the idea of my team. It's just to be challenged and just because we're gonna be challenged, we're gonna have the tricky scenarios that we never see before. And then it's gonna be the better way to give feedback to the engineering team and to have the best product on the market. So that's how we work on my team. So thank you very much for your time guys. And yeah, hope to see you after.