 So now it's time to speak about C-Pass again with my colleague Mathieu Dupré and Florian Califromerti, and it will be a bit more technical how to use C-Pass to deploy low latency virtual machines with redundancy. So let's go guys. Thank you. Hi everyone. I hope you were there this morning for the C-Pass presentation. So this won't be a presentation or explanation about what C-Pass is. We are going to show you how to install it, configure it and use it hopefully in 40 minutes. So I'm Florent. I'm with RTE. Of course the slide doesn't work. Okay. Let's try this way. So RTE is the TSO, the electricity transmission system operator in France. And Mathieu here is from Salafa Inux. We're both working on the C-Pass project with the LF Energy. So C-Pass is the configuration project to build the virtualization platform for a latency sensitive application, I should say. Brief reminder of the architecture. This will be useful to have that in mind for the demonstrations. Usually we think that C-Pass has at least three nodes since it's a cluster and that number of machines. At least two hyperverses which you can have three, you can have five. And inside each hypervita we have loads of technologies that are used to deploy the VM and manage the VMs. So basically KVM, everything running on real-time Linux. And for networking purposes open the switch. That would be interesting to understand that we use open the switch to extend the network from one host to another so that VMs can talk to each other even if they are on different hosts. And also that if they have to move, if they have to migrate from one host to another, for them it's the same. Their network doesn't change. Yeah, C-Pass uses both Debian and Yachto. It can be run on top of Debian or on a custom Yachto distribution. So C-Pass uses Defer in the image creation. For Debian we use a full automatic installation to create the Debian iso machine which will be flash on the disk later. And on Yachto you use of course Yachto to create the image. The setup is done with on C-Ball or on C-Pass. And the on C-Ball setup is similar for both versions. But actually there is two branches, Debian main branch and main branch which is actually the Yachto branch. We plan to merge the both branches in the future. So the setup will be the same for on C-Ball and Debian. But for the moment it differs. For the users it's similar and uses both C-Pass version use the same tools and can use the same on C-Ball playbook. So yeah, so of course all of the following demo will be except for the image creation part will be can be used for both Debian and Yachto version. So to use C-Pass you need of course machine. It is important to have a node number of machine because in C-Pass you have a cluster and we need to have a quorum to vote and a quorum occur on a node number. You also need of course a flash image. You can use a flash image in the USB drive. And all the setup is done by on C-Ball. We made the on C-Ball playbook to do the setup but the inventory need to be customized by each installation. I will present you the media creation. I will begin with the Yachto version. For the Yachto version it is a regular Yachto installation. You have to fetch the source. We use repo to do that so you have to use the repo manifest and use repo sync to fetch the source. In C-Pass you have to put your SSH key during the image question step. So the SSH key will be used for the setup in the setup step by on C-Ball. So you have to put it in the keys directory. You can also if you want customize some little aspect using the C-Pass.com file. You can base on the C-Pass.com sample to do that. You can change for instance the key map and little small thing. After you do that you can build your Yachto distribution using big bake or we made a custom tool named CQD. We can help to do that. It's just two commands. CQD it will initiate and create the curve. All you need to build a Yachto distribution and C-Pass and after you are just to one CQD run to build the image. We do not have a demo for the C-Pass version because it's a very long building. But I can show you what we do for the devian base image. I'll try to do the demonstration live as much as possible. Of course many times it takes some time so we have also videos that will help to show things that take more time. But for building a devian version it's as simple as going to any Linux machine with basically docker on it. You find a repo. You clone the repository. Before I should say that everything is documented here. So if you start your C-Pass journey at the C-Pass architecture repo it will explain that there is a Yachto way of doing things or a devian way of doing things you have to choose. If you choose a Yachto then go to this repo and that will help you use big bake to build the image. If you choose devian you go to this repo and that will help you also create the installation media. It's as simple as running one command. Just before you need to do some mandatory customization that I will show you. So you clone the repo. You go to the folder. I hope it's big enough. Then it will tell you that you need to copy the default variable to the actual variable file. So you copy the default variable here to the actual variable file and you edit it with your own settings. You can change the key map, the time zone. You can change the ashes of the default password and the default username. You can set and it's better to set the SSH key that you will use to connect to the machine. That way you won't even have to put a keyboard or a console to the server. You can directly access the deployed server with networking. For that you have to set up some networking with a default interface, some IP address. And everything will be built in the image. Once you deploy the server it will already have networking. It will already have credentials and this is what I will show you right now. You go to the folder you want to build the image in. You run the command that's written in the read me. And basically it will create the Docker container for the environment. It will download all the package. It will build the image. And it will provide you with an ISO file. It takes about five to ten minutes. So I won't show you this. And like I said I have a video for the next step that we'll show you. Let's go at the end of the video. So at the end of the video you have your ISO file. To test it we're going to create a VM. Of course in the real world you will use real hardware. You will get the ISO file. You will set it as a virtual disk. And then you will run the machine. And what will happen is FAI is going to boot. You don't have to touch anything. It will deploy Debian version with the CPAS specs which is basically a list of packages. There is no configuration done beside having a list of the needed software. And like I said some networking and some default credentials so that you can access the machine easily after deployment. So if I go to the end of the event once again it will take like five to six minutes to deploy everything automatically. Download the package. And in the end it will reboot the machine. I'm going to show you. If you go to the end of the process it will reboot the machine. Boot the new Debian system which is once again completely empty. It only has a list of installed packages. And once the machine has been booted since you have configured your network and your credentials it will be as easy as you can check in the console that the IP has been set as you wanted. And then you will go to your favorite SSH client and connect directly to the server. Of course you will have to do that three times for each of your server. Like it's written in the documentation there. Once you have done that you connect, you interconnect your nodes. There is a documentation on how you should do that with some STP configuration and so on. We won't go into details here. Once your cluster network is created then you can go to the appropriate Ansible branch. So if you're using Debian it's this branch. And this documentation will help you configure the servers. That's what Mathieu is going to show you right now. Yes, now we have all the machine is fresh. You can set up Ansible. You can set up the closer with Ansible. So you can use a playbook in the Ansible repository. Playbook will perform action but you have to tell on which machine you have to apply the action, the task and you have to set parameter. With Ansible it is in the inventory. You can do that. There is several ways to write inventory in Ansible. We choose to do it in a YAML and to bring together machine and associated variable. So all the machine needs to be placed in the cluster machine group and hypervisor in the hypervisor group. Here it is an example which is kind of having the Ansible repository. We have three machines on it. Yes, node one, node two, node three. On each node we define the variable. For instance Ansible OS is a default Ansible variable in which Ansible will use to connect in SSH and connect to the machine and run the task. The other is C path custom variable for instance to define network IP or other parameter. Variable can be placed just in the machine, near the machine if it is machine specific. But it can also be a global if it applies for all machine to the cluster path you also need to place the machine and just scroll up a little. You have to set the Oath in the main group for all the machine will be in the cluster. And in the client group and the machine will store data need to be placed in the ODS group. OZ group. Can now show you a demo of the setup. It is an excellent Android time demo. To call Ansible you need to call Ansible playbook command. You have to pass the playbook you want. Here is a cluster path. And the inventory in which the action will be set up. It is your inventory you have just awaited. Yes. It is how Ansible works. You can see all the tasks which is performed and on which machine it's running. You have a state to if the task is keeping or if it is just run and strange looking. It is display in green and okay. If it changes something it is in yellow and it is white and change. And if there is an error it is in a way fail. At the end you have a little recap of all the steps. Yes. It is not very important. Yeah. And the tab. Good. Next step is yes network. Yes. So like I said before we are using OpenVswitch to extend the bridges from one host to another and to make the hosts communicate with each other in the most simple way. Of course once again we are using Ansible for that. So we are using Ansible to configure. Ansible to manage guest networking. We are also using Ansible for security and for basically everything. So I'm going to try to do this demo live. It's going to be interesting. So the idea is to go to an EPAVSR. So I have three EPAVSR. They are called Elabo 1, 2 and 3. I'm going to call them Node 1, Node 2 and Node 3. And they are not running VMs. Right now the VM are shut down. I'm going to show you how we're going to try to make a DBM, this VM, which is actually not running, communicate with a DBM 2. So we are with DBM, DBM 2. We're going to have DBM running on Node 1, DBM 2 running on Node 2. We can also have DBM 3 running on Node 3. And we're going to do what we need to make them communicate. So the first thing we need to show you is that if we run OBS control show, we can see all the configuration. And we can see that there is no DBM interface. There is no DEB interface. So we're going to create them, use Ansible to push them on the server and then configure the VMs so that they can connect to those interfaces and then run the VMs and see if they can communicate to each other. Let's do that. So for that we're going to go to our Ansible server because everything will be done not online, not on the server, but with infra as code. Let's go to Ansible. And there is an inventory for the cluster, which I showed you. There is also an inventory for the networking, for the guest networking. Let me find one, sorry. So we're going to use this one. And this inventory, this OBS inventory will be the topology, the open-v-switch topology that the cluster will emulate. So you can create bridges, you can extend bridges from one host to another and you can connect your bridge, you can create interfaces for your VM to connect to the bridge and you can connect your bridge to physical interfaces if you want to extend your bridge to the outside of the cluster. So what I'm going to show you is that I've prepared this. So we're going to create three interfaces in the bridge, which is called the IPO. And we are going to just add three interfaces on the topology. And once you've done that, we're going to run the playbook, which is called network. So once again, there is no, right now, no interface to that name. We're going to run the playbook and we'll take the configuration in the inventory and create this configuration on the clusters. So that's this step, basically. So once this step is done, okay, we should, yeah, go and see that the interfaces have been created and they have been created on all servers. So wherever the VM will be, it will have an interface that it can connect to. So the next step is to go to the VM configuration and tell it to connect to the interface. So we have custom tools to do that. For instance, this is our tool where we can edit all the metadata of the VM. And if we go to Debian 1, we can go to the XML of the VM. So this is everything, this is the VM configuration, basically. And if I, if I go there, I see I need to add this kind of XML that say basically create an interface on bus 0, PCI bus 0, slot 2, that will be important for later. It's not managed because it has been created by OpenVswitch and use Debian 1.1. And on the second machine, Debian 2, we do the same thing. Here, yeah, same thing. So use the PSI slot 2, Debian 2.1, not managed. Once we have done that, we just have to run the VM, basically. So let me show you that the VM is not running. I'm going to use a tool that Matthew will show you a little more in detail later. And we're going to run Debian. This is warning. Don't worry. It works. And we're going to run Debian 2. And we're also going to run Debian 3 because why not? We'll do live migration test later. That will be useful. And since we are in Elabor 1, we can check that the VM Debian is started in Elabor 1. So if we run libvirt, we can see that the VM is actually running. And if we are curious of what's happening on the VM, we can also connect to the console and check that it's properly booting. So we can, this is the VM booting, basically. And once it's been booted, we can connect to it. And we can check that there is an ESP0 S2. That was the bus and slot, the PCI information. And we can, we have an IP. This has been configured before. And we can connect to the machine. And since we have the same thing with, let's take a few minutes, a few seconds. And if we go to Elabor 2, the second node, and we can check once again that the Debian 2 VM is running on Elabor 2, we can verify it's the case. So basically we have two nodes, one VM on each node. And the idea is to see whether we can ping one from the other. And it works. So basically we have two VMs and they are discussing together through an OVS bridge that I extended using the XLAN. Let's talk about introspection. So now for this one, I probably use the video because it's a bit more complicated to show live. The idea is that you understand that we use basically Ansible for everything. And we also, since it's very important, we all very often get the question of how we manage cyber security. Since on this version we're laying on Debian for the patches, this is an opener. We're dedicating to Debian. But for the configuration, we might have to change things because the national agency says that we have to apply specific settings. That is a vulnerability. So let me go to the video. We're going to go a bit quick on the video. But the example I chose is to have, for instance, a secret file. Let's say we call it secret. And this file is readable by everyone on the server. And let's say we need, because we heard from somewhere that this is a vulnerability and that we need to change the permission on this file. Well, since we're using Anthras code, it's pretty easy to go to the Ansible playbook to add a task that will say, okay, this file, it needs to belong to root and to have 6.0.0 mode. So basically only readable and readable by root. And once we've done that, we apply the cluster. We apply the paybook. We reload and we see that everything has been done and we won't even have to connect to the machine. So if we have many clusters to manage, it's pretty easy. And the second thing we want to show is that it can also help with compliance. Because the idea is that Ansible is pretty good at showing you that it had to change something. Which is not the case if the settings were correct in the first place. So this is normal. This is a new setting. So it changed. If I reapply the playbook, it shouldn't have to change. But if there is some malicious threat actor that, for instance, changed the permission, I'm going to show you first that if you reapply the playbook, Ansible will show you that nothing has changed. Everything, sorry. Sorry. I say again. If the malicious actor changed, for instance, the mode of the file and put it to 6.0, or on another server changed the owner of the file, then I will say that this mode is not correct. Sorry. This mode is not correct. This owner is not correct. And if I reapply the playbook, first I can check that the playbook will correct once again. It's the job. But it can also show me that on virtue level 2, everything was fine. But on virtue level 1 and 3, they had to be changed. So we can use Ansible to detect any cybersecurity issue or problem. So it would be a recommendation, even if there is no change, to reapply the playbook regularly just to monitor that the number of changed items doesn't increase, which would mean that someone is playing with the servers. And yeah, we put this to the next level by implementing a playbook that does a lot more than that. So we use this to implement the French agency, cybersecurity agency rules, BP28, on the version. On the 2 versions, otherwise during the build time, I will detail that in the latest call with Gaon. So we made a playbook with Arden, Debian, following this guide. You can run the demo. So the playbook is running the same as the setup playbook. You can see it changes. The interesting part of this playbook, it is you can check the rules applied by Ansible, but you can also revert all the Arden in the playbook. It is useful if you, to play with the servers, you have to be able to play with the servers. So you can run the Arden in the playbook. It is useful if you, to place all your clusters in the debug mode, because the Arden rules, with Arden you can't use regular debug tools. It's not a mode you use for development. It's really a production mode. Since you have to work, we still have to work on the server. We like to be able to un-harden the system if needed. This is how we un-harden. A simple asset. So one playbook to un-harden. So once again, playbooks for everything, playbook for cybersecurity, playbook for compliance. And now, so we thought about installing CPAS, we thought about configuring CPAS, hardening CPAS, managing security. Now we're going to talk about using CPAS. And Mathieu will tell you about the VMs. Yes, because the VM part is complex in CPAS. We use multiple tools, because there is the virtualization, but also the cluster part. It is done with different tools. The virtualization is done by KVM, KVM and QMU together. It can be managed with LibVert. We use LibVert to control KVM and QMU. But KVM and QMU rely on SAFE to use it. Because SAFE stores the VM disk inside it and distributes it in all the clusters. The distribution storage is done by SAFE. So QMU and KVM have to read and dialogue with SAFE. But you cannot directly manage SAFE with LibVert and QMU. And there is also the failover part, the cluster part, which is done by KVM and PESMaker. And KVM and PESMaker have limited control on LibVert, but have no control on SAFE. So we made tools above all that. We made three tools, a pattern module, which is called VMManager, which is a command line interface, VMMGR, and a non-several plugin, which we rely on VMManager. And of course, the associative playbook, which we use on this program. So the idea is you can use VMManager to manually deal with the VMs. But since we are in love with Uncivil, we expose all these services to Uncivil module that means you can use Uncivil to manage the VM as well. Yeah, definitely. Yeah, we have a demo of the command line. Just a bit of the command line interface, VMMGR. Here, this is the helper. You can do all the basic things, create VM, remove VM, list of VM. Some advancing options are also here. You can create a snapshot, a reverse snapshot. And you can also define some cluster rules. For instance, you can add collocation rules to tell the VM to always be one of the same machine on the server. This can be useful if you do not have an extended network on all the cluster, but you have just a private network to communicate between the two VMs for performance present, for instance. Yes, I can... This is an example of how to create a VM. You just need to have a name. Yes, okay. You just... Is it running? Okay. You have to set the name, the disk you want to use, and the libvert XML configuration. And the command will deploy the disk image on-safe and create the VM and configure the pacemaker to do all the cluster configuration. Yes, now the VM is created. You can see with the VM set, this is the VM is started. You can do all the... Yeah, this is just another example with another tools, which is pacemaker control tools, CRM. You can see how demo VM is created. You can also see where the machine is running the VM. On the machine, the VM is running. Here is the AMD machine. Now, for the demo, I will remove the machine, stop it before and remove after. So you can stop the VM, which it will be still on the cluster, but it will be in the stop state. You can disable the VM, the VM is still in the safe cluster, ready to be enabled, but it's not on the pacemaker cluster, or you can remove the VM and it will remove it completely from Cpass. And the disk is here. So, this is the demo. Okay, that's fine. In Cpass, we have to deal with the real-time constraint in order to have low latency network, for instance. This is a particular VM. They have to be pinned on a specific CPU because we have to use CPU isolation if you want to have a good real-time. We also need to use the real-time scheduler. Okay. And some Libertwick. All this configuration is already done by Cpass. I have just a quick demo. We showed the secret test tools. And how it differs between real-time VM and a regular VM. On the top, this is the max latency on the regular VM. And on the bottom, the latency on the real-time VM. This is the kind of latency we were expecting from the guests. And, yeah, let's talk about live migration and also fail-off immigration. Once again, I'm going to try to do it live because it's fine that way. So let's go back to our cluster. Same as before, same VMs. No cheating. So we have three VMs running on different interfaces. Let's connect to the first one, for instance. So the first one is Debian. And so we have booted it 13 minutes ago. That was for the last demonstration. And we're going to have a ping it Debian 2. So both VMs are there and they're pinging each other. Let's say you want to move a VM because you want to better balance your cluster. You want to do a test or whatever. So it's very easy to manually move a VM using a pacemaker. You use the CRM tool, and you say you want to move Debian 2, for instance, E03. And if you check what's happening in CRM at that time, you're going to see that the VM Debian is migrating. So the memory of the VM is going to be transferred from one host to another. And then once the memory is synced on both hosts, then pacemaker will switch. And so we can see that now the VM is running on the third node, and the VM hasn't stopped pinging. It doesn't even notice that it has been live-migrated. So that's pretty nice. And of course you can move it back. But before we move back, we're going to try something. We're going to say, okay, we have now two VMs running on E03. Let's say we want to do an operation on E03. You want to reboot it. You want to apply security updates. You want to do some serious maintenance, and you don't want any VM running on it. Once again, with CRM it's pretty easy to ask it to put the node in standby. So what I'm going to do is I'm going to put it in standby. And at the meantime, I'm going to show you what's happening at the cluster level. So I'm going to say, okay, put node three in standby. So node three is going to be standby. And all the resources running on node three are going to be migrated. That's it. So we're going to wait a few seconds for the VMs to be migrated. And we're migrating dbn and dbn3. And once again, we can see that dbn1 and 2 will be pinging each other with no impact. And then, okay, that's it. It's done. So now, no VMs are running on E03. We can check. We can go to E03. And we can say, okay, do you have any VM running? Or say no. And then we're going to say, okay, I'm done with my operation. I want to node back online. So if I put the node back online, then the cluster will automatically say, okay, I'm going to migrate the VMs back there because you asked for the VM to be running on E03. So E03 is up and running. So basically, it's going to migrate there. And once it's done, the final test I want to show you is what happens if we crash, basically, E03. So let's have the VM still running on E03. And we're going to crash the node. So what will happen? Of course, you won't be able to migrate the VM because the memory will be lost. But at least it can restart the VM somewhere else. So to crash E03, I'm going to go to E03. And I'm going to do something that you should never do. I'm going to sync the disk so that... This is the only real way to reboot the server. So I'm going to reboot the server very, very badly. So at that moment, everything is crashed. The server is down. The VM is down. The cluster will very promptly notice that the host is down. And it will, as soon as possible, start the VM somewhere else. And it had decided to run the VM on E03 and E03 on E03 once again. Of course, the VM has crashed. That's normal. But they will be restarted in a timely manner. At least in 30 seconds. And so if we wait 30 seconds, then we will be able to connect back to the machines. We can even monitor the boot process if we like and connect to it as soon as we boot it. And during that time, I'm going to end the presentation or the failover that's what we've just done. So this was our last demonstration. The VM is still booting. In the meantime, thank you for your invitation. With the C++ project, we have many resources. We have a Wiki, a website, a github. We have a Slack. And very recently, we have a YouTube channel where we'll be uploading all the videos we've seen today. And more. And hopefully the community will continue to upload videos and tutorials and explanation of how C++ works to this channel. So don't hesitate to go there. And we are available for some questions if you have any. Thank you. Thank you. So it was quite technical. I guess you've got a lot of questions. Again, most of the videos are available on YouTube and we make sure to upload as much content as we could do. So if you have any questions? Yes, sure. Thank you for the presentation. Exist also a life cycle for a disk upgrade? Disk upgrade? Yes. You use Debian. And the Debian life cycle is around three, four years. So you have to make a disk upgrade for security reasons. Exist also some disk upgrade mechanism? Well, I'm not sure I understand the question. But the upgrade mechanism we are relying on. The host itself. The host itself is running Debian. So basically we are relying on Debian. So if we want to upgrade, then we're going to use Debian to secure the upgrade. We are using LVM snapshots. So our recommendation. And there is a documentation on the GitHub also. It's if you want to do some... Well, I'm going to show you the best way. Well, well, yeah. So if you want to upgrade, what we recommend is to do a snapshot of the root file system. Then to check the snapshot is working, to do the upgrade you want to do or any change you want to do. And if it goes wrong for any reason, then you can also always roll back. If it does not, you can discard the snapshot. I'm not sure that I answered your question, but... Okay, thanks. And just in the meantime, just to show you that the host has been rebooted and so that the cluster has decided to move the VM back on the node since it's back online and clean. So thank you very much for the presentation. So it's great to see the guidelines for Debian. Do you have also the same recommendation for Yocto using a packet server or elsewhere? In Yocto, all the other languages are done during the image creation. We also apply a VP28 rules. I will present in more detail with Anguillon in the last session. But in Yocto, you can also have the package management. That's why I was asking what... No, no, no. In Yocto, you use software update and PC active slot. System AP. Thank you. Last quick question, maybe? Thank you. Thank you.