 So, hello everyone, I think we're good to go, so glad to see you all here. So today I will present you bare metal migrating, a bare metal server into OpenStack, so obviously it's possible, so I would not be here. So I will present you the how tos and lesson learned that we get from that process. So who we are, you might have heard of us, so we are internet, so we're public infrastructure as a service provider, so we're providing our services through OpenStack. We're providing as well our bare metal, our provision through bare metal, so we're using IRONIC to provision our bare metal. We were one of the first to do it as a public cloud, after Rackspace obviously. So we are also infrastructure donor, official infrastructure donor for the community, so it's a good thing. So we offer like 150 VMs to the CI infra of OpenStack. So internet in numbers, it's 15 data centers, six OpenStack region. We have more than 30,000 nodes across all those data centers, and we have close to 2,000 bare metal currently in our inventory. Me, I'm Maxime Bellanger, I'm working for internet public hosting business unit in Montreal, and I'm a senior software developer and technical lead of the business-facing software development team. So previously known, the business unit hosting was previously known as iWeb, so we were there at the beginning of OpenStack back in 2012. So our mission was fairly simple. So we have a bunch of customers that are using our legacy platform to provision their bare metal and their cloud, of course, and we have also the same customer that are using our new platform, so OpenStack. So basically what we have, we have that old rusty legacy platform that we use. It's working, but there's no new feature development on it. We lost a bit of knowledge about that platform, so for our customer it was better for them to have a unified experience with that shiny new OpenStack feature. So basically there's a lot of development feature in OpenStack with the biggest community, and also the API in OpenStack, in our case, it's open, so it's not close to anyone, and in the other case, the API was closed and it was not known of a lot of our customer because it was only on demand for them. So we'll show you a simple customer setup. So basically they have, let's say, one server will simplify that. So an OS name, an OS, an IP, and for us, let's say a presentation of the machine in our inventory, so the bare metal. Let's say that bare metal is in New York. So what we want to do, fairly simple, we want to represent the hardware in ironing, so creating the node in ironing. We want to represent it in Nova, so creating the instance with the name of the OS name, moving the IPs in neutral, and in glance, of course, we have the OS. So what could go wrong with that? So simple steps. First thing, you create the node in ironing. Second step, you move the network in neutral. Third step, you create a flavor in Nova to represent that hardware. Next step, you fake boot the server because the customer don't want to lose his data on the server, obviously. So what we have with that, we have an IP customer. So here's what really happened. So you see the buzz there? It's our plan. So basically, at the beginning, we just failed because what we added was that. So a simple script that worked for a simple customer setup. So it was working fine until we discovered that the customer has more complex stuff. So we end up with a fully managed project inside our development team. So we worked a year, almost a year, with a full development, a scrum development team with the help of our ops and a Tiger team as well to prepare everything to be migrated, communication with the customer, et cetera, et cetera. So here is why our plan was a bit oversimplified because we have some customer with complex network setup in our infrastructure because we're currently not using virtualized network. We're using real appliance for the network. So we have customer with special needs. So they have complex setup, and of course, we didn't foresaw that. So let's dive in. So I will present to you the challenge and problems that we encountered and the solution for those problems. And at the end, I will present to you the lesson that we learned basically. So the first thing for us that we foresaw was we need a representation in OpenStack. So in OpenStack, an instance basically is represented in NOVA. But the first problem that or the first challenge was we don't want the customer losing his data because his server is already working. We don't want to reimage that server because once you boot something in NOVA, you boot a bare metal with NOVA, it will install the OS. We don't want to do that. So the first thing that we realized, hey, there's a beautiful fake driver in Aronic. So let's use that. Fake driver is doing basically nothing. It faked the boot, and there you are. You have your representation in OpenStack. And at the end of the process, once the instance will be booted, we'll just remove the fake driver. We will replace it with a real driver. And now the customer can manage his bare metal to OpenStack now. But this brings up a new problem. So once the server is visible in Aronic, NOVA will see that server with an hypervisor. So if the server is available in NOVA, anyone could fall on it. If you want to boot that specific flavor, you can fall on that server because we're public cloud, of course. And even the customer itself could boot his own server erasing his own data. So what we did was fairly simple. We added the capability to reserve the node or the server for ourselves only. So through our process of migration, we reserve the node, and only us can do something with that server. So we make sure with the fake driver that we don't erase the data. So the next thing, once we have the solution for those two things, so we don't erase the customer data, we need to represent every server in our inventory in Aronic. So we have all the data. Yes, we have an ERP, but the problem is the data is a bunch of crap because there is in the past with our legacy platform, there was a lot of manual step to it. It was not fully automated. There was part that was that were automated, but there were there was part that were not automated. So basically what happened is that the customer order server to the API, let's say, the server is delivered. That same customer calls us and say, hey, I want more RAM in my server. No problem. Mr. Customer will put more RAM in your server, but we don't update the data in our ERP. So we had to do a bunch of cleanup regarding to that. So we clean up. We wrote a script basically to clean up the data in our ERP. So we had like an example that I can show you is let's say we have the OS of the customer in the ERP, but sometimes that's it for CentOS. It was written with a capital C, sometimes with a lowercase c, and sometimes it was all together. Sometimes there was a space between let's say the version and the OS and so on and so on. We had the same problem with the hard drive, et cetera, et cetera. So it was a mess, but we cleaned that up. But next thing, next thing, so we had standard flavor, no, we don't. We had a beautiful rainbow of flavors that we offered to our customer. Again, at first it was fairly, fairly simple. We started with a couple of flavors and we increment that. But since the customer can modify his own flavor, so we end up with like 300 plus flavor to, to modelize. So basically we, what we did is to make sure that we have every data in our ERP. We just rescan every server hardware. It was seamless for the customer, obviously. So we wrote a script to rescan the hardware in our inventory to update our ERP data. So we make sure that the data is coherent from the data center in our ERP. Next thing, so unsupported power distribution unit. As you may know, server need powers. And those PDUs are controlled with some interface. And in our legacy platform, those power distribution unit was not supported by IRONIC. So basically it's fairly simple because for us, we decided to migrate the server that were supported already in IRONIC. So we decided to do a phase migration basically. So start with the server that can be migrated now and implement the solution for the support of those power distribution unit in IRONIC and after that migrate those server. So we just implemented a new power distribution unit in IRONIC and I think we will submit the patch soon to the community. Next thing, as you may know, IRONIC doesn't come with a graphical console. But our customer, again, that they were having to interface to communicate with their server. Either it's SSH or the graphical console. In our previous system, they were having that graphical console. You know the console when you go in Horizon, you have the console of your VM. In IRONIC, it's not possible. So we implemented two solutions. We implemented the IPMI console and we implemented the serial console. Those are not submitted yet to IRONIC upstream. We have it downstream. But there is two blueprint about those serial consoles. So if you are interested, go vote for them and we will be happy to see you contribute to that. So now network management. So like I said, our customer has special needs and our way to modelize the network is with hardware appliances. So we don't have everything in Neutron. The only thing we have in Neutron is the represent the IP pool basically of all the customer. And each customer in every region has basically, to simplify, let's say two VLAN. One VLAN for the one and one VLAN for the internal LAN. They can scale up that, but to simplify, let's say we have two. So we have to move that data in Neutron. So we need a single source of truth. Again, our ERP was doing it. But now we want Neutron to manage that because if the customer needs to spin up a new VM or new bare metal, it needs to get the IP from somewhere. It will be from Neutron, not our ERP because we're fully open stack. So we have to move the data in Neutron. But once we do it, we have to do it for the whole region. Because if we migrate, let's say one server, we will have to migrate its subnet attached to it. And if we don't migrate every server and every IP in Neutron, we will have a problem because the customer can take his own IP and you will have a network conflict, basically. So Nova, yes, we had a problem with Nova a bit. What happened is that some of the server of the customers has multiple IP attached to it. That's fairly normal. But some IP were in the same network. And once you boot with Nova, you pass Nova, two IPs in the same network. Here it goes. Nova just exploded. I'm exaggerating there. It just doesn't boot the VM. But yeah. So what we had to do is we fake boot the server, again, using the ironic fake driver, with the first IP of the subnet. And what Nova will do, it will create a port in Neutron. And once we have that port, we can reassign the rest of the IPs to that same port because it's one port per subnet. So that's our scheme. So basically, that's what we do. So we didn't invest time to fix that directly in Nova. We know there's discussion to support multiple IPs in Nova, et cetera, et cetera. So that's what we decided to do. It was a simple solution for us to do that this way. So IP ordering. We didn't know that for Nova, it was really important to have a proper IP ordering. So we discovered that by pure luck, basically, because we were testing with test server, basically in our production. We were trying to migrate them. And once we succeed, we say, hey, now we can manage the server with open stacks. So let's re-image that server. So what happened? No signal. So we completely lose the access to the server. And we were scratching our head to the wall. So what just happened? So we discovered that the cloud in it, what it does, it takes the first IP that you passed to Nova, and it will try to add it as the gateway of the VM or the instance. And what it does, it took a private IP of the customer, not the one IP of the customer. And it re-assigned the gateway to it. And we just lost the server. So what we did, again, fairly simple, must pass Nova the one IP first. So that's what we did. And after that, everything was OK. So customers have other appliances. Because like I said, they have complex network settings. And some of their network infrastructure requires load balancer, firewall. And currently, we don't provide load balancer as a service or firewall as a service, et cetera. So it's really physical machines. So what we do with those other appliances, they have network. So for us, we consider them as a toaster, basically, but with an IP, or with multiple IPs. So again, very simple. Because like I said, we want single source of true. And it's open stack, not RERP. So we just create the port in Neutron. And we assign all the IPs to it. So if there is multiple subnet for, let's say, a firewall or a load balancer, we just create multiple ports, et cetera, et cetera. So it's really simple for us to do it that way. And again, the customer cannot take his own IP in open stack regarding the legacy platform and so on. So you think customer upgrade their operating system, right? Nope, they don't. They still use the final beta release. They are not even on 3.11. That's a bit lame, but again. So no, just joking there. But we cannot force the customer to upgrade their OS to the latest and greatest version. So we had to take a decision there. So all to know that a customer is still on, since like I said, we had all those beautiful manual process. So let's say a customer order server with Ubuntu 1204. And let's say we don't support it anymore. So what we do, we don't know if he upgraded to 14.04 or 16.04, we don't know. So what we decided to do is if we don't find a matching OS in our current offering with OpenStack, we just create a fake image, we previously created the fake image. We upload it to Glance and we boot with it. What does it mean for a customer? It means that the customer cannot rebuild his server with his existing OS unless you upload himself the OS you want in Glance. So we can reuse his own OS to boot his two image server, let's say. And if it matches, so basically we boot with it. So again, we fake a boot. So yeah, timing is everything. So again, it's a small issue in Nova between ironic, I know that's a discussion. So you probably remember that thing. So you know that the timing matters for a telegraph. So in Nova, we had a problem where we created the node in ironic and Nova will scan periodically as hypervisor. And once a new hypervisor come up every 32nd, basically will update its database with it with the capabilities of the server, et cetera, et cetera. So Nova can boot with those hypervisor, but it's ironic that managing those. The problem is the more you have server managed by ironic, the more time Nova takes to update itself. Because at some point, it takes more time to update Nova than the refresh loop that Nova has to refresh the hypervisor. So you will hit the same problem with a big cloud deployment as well. So more than, let's say, 800 or 900 nodes per deployment. And there is another problem that we don't know why it happened. So basically, once you query Nova, Nova tells you that the node is completely ready. The hypervisor is ready to boot. But when you boot it, you have a 500 error, and you cannot boot on it. So it's a bit lame, but we had a sleep of 300 seconds in our code to make sure that, not to make sure, but we take the chance that it won't take more than 300 seconds to Nova to be ready to really boot on that hypervisor. We also had the request timeout. Again, for a bit of the same reason, the more node you have in ironic, the more hypervisor you will have in Nova. So when you query hypervisor in Nova, let's say hypervisor list, for example, the call can take up to two minutes and a half to come up. So we had the problem in our own infrastructure where our load balancer timeout request was set to 30 seconds. So we had to debug that. And basically, we just bump out the timeout. Because we know that some discussions that are happening currently between Nova and ironic have to manage those problems. And we know that we can lower that with proper fix directly contributing in OpenStack. So here's the part of what you should do if you have the same process as us. So here's the lesson learned, basically. So the first one is obvious. As a development team, our job is to automate everything, basically. So we decided to do our process of migrating customers automatically. And the more automatic as possible. So we decided to automate every task that could be automated. So it really makes the process repeatable. So you have less error and less problem comparing with manual step to migrating customer. And remember that we always use the API. We don't play directly with the database of OpenStack to change some property on nodes or to modify something. We always use the API of OpenStack. Next thing, it's based on the same principle. So we decided to hire those guys to do our testing. No, I'm kidding. So we have a way to virtualize everything that we have in our production. So we developed a virtualized virtual switch. We developed a way to virtualize as well the PDUs. And so basically, in our dev environment, we can spin up our virtual data center. So we even virtualized the bare metal itself. I know it's not fully representing the production, but it's as close as we could get. And those projects are present on our GitHub. If you want to look at them, it's free. So it's open source, basically. So we have the virtual PDU, virtual switches. So it's easy to install. It's Python code. It's following the same philosophy as OpenStack. And basically, we test and retest. Because that process can harm some customer. So we know it. So basically, we took a lot of time to add unit tests to our project, system tests, functional tests, et cetera. It's all run automatically. And we even use the same infrastructure as OpenStack. So we use Garrett with Zool, if you're familiar with it. So we use the same infrastructure as the CI and for OpenStack. But we have it internally. So this one, a bit funny. So remember, I said we use the fake driver. So we hit the problem where we just, hey, ironic, put the fake driver on that node. We just passed the next operation. But the thing is, we just wiped our server. So we decided to add a double check to make sure that ironic really set the fake driver in the node before booting on it. Because fortunately, it was our test server. So we didn't harm any customer on that. But let's say we tried to boot. And it's not a fake driver. We just stopped our process there because it could be a problem. Next one, remember that we have the beautiful flavor party mix. So for this one, it depends on your case. On our side, since we're a club business, basically public. So we built by the flavor, like the VMs. So it's the same thing for the bare metal, basically. So we had two choices. So either you do a bucket, basically. So you can create a generic flavor. If you don't care about the flavor of the server, basically the hardware, you can create a generic flavor. You boot with it. Everyone is happy. But in our case, it doesn't match our business model. So we had to create a specific flavor for each and every specific hardware that we had in our inventory. So we end up with like 300 plus flavor per region. Involve your expert. In our case, our expert were the ops. We had the customer service expert that know the legacy platform. And we had the provisioning agent as well that knows the legacy platform. So those guys probably know the system better than you. And when I'm seeing you, I'm seeing developer. And they can help us with testing. So basically we were sending them our software to migrate a bare metal. And basically they will try to test edge cases that we didn't think of. And when there is a problem, we have a new requirement. We develop it. We have a functional test to make sure that it doesn't happen anymore and so on and so on. So it really helps. So clean up your data. This one is really obvious. In our case, it was a mess. So we decided to clean it up before doing anything. We didn't migrate any server before the data was fully cleaned up in our ERP. This one, you probably know Sunzoo, know your enemy. In our case, we didn't have a glimpse of what we had in our inventory. So basically our inventory was our own enemy. So we didn't clearly know it. So with the cleanup of the data, with a lot of effort, we managed to understand completely our inventory. And we were able to move on with some migration of customer. Like I said, things you don't control. So we had some problem that we didn't foresee and that we don't control as well. But I only have one recommendation for you, is deal with it and contribute to OpenStack. Because I think it's the goal, that's why we are gathering here. So we decided to contribute as much as possible with that in OpenStack, so missing feature that our customer required. So we want to offer them to the community so they can use them as well. So to show you that I'm not saying any BS, I will show a little video of migrating a server in OpenStack. And you will see that you will hear terms in the video that doesn't really quite match in OpenStack. You will hear enrollment. You will hear enroll. So basically in Ironic, enroll is just putting a new server and make it him manageable by Ironic. In our case, enrolling is migration of the old customer to OpenStack. So I know it's not the proper term in OpenStack, but just think with that video that enrollment is clearly a migration to OpenStack. Hello, and welcome to our presentation of an Ironic bare metal migration to OpenStack. As you can see in Verizon, our test customer starts with no instances. Our server delivers a web page with its uptime. Contents of the server will not be disturbed during the bare metal migration. To prove this, we will ping the server throughout the migration and check the uptime at the end of the migration. In our migration application, we first validate the client against a set of business rules to see if they are a candidate for migration. Great, our client is ready for migration. You can see that the application performs a series of checks to validate the client's network, including VLANs. The client is given the green light by the enroller to migrate. Next, the client's device is validated against a set of rules. The application examines the server's PDU device type IP assignment and OpenStack flavor. The server passes the migration's checks and is also green light for migration. The application then migrates the client and device after validating it. After a few moments as part of the migration, we create the ports in Neutron. Next, the migration tool checks for a driver update. It switches to our fake driver and then returns to our bare metal driver. We are almost at the end. An instance then appears in Horizon for the migrated device. Migration is now done. Migration operations show a series of checks that have passed for the client and the server. For example, we validate that the client's migration steps were well executed. As the client is now in OpenStack, its network is well configured and the OpenStack quotas are updated. For the device, a series of checks were also passed. The ironic node was created and reserved. The fake driver was applied during the migration. The Novel Boot was completed and we had a working instance. Finally, our ERP had all the information needed for our migrated client to ensure the migration had no impact on the customer's server. Let's check again, it's uptime. As you can see, the server never stopped working. Now our customer commands his server with OpenStack. Thank you. So yeah, we achieved that by the little application you saw is basically very simple. It passed a couple of steps to validate that the server is migratable to OpenStack. So based on some business rule that I explained, let's say we were talking about the PDUs and so on and so on. So we validate that and once it's done, we can, like we said in the video, enroll them or migrating them and we add a series of operations that we do. So basically it takes one server after the other for one customer and it will migrate them. So at any time in the migration, if something just fails, we can just restart migrating one server and it will have no impact on it and it will take the rest of the server and the customer and migrate them as well to OpenStack. And if we want to let's say migrate multiple server at the same time, we have to do it per client. So basically it's a simple Docker container. We just spin up multiple container and start the process for different clients or the same client but with different region. And we can migrate multiple server at the same time. So for the statistics, because we added the 300 second sleep in our code, it takes about 10 minutes to migrate one server. So it's fairly long, but at least it's automated. So we don't need a guy to wait for that 10 minutes to pass. We just start the process and we can look at it, let's say in a couple of hours if it's done or not because we have customer with 100s and 100s of server. So currently we migrated the easy one. Like I said, we do phase migration. So we have close to 600 server out of 3,000 or 4,000 server that we have to migrate. And we have 32 toasters. So basically we have low balancer firewall and we even have our old platform provided cloud as well. So we migrate, like I said, we migrate only the IPs and we have out of that 179 customers that were migrated right now. So they can fully manage their old inventory for them with OpenStack now. So that's it. Thank you. I hope you enjoyed that presentation. So if you have any question, please stand up to the mic because it's recorded. No question? Yeah, we're public, yeah. It's done with it. So the question is what do we do when the customer is done with it? But basically the customer will be done with it. Right, how do you still think driver? No, no, no, no, no, no. Like I said, and like we saw in the video, once we migrated, we switch it to fake driver to boot the instance in Nova. So it doesn't affect the server at all. And after that, what we do, we put a new driver, not a new driver, but our own driver that manage our ironic bare metal. And once it's done, the customer can manage the server. So he can put it down if he wants. It doesn't matter for us, he's done with the service and he can reimage it or he can shut it down. But now it opens back instead of our legacy system. The question is if we have our own driver, yes, we have our own driver because we have specific stuff to manage related to the network. So yes, we have our own driver. We don't use the vanilla ironic driver right now. The question is, have we looked at the adoption of an ironic? Currently, our ironic version is Liberty. So the adopting mode was not in it. But again, we looked at it afterward to see what it does. And basically it does part of what we need to do. But it doesn't manage the Nova side. So that's the problem. So we cannot use the adopting mode of ironic to migrate those server because the Nova part is not managed. But that could be an interesting contribution to ironic or to Nova to, if you want a server to be adept by OpenStack, it could manage more than just ironic but the old OpenStack services as well that relates to a server. Yes. Thanks a lot for sharing your experiences. No problem. Now that you have all this nice toolkit and I guess you have built up some trust in all your tools, do we actually enroll servers in parallel or one after the other? Is it not possible to do this in parallel? Because if you're talking, I guess, like thousands of servers, you don't want to do one after the other, right? Yeah, like I said, we do one customer, one region at the same time, but we don't parallelize the region itself because it might be a bit too dangerous to do it. We tried it with our own test server and we saw that we had some problem with our ERP. So we had, in our infrastructure, our ERP, we did too many API call to our ERP and it went almost down. So we decided to do one at a time but we also spin up, like I said, it's containerized application. So we decided to spin up multiple containers to do multiple clients at the same time. Right now I think we can do six to 10 at the same time and it's not too problematic for our ERP. So yes, we don't do it in a series. Any other question? So I guess that's it. So thank you for joining.