 Good morning everybody, the microphone is working. Welcome back. The third day of DEF CONF is about to start. I hope you are enjoying the conference and that you enjoyed the party yesterday. Before we start a little bit of housekeeping, please when you leave the room, come to the room, be careful about slamming the door. It's really, really distracting. And also respect the full sign. When you see a sign on the door that says it's full, well surprisingly it means the room is full. Please don't enter. We'll be happy if you tweet or if you write blog posts. So go ahead to do that. And today there's a grand finale, a competition for cool prizes in room D105105 at 4.30. With this, let me pass the floor to Fabian Eretin with his talk, Sentos Infra Revealed. All yours. Hopefully the microphone is working for me. Yeah, well I can scream anyway. So good morning everybody. I was not expecting many people in the room on a Sunday morning for the first time slot after the party. But here we go. First thing, who am I? For people not knowing me, just quick summary. I'm a Belgian guy, meaning that obviously we can speak to me about Belgian beer, French or Belgian fries or chocolates if you want, but that's a little bit off topic obviously. Apart from that, I'm a season mean by choice because I really like what I'm doing. Season mean for the Sentos infrastructure. Sentos user, I would say even abuser for a long time. And I'm a member from the Sentos project as well. So what do we want to cover today? A little bit of history first, where we are coming from and hopefully where we are going to. A quick overview of the infrastructure and the way we are doing more and more interaction with other distribution like obviously Fedora, because we share the same routes and so working with the Fedora infrastructure team is really something we are doing at the moment. So back a long time ago, when there was the big split between Red Enterprise and Fedora, some people said that oh, there is suddenly a gap that some people have to fill. And so more than 10 years ago, some people decide somewhere in a basement just to say, oh, let's start something for fun, let's try to rebuild the social PM. And it was just a few private notes, right? So KB is just at the back. He would, you know everything about the machine that were maybe running in his garage or basement just to do that and Johnny in the US. So it was really, really, really small. And I don't think a lot of people were expecting the project to become so successful at a time. But you know, it was something all made. It was really just people doing that on their spare time because they like it, but we had all our jobs and we had just some free spare time to spend on it. But there was certainly a need for automation because everything was running from almost one or two machine like the main website was running on the machine, the forums, the main mirror, everything was on single machine. Then people from the community, and that's the beauty of the things, the C in Santa is community, right? So people from community said, we like you because we use it. So what can we do to help? Well, we need machine in bandwidth. So we add one machine here, one machine there, one machine there, another machine there, and it's, well, microphone is stretching a little bit. It's smart. Okay. Let's try to not move. That will be the... It's a little bit easier. Yeah, let me just try to... That Sunday morning, take it easy. Hopefully. No, not better. I can scream instead if needed, but let's try not to move. So the current situation is that we have, obviously we are just eating our own dog food, so we are still running Santa's five, six or seven. But in fact, the Santa's five note, yeah, there was only one note at the moment, still running Santa's five, which is Xendom zero, we're seeing some virtual machines, so we just need to move that somewhere else. But we still have some machine running Santa's six, and obviously the new machine that were given to us were donated to us, our running Santa's seven. We started to reinstall because some machine were donated almost 10 years ago, so we started with the machine running Santa's three, then reinstalling Santa's four, then in five, and we just continuously reinstalling those machines, because you would be surprised how many Pentium four we still have in production at the moment, behind the board of Santa's log, with one gig of RAM, only in a single SATA disk. So it's always fun to deal with those machines. So those machines at the moment, we can't even reinstall those machines with Santa's seven. Yeah, let's, yeah, maybe, otherwise it will be annoying. Let's, oh, one, two, one, two. So back to the thing, it was still running on a lot of donated machine everywhere. So we had most of the machine coming from company in the US, some in Europe, in the Europe, and one machine in Brazil, one machine in Malaysia, one machine in Australia, something like that. So we were just, in fact, reinventing the cloud before it was a thing, in the sense that we had suddenly to, you know, automate a lot, a lot of things, because we had no time to work on that. So, puppet was still a thing. So I even remember in 2007 at FOSDM, we had Luke Kenes in the Santa's destroyer room, just giving a talk about puppet, which was beginning to start to be a thing. So we came with, I migrate from puppet to puppet. I think I was using from puppet zero to the three, two, yeah, multiple version, and at the moment we're running three dot six. The thing we did was to switch from plain puppet, most of the, how many people are using puppet in the room? Oh, not, so the rest is using, and say, well, maybe, yeah, or salt or whatever you want. So we switched from plain puppet master D to puppet and foreman. We use foreman as an external note classifier. It was mainly for a puppet dashboard, and then we are just separating everything, variable from the modules, so everything is coming from foreman. And still, we started to use a little bit of Ansible for specific ad hoc task between machines, or even what Brian will explain that in the next talk in the CI environment. About monitoring, because when you have machine running, well, all around the world, you still need to do some checking. So we decided a long time ago to use Zabix. The main reason was that, well, KB and myself, we were using Zabix for our daily job, so we just decided to use it because it was, it fits the bill. So we have Zabix agents install all the notes. The only problem we had was you, that we had to use Zabix proxy. If you are familiar with Zabix, you know that you have latency issue, so we have to use Zabix proxy sometimes to get information from those machines because the topology of the network is geo distributed. One thing we do, what we are still doing is, for DNS, nothing fancy. We are just using bind, like almost everybody does, except that we have geo redirection. So we have a delegation for some of the notes, some of the record, like, if you try to it mirror.centers.org or msync.centers.org, you will be redirected to your geo IP, the nearest machine for your location. So that's transparent for a user, but that's really something efficient at the moment. And even if people are not really supposed to eat those machines quite a lot because normally you just eat mirror list and you download the package from your mirror, one of the external mirror. Some people are just using plain mirror.centers.org and we have, on average, 400 requests per second for the DNS record. And yeah, we obviously have some of the things. Let me just put some notes. Yeah, that's better like this, for me at least. We have other machine, obviously website, forums, bug tracker, mailing list, and torrent cedar for people willing to use the torrent tracker. One of the note that is really crucial for us and also for you is the msync role. That's how we distribute all the updates and all the new release to external mirror. We have, at the moment, 69 donated machine dedicated for that role. We are just using a pyramidal setup where from the master we just distributed to the level zero, the ring zero, and then the other nodes. From a statistic point of view, when we're pushing new release, the aggregated value for those who know this is actually pushing to four gigabit per second during two, three, or four days, depending. And they are just used to seed all the external mirror. We have, at the moment, we have something like more than 600 external mirror that most of the people are using, not counting the people using an internal mirror for production, obviously. So we have, as I said, we have a lot of machine in the US, a few in Europe, but some part of the wall are not really well covered. Like I think we have just one machine in South America and the machine is in Brazil. The machine has not a good connectivity, but on the other hand, internal to the country, the machine, that machine has a really good speed. So it's really quite slow for that machine to get everything from us. But when the machine has everything, it serves all the other machine in South America. Same like Malaysia, you know, we are in 2016, but we still have some machine connected at 10 megabit per second. It's not even limited at the router, I mean really connected at 10 megabit if you look at a DH tool on the machine because it's really limited at the switch level. So why do we want to keep those machines for the same reason? Because 10 megabit per second is really slow here in Europe, but in Malaysia it's still a thing to have a machine in the data center connected at 10 megabit per second. So it's quite an nightmare for us to manage to have that machine up to date, to distribute the update to the other machine, but it's still something that when you are a user, you are happy to face that machine in that region. How do we verify that? We have a process, we are not using Mirror Manager. A lot of people are expecting us to run Mirror Manager, which we don't at the moment. That's something we can eventually do in the future, but if we speak to the federal people to see if we have the same problem to solve. But at the moment, we have a process that check in loop every mirror for every release, every repository, and also the ease of file are checked per country. Meaning that if you, well, let's run parents, but we have no base URL in the YAM repo file that's the mirror release thing. So you are once again redirected to your Nero's mirror. And same for, is already directed. If you want just to download a Nero file from us, you're also redirected transparently and it's check in loop. The fun begins with the donate machine because everything is fine. A company said, oh, I want to give you a machine because I like you. And obviously I like free gift, so I accept the machine. And almost all the machine are running on those donated machine. But it's sometimes, well, we had to migrate the role multiple times because you receive a machine, but you don't know if the machine will stay for one month, one year, or 10 years. So we just have to automate everything because we will switch the role. Sometimes we add to, well, there is other issue because machine is maybe 10 years. We had last, for example, in two months time we had to migrate the wiki that's just a fact from last year. We had to migrate the wiki instance three times in a two months window because the machine was moving from one DC to another or the machine disappear or add another issue. That was really some of the problem we are facing too with. Same for the mailing list, we had to automate all the things because you don't know when it happened. The machine can just disappear. And the main reason why a machine disappear is not hardware issue. It's just that the company doesn't exist anymore. That's really something we had a lot. You know, you have bankrupt or even something more fun. You have company A giving you a machine. Someone inside of company A give you a machine. The guy left. Management is not aware. Company B acquires company A. And sometimes company C acquire company B. And suddenly you have no contact anymore with the people giving you the machine. The machine still runs. Sometimes they are just, well, they do an inventory and they say, what the hell is the machine in the DC? So they have a name. And we have notification. If they are smart enough, we have notification at machine, you know, well, we are not interested in sponsoring you anymore. So we don't care. Sometimes they don't. So we just discovered by was that machine is not reachable and we try to contact people. Nobody's answering the phone or something like that. So it happens to us quite a lot. The more interesting thing is that when you know that company went bankrupt, but the machine is still running. So we don't know who paid the bills, but we have machine in DC still running. We just crossed the finger to hope that there is no, there will be no electricity outage in the DC or out of issue because we know that we'll lose machine but machine still runs. And that scares me when the machine runs without any inventory but that's a good, that's a fun part of managing those machine. So what we do with those donated machine, the first thing we do is obviously just reinstalling those machine because I don't know if you have already tried to use a dedicated and donated server, well, a dedicated server somewhere. Sometimes they are smart. You know, they provide you Fedora, Ubuntu, CentOS image, official one. Sometimes it's a bastardized version of CentOS. Like I don't want to name OVH, which I did, but they just have a different kernel they force you to use with GRSEC instead of S&NX for example. Some other DC guys are just injecting directly their own SSH key into root account or things like that. So I'm a lazy guy. I don't want to audit the machine. I just reinstall it directly. So how do we do that? Because obviously we have no, you know, it's a single machine in a DC. So we just, with a simple script, we just reinstall the machine. So we just on the machine, as long as we have a S&H connection, we just don't know the kernel, the initrdee. We just install kyexec tool. We inject the kickstart. I guess that's a lot of, well, you are all familiar with kickstart. We inject the kickstart in the initrdee and we just chain and we group it on the kernel and the machine auto reinstall itself without any IPMI or VGA connection at all. When the machine reboots, Puppet is called for certificates and yeah, it starts configuring the machine for us. Meaning it's just allowing source on IP to just SSH into et cetera. What we do is that we start small because the first thing we have to do is try to build a relationship with the company willing to sponsor us. Because I said sometimes you don't know the company. So we have some random guy somewhere willing to sponsor the project. So we have a machine. We start with something really non-crucial. Like we have, I mentioned the emsing role, so mirror machine. We start with that because what can happen? There is no data, there is nothing in it except package, which are GPG sign, right? So at the moment we start with something like that. And what we do usually is that we just, before putting the machine officially into the mirror network or into production for something else, we just test their support response time. Because some DC are really fast. It's cheap, it's free, right? It's donated, so there is no SLA on the machine. But still some people like us and so are giving answer to tickets we create quite fast. Sometimes it can take days or weeks because of that donated machine. So we just start with something really small and with the time, we know if we can trust those people or not. Because they prove that we can use their services. Sometimes we are lucky in the sense that they want just to sponsor two machine in the same DC, which is, because we are not the traditional on-premise thing, we have machine everywhere. So sometimes we have two machine. So we can do some kind of fellowver thing like for the DNS at the moment. We have machine where we can just switch from one to the other if needed. And we know those people, so we know that we can use their support. Apart from those machine, you know that people are facing for mirror. We have also other resource like in DC at the moment, we have four physical node that we use for what we call the DevCloud. So people willing to work in a special interest group in Santos can abuse resource in that DevCloud environment. It's at the moment still based on OpenEvola. So people can just spin a VM quite easily. And we are just using Luster with InfiniBand for the storage part of that. If you are interested in details, why we had to switch to InfiniBand instead of using standard internet, feel free to ask me at the end of the presentation. It's running at the moment quite fast and we are even thinking about expanding that because we have more and more people interested in abusing that environment. Apart from that, when I mentioned donated machine, I should say that obviously two years ago something happened with Red Hat. I guess that people are aware of that. So the biggest sponsor at the moment is in fact Red Hat because they sponsored some IBM Blade Center in IBM facility, in Red Hat facility. So we have a dedicated build farm, the Koji build farm, running in that environment. And it's basically at the moment just, it was only Intel X8664, but we added recently ARM64 as well, builders in that environment. And in a remote environment, we have also PPC64 and PPC64 LE because probably you are aware that we release support for those architecture in December. Same for ARM HFP. So ARM V7, if you have Raspberry Pi 2, feel free to abuse CentOS 7 on it. It works. And next to that build environment, really next close because it's in the same rack. We have our CI environment setup. If you are interested in the CI environment, Brian will explain all the details about how you can use it if you are a member of a project in a special interest group. At the moment, it's Jenkins driven. If you just visit HTTPS, CI.Center.Zor, you will recognize automatically Jenkins UI. We have some chassis that we are using with compute node inside. So it's a 256 compute node that we can abuse. For that, we do bare metal for CI. Brian will explain that, but we do bare metal because we want to be as close as possible to what happened in reality. Because we have RDO people building on top of CI, so on top of CBS. So if, for example, you just want to, you have install OpenStack, CentOS release OpenStack, you can. It's built on CentOS, for CentOS, and distributed through the mirror. They want obviously to test the whole thing. So they want bare metal because then they can simulate everything on top, including the VM. And nested VM is maybe not that good. In the back, it's just Ansible that is provisioning and re-installing all the physical machine as fast as possible. So the roadmap for the CentOS infra is extending the authentication because Puppet takes care of the accounts of all the machine at the moment for the infra team. But still, we add more and more people interested in contributing to the CentOS ecosystem. So including for the special interest groups. So they need access to the Koji build farm. Instead of using our own, well, it was a custom made CA script and wrapper script. We decided to move to something that a lot of Fedora package are known already. So it's, we have accounts.centos.org, which is backed by FAS, but not the Fedora FAS. That's the FAS backend, but on our own system. There is no federation at the moment between the two. Something that can be maybe done in the future. We don't know. We obviously want to start to migrate more and more web services to that centralized authentication because at the moment the forum are still using their own local database for a user, et cetera. What we want to do also is have faster data going to mirror because we are faced with an interesting problem. We have a lot of machines with limited capacity and we have more and more and more package and update of package and cloud image. And it starts to become a problem for us on those donated machine. So we are just rethinking the way we will distribute all the machine through some kind of caching and CDN for the infra, based still based on the donated machine and some kind of message bus. We are still to think about it. If you have ID, feel free to share about how we can speed up communication with the external mirror. That's basically already the end of my talk. So if you have question, feel free to raise your hand. Yes. Not reintroduction, no, what? Yeah, so the question is, do you run some kind of intrusion detection system? More or less, well through the audit, because if we see that, for example, it happens to us that some people in the DC just had to do maintenance on the machine without notification for us. So what we just do is just reinstall the machine completely automatically. Because everything is automated, right? So it just after the time, the time is just needed for the machine just to sync everything back from another machine nearby. So we consider those machines like, you know, pets and cattle. So those machines are cattle. We have the machine, we are happy. The machine is gone. Nobody regrets it, right? It's not like if we were using the, well, the main machine like Git.center.org, which is not running on a donated machine, obviously. But for those kind of machines, no, we don't. We just reinstall the machine directly. Other question, yes? That's an interesting question. Puppet, because you describe the state of the machine and we started with Puppet a long time ago. So we continue to use Puppet for that reason. But when you want to do some orchestration, Puppet doesn't do that, you know. The Puppet MasterD compiles a catalog and it's just for that node. If you want that node to interact with another one, then it begins to fun. Especially for example, consider the main issue that I had myself with Puppet and for example the configuration management in general is the fact that you don't have a clear view between Puppet MasterD and the monitoring system. The only way to configure the Zabixin surface, maybe, well for us, directly is to use something else that, you know, traditional way of Puppet, of doing things is you use exported resource and then you reuse all those exported resource on the monitoring machine to just apply all those configuration. Sometimes it's really crazy because the more node you have, the biggest catalogs it is and the biggest time, well, sometimes it's 30, 35 minutes just to verify that nothing changed and that they are monitoring the correct thing. So I'm just using a kind of, you know, workaround at the moment. I said that we are using Forman. In Forman, they have hooks. So it's even driven. So if I update or create or delete host at Forman, I have a script that says, I have a matching table that said, oh, that's Puppet roles as that monitoring template in Zabix. So if a machine is changed with added or removed role, it automatically use the hooks to just use the Zabix API to add or remove the roles at Zabix. And using the same thing in both ways, if Zabix said that, oh, that machine is gone, for example, it reflects that directly through the Forman API. The way I use Ansible is like, how do you install Puppet, for example? You have to start from somewhere. So at the moment, it's fun that you can just use Ansible Playbook to bootstrap Puppet itself or even set up Puppet MasterD. So and some other tasks like we use, we abuse Ansible for like, as I said, re-installing the machine automatically with the Jinja 2 thing for the kickstart in the C environment. The machine are re-installed on average 100 times per day. That was me, what would be? So it just, well, it's transparent in the back end. Puppet doesn't do that. So, you know, that's really different. If you like, you can use the two. If you, you can discuss with the Forman guys, they just did the integration with Ansible at the moment, meaning that you can just use, you can still continue to use Forman as external load classifier for Puppet. But at the same time, you can have an inventory script to use Ansible in point to exactly the same inventory. And you can even with a callback plugin, have all the reports from all the Ansible Playbook calls directly back into Forman as well, like you do for Puppet. It's quite cool if you want to use it. So, other question? Yeah. Okay, so the question is, I mentioned Gluster and Infinity Band. Why? We had a space constraint in the rack where those machines are hosted. So it's just a four machine setup at the moment for the DevCloud. And we need a shared storage. So at the moment, every machine is at the same time a supervisor and Gluster storage and client. We started with, you know, dedicated Gigabit Ethernet connection for the storage. But then I realized something. I don't know if people have played with Gluster in the room. No, Gluster is really damn easy to set up. But it's so easy to set up that it hides some problem. Something you have to understand with Gluster, it's that, well, it's turning now into a Gluster session, but I don't mind. If you are running in distributed and replicated setup, your Gluster, in fact, it's just a metadata server, meaning that if you want to replicate it, it's the client's responsibility to send the data twice. So if you are just doing that over Ethernet, you're bent with it automatically divided by two. And can you imagine the fact that your virtual machine is in fact, you know, the ION virtual machine is really bad? So we try to optimize that just by switching to Infinity Band. If you look on eBay, you can find 20 gigabit or 40 gigabit Infinity Band for almost nothing those days, including for HBA. And instead, you can use Infinity Band in two way, like the traditional RDMA thing, or you can just use that with IP over Infinity Band. So you treat your Infinity Band HBA like a normal 20 gigabit Ethernet card, in fact. And the only thing you can do to speed up that is, well, it's the same for storage. If you are using iSCSI, you know that if you want to tune iSCSI, you are going to jumbo frame. So you can do the same in Infinity Band. You have connected mode versus data gram mode. You switch from one to the other. It's more or less the same thing as jumbo frame. Once you do that, and if you are using Glossier for your virtual machine, you just need also to switch to libjfapi and not using fuse, please. Because fuse is really, really everything but speed. So switch to that. And then everything starts to fly again. So, but it, the reason why you decided Glossier was, you know, we had no storage storage and we need a solution on, so using the machine of both as a supervisor and storage node. Does that answer the question? Okay. Other question maybe? Yeah. We can discuss about that later maybe. But yeah, some people sometimes try strange things. It's true. But yeah, we can discuss about that if you want after. That will be, yeah, I will not have time enough maybe to discuss that. But you still need a pair of gloves. Another question. Last chance to win a pair of gloves. Center's gloves. No question. Okay. Thank you. Oh, yeah. You want gloves, right? Okay. Well, the torrent for the YAM operation, no. But that would be interesting. Yeah. Yeah. Well, we discussed with those guys because as I said, we have also a center seven arm HFP, which is there, but we decided to go with arm HFP, so arm V seven. So our target was not arm V six. So it's only works on a hundred by two and a cubic truck and all the oddball, but not one has very by one red sleeve guy decided to be compatible with their own legacy thing, meaning that they were targeting arm V five, which is really older. It's still running on a Raspberry Pi one. So I, yeah, if they want to join the fun and benefit automatically from the army or they can just be part of centers and build arm. Well, the NF doesn't exist at the moment in real seven, but yeah, you said torrent. Sorry, torrent, I meant torrent. Yeah. For the YAM part, I don't think that you will, well, YAM itself has plugins, so maybe if someone has a crazy idea like that, but I don't think that it exists at the moment. And that would be quite interesting to YAM install something and, and starting a torrent client in the back. Yeah, that means that you have a, yeah, you have maybe a big, big mirror. Well, no, you have a, you have no mirror and you have a big, big infra because a wall, a wall set of package for red sleeve, maybe something, well, for center seven is the same thing. It's not that big. And if you, if it's starting to become big, only if you have, you know, a lot, a lot of release in parallel and a lot of architecture. So pretty sure that they, they have some mirror to answer your question, they have some. So I don't think that using torrent would be good for that. Just to distribute to other machine, but not for the client part. You still have, you still have some gloves. No question. Thank you. You know, just, you know, a simple process on the machine. So there's no real, I don't get you about it. Yeah. Yeah. That's something you have to think about. Yeah. Yeah. Not, not a little bit. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Okay. Cool. Yeah. We have some. Thanks. Thanks Good. Yeah. Yeah. Me too. Just switch your slides on this flat drive of nothin' with now that you have time and arrived me back home. You are, you need thaosegit now. Yeah. Yes. There you are. Yeah. I'll show you a sign, like, 10 minutes left, then it's up to you. Okay. You can hand out the scarves for questions. You have three. You can just leave away three. When somebody asks questions, we'll just repeat it for the, you know, for the camera and for the mic. And you might want to take the mic, so this is where you mute it. Okay. And I mute it, but we had some problems, so I'm going to use this one. I think it's fine. It's just, you know, sometimes it's just... Yeah, it's fine. I can use this one. Okay, thanks.