 So, hello everybody. I'm Mehdi Doghi, Debian contributor. I'm here to speak about Saibyan, which is a distro for industrial R&D needs and engineers. Another way to put it is distro for scientific computing. So, we are mainly interested at the HPC clusters and the workstations for scientific engineers. And another simple way to look at it is just Debian derivative. So, why we did Debian derivative, what it's meant for, what we did integrate it, et cetera. So, a bit of history. So, I work for EDF, which is Electricity de France. So, you can see here France. And since 2003, we are using an internal distro based on Debian for our needs. And we just thought that it should be useful to the community to publish it publicly and try to gather a wider community around this project and contribute on the same basis. So, in my presentation, I will briefly introduce our business context, why we use it for and what do we deploy using Saibyan. So, our scientific needs, IT needs, and what we did in Saibyan and how, basically. So, just one little slide about EDF for those who don't know the company. It's a large electricity producer and seller. It's a world nuclear energy leader, leader in the world and European hydro power leader. We are not only implemented in France, but all over the world. So, we do also business in Asia and the United States. It has a big annual revenue, but it's more impressive in numbers, but we are organizing small teams. Another way to look at it is like many small business units with their own budget, resources, etc. So, we don't have, sadly for Debian, we don't have that budget for Saibyan. We have a large R&D division which does a lot of work on all things related to energy and how to implement that, how to make it more, I mean, safer and cheaper. And you have a nice webpage if you're interested in more, to have more details about the company, I put the link. So, our scientific computing needs. So, at EDF, we have the R&D which works on the modeling staff, on the conception, on the renewable energies and electrical networks. All that is necessary to produce electricity in a safer and cheaper way and make it controllable by the engineering to make it efficient. One of the things that Saibyan is used for at EDF is for the power management. We have computing chains which make simulations to make us able to reduce downtime of nuclear power plants during the maintenance. And we also have all the simulations of the renewable energies but it's less computing intensive. We are also trying to plan for the conception and prediction weeks in advance because if you have a big delta between the two, you're losing much more money and it's not good for the business. So, we are trying to make that predictable and adjust either way to reduce the delta. So, more concretely, we do some modeling. So, we try to approximate reality with models and then we use those models to implement scientific codes to simulate what could happen in a system. And then try to visualize the results on the hardware with some big GPUs because you have many big simulations to visualize. It takes a lot of RAM, etc. So, you need the IT that comes with it. It's necessary for that. So, what are our typical IT user needs? They use workstations, laptops. So, for modeling, visualizing, for developing codes and accessing all the IT infrastructures, we have the high performance clusters deployed internally which contain thousands of nodes, of compute nodes, and we have some small computing chains which are used for small simulations which doesn't necessarily need a big HPC cluster. So, all those are using Saiban. All those are using the same packages, the same kernel. And we have a single solution to deploy all those environments. It's Saiban. So, Saiban is heavily based on Debian. We actually don't fork the kernel. It's Debian's kernel. We don't recompile the kernel modules. We don't fork Debian. We take Debian, Debian stable, actually, or old stable, and we add on top of that some customizations to make it suitable for our IT needs. To make it, for example, connected to the LDAP server, to make it with the printers, to be able to deploy it on the HPC clusters with the newer drivers because it's much more interesting to run new hardware for the computing because it consumes less energy. So, you have to follow the technology somehow and we do that with the same environment which is Saiban. So, we have on one side our users and some applications that do some computation regularly. Like, for example, for predicting the consumption and production, there is some codes running at different times of the day which tries to make that, makes the numbers. On the other side, we have the IT infrastructure deployed. So, a few servers, few workstations, and a few clusters. To give you a rough idea, we have, I think, like 1,500 workstations and something like 4,000 compute nodes deployed using Debian and in production. So, why did we choose Debian? We are firmly attached to Debian. I wouldn't have worked on Saiban if it wasn't based on Debian. So, we use one OS with the same stack on all the environments which gives us a full binary compatibility between all the environments. So, engineers can make their codes on their workstation and then go on servers and HPC clusters and just run their code. They don't have to port it or test it otherwise because they are guaranteed to have binary compatibility and have the same software stack. So, that's interesting for us. We don't want to have separate environments and separate and different OSes between the clusters, for example. The RISC cycle, we have one major version every two or three years, which is OK for our IT planning. Debian has just the largest scientific offering, largest package repository with a lot of scientific software. So, it's very convenient for us because when you need something, you have 80% chance of finding it in the archive. So far, there is only Ubuntu which matches this criteria, but basically by taking what Debian did. One interesting question could be why Debian instead of Ubuntu, which has some security, not security ports, not only security ports, but also contractual support, which might be interesting for industrial users. So, in Debian, all what you find in main is maintained with the same criteria, with the same QA, with the same expectations, and with the same workflow. While it's different in Ubuntu, Ubuntu focuses its main on some core components which they maintain, and then the community takes care of the rest of the packages. So, it makes somehow some second class packages, and we don't find this solution suitable for us, so we picked and stayed on Debian. Debian is also designed for customization. The installer is very modular. It's easy to get your stuff done in the installer simply. At least for my experience, simpler than in Kickstarter, for example, if you want to adapt it to your needs. You also have easy ways and many documentation to make your deployments and configuration easy. The community is also quite open. How do you see it? It's a fully benevolent, driven project, so whenever you need something to get done in Debian, you can just contribute it. It's really a dual-currency, so if you are interested in something, and you have the resources and the skills to do it, it's very easy to get it integrated in Debian. That's very convenient for us, because whenever we stabilize some packages or development, we can make sure to integrate it and have it also available to the community and have the community contribute to its maintenance. We share the load with all the actors. It's easy to integrate new applications, because packaging helpers are really cool these days. It used to be complicated and it's easier and easier with time. When you look at Debian rules today, it's very simple. You only need a few lines to make your package work, and it's not the case for other distributions. You have also abundant documentation, which gets it very easy to start doing some stuff and contribute. All those are very reasons to use Debian and stay on Debian. You might ask me why did you choose to make a derivative? Why not just using Debian proper? We used to have a six-year support on each Debian release, stable release or old stable release. Debian catched up a bit with Debian LTS, so we are able to do five or six-year support now, but it was not the case before. By doing it internally, we have the choice to make it longer. For example, for our IT needs, we still support Debian squeeze. In the total, we will support it for eight years, because there are some business applications that are still needed. We cannot port the servers to newer versions of Debian, so we still maintain the support. We have also our own security team inside Debian. It's not a big team, but enough for our needs. So it has two advantages. You might make different choices than Debian. For example, if Debian didn't qualify some vulnerability as critical, but at the same time, it's important for us, we can fix it internally. We don't have to force Debian to update stable for that. On the other side, if you are proactive and wanted to fix a security bug that is also important for Debian, we can contribute it. We actually contribute to the LTS effort, so it's one way also to give back to the community. Another big reason is because in the scientific context, when you buy hardware, you want to get most of it. You want to take advantage of all the performance that the hardware is capable of doing. So you have to deploy the new drivers with all the new versions, and it's not very easy in Debian to make that happen for stable releases whenever the versions come out. So it's mainly used for InfiniBand, Omnipath, Nvidia and some other kernel drivers and Intel micro architectures. That's basically it, so the hardware part of the infrastructure and some storage stuff. But at least we have that possibility. It's flexible enough to be able to integrate anything in the repositories. So it's not always easy also. Like I said, we still support Screes because some business applications were not ported to newer versions of Saebian. It's also because there were some libs removed in Debian. So we keep them around to make the applications still run because they are critical. And we have more time to... Yeah, there is LibTau, if I recall correctly. Tau, T-A-O, which got removed from Weezy, if I recall correctly. And so the question was, can you give some example of some removed library? And the most recent one that I recall is LibTau. It's a crappy thing. You don't want to use it. But yeah, we have some business applications that use it. And the scientific software is not known for using all new kind of technologies. And we have the choice to agree to new major versions of some software. We are not forced to strictly comply with the backports policy, which force you, for example, to use testing versions or stable versions if you want to backport to old stable. We can pick whenever versions is useful for us, package it, and just use it in our repositories. So all those are the reasons why we did the derivative. All those reasons are equally important for us. So the life cycle of distribution. So each version is supported for some years, let's say X. So we have first version, two years later there is another version, and so on. Each time we receive a new hardware, we qualify it with the latest stable release, or the much recent one available. So this one was qualified with Sabian 6, and this one happened in 2015, so it qualified with Sabian 7. And then there is our users which develop and use some business applications. And when they start writing a business application, they also are recommended, at least, to use the latest stable release available so that the life cycle of their business application lasts longer. When the basic, I mean, when the OS which was built on gets an end of support, we have some mechanisms to make them available on the newer versions of Sabian with some short mechanisms. So whenever you upload, for example, a business app for Sabian X, it gets actually compiled for Sabian X, and then the package is... So there is somehow a compatibility package which is generated for all other Sabian versions. So it's very convenient for users because they upload once, and they can use it on every environment. They don't care about which versions they are using on their, of the OS on their workstation. They just know that the application is available. They might not have all the performance that they could possibly get with the newer hardware, but at least they can run the application. So our engineers can upload business applications. They can do it in an autonomous way. And we wrote some specification for those applications. So we enforced them, for example, to not have maintenance scripts, no services, to put all the files in something under slash opt, to not collide with systems, system or Debian packages. So they just follow those rules, and the build infrastructure runs automatically some tests to say if the business application complies with the spec or not. So whether it complies, it gets uploaded and published, and if it doesn't comply, it gets removed. And each, like I said, each one is made available for each, for every available version of Sabian. So this is what Sabian is about, what we use it for, and how we do mainly stuff. So now I'm just focused on the HPC cluster side or the computing chains. So there are some problems when you try to deploy HPC clusters with Debian. So when you buy a solution of this size, of thousands of nodes, each manufacturer will come with its own solution. There is no way to ensure any compatibility between all the clusters you have. And for us, it's a big business issue, because you have to port each application for each cluster, and it takes just a crazy amount of time. So for example, for our engineers, spend at least 20 or 30 days to qualify a business application for each Debian version, and they have like 500 applications. So if they do that for every cluster, they just can't follow. So it's very critical for us. So yeah, like I described the point, there is no binary compatibility between clusters, and you can even have some conflicting stacks. So for example, with networking stuff with InfiniBand and Omnifat, sorry, each manufacturer comes with its own stack. So it's optimized for their hardware. They use the same libraries of the other manufacturer, but they integrate their own patches that are not directly put into the community. So it takes time. And you are forced to deploy the manufacturer version if you are expecting full performance from the hardware. So it's quite problematic because you are not, I mean, as a company with public money, we cannot enforce some specific technology. We receive whatever technology qualifies for some technical parameters. So first, it makes things more complicated. It gets better, but it's not there yet. There is also a large number of compute nodes. So it's nowhere near the scale of Google or Amazon or whatever, but it's large enough to be complicated to deploy in a fast way and reliable way. It's also not easy to get Debian support, so usually it's Red Hat or Suzy. So it's very common to find support for a hell, but for Debian, the first answer from a manufacturer is Deboit, sorry, they never heard of it. And it was even more complicated in the past because we had our own version of, I mean, Debian internally, which wasn't published. So they had to qualify their hardware, which an OS that is private. So we had to make some OS disks for them and solution maybe, et cetera, but that's also something that we try to fix by publishing Sabian to make it publicly available so that manufacturers and integrators can use it and test it on their own hardware in their labs to qualify it and use it. So the question is, do you know manufacturers actually doing that, doing what? Sorry. Well, you said you've published Sabian so they can test it and or check it. Do you know actually when they're doing that or is it just an offer that you send out and nobody picked up yet? So there are multiple things to say on this subject. So for now, Sabian is not fully published yet. We don't have the repository published, it's work in progress. It should be fully published by September, October. So we are almost there. So manufacturers cannot use it yet. But in the past, we made a lot of efforts to fully update and make OFED, so the software stack for InfiniBand functional in Debian, thanks to the work made by Anna here. And in the last bid, we just told them to use Debian 8 to see if their hardware is qualified for us or not. And it was okay for them. They were able to test it and propose some patches to make some newer drivers available. And we hope to make it even more easier in the future to make the, with, by making the software stacks of those manufacturers integrated in some repos of Sabian. So it will be much more easier to deploy clusters with it. And so I was saying that, yeah, with those big infrastructures, so the business users invest some amount of money to make it to 15 minutes. Okay, thank you. To buy that hardware. And they expected to use it rather quickly. Rather quickly means that from the time that the hardware comes to the data center to the day that I use, that are capable of running the first simulations, there is at most three or four months delay. So we have to prepare and advance some stuff and make it the more, make it more automatic and make it more easy to deploy and to test. And at the end, the ultimate goal is to get the, I mean, the highest performance possible. So you have to tune the system. You have to get the drivers right. You have to have the new, I mean, the correct versions of the stacks. And integrating them takes time and energy. Can I ask? So if you say three months, does it mean that the vendor, the hardware vendor, is supposed to install Sabian on it and just hand it over to you? Or are you going to deploy it on all the compute nodes? And if so, what are you using actually for that fully automatic installation or something else? We are using our own system deployment system, which I will talk about. And we don't ask them to install Debian. We ask them to make sure that the hardware is qualified for Debian or tell us where to find the driver. And we do all the software installation and the software integration. We only ask them to do the hardware installation because it can be tricky. But we take care of the software installation ourselves and make sure that before the driver, the hardware comes, it's already tested with Debian. So with Sabian, it's not only on OS. We try to standardize how HPC clusters are installed and how. So we define the pattern of architecture for a generic HPC cluster, on which we describe how to install it using packaged tools in Sabian, and then how to configure it to make it usable by your users. So we try to document that on this installation guide, which will be part of Sabian. It will also be moved to the Sabian namespace in GitHub, but it's not there yet. Anyway, it's available. And with all that, we have our deployment system. We call it Puppet HPC. The name is not... I mean, it's a deployment solution. For us, it's meant to deploy HPC clusters, but it can be used to deploy anything. You have many generic... When I say generic, it's very generic. Like, for example, there is a module for apetication in G. It only configures apetication G, and then you have another layer in Puppet to make it adaptable to your infrastructure. So it gets in two layers. You have a module to represent the HPC configuration in Yarra. So you have, on one side, all the code that configures your infrastructure. On the other side, all the data needed to configure it. And that's what allows us today to publish Puppet HPC in open source, but keep the private data internally. So it can be used by everyone, actually. And for us, it's another way to contribute to generic system installation for Debian and Cybian systems. It should work... I mean, it works without any change with Debian systems, because it's tested first with Debian. And then there are some stuff that are packaged only in Cybian, so they're working there, so no surprises there. For us, it's another way to contribute to build a community around Cybian. And that's all. It's fully... Well, yes, 70% of the modules are documented, but they are easy enough for a system administration to have a look at the code and understand what it does. And it's fully published on GitHub, so you can have a look at it and get you started quite easily. So why should you use Cybian? So first, it's still Debian. So we don't fork Debian, it's still Debian. We don't even change the kernel. So you stay compatible with the other machines installed in Debian. Then, since it's a derivative, we have our own way, our own rules to publish, to integrate stuff, etc. So we don't have to follow the policy of Debian backwards, for example. You have more relaxed rules. All the tools for deployment and other business applications are integrated in Cybian. So it's not the first open-source project published by EDF. We have also some business applications that are published like Code Saturn, Code Aster, Open Turns, Salome, etc., which are very popular in their field. And our aim is to also package those business applications inside Cybian. It's also an opportunity to meet other industrial users and have a base to contribute around. It's a solution that we tested on production. I mean, we actually use all what is presented today in projection. So it's not some pet project, but some IT team in their corner. And it's also an area where we can integrate some specific proprietary software stacks that is not possible to integrate in Debian because you are making choice and preferring one manufacturer over the other. So, for example, for the Infinity Band, etc., we have to deploy the community stack and not metanoxes or Intel stack to make it stay neutral and available for all the type of hardware. And you can find your own reasons to contribute to Cybian. Those are the ones that were obvious for me, but you can have your own ones. Okay. So what's next? We have a lot of things to do. We still have to update our website. We have to publish the packages. We have to still discuss with the manufacturers to convince them to publish their software stack as repositories and not only as tar balls to make it easier for everyone to deploy clusters. And we want to create some community around all that. So it can be integrators. It can be manufacturers. It can be industrial users. Everyone is just welcome to join the efforts and contribute to Cybian and, most importantly, to make Debian used on this kind of IT infrastructures. That was all from me. Thank you for listening. And if you have questions, I'll be happy to reply. Thank you for the talk. And I have one question about from the business side. Is it possible for the companies that have hardware, infinite hardware, for example, to test Cybian in beta mode? Like I said earlier, we will publish all our packages quite soon. I can keep you touched whenever it's done. When it's published, you can grab it and install it and test it. We have some Twitter account that is not very used, but all announcements will go through it. So you can still keep it. Okay. Another question in regards to the kernel side. So why you have to use the kernel of Debian instead of upstream kernel, plus the security patches from Debian? I'm not sure I really understand the question. I mean, Debian's kernel is upstream kernel with some security patches and some bug fixes. One thing that is important for us is to take into accounts all the environments. So that includes servers and workstations. So manufacturers might recommend some specific kernel version for HPC clusters, but it's not suitable for workstations, for example. And we want to keep full binary compatibility between all the environments so we cannot diverge from Debian's kernel. And yeah, and sometimes we take manufacturers stacks and we backboard them or port them to the Debian kernel because I already did that. It's doable either for the storage software like GPFS or Luster or for OFED stuff. Okay, I have a couple of questions. So you said Debian has a three-year release cycle, but I think in the last couple of years it was mostly two years. So what would EDF want Debian to have as a release cycle and or would it make sense to skip every second release? Have LTS for if you have two release, two-year release cycle, skip every second release and then have a seven to 10-year support for that and so not to have LTS for every release. Do you have any opinion on that? We tried to skip some releases in the past. We skipped WISI. We went from squeeze with the 3.2 kernel to jesse. It was a bad experience because when you're trying to port business applications from Debian X to Debian X plus two, it's harder than doing it gradually and for every release. So if you want to make the maintenance and the portage easier, you have somehow to follow Debian releases. That could be done in development. I mean, you don't need to deploy it that much or maybe I don't know. Yeah, but they deploy new versions quite often, at least every couple of years. So for now the release cycle of Debian is well suited for our needs and with what I showed earlier about this life cycle, we found our ways to work around the few things that weren't suitable for us. I mean, my main point is that I talked to several other companies and they also need LTS and if it's way longer than five years, telling the LTS team to support every Debian release for like seven years or something is not going to work. So maybe it's useful that some companies come together and pick a Debian release or every second Debian release and say, okay, we want to have LTS for that, but longer. But I don't know. Okay, we can discuss it. I have another question, actually. It actually reminds me of Thomson. That's another reason why people can take advantage of Sabian. I'm not pushing for using it. But when LTS stops, for example, when it stops for squeeze, we are still maintaining the security support for squeeze. And all that will come into Sabian. So it's another also reason to use it because it's still security supported and it's still suitable for using production. Right. Well, okay. My other question is more about the hyperphones computing in a sense that do you do any recompilation of packages? I mean, the problem sometimes I think is with Debian that, okay, we have AMD 64, but still we don't we don't use MTune native. So basically it's a low-scum denominator. Is there any performance problems so that your people say, no, we have to recompile BLAS? I don't know exactly what kind of scientific libraries you use, but do you have to recompile them or are people just happy with it? It's less and less true. It is less and less true that you need to recompile packages to get all the performance you can get. So some are in the engineers do it because they want to test all things, some weird compilation flags or some stuff like that, but it's mainly for R&D purposes. Our engineers use, so the packaged business applications. We are not even compiled on real hardware. It's compiled on a VM and not even with the HPC stack. And it works fine. And we made some comparison in terms of benchmarked. The same applications compiled locally on the cluster and with the version package. And we didn't find much difference between the two versions. So it's actually quite okay to use some... Anybody else have a question before I totally take over the Q and A? Okay, last question. What do you use for APC management? I mean, to schedule the workloads? We use SLURM on all our HPC clusters. And we contribute to SLURM a bit with new tools and code inside SLURM. Okay. I think we're running out of time anyway. So let's thank me again. Thank you for your attention.