 Hi everyone, welcome to the Fedora CoreOS talk. I am Sini and I am working in Fedora CoreOS and I am contributing to Fedora from like maybe four years or something like that. My name is Yakup, I am software engineer in Fedora multi- branch team at Red Hat. I am contributing to Fedora since I started at Red Hat like five years ago and you will see how my work fits into the Fedora CoreOS. Okay, so can you tell me how many of you have already used or know about Fedora CoreOS, some context? Can you raise your hand if you know about Fedora CoreOS? Okay, then yeah, I think we'll give some introduction and we'll proceed with the talk. So here's the agenda for today's talk and yeah, let's get started. So what is Fedora CoreOS? It's an operating system and it automatically updates and it's a minimal operating system for running containerized applications and it's aim is to run the container workloads securely and at scale. You can run the operating system in the cluster or it's also works well in the standalone. So yeah, it depends upon what kind of use case you want to use it. Also, it's a new Fedora edition. So it has all the underlying things is RPM and it's coming from the Fedora RPM which you have all signed and everything is from the Trusted Place. We are not using anything additionally here. Also, it is upstream for REL CoreOS but it brought a scope because REL CoreOS is mostly right now in the OpenShift but Fedora CoreOS can be run standalone or with other use cases. So a little bit of history about the Fedora CoreOS. So we have Fedora Atomic Host which is currently getting released now as well with two weeks of release and there is container Linux from CoreOS. This came from, I mean, after the acquisition, these two were together here and we were like, we have two awesome distribution which is both have common goal running containers and we really want to have something even better. Like, it's better to use both and get something new which has better benefits and all the awesomeness from both of them. So here we are with the Fedora CoreOS, a new edition within Fedora. So it's like, we are trying to get this from last year, this Fedora CoreOS and it's taking time because we are maintaining Fedora Atomic Host and as well as we are maintaining container Linux, we have users for both and we want to maintain them and we want to develop something new, including both. So we have a lot of design, design decision to do what we want and what is best for the new operating system. So yeah, we are here with the new operating system with the awesomeness of both of them. Features. So if you already know about RPMOS tree or OS tree, it also uses the same technology in underlying. It has OS tree and RPMOS tree on top of it. So we have immutable host, which means the slash OSR is read-only file system. So like, if you have used the silver, blue or Fedora Atomic host, you already know that the slash OSR is read-only mounted. So really you can't change or anything, which keeps the host immutable. Also, it uses RPMOS trees, which gives the features of automatic update and rollback. We'll get to know more about the updates mechanism later on. But here we have atomic update, which means that the entire operating system receives the update together in one transaction. So if you know about Git, it follows the same mechanism, similar kind of functionality, like everything in your entire OS is kind of composed together and it forms a single commit. So everything can be tracked here. And if whenever a new update happens, it's like a single transaction update. And if something goes wrong, we have a very nice feature rollback, which you can use to go back to the previous working system. And everything is tracked, so it's really easy. And Fedora Core OS have Ignition. So Ignition uses Ignition config. So during the first boot, Ignition runs the Ignition config and it configures the system, basic things like user creation, the system partitioning or anything you want to do. Or if you want to add something, you system the unit, you can add it in the Ignition config and system will take care of during the first boot. So what happens is, you either get a working system when you use Ignition during the provisioning, so whenever your system runs, your Ignition runs successfully, then you have a working system. Otherwise, your system is not running. So yeah, it's not, it doesn't leave you in between intermediate state, like it's not working half of the thing. So unless everything is working fine, it won't give you a working system. So that Ignition really helps a lot here. And it only runs on the first boot. And it provides you immutable infrastructure. So what we mean here is, in Fedora Core OS, what we want is, you don't change anything here on the host. Let us manage everything to update and you only have to run your application there on the host inside container. So everything will be running smoothly and we'll take care of the host, updating it with all the security fixes, etc. So suppose you want to add something in, change something in the configuration in any of the ETC file. Because yeah, it's the OS2 base system, ETC, you can change things and slash what is also like writable. So if you want to change something, please don't do it here. The recommended way is add something into the config and provision a new system with the updated config and discard the old one. That's how you maintain the infrastructure immutable. Like you're not changing anything. It's the same thing, whatever you have provisioned in the beginning. Also it provides you rolling release. Rolling release is like very interesting. It came from, I'd say container Linux. They have different channels called alpha, beta and stable. Here we are using rolling release thing. We have different streams. So like, if you are in, there is a single ref, which we call in Austrian way, or if you are using Git, you call it branch. So yeah, you have a single branch and you will continue receiving the update there for the entire lifetime of that particular system. We'll get to know more in detail in future slides. Also, Fridericorias has no Python on the host. We were able to get it after some effort and discussion on the community channel, how we want to get all the benefits and still remove Python because we are Python. It's a lot of things. So we really worked with maintainers and they were really friendly. We were able to do sub-packaging and get whatever we wanted in the host and without Python. Whatever was Python there, it was moved to either another sub-package or we figure out we'll remove that from the base. And we'll try to get that functionality from the container. So yeah, we have no Python on the host. Also, since it's a minimal operating system and it gives you a containerized workflow, we have all the container tools available in the host, like Podman and Mobi, which gives you Docker. So they are pre-installed, so you can try whatever you want. But yeah, Podman is the preferred way, I would say. Okay, so release stream. Let's talk in detail what the release stream is. This is a new thing, I'd say, at least in the Fedora side, because I haven't seen these in Fedora so far. So we have multiple streams here in Fedora CoreOS. So we have categorized it into three. First is production. So there are three refs in production, three refs, you say, three streams, you say, or three branches, whatever you can say. Like if you have used atomic host or silver blue, you would have seen there is a ref when you do RBMOS tree status. There is a thing called x8664 slash 30 and something like that, which keeps changing every time you update to a new version. Like if it's a 31 base, then the ref will change and you will have to rebase your system to the newer ref if you want to update your system based on the latest release. So here we have these only streams and it will receive the update for the entire lifetime. So one is testing. So what testing tracks is the current Fedora release. For example, we have currently Fedora 30, so there will be Fedora 30 content and the body updates which we have in the Fedora 30 and any additional fixes. Like if you're not able to get something in the current release and we want to get it as soon as possible, then we try to get that additional fixes in the testing. And then stable branch. So stable is basically a promotion of testing because we don't want to add any regression there. So testing is already like people are trying out from a few days or something like that. So we know that these are working fine. So it gives you extra stability and the testing after a few days of baking, it goes to the stable release with any additional fixes. If any end-moment fixes comes out, then it will definitely go to the stable stream. And then there's a next stream. So next is for tracking the new upcoming features. So Fedora has like right now Fedora 31 upcoming. So next should track Fedora 31 after the body, the branching happens and body updates is enabled. Otherwise, it's of not much use because we don't have any next coming. So it will be mostly testing, you can say. Yeah, these are the three streams which is like recommended to run in production. And the recommended way is to have some of your nodes on testing, some of the nodes on next and majority on the stable because that's the stable. And why testing? Because you can get and see the regression coming in your system if any in advance. And you can notify us and I don't want your entire nodes or cluster getting down. So that's the recommended way, have some on testing, some on next and remaining on the stable release. And yeah, the source of the packages coming here is from CoroS pool repo. So what we have here is whatever packages are coming in the Fedora CoroS, we want to keep track of everything. So we have a new tags created by Relinch which is called CoroS pool. And all those packages goes there and we have a big giant repo there and everything comes from there. So the reason of doing this is we want to keep, we want to make sure that we can rebuild anything we want in future. Like suppose something was done two days back and if there is a new body updates, the package will get updated and you will never be able to get the same compose or anything using the current version. So what we have here is in CoroS pool, we have a log file mechanism which will help you to keep track of in which version, which all packages with NVR has been there. So that's how you can reproduce your build and it helps in testing or any other things. And these are the development streams, testing level and next level. So testing level is the night list snapshot of current Fedora release which will be 30 and the body updates and periodically it gets snapshotted to the testing stream. So the testing what we have in the production testing stream that comes from the testing level and next level is for tracking the next thing. So whatever comes in next, it will get snapshotted from the next level and it contains the upcoming Fedora release which will be Fedora 31 in this case. And this is not recommended for production use. This is mainly for testing or anything developer testing or something if we want or mostly things will be running in the CI so we catch any regression in advance and before getting anything into the testing. Also the source of the packages are in CoroS pool here. And these are few refs which we call branches or streams we call as mechanical streams. These are branched body updates, body update testing and dried. We try to keep the name similar to what we have in Fedora so that people don't get confused. So branched is then the next upcoming Fedora which is like once it get branched like after sometime when a stable version release we branch from Rohit so that time the branched will be in use and that will contain the night list snapshot of the packages which are in the branched. And body updates is like night list snapshot of current Fedora release, body updates included and similarly body update testing which includes currently body updates and body update testing. So the reason is to have these refs so that we keep at least CI for everything checks so we can easily go back and see when something goes wrong and if you want to test something new coming in Fedora we can quickly have VM or instance launched from these and check it out there rather than realizing later and getting sad that these things came into the production or testing and people are going crazy because things are broken and we want to keep everything in rolling release so we really want the system to be stable. So yeah these all the things are hopefully like extra precaution taking and everything to keep things stable here and these are also not for production use so production you only use the refs which are mentioned in the production section which is testing stable and next and majority on the stable. Also the source packages are here directly from the Fedora repose so whatever we have in the Fedora repose the current one is just using directly that nothing else. So now we talked some components which we use in Fedora Core some of them are already available in the Fedora Core host itself and some are like we use to develop the Fedora Core itself. So for example Core is a similar this is the building tool which is responsible for creating all kinds of image artifacts which we have in Fedora Core OS. For example ISO QCAU2 or OpenStack image or AWS image anything we want to develop everything is coming from Core as a similar. There is a single tool which is really easy to develop also because you can quickly and easily run in your system. It's really very easy just you can go to the GitHub page and follow the documentation. It's really very easy and RP Moistree is the heart. It's like it used to compose the Austrian repo for the Fedora Core also it's responsible for updating the system rollbacking everything is done through the RP Moistree and then Ignition which is for initial configuration of the system during the first port so it runs during the initial MFS during the whenever the system first port and once the initial config run everything gets updated and then it switches to the real root system. And we have Mental. Mental is for running tests like it has a sub component called COLA which runs the which launches in the AWS or other stuff images and it tests really quickly things and it also do the religious stuff uploading to the different cloud providers that's really pretty cool application as well and Zincati. Zincati is a really new tool. It has been developed recently for coordinating the automatic updating so it is in the client which listens to the Cincinnati another tool and if it sees there is a new update it says to RP Moistree to update the system and then reboot it and I think if system doesn't would find then it will automatically rollback to the previous version and there is an FCCT so this is called Fedora Core as config transpiler this is also a new I'd say in Fedora is from container Linux they were using like we don't directly write ignition config because that is in JSON we provide something a bit more human readable format called YAML and so we write YAML config and FCCT what it does is it converts it into ignition config and on top of it it also checks whether your config is right or not if something is if the config doesn't look good then it will throw you error and it will fix it then you fix it and then you get a new config written and then convert it to ignition config so that's because it's really important because ignition config runs on the ones and it prevents the system so that needs to be the correct format so FCCT really helps a bit so it's recommended don't try ignition config yourself use FCCT tool and where you write Fedora Core as config okay so since we are talking you can see how Fedora Core config look so yeah it's a YAML syntax where right now it's 1.0.0 so that's the version defined and these are the syntax like here is a name of the user which we're creating and the password hash this is a solve password which we create whatever you want to give and then SSH keys which you want to add which all groups this is a simple FCC config file there are a lot more things you can add their system unit and file system configuration something like that so that you can check the whole things on the ignition or the FCC documentation page and to run this tool you simply do the FCCT this command to run it and it converts you into the ignition config the last command which is in there okay and there is a really very interesting which was very very recently we were able to do that and this really important auto update thing so really I really enjoyed it because it was done recently by Lukahi sitting here and what it does is it's auto update your system and in a rollout way so what happens you can if you see this file it says updates and there are some barriers and dead ends so what happens that since it's a rolling release and automatic updating system it happens that the user can't be on any of any of the previous version nodes or anything so suppose it can happen that sometimes your system is on updated system and due to some reason that particular version was not good or something like that then we would like or any other issue so we want users to not update from that system to another that's kind of a bad kind of version so that we call dead ends because it ends there so we keep track of those kind of version here for example recently when we did the release of federal corollies this particular version which was 30 dot 2019, 7, 16 dot 1 there were some issues related to RPMO stream and because of which we're not able to auto update so we kept it in the dead end section here and this is the most interesting thing rollout so what happens is there is a rollout period defined in which all the nodes which is available in the internet they will receive the updates so we define here a version number so which is the latest version which we are going to release so that the system should be updated to that particular version and this is start epoch so the time at which the rollout will start and there's start value so this is the percentage which is 0 in the beginning and the duration which is 120 in minutes so what happens is in 2 hours all the system will be updated to the latest version automatically using Zingati or RPMO stream which is on their host so the update will start from 0 dot 0 and the percentage keeps increasing as the time goes on and at the end of 2 hours from the start time which is mentioned in the start epoch all the system will automatically update it to the latest one and suppose something goes wrong then we also have a feature to pause the rollout and do the fixes and we can roll out a new release that's really interesting here okay now I'll hand it to you so currently you can get CoreOS only for 64-bit Intel it targets bare metal environments so it will server and other boxes it targets the mainstream virtualization environments PMU, VMware, OpenStack and VirtualBox you can also run CoreOS on the mainstream cloud providers like AWS, Azure, DigitalOcean on Google and the packet here comes my work currently it's just a 64-bit Intel but I'm looking into way with help of others in CoreOS to get other 64-bit Fedora architectures like 64-bit ARM, PowerPC, little NDA and 64-bit mainframe currently I'm looking in direction of bare metal OpenStack and PMU basically virtualization and bare boxes I'm looking into direction of the cloud providers I think that's really open questions if there are some that would be interested in that current state of CoreOS for multi-arch it's actually building and running but just locally we do not have infrastructure that currently the Intel version has and as the whole CoreOS is built or Fedora CoreOS is built on top of Fedora we could leverage a lot of work that CoreOS folks did and that was done in Fedora so it was fairly straightforward to fix the individual issues and really big thanks to the community and to CoreOS and Fedora CoreOS folks so currently the next big goal is to set up build pipeline similar to the Intel one we'll probably need OSBS and the rest of the bits as an OpenShift cluster that's used in build of CoreOS Fedora CoreOS so I will... Okay, so we talked a lot so let's see where we are currently so we had the preview release on 16th of July which we announced and so in this release what are things are really already available so we have x864 images building and we are currently having testing stream and you can download the artifacts available from getfidora.org slash en slash coreos slash download so all the available images are there and once every time we do update things will be updated on the website and the new artifacts will be automatically be available there in the website so do try it from there also in this release we have the bare metal support available so you can install it on your bare metal in the PXC or using directly ISO and the QMU also works for us and OpenStack images are also there VMware images are also there so you can try on these platforms from the preview release and we have also in the cloud provider AWS it's available in AWS, the AMI is available so you can try launch something there and try quickly also the no python on host is there so we were able to remove the python dependency from the host and the automatic update is also in effect so if you have like the latest version of Fedora CoreS preview release installed on your system so whenever we do an extra release it will automatically update your system reboot it and you will have the latest one so these are all things that are already working and in place yeah so let's see some demo so that it's not like I'm just talking here okay so what I will do I will do a quick demo with the QCOR2 image because it's easy to launch and okay so let's see first as we talked we don't write Ignition config directly we use Fedora CoreS config which is in YAML format and we write that and we let FCCT tool to generate the Ignition config from there so I have this FCC config available okay so you can see it's just a simple one with the password and the username this is the solved password you write it here and the SSH key and I added it in the wheel group so that I have the pseudo access now I have the binary of FCCT tool for the Linux downloaded here so I will use that to convert it into into Ignition config okay so now we see what we have yeah so it basically generates a lot of things since we haven't defined these stuff so it's blank for now the version is 3.0 Ignition here we can see the latest one is 3.0 the password the section which we added is here groups, users, name password has SSH key and storage and system de-units we haven't defined it so it's blank for now but you can definitely add your own system de-unit and other stuff whatever you want to however you want to configure your system so let's run key omu command so this is the simple key omu command we are giving two geeks of ramup and cpu and this is the key omu image this is the latest image which we have from the first of August and yeah the Ignition config you provide it in this way in the key omu so it's booting it will be very quick yeah so the user name was core and the password was test right now it's everything it's getting printed out here so it's fine let's see the first command which we usually run is atm status so that we see where we are okay so here you can see we were talking about a lot of streams so this is what it exactly looks this is the stream name here slash x8664 core os slash testing so this is the exact name of the ref which we use and if you see it's testing and we are not specifying any fedora version here because it's going to be a rolling release and it will be always the same testing which will receive updates every time so in case if you use atomic host or silver blue you see that whenever a new release happen on like fedora 30 you do rpm os2 rebase and the new ref and we don't need to do anything here also yeah also yeah the commit version where we are and the gpg key which we signed and now let's see the python so I will do rpm and qa python there is no python see no python no python 3 yeah I think that's pretty much for demo it's running and now if you want to run something you have podman or docker available so you can do all the container things from here I think Jakub will show the demo for power pc so as corus assembler was mentioned for corus it's extremely easy even if you are not on some more exotic architecture like 64-bit arm or power pc it's extremely easy to hack on building the os itself compared to for regular fedora so go there and check out the corus assembler if you want to try to build your own corus and based on that you basically get the assembler, you build the assembler container and then you run the assembly in the container there are a few stages you fetch all the configuration information that goes into the build, you fetch the packages and then you build the artifacts that's what bits that I did here and I can show you that you currently have the master branch of both of the config and corus assembly you can actually run it on power if we have not working presentation goes on not here but basically you can you can run it even on your you can build and run the images if you for whatever reason want to build it and hack it on, it's really easy and really check it out it's really cool and there is built functionality to the assembler container that you can run the resulting images which I will show you here but unfortunately the network is not here so back to Cine that's bad internet is really flaky maybe if you have a power pc system you can go and try it yourself almost 64 I would be really happy to hear about it how it went we can try it together and the community around federa core is great so if you used to buy on RC or other channels they will help you with any crazy stuff you are doing so the future plans the upcoming things in federa core we have a lot of things already in place but few things we really want to get done which is like the next and the stable stream so right now we have only testing it's also intentional because it's a preview release right so we want people to test and we want to give them feedback file issues or anything and we don't want them to run in production so that's why we don't have stable for now so we would like to get the next and stable stream soon also we have ISO PXC booting running but we want the live ISO PXC as well so that people don't have to really install they can just try out and see how it goes and then they can install it whenever they want also support for other cloud providers there will be a lot of cloud providers and right now we are on AWS so we would like to go to other azure digital ocean many else whoever is interested to want to add Fedora CoreOS on their cloud so integration with the Kubernetes distribution and Okadee things are already in progress for Okadee there are lots of discussion going on so if you want to really get involved and get more information you can follow the Okadee mailing list or the ISE channel and see where we are for Fedora CoreOS on Okadee also support for architectures different architectures are already in going on everyone is so friendly and people are doing the changes and sending patches so more input here would be really welcome also the toolbox so there was a talk about toolbox in another room I think few hours back and that's really nice I have Fedora Silver Blue system and I don't have a toolbox for anything because I say RPMOS tree based immutable host so you really need DNA for anything to do so you need toolbox similarly if you want to do some debugging or anything toolbox is really handy so we would like to get toolbox in place in Fedora CoreOS as well but to get it on Fedora CoreOS we want to I think remove some dependencies which is part of toolbox so far and maybe there are few additional details also documentation it's very important right now we have this new Fedora CoreOS which is like the loss of health new things we don't know even I don't know lots of things still so lots of documentation we have few documentation but more documentation for each tooling and how you want to use what you can do what things can be done it will be really nice to improve documentation there anyone can contribute to documentation if you're doing something and want to add it's really everything on GitHub also preview release we want to continue it for few months and then we would like to get a stable release coming soon so that people can run in the production and that's the summary of the whole talk it's like the Fedora CoreOS itself is a minimal OS with focus for the containers also it provides immutable infrastructure which we talked like we don't want anything to change on the operating system itself if you want something to add any config then you change your FCC config you provision a new host there and that's how you keep your system without any manual changes there also we have multiple streams available in Fedora CoreOS which helps to run things smoothly test coming features in advance catch any regression so that really helps Fedora CoreOS to be a stable operating system focused for containers yeah that's all and if you want to get involved these are few things I'd recommend first try to try Fedora CoreOS preview from the link getFedora.org and if you encounter an issue file it on the GitHub Fedora CoreOS tracker also we have some documentation in place to get started how you can try out Fedora CoreOS so you can get that and yeah we have IHC channel has HashFedora and CoreOS to join mailing list for more updates that's all thank you and there is an interesting talk from Leuka tomorrow at 11 about auto updates design so he will dive into that and that if you are interested in that let us know how Fedora CoreOS went for you on your platform of preference whatever is Intel or whatever through IHC channel or discussion or through mailing list or file issue tracker if you don't say something there we will try to get it in from your support any question who goes first yeah as far as I know there is going to be a separate side release for OKD at least initially so that people can try maybe the first kind of preview release how we have for Fedora CoreOS so there is a different working group set up for OKD on Fedora CoreOS and once it's like it goes a bit stable maybe in future we can have something together in place I can't talk about that much future but that's how it's currently it is yeah definitely yeah yeah Leuka you can take so we want to run everything as a container including the cluster management and stuff itself but the problem is that the agent that Kubernetes uses which is called the Cubelet is not something that can be containerized easily and from that on there are requirements that stems from the Cubelet for example like some specific version of Cubelet can be used on some specific version of Kubernetes and only with some specific version of Docker or Cry or whatever which means that as soon as you fix the version of Kubernetes that you are running you cannot update the rest of the system independently so we have this kind of struggle with we want to provide a minimal OS which out updates to the latest version stable, secure, whatever version of Kubernetes you are running which we don't want to do because it's an application it should be containerized so there is a bit of friction that we need to solve in that regard and OpenShift doesn't have this friction because the Cubelet which is the OpenShift Cubelet and Cry or they are part of the operating system and the operating system version is part of the OpenShift product so everything comes as a single stuff Peter, automation you mean the host automation or something? I think the Cub will be better too for this I can take that one as well you will not like the answer, I can provide three different answers the first one is like you don't because it's an immutable infrastructure so if you are changing something in the OS you redeploy it that's the first answer, the second answer is in fact there are people running Ansible on container Linux in two ways the first one is like trying to containerize Ansible itself and then bind mount stuff into the container if you are just like touching files and the other one is like run Ansible but without depending on libraries that are part of the OS which is in general a better answer because if we update let's say that you depend on some OpenSSL version and we update OpenSSL for you then we break Ansible that you are providing because you depend on a different version of OpenSSL so as soon as you manage to split your application from the OS you can do whatever you want you don't and you just redeploy the node because any way the node could be could go down and be redeployed at any point because AWS decided to kill your node, for example Thanks again I'm taking a lot of answers, sorry about this most of this is what we were doing with container Linux and we just ported it to Fedora Core S I can give like a bit of historical perspective and like from our point of view the server is dumb it's just providing the ignition configuration and it could be just like whatever server you want and then the client is applying that on first boot only which means that you are redeploying the switch from our point of view which could take like a lot of time if your infrastructure is large but it's not putting stress on the server itself it's putting stress on specific portion of your network and the other kind of like approach that we have is that we define, we build our tools so that you can define maintenance window per cluster and so you can say, for example you can say I want exactly these four nodes to be in the same group for out update and only be rebooting two of them at a time because I want to have some kind of like HA continuously and if something goes wrong we stop the roll out of up update so this kind of like approach to you define how you want to manage your updates and stuff but from the point of view of the infrastructure like the nodes themselves they are really cut off they could disappear at any point it could be just be redeployed from scratch Any more question? So the package name is Moby Half an Engine, it's not Docker Docker is a different package Moby Half an Engine, yeah I believe there are some I believe even like VMs and bare metal hosts but I have not located direction yet, fully and some smaller ones like that's more one or two but I cannot remember from top of my head and I believe of course IBM Yeah, so you can see like I'm in the system and if I do which Docker it's Moby Half an Engine package. Any more question? Thank you guys, do try Fedora Chorus and let us know how it goes for you