 Hello, good. Well, it's not morning anymore, but good, exactly good noon. To my talk, Fedora Chora is an action in the context of operating system for virtual appliances. And when I started to think about giving a talk at DefConf in general, I thought, well, Fedora Chora is surely I will be the first one. Turns out, absolutely no. If you just look at the talks given at DefConf about Chora in general, there are a lot of talks. And so a lot of topics were already covered. Surprisingly enough, when I looked at the talks or skipped through them one by one, I saw that not a lot of practice experience was already shared. Only for example, 2021 up and running with Fedora Chora S or getting started with Fedora Chora S are really intended to give you an introduction to Fedora Chora S. A lot of the other ones are just deep dives into certain features or certain aspects or yada, yada, yada of Fedora Chora S. So that was a bit surprising. So I thought, well, that's a good gap for me, because I have a little bit of a particular experience I can give you about Fedora Chora S. And this is what we are talking about today. So what do I mean with inaction? Because that could mean anything. This is the concluded experience from mainly me, but also some colleagues in current company or in former companies. And when I started working with Fedora Chora S, we basically then started from scratch. So there wasn't really anything. Also means the last point, limited experience. We also had a scratch experience. So Fedora Chora S at that point was a new word for me. And also virtual appliances were a new word for me. And in concluded now, it's about one year, maybe a little bit more of experience developing and supporting a virtual appliance or other use cases on Fedora Chora S. So I have some things and I matured in a way, but at that point we will come. Good table of content. I will give a very quick basic introduction to what a virtual appliance is, what a Fedora Chora OS operating system is. If this will not be enough information to you to fully understand the whole slides, well, good thing. Hopefully there will be a YouTube video of this. If nothing goes wrong. And so we just recheck the links that I put into the slides and then just look at the talk again. Then we will come to, okay, I don't understand what a virtual appliance is, what a Fedora Chora is meant to be. How do we actually get started developing? Again, here we will provide some links for further reading and testing and tutorializing. And once you are over the proof-of-concept phase, now it gets into the really interesting problems. How do you actually get production ready? How do you make now this proof-of-concept into a real product that your customer can actually use? And after that, my personal experience or more personal experience, especially points that I think I did not use as intended. Then, of course, the normal last chapter's conclusion questions, thanks and the appendings if we have the time. If not, we will have to look in the slides. Also, note that please put your answer questions directly into the chat so they are not forgotten, would be a shame. And then I will go over it at the end. And furthermore, as mentioned already, I will have links all across the slides. And they have cryptic numbers just on them, like you see in scientific writing. So, be sure to access the slides as a PDF in the schedule thingy website where the slides are uploaded so you can actually access them for further reading. Good. And I am trying out a lot of stuff today with this presentation. So please give me feedback with anything that occurs to you as feedback worthy. Good. So, let's start finally with the talk itself, the basics. So, what is Fedora-Cora? Fedora-Cora, at first, is a quite minimalistic operating system that has one major focus or one of the major focus is container execution. So, this, for example, means you do not have to think about which container execution, how it's called, container runtime, do I need, which one do I should I install, which one can I install, which actually is supported with Fedora-Cora. You just get Podman out of the box and it runs really well. And it's also really well supported. I never had problems with Podman on it. Another interesting point is Fedora-Cora is not like your usual Ubuntu Manjaro Arch Linux image. So, when you first boot Fedora-Cora, you will see that there's no such thing as an install script, as an install graphical user interface, nothing, it just boots up. Basically, what you have for Fedora-Cora is always a disk, more or less, with the full operating system already on it. So, the startup is also pretty easy in the first setup. Beyond that, we have that first setup, but that is just more or less a skeleton. You fill that skeleton with something called ignition. And this ignition uses, if I'm not wrong, cloud init to do certain stuff in the operating system during boot. For example, creating system d-units, creating files, creating folders, whatever you like, has a lot of things you can do. And also a note, this is mainly where I developed Fedora-Cora as I did not touch Fedora-Cora as itself. I mainly interacted with the ignition and which files were put where and scripts and what, yada, yada, yada. Good. And also very interesting about Fedora-Cora, which will get important later also, is some parts of Fedora-Cora are immutable. There's also a quite interesting block entry. I think it's linked number two. But also I have a look at that because often it's referenced as an immutable operating system, which it is not. Otherwise, you cannot write anything anywhere. You can write to certain folders, so it's just partly immutable. But even though just some parts are immutable, that can greatly help later on. And then what is appliance or virtual appliance? Appliance is a package that contains all your software in one package. Everything is included to run it. So the virtual appliance is just the same but with the operating system. So basically it's a virtual machine. But your application is on their default configuration, maybe default resource files, whatever is needed, that your application as a default instance of what you want to deliver your customer is already there and just needs a hypervisor or bare metal and then go. So that's another, not bare metal type of visor in that case. So that's a virtual appliance. And now why should you choose exactly this combination? Because you can deliver virtual appliance more or less with any operating system, as I said, with Ubuntu, with Arch Linux, whatever you like. I think Fedora-Cora is a good match for the reasons just set with the minimalistic idea of the operating system. But also it's quite easy to maintain for certain reasons. And by that, I mean, it has a feature that automatically applies updates. And they are very careful about applying updates in such a manner that it does not break any system. So it's just some update and well, if it breaks something at the operating system, we don't care. It's carefully done. Of course, there will be breaking changes, but that's another matter. And also it's just another way to ship your application, especially is I have the feeling that software engineers in my generation of software engineers always think about containers and there are only containers. But of course, it's not only containers. You can ship your application if it's a Java application, just as a jar. That fits the exact use case you have and the customers you have and compliance and whatever controls. Maybe that's a good solution. Maybe also the virtual appliance is a good solution to also deliver a virtual machine with it. It could be a good fit. And of course, there are official use cases and also what I like, anti-use cases for Fedora-Cora, so that's pretty nice. And I see that I'm talking too much about the basics. So I just make it quick with the next point. Why did I choose it? Well, I did not have a big choice. It was a legacy when I started working with Fedora-Cora as in virtual appliances. There was already a proof of concept, so that's it. And currently, I'm working with Fedora-Cora just because I now am used to it and I enjoy it very much. Good. How do you start developing? That's a very short slide. At the beginning, initially, I thought, well, I want to also show you or present you or at least give you some scripts due to time limitation that was not possible. If you want scripts, I want to see how I would approach a script, a build script, for example. Let me know and I will create one. Besides that, have a look at the linked resources, because they spend their main point of the video or the talk is to give you an introduction. So this is way more than I could give you now in these 20 minutes. And also, besides that, I strongly recommend VirtualBox, but that's just a personal preference. VirtualBox just always worked on any platform, so on Linux machines and Mac machines and one or two times also Windows machines. VirtualBox just worked for this combination of Fedora-Cora as in virtual appliances. I also tried Kimu, for example, on Mac in preparation of this talk and somehow I wasn't able to fully get it to work. Not sure why. And also VMware, for example, needs another step until you can import a virtual appliance. Yeah, so that's it. So, okay, now to get to the important part. So when you are thinking and you have your proof of concept already there and it works to some degree pretty good and you think, well, we shall continue with that one. And then there's a question, okay, now how do we get this to production? How do we make, how do we actually solve the problem of our customer? And you may have certain constraints while thinking about that. One of the more important constraints is, will your virtual appliance be online? So has it a connection to the internet? And can it also be connected from the internet? Or not? Because if you're thinking about, well, we have upgrades and, I don't know, we just put some upgrade files for our virtual appliance somewhere central, like Artifactory or something, with a public IP and that's fine. If some of your customers only want to apply or run your virtual appliance in a complete offline environment, what do you got to do? So you also have to think about how to cover these cases. And also then when we're already talking about updating, oh, by the way, I'm using upgrading and updating interchangeably. So the update path is the same as the Fedora CoreOS, I think more or less. At some point, you always have breaking changes. That's at some points in the development that just, I think, what it sometimes happened. And then to not break the systems of your customers, you need to follow a certain update path. So for this version and this version, and then they're fine to do whatever they want. But they have to follow this path first. Otherwise, something goes very wrong. And you get a call at 3am, which nobody wants. And so that's also a thing. Do you really, do you want to maybe to prepare these kind of things? So for example, implement a logic that the user calls to update the virtual appliance. And then in this logic, there's also a check, okay, which version am I am a certain update path or not with the version that would now be the one that updating to, would that leave the path but not. And then you could message the customer or the user that's currently updating, well, do not update to this version because reasons. So yeah, and another problem is limited access to the virtual appliances. As I said, maybe he or she, the admin of the customer, only wants in an offline environment, the virtual appliance. So if something goes wrong, and they need support, what now you cannot access. It's because maybe they do not have a how it's called a jump host of some sorts. So you cannot access your product, you're supposed to debug what you're gonna do. So that's also another thing that you have to think about. Also, what this also implies is, if you need to be, what I also mentioned, if you need to do an update, it cannot be you who does the update because you cannot access this. So the user must do the update, he must interact with the machine. And that can need to just usability questions, but we come to that now as a good transition to the second point of customer expectations. That's okay, the user has now to interface with the virtual machine and have to do stuff. So why not make it usable for him? Why not make it nice, make it easy. So what could be an example is a script or some program you created that offers the normal things you have to do on the virtual appliance, like an update, or maybe also something very easy like a restart. So they don't have to type in sudo, what is it, shutdown now or reboot. So just make it a little bit nicer to him. Of course, this is then maybe not the highest priority of things you should take care about, but also certain things. And also with virtual appliances, like for any project, you should always think about guaranteeing continuity, not in terms of availability of your application, but rather, if something goes wrong, can their customer in context of the contract we have with them, can he contact someone of tech? If someone of the team gets sick, can someone take over, especially if some critical bug is occurring, and so back and so forth, or the all well known truck factor or lottery factor, bus factor, if someone gets hit by a truck. Next day, you lost your virtual appliance guy, what now? So of course, as in any other project, you need documentation, you need testing, basic, and more people need to know about the stuff, but that should be quite the basics. Yeah, and support considerations, that's what I meant. If it's completely locked and you have limited access in terms of maybe no access, then you cannot get firsthand access about bugs at the customer. So you cannot directly access a virtual appliance and see, is this file still there? What does this file contain? What version is blah blah running? You have to think about what can your customer do to send you certain parts of the virtual appliance, so you can then debug these parts. So what you could do, for example, you can make like a zip file or tar file or whatever that contains specific configuration files or folders or elements of files that are now then packed together, and then this file is then somewhere on a well known path on the virtual appliance, or maybe even retrievable via HTTPS. I'm sorry, Michael, to interrupt you, but we hit 20 minutes mark, so we should slowly go to Q&A. So please finish the idea. Good, then I'll talk way too much. So then to finish this very important point is here comes the prose of the immutable part of your system, because if the virtual appliance at the customer is the same version as you're in a development machine, then this means that it looks the same. These parts of the system look exactly the same. You don't need to debug those. What you need to debug is immutable parts, and that's exactly what this package then would be for to debug what they changed. Besides that, good points. One thing still to mention before the Q&A is the ignition file, because we are righted horribly wrong. Ignition is meant to be used once at the very first boot of the virtual appliance. That's the idea. What I did is executed every boot. So this is absolutely not how it's meant to be used, but on the other hand, you know that certain files and folders after restart look a certain way. So you more or less extend the term immutable is not really right here, but you know how certain things look like after restart. So this extends the immutable part of the system. So that's actually in this case, we had I had like it worked out well. Good. And yeah, said have to skip these things. Yeah, okay, we can come to the Q&A. Thank you for the interesting talk, Michael. So it seems there is one question for now from Clement Verna. Do you have virtual appliances using Fedora CoreOS testing and the next stream to test early the OS updates? Oh, yeah, good point. I also saw that there's a CoreOS engineer, at least it was an attendees list. So I was afraid of tricky questions. And this is a good question, because when I remember correctly, Fedora CoreOS community as explicitly tells you, not tell you, but want you to also use the testing in extremes to report bugs. I guess as a way of paying them back or contributes also to the project, which I think is very fair. But I did not do it because I was not really familiar with all the setup. And I just read about that in the last two months. And we're thinking of ways to implement that currently. Thank you for the answer. There is another question from Andreas. Is there a documentation or how to deploy a micro shift on Fedora CoreOS? Oh, good point. Because OpenShift is one of the main use cases for Fedora CoreOS. But sadly, I do not have experience with OpenShift or I guess MicroShift is a smaller version of OpenShift. Sadly, no experience with that. But also have to Google, like anyone else.