 Good morning, everybody And if the video is all good stop me Good morning, everybody. This is my First that come my name is cosimo shakey and I'm going to be talking to you about something that we did with that end of the company that I work at endless we built a Debian derivative Different that in distribution. They use some technologies that Are typically not exactly what you think when you think of that name Lee OS 3 and flat 5 So Alright, so a little bit about me You can find me on Twitter or other places with the cosimo see handle I am a long time GNOME contributor and no aficionado So I've been maintaining a lot of modules to know the model of the file manager a whole bunch of other things now I don't do that as much anymore, but I serve on the GNOME board of directors and For any no related inquiries you can find me impressive know the org and my day job is managing the engineering team at endless that built this operating system so Basically three things that I would like you to get at the end of the stock First of all what endless OS is if you don't know what it is So hopefully you'll know that at the end of it how to deploy Well, first of all, what is our street and how to deploy that end that end file system tree using go Street? And then finally I have a super modest straw man for how that mean can adopt or street a little bit more and Why that would be useful to a bunch of people So but before everything a little bit about endless and a little bit about endless OS I want to talk just for a few moments about the company. This is, you know, not a talk about endless at all But endless was founded for five five years ago now to bring technology To people that don't have it so to kind of the next billion computer users So we target really people like this, you know, maybe in people in Indonesia people that are first-time computer buyer and You know, this is a slide that my My our CEO always uses this is you know, and that it looks a bit photoshop But it's an ad that we actually find in San Francisco So, you know, it's an ad by Mozilla. The internet should be accessible to all what are the you know in 2017 what are the four really Fundamental needs of human beings food water shelter and the internet and you know, Mozilla our friends and Mozilla keep it fair and open So we'll love them for that But the reality is that the internet is just not a thing for most people in the world actually for about half of the world population And you know mobile internet and mobile penetration has helped a lot With getting some data to places that previously didn't have it However, there is a big disparity between what? somebody can afford In terms of an internet plan on their phone 300 megabytes. I think it's the average data plan size in emerging countries and 60 gigabytes, which you know 300 megabytes versus 60 gigabytes doesn't even show on the graph, right? It's actually how much data a typical desktop laptop Computer consumes so endless, you know, the product that we build wants to solve this problem Solving the availability of internet and effectively this is a very very simplified, you know Diagram what we do, but we preload a lot of content on the device itself when you buy it Windows computer or a you know Chromebook or whatever other computer most of the things that you have on the computer will need internet So it's pretty much useless. Actually, I went to a shop a Few few weeks ago and tried a brand new Windows 10 That laptop the only application that I was able to run was camera I think even solitaire in Windows 10 requires an internet connection, which is insane So what we do is we preload a lot of content on the device itself We refresh this content whenever internet is available or really whenever anything that can refresh the content is available So we can refresh content using a USB key or using, you know Other computers in your local network if they have content that we are interested in and then all this content is accessed through Native applications that we ship on the device itself. So Yeah, it's This is another way of looking at it. Everything is searchable. It is very very intuitive like for people that Have not typically used a computer before but maybe are used to a tablet or a smartphone This will look very familiar apps are front and center And you know the way that you install or remove applications is through an application store that kind of takes care of everything So, you know, we are really aiming not at obviously You guys, you know, this is this population of very experts Distributors and you know operating system engineers But we have tested this user interface and we built it over, you know A few years of research on the field and we found that it behaves very well for people that are used to handheld device which is You know primarily It's not Windows anymore. The you know the contender for for our software, it's actually Android And what we started doing at endless very recently is we started shipping with these Two big names in many regions of the world. So you can buy right now in Indonesia or in Thailand or in Vietnam An endless well an ASOS laptop or an ISR laptop that has endless preloaded So, you know, this is kind of to give you The context of the users that we try to address and why we have made certain decisions when architecting the operating system So as I mentioned, this is an operating system based on GNOME so the first definition you can have for endless it is a Uses a fairly stock GNOME 3.22 base. It has a custom shell and other pieces of Of interface on top such as for instance a very simplified initial setup for people that You know don't want to spend too much time configuring their computer when they first get it kind of works out of the box As I said before GNOME software so an application center kind of experience is the the main way that you get other software That's not on the device or the main way that you manage the software on the device and You know the technology stuff is just not very interesting because it's the same thing that a modern Deviant or any other distribution really uses the system the all the various use your power you this Luzi bulls audio a la that kind of stuff Endless is also the Linux distribution based on that. Yeah so our Base OS is that the end Jesse for the person that we have released right now. We are in the process of updating to stretch Right now in for our next release and Of course, we don't take Everything stock from that yet. We have our custom set of packages that we overlay on top of that in itself And those are managed with an OBS instance. So we use the I think it's open build server It's a it's an open Susan project that also works very well with that packages The kernels right now is based on the moon tube This was a decision that we made a while ago and it's kind of hard to change once you have it But it also has a Lot more support for more, you know other kind of hardware So we don't mind that and we are available right now for x86 before and also for our V7 Because one of the first I didn't mention it, but we used to make Hardware devices and one of them we still do is Based on am logic s805. So we have a whole version of the distribution that just you know built for army 7 and Right now we have a predictable release cycle. This is you know kind of a difference with that in so we follow a little more or less what You know other distributions do we try to align with upstreams such as you know So we have a major version every six months that you know Rebases the whole staff on top of the version of no and every month We make a minor release that is just bug fixes or you know other minor features and You know we I don't go into that in more detail if you're interested later, but it's really harder to get it less than a month Sorry to more than a month. So Many times we you know our hand is forced to make releases very often because of hardware support On the other hand so this feels pretty standard so far, but it's not a traditional distribution because the I think the main difference with what I consider a traditional distribution is that Packages are not exposed to the end user at all. There is no package manager available If you try to use up get it will say no, you cannot do that if you try to use the package You can you know Search for things because we ship the package database where you can only store anything. You're like, but that's kind of it's kind of weird So the US is really a single deliverable It is a set of bits kind of how it is on Android or iOS and you take it as it is it goes forward forever and You cannot change it So you do not as a user have fine control over what's installed in the operating system You can install applications. You can remove applications. You can install maybe other small things like printer drivers or codecs But you do not control the operating system itself as a user big difference And you know as a corollary of that OS and applications are separate so the operating system You know, I'll go through that a Little bit later, but the applications are managed to flat back and the operating system is managed with OS 3 these two things have Really no relation to each other in a way. They're just different layers So the OS can update independently of the applications independently of the runtimes that the applications use And almost every bit is delivered like that. So there are a few exceptions as I mentioned printer drivers But pretty much So, so how do we do that? How do we do it? And Mostly why did we choose first of all why do we choose to do that? That's a pretty, you know, different decision that what most Desktop distributions do and the first reason is that Packages are hard to test all the configurations That a given system can be especially given the number of packages that are available in the distributions like that Yeah, makes it effectively impossible to test all of them especially with a small QAT so You know the first reason is kind of is to test or we kind of you know Decided to favor the quality of the bits that we're shipping over the number of bits The second reason is that we really want to deliver the same bits to every user So we do not want to you know think oh, do you have This version of this package this other version of this patch did something happen before did you upgrade from you know an old version and the migration script didn't work This you know if you if your requirement is you actually have to deliver always the same bits to everybody These are not problems. You can you know First compose your system on our end and then deliver it to everybody in the same way You're insured that everybody gets exactly the same thing that you have passed it Also upgrades can break easily. I mean I can I'm sure that at least some of you here Have had my same experience where you know you install a new version of a package It just doesn't work in breaks. You have to resort to the terminal You have to do some magic you have to remove go back that happens. I mean it's it's fine if you are a Somebody like me or somebody like, you know probably most of you in the audience But it's really not good if you are the person that I showed at the beginning in Indonesia Our loss is also a real issue Right that's we deploy computers in situations that don't have great power supply and you know, maybe power is intermittent We really wanted a system where we can deliver bits we can deliver updates that is very resilient very resistant to things just exploding at any time and Packages are really not atomic in a way So, you know, you have you install it Maybe the power goes out during your the middle of your like post install script or something and who knows What state it is at that point It's also very easy to show yourself in In the feet like if you remove the wrong thing if you find the wrong command you're kind of screwed You know, you need to know what you're doing And mostly just an unfamiliar concept for new users So, you know There are many cases what many cases the one can make why packages are a great idea for these reasons we chose Not to use them for for this particular deployment What we chose instead is OS tree so I took this kind of from the website, so it's a Effectively, it's a gift-like model for binaries like that. I think that's that's the easiest way to describe it So you take a whole file system tree big binary blocks You commit it to a repository repository can have many branches can have many, you know remotes and things like that and then you download those bits again on the client side and this happens so You know, not a chance that it's part of project atomic, which is a red hat Fedora base way of using this That's kind of where it originated Atomic means that you are Guaranteed by the system that you are either at any given time You are either fully in the old version of the system or fully the new version of the system There are no in-between so you can go forward you can roll back At any point you are completely the old one or completely the new one Which you know fits exactly the kind of requirements that we wanted The data itself is it also has you know a way of It can be distributed pretty simply so All you need is an HTTPS server. It has kind of a gift-like object Repository behind the scenes so if you look at it, you know The files are not stored by their file system But they are kind of content hashed in that content address in this repository And that's what you serve to your clients And each kind of commit that you make to this binary tree It can be GPG sign which can also of course be verified once you download it So you are sure that you get the same bits You know as the vendor intended Something that can kind of comes from what I mentioned before about the object model repository Is that the data is de-douplicated so each file You know traditionally if the same file occurs to packages maybe in different directories That really occupies twice the space on this Here it's de-douplicated so Being content address means that there's only one copy of it and there are hard links from that copy to The various actual locations on the file in the file system tree once it's deployed That also means that you can very cheaply store different versions of the same tree right because You know you change a package Maybe you change a program in there. You won't change the whole file system You can make effectively two versions of it that whose only difference is this new file or this new set of files All the other files will not be Stored twice on this even though you can choose to boot in one or the other so it's very powerful And another way of delivering updates is doing deltas between commit This is supported as a native operation, you know a street So instead of delivering the files individually you can deliver a blog in a single request Which is a lot more efficient. That's also compressed And it's just the binary differences between the old commit and then you commit to your point of the blog So that's really good It supports integration with bootloaders such as grab a new boot, which is perfect for us because you support R and x64 And finally it's built in a way that makes it easy to integrate with the operating system So, you know, I really like this project. Yes, you check it out A flatback, I won't talk a lot about flatback because Simon has a talk about flatback right after this Should go to his talk, but I just want to mention it here because it's what we Used to deploy our applications. So You can build Basically flatback is a building and distribution system for apps on Linux Similar to snappy similar to up image. It's that kind of, you know Mindset Each application is a bundle that's independent from all the other bundles. You can install them remove them kind of independently And it's you know, it supports a very secure model of Application confinement through the use of technologies such as namespaces and cgroups in the kernel that really you know Give the developer of the app very fine brain control Over what the application can or cannot do So I was making you know, like he's before so case before as to why not use packages for our distro but Everybody uses packages when they're building a distro and for good reasons because they're a very established concept There are very high quality teams that maintain these packages typically including most of it here And thank you for that There are lots of tooling around how to build a package how to deploy a package how to install and install There's tons of documentation. It's really the perfect system to build a distro in my mind You know, there's others that have tried doing, you know Yonkto style and stuff Jason stuff But I mean this is this works very very well before it for two is building a distro So we really, you know our goal when thinking about this tool was really to bridge this to work these two worlds We would like to deploy packages Demian style to a file system tree and then use that as an input to create an OS tree to deliver that for users It has to be an automatable operation So we don't want to be typing commands every time that we want to do it. It has to be reproducible, you know, this is kind of a loaded term, but If you run it twice, it has to, you know work every time effectively and ideally not add any complications By the process itself. I know that not all the packages are sort of perfectly reproducible So if you are pack a package twice, maybe you'll get slightly different output because of timestamp. So why not? But within those bounds, it should be reproducible. It should work with existing packages, like we're saying And it should support Customizations outside of the package system itself And you'll see why And this is the project that we've built to To do that so a little bit of history to this this is We use it mostly as an internal tool. This is a Public version of the tool that I that a colleague of mine made last year I updated it for this conference. So you, you know, please try it. Please download it. Check it out. However, it is Kind of untested in this specific configuration because we have our own hooks and our own, you know Internal add-ons to that. So I haven't Tried it fully, but I think it's it's a it's a good start So That was rebuilder. What does it do? It's designed in Stages and it's mostly like really four stages that you need you need to do to Deploy the system. The first one is a check for updates So should any OS3 be built given the set of packages that I am giving you right now If so, let's assume that the answer is yes The first time will be as for sure. There's the OS stage which creates using the bootstrap And then basically taking a meta package, which is a list of dependencies just simply a list of dependencies That should go in the file system. It deploys everything So it makes a demo strap into a temporary directory that installs all these other packages into a temporary directory as well That effectively will have the fully deployed file system tree as if it was, you know, a regular distro takes that creates a commit to a local OS3 repository during the OS3 stage and then signs it with the GPG header you give it And then finally it publishes that OS3 to another server or to a set of different servers and reality the publish stage like in at least in the copy that is there is Basically empty like up to you where do you want to publish it? If you don't want to publish it anywhere You have the local copy and that's fine. And you know first you have an error stage if something was wrong maybe you want to put you know a notification to your Jenkins to your github to your, you know, whatever CI you think you have to notify that something went wrong So each stage here can be customized there is a core logic to the stage So, you know how you deploy the deep-boost how do you do the deep-boost trap how you deploy the meta package? Those are core logic and then there's a number of hooks that you can you you know distribution builder can add Each stage so things like I want to publish to this other server I want you know to add these new files that are custom or I want to I don't know Yeah, like maybe change a configuration in a specific environment or things like that the configuration of how all these things are put together is separate in the food base and You can you know, it has usable defaults, but you can customize it per product which is Effectively the kind of the top-level configuration of where you're building like I could build a Demian stretch or a sweet product and a Debbie and sit or a street Those would be likely to different products because they have different packages a branch Which is you know different versions of the same per architecture per platform platform is kind of a weird concept But I will go into it and each of these things can be customized. So it's very very flexible But kind of useful from the guy go Just as a as an aside Making these trees has a cost so especially maintaining them has a cost So, you know one could be inclined looking at this thing. Oh, I'm gonna make you know a hundred different versions of OS 3 Well Making an OS 3 is not exactly like maintaining a distro, but it's similar because you you know You will end up being you end up facing the same problems They are trying to solve as in to many configurations to test combinatorial explosions and stuff like that So we have limited, you know kind of intentionally Trying to limit ourselves to just making OS 3 is when needed. So per architecture per product or platform and next hardware is An OS 3 that we make especially for people that have bleeding edge hardware. So it just has like a different kernel or something like that And people can opt into it if it doesn't work Doesn't work with the regular one So again very simple the Jenkins CI runner Calls this demo S3 builder invokes this on a timer for instance every night if you want to build You know a snapshot of your Distribution every night. It pulls the packages from OBS builds it Commits it to the OS 3 server. This can you know be run manually on a timer however you want it What happens after that? The In our deployment, you know this this you can do it pretty much however you want it in our deployment. We have a three stage Server policy basically when you build an OS 3 first published to a staging server that is internal So we use that to do our QA we push from the staging to a tree that we call demo Which is public beta kind of whenever bi-weekly or roughly bi-weekly Then I really at release time when you know the beta passes the public QA the staging passes internal QA We push everything to production and this happens again once a month for minor and once every six months for Major releases updates are installed automatically on plane machine So we feel so confident and this one break that we just install it on users machines Provided that they are not connected with expensive connections or mobile kind of internet because you know that would upset people because it costs their data But if you're connected to like fire to internet We just downloaded and install it and then we ask you to reboot whenever you like to Get into the new version of the OS and of course you can manage to install it So this is how it works on the client. You have this component US of data who is responsible to kind of like Paul and You know fetch the latest data from the OS 3 server and then again You are either in the new deployment if everything goes well, or you're staying the old there's never an in-between broken situation and You know it actually never happened so far that you would that That this didn't work so it's very reliable However sometimes you really need packages and this is you know one of the one of the things that I will discuss at the end To develop the distribution we sweet, you know endless developers. We still use packages internally. So we have kind of you know a few hacks that We built to make this possible. One is US convert system, which basically takes everything that I've told you about OS 3 trashes it completely and restores the whole normal package system. So you can use apt There's voice 3 admin unlock, which is a really interesting thing. It makes your user Temporarily read write and Overlays some file system on top of it so that you can actually solve things temporarily until you reboot and I think you can even pass it a Flat to keep it after the reboot But anyway, it's still a bit like what am I doing here? We're lay fs Sure, I feel about it. And right now this is strictly for internal use So we don't advertise this. I mean, of course like we ship this command. So if you find it on an endless system, you know Good job. We don't we don't tell you to use it So like this doesn't really feel like the right solution to me I'm not gonna talk about flatback But the one thing I want to mention is that we try to build flatbacks using a very similar kind of death-based builder It didn't go very well. It's it's complicated Yeah, if you're interested later, we have time in the Q&A So how can we prove this? I think you know before Before I go to that, I have I have a simple idea, but Some of the lessons that we have learned from doing this, right? One is that Software can only really be tested as a whole like many times we have, you know tried to Change the way we test things to just test the individual change the individual package things like that It never really worked like there are many cases where the integration is really the difficult part And testing as a whole has a ton of value Terminals are a nightmare for users. So anything that requires Going to the terminal basically for the kind of users that we're trying to target We lost them already as smartphone OS is the both standard is what people compare the software that you put in in front of them too because you know, we grew up with Desktops or laptops or whatnot Many people in other areas of the world. They just grew up with Phones and tablets and that's that's the digital experience that they have and that's what they're compared to Another lesson the coupling deliverables has a ton of value for us as well Like the moment that we were able to say, you know, all the applications don't need to follow the OS We don't need to keep them up to date and working with the libraries that are provided by the operating system They can move at their own pace. They can be released whenever And you know and you ask and evolve this other way once you have a figure out the interfaces They need to do that. That is absolutely super valuable and I think it would be valuable for You know for for other distributions as well Predictable quick release cycles, especially if you work with hardware manufacturers are Very very important. So, you know for us the ability to you know, make releases of the operating system, especially very quickly has Has a lot of value and working with hardware is hard, you know, not a It's it's not just fun. It's true there are many requirements that come from working with hardware vendors and Yeah, there's always something else they haven't thought of so it's it's quite it's quite a challenge so In any event they follow this what I want to say with a grain of salt I don't know really how Debbie and works as I mentioned This is my first step comp and I'm not a devian developer. So, you know, I don't know how things work but for me the Main issue with the setup At least for how I see with the set up that I've described if I Seriously wanted to propose it to the devian audience is that you cannot install packages like, you know, all of us All of you will want to install packages there This is actually a problem that's solved on Fedora. So Fedora another working on a system called Fedora atomic Which tries to bring some of this technology into the main line of the distribution and they've developed this technological RPMOS 3 that allows you to layer packages on top of the Distro itself so you have some tooling to rebase the package on top of the tree whenever you update the tree Or upgrade the package together with the tree That's all, you know fairly transparent and mostly it doesn't Lose some of the guarantees and nice things that OS 3 gives you for instance that updates are still guaranteed to be atomic So I I don't know it feels like I'm not an expert in The package or that format, but this feels like it should be something as she will Especially if you know if he was done with RBM on Fedora And I think that with this, you know, if you had personally personally if I had something like Like this where you know where the system is managed by OS 3 I have something like that voice tree to allow installing your packages on top when I need it I don't want my desktop system to be managed like that, you know now. I'm running I'm running Fedora on this because I needed virtual machines and we don't have a virtual machine out on endless But when I used to run endless on it, it was amazing because I didn't need to care about anything like, you know The OS always gets updated. It's tested Many people used exactly the same thing I'm using so if I have a problem for sure others will if I don't other people won't Like, you know, it's very nice. It's hard to go back to Managing your own packages up to that. So I would love if that didn't create an OS 3 version of the base desktop system basically, you know Very similar meta package definition to the one that we have at the end of the day unless it's a desktop as well And you know that it could build a reference OS 3 for Base desktop system just like that. There could even be different meta packages for different desktops. That's what people want And I don't know if that has a concept of package groups This is something that they have of Fedora where, you know, these are there are already these kind of meta packages They install and it grabs all the dependencies like, you know, you do install Katie Just has everything you don't need to know all the defenses manually Also, if I know that that means inside them and there is a project for There is a project for reproducible builds And with that you could really help share the bottom layers, right? Because whenever, you know, you Yeah, basically all this different desktop configurations could use the data application technology of OS 3 and share the same data And users could even very easily switch between different Trees without the complication of oh, how does the KDE package interact with the GNOME package whenever I have both of them installed Which always opens like huge discussions and I've, you know, I've seen it from both the perspective of an upstream developer and this redeveloper And you know, if if deviant did this Then there's a good chance that how to build and deploy this deviant trees pushed to to an OS 3 Would become a standard and not something that, you know, we have made on that endless And for us or somebody else building on top of that yet, there would be many advantages, for instance, you know I mean, we wouldn't need to do most of the things that we do But we would just host our binary builds and adding our packages on top of, you know, a base that is shared with the rest of that Yeah, so we would actually do most of the package development work Directly that yeah so few conclusions, um, I think, you know the The world of OS trees Is very inflexible the word of distributions is very flexible, but I think that we can We can find a good middle ground Where we get the best of both worlds. So I think that we can learn Something from each other and teach something to each other Right now integrators and their users would be those who would benefit the most From, you know, a tree-based approach writing Debian But I think that with something else on top like demo S3 This has a lot of potential to cater to Debian users as well And you know distributing OS 3 binaries can be achieved quite easily like the on the server side You know the infrastructure required for that is not really a big deal and I'm sure that, you know, Debian has A lot more complicated things that That occupy the infrastructure And, you know, if there is interest for a project like this in the community We are Here, you know, we as in as an endless we're here to help like we would like to get involved in a project like that If there is an interest So So that's it. I like your overall approach One thing that I'd like to see or hear from you is I've I've read a lot of what seemed to me like overheated advocacy in favor of Snappy flat pack things like this It sounds to me like you've identified pretty much three different types Or three different approaches to system integration the package model the os3 model and let's call it the flat pack model And I think your presentation goes a long way towards Shedding light on this issue because it's not like well one of these systems is just going to kill the other two off So could you distinguish in particular? I know you wanted to Leave some space for Simon's talk on flat pack, but uh, you very nicely contrasted Where os3 is useful versus where packages are useful Could you shed some light on where you think the flat pack app distribution model Has advantages and disadvantages relative to the package Orientation because that seems to me like the the real point of friction right now in the wider community Yeah, um, and you know, this is my I don't I'm wearing a flat back t-shirt, but this is like very soft I think Flat back Basically right now if you want to develop an application you want to distribute it to as many users as you can There's really no way around it No way around you going to Every distribution and convincing them that they have to distribute your application So either they will do it because they like the application or you have to kind of advocate for your app to go there That's actually not how it works on pretty much any other platform that I know like usually people, you know, there are um You know applications just exist somewhere and they either exist in a store or they exist on a website and people download them from that Um Flatback makes both of these things possible in a way that is not dependent on any distribution So that is a big deal because uh, you know, it hasn't It hasn't been done before and I think it's something that in general would You know incentivize the production and distribution of applications on linux It doesn't do that for free though It does that by taking away some of the things that distributions are really Good at and then right now they're in charge of for their applications such as You know, uh making sure that security updates are always there being in control of the binaries that are distributed Being in control of the version of the library that you know Something links to making sure that there's nothing application of libraries between different applications that may require It's a trade-off Personally, I am, you know amazed and kind of mind blown by the fact that devian can Manage such a large collection of packages and always rebase all the applications that they have to work with all these other libraries um But you know, it comes at a huge cost and I personally think that you know, uh if if if those efforts were focused on something like You know making these flatbacks, for instance the whole community would benefit And not just the people they use them because it would work across across different distributions So, you know, it depends really in my mind on how much value you put on the things that I that I mentioned Um, and how it gets updated in the field, but you'd sort of left out the middle bit or how that gets installed on the End user machine in the first place. What are you using for install technology? Oh, yeah. Uh, good question I didn't I didn't talk about that. Um, we Take us we we make images. So that's um, how you know, if you go through the endless website, you can download an iso image and install it We have two installers effectively one for windows And one for linux. So the one for linux is super super simple. We don't actually Uh, do any Uh, you know Dual boot or partitioning or anything like that. We just basically take the file system Overwrite your whole drive that you decide to to use endless with And that's mostly because like, you know, a lot of endless users Won't come from linux or if they come from linux, they Probably don't care about keeping both the uh windows in store is actually a lot more interesting Because it doesn't touch the partition scheme of your hard drive like differently from many other windows installers in the past It instead installs endless in a single file in a windows directory and then it has a driver in the Basically indeed in rd on the other side to create a device mapper device inside the You know inside the system when you boot it that knows where that file is inside the efts file system that you have on windows So, uh, it basically maps all the extents of the file and which creates the device And by doing that you can install or uninstall and then it adds like a little eft file order thing to the window folder for dual boot And by doing that you can basically install or uninstall Endless as if it was an app or it was like you install it You can boot into it if you want if you don't like it you you can Regarding trying os3 and devian, uh, i'm doing that as a part of the tango devian derivative and one problem I'm struggling with though is uh Basically getting around the concepts of os3 and get finding enough documentation on how to build things properly And what I do at the moment is just look at how fedora did it and try to figure out how they managed to build this system And yeah, this is hard. So thank you for your talk and maybe we can meet later and discuss how how you do it because I think you're a little more experienced with it Yeah, uh, feel free to find out be here today and tomorrow for sure Yeah, the the project that uh that I linked to it has you know the way that we use os3 You know, you're right the documentation can probably be improved You know, I I know the maintainer calling is is always very open to improvements along those lines But you know sometimes I guess things move too fast to to keep up Other questions On that subject, um, the os3 package in devian, uh patches, please send them there are box open tagged help Thanks. I was cheating in reading some os3 documentation as you were speaking so It appears that var and nc management is somewhat separate Those are rewrite director. So for novice users, what do you do to manage that and recovery situations and things like that? Yeah, so var is It's basically taken as it is and always mounted as it is with your deployment. So Um, that's actually useful for us because it's where things where the flatbacks are installed for instance, they go into barley flatback and You know other things like drivers that are loaded at runtime. We haven't really had you know Something you need to be Mindful not to send out an update that would not work with some stuff that you have in var But another thing that we do is we have this little package for us to help her And systemd allows you to run Certain units after an update. So, uh, but before the actual system boots So there is a way for us that we have had situations in the past where we actually had to Do some surgery on a directory or on a file Before the new update could actually, you know be booted And we have used the systemd units for for that. It has worked out Yeah, and etc. So os3 does something which is so we don't expect people to manually change stuff in etc But if they do os3 does a three-way merge So it will take the new version the one you have the old one and try to make the most of it Um, I think it's an area where you know, uh, there are other solutions there and you know It's an area that may need improvement in os3 itself But so far it hasn't it hasn't given us, you know paired with Some ability to do some manual stuff. It hasn't it hasn't given us many days I would say on I think on etc a lot of the work that the Geisen project Tomica doing is about Sort of basically emptying the etc directory, isn't it right? So In fact, in normal situations that directory Ideally would be completely empty and then anything that's contained in there would actually be stuff that the user themselves has changed And then presumably if they changed it, you know, they know what's going on there. So, you know Ideally this, you know, there's definitely work to be done there, but It becomes less of a problem over time if we if we go down this direction Any other questions? I wanted to get clarification on what the person who just spoke mentioned So the idea I guess would be something like Um Since so many applications look into files in etc For configuration defaults, the idea would be what to Maybe migrate a lot of this stuff over into a user-lib package name and and you know just have Have an approach to config file reading where look for the faults there and look for any overrides in etc Yeah, that kind of thing. That's that's the kind of approach Okay, I guess I have user tools writing to other things other than etc and then having applications be able to manage up with themselves There's another I think thing that Somebody has tried to do which is to separate the Like for instance for for users and groups, right? Like you have etc shadow group past past wd and all that Those can actually be generated like so you have your definition of the users Somewhere else like you know in dot these nippets or whatever And then those files are composed when when you boot If something has changed in those configuration pieces and Uh, I think it's for like user generator or something. It's a system d concept. Um and uh Yeah Otherwise you do have the problem like one problem that we had with groups Specifically is that if you deploy the same reboot strap twice Maybe the same group will get a different gid depending on whether other packages got installed before or after And that was a big headache for us. We got this bug that we couldn't understand like oh wait like oh the user here You know the group change id Uh, so we right now have just a static list of all the groups that we expect to be created and we gave them static IDs But using something like a generator would be much much nicer and more upstream way of doing it, I guess For questions. Thanks everybody