 Okay, hello everyone, we've made it to the end of the conference My name is Davis Roman and I am so excited to be here in bobao to talk to you about OS tree for the uninitiated what you need to get up and running with OS tree on your next project Okay, so who am I I'm an embedded links engineer I have 16 years of experience on home security security products and access control systems and now I'm in the automotive space Previously I was employed at Honeywone Residio and now I'm at Lucid Motors in February Originally, I'm from Queens, New York and now I'm based out of Fremont, California I've been using OS tree for many years across multiple projects one of which is a product with over a million devices So let's imagine a scenario It's your first day at mega corp and your boss hands you a next-generation device bound to change the world It's the eggminder the device the smart egg tray when you forget how many eggs are in the refrigerator So you're not one egg short for tomorrow's omelette You're told it runs in bed of Linux But before you can begin developing your collapse on this device you'll need to first update to the latest and greatest It's been on the show for a while So it's probably very out of date out of habit you'll reach out for your USB thumb drive But you notice you can SSH into the eggminder because why wouldn't the eggminder support SSH, right? You SSH into the eggminder and as you begin to type a bunch of commands apt-get install your boss says to you Don't be silly just type the following OS tree admin upgrade dash R And you're done a few a few moments later the system reboots and the update is done What just happened? So let's take a bird's-eye view of OS trees update process After typing OS tree admin upgrade dash R OS tree determined the required remote and ref just think branch for a moment using the origin file The summary file was pulled from the OS tree server, which specifies the latest remote and commit a Diff of the commit metadata that was pulled and stored onto the local OS tree repository and the commit was deployed Meaning a hardlink form was created in the deployed directory pointing back to the OS tree repository The uemv.txt file was updated Reflecting the deployment as well as which kernel and in it remfs to use a 3a merge occurred in slash etc Slash var was selected according to the state route After the reboot the uboot bootloader mounted the file system and imported the uemv.txt Into its environment The deployment file was passed via the kernel args and the kernel was booted with the init remfs The init remfs determined which deployment to use via the kernel args and mounted it onto the physical sys route The init process intruded into the deployment becoming its new slash and Proceeded with booting into user space as usual You got all that right a lot just happened. So let's take take one step at a time Let's start with a few objectives We'll begin by discussing prerequisites for understanding OS tree We'll discuss OS tree versus lib OS tree and why it's hard to search on YouTube and Google for OS tree related content We'll discuss package versus image versus differential upgrades We'll discuss some different OS tree architectures We'll get started with an OS tree enabled Raspberry Pi 4 build using the yachter project and We'll discuss pushing to an OS tree server, even though it's not natively supported. We'll discuss that later We'll talk about basic OS tree command line basics and booting into an OS tree deployment and finally yachter integration So as far as the prerequisites if you're a user in OS tree meaning an application developer or testers There's none. Just learn a few OS tree command line basics, and you'll be fine However, if you're a system integrator BSP developer or system engineer, there's heck of a lot more to learn You have to know about overlay fs's hard links to roots bind mounts and it ramifaces you boot and the yachter Build system. So now's probably a good time to take a quick snap of the cure code I forgot to mention that in my intro slide So because I have got a lot of content and spring and links sprinkled all over the place so I'll just Wait a moment Okay, we from got it. Oh, I see one more and All right, we'll dive into OS tree now So originally call it created by Colin Walters from Red Hat originally for the good old continuous project You can think of OS tree is git for file system trees. So some of the features Transactional upgrades either it's fully applied or not at all Rollback so once you're on the latest commit if for some reason you decide that this commit is has some issue You can roll back to a previous version Also, Austria allows for replicating content incrementally over HTTP Also, you can you can do parallel installs of more than two bootable deployments and you have the binary history on the client and the server You have flexible support for multiple branches and repositories and it's designed to be resistant to certain power laws Austria was initially driven by server needs both later ported to the embedded Linux base Austria is primarily focused on the delivery and deployment of file system trees So Austria was originally called Austria, but has since changed its name to live Austria Basically to get away from putting too much emphasis on the command line tool itself Austria refers to the command line tool itself while live Austria. Obviously, it's the library many blogs and tutorials use the terms Interchanging really which can be very confusing. I know that firsthand Austria began in the desktop Operating system space, but it's since been ported into the embedded Linux use cases Most videos and online documentation is skewed towards the desktop at least for now So see here are some of the projects that have that that have OS tree I was initially I initially became familiar with Austria via the automotive grade links project Here's some other great projects that I encourage you to check out So let's discuss about package versus image versus differential upgrades So you're probably all familiar with package based upgrades rpm apt yum So some of the advantages so requires low bandwidth easy to use lots of lots of information available online As far as the disadvantages every system is slightly custom since there's no strict control of what can be installed on specific versions And it's unreliable in the absence of a human operator It's not power safe and could corrupt the system at any time As far as full system upgrades The advantages it can be very it can be tested exhaustively because we're dealing with just one image EXT4 or squash a fast that gets downloaded onto a device It's easy to reason about because like I said, it's really just one file that you're downloading and then flashing onto a partition And it can be far safe with innate with a be configuration Disadvantages, of course, it consumes a lot of network data Even if you change one config file, you're still downloading an entire file system And it decreased the lifetime of store of the storage medium due to excessive writing But now atomic differential upgrades or OS tree So the advantages requires minimal network bandwidth as only the difference is downloaded It's power safe due to its transactional nature and the smaller the difference the faster the update takes But here's it here the disadvantages it requires a reboot most folks complain about this one HTTP request for each file that we don't have and We can't perform cryptographic verification on the block level here I've provided a link from the talk that Colin Walters gave as far as a compromise between image and package based I think he makes a very good argument and I encourage you to watch this which watches video Okay, so we've discussed the prerequisites Package versus image versus differential updates and now let's talk about the different OS tree architectures so They're not necessarily called this in the documentation, but overuse. This is this is kind of what I've converged to So push pull offline distributed. Let's start with the poll So for the poll you can think of you can think of three in this case three embedded Linux devices that are all Austria-enabled they each have their own copy of the oyster repository in the file system They have some sort of OTA OTA update client That that is internet enabled and connects to some public Austria server This public oyster server is populated by this fictitious company called megacorp with apparently only three developers their CICD process commits to a local oyster repository and those commits eventually get promoted to the publicly available oyster repository So some some of the advantages for the for for the scenario So it relies on oyster server publishing a summary file that all clients can periodically monitor if you simply care of just about just being On the latest then this is a very straightforward path But the disadvantage is is that you don't have strict control over which commit gets downloaded onto specific devices Here's a quick tutorial just on this So we start with a raspberry pi that has an oyster-enabled image Like I said for those in the back. I apologize you can't read this But I promise it should come out better in the recording So in this case what we're gonna do here is we're gonna add a remote the command is OS tree remote add We give it a a remote name in this case I've chosen pocky then we give it a URL to an oyster server in this case. It's just a server on my local network Then once the remote is defined we use oyster remote summary pocky The the raspberry pi goes out to the oyster server queries the summary file to to get a list of refs and available commits So in this case is f621 Then we try to do an oyster upgrade, but that commit the commit on the raspberry pi is the same as the commit on the server So there's nothing to do at this point. No update is available But now let's try something in the background will add the h top package to our yachto build and Then push to an oyster server. Now. I know I haven't mentioned what yachto is in two sentences what I can say is yachto is a System with metadata that allows you to create your own custom Linux distributions. It is not a Linux distribution. It is a distribution maker so now At this point after adding the h top package pushing to our server We call the oyster remote summary pocky command again, but now For the raspberry pi for ref the commit has changed DC f8 So now on the raspberry pi we say oyster admin upgrade Po only and we see we get we get the metadata plus additional content objects And we do a diff now we can diff the previous commit of the ref with the current diff of the ref and now What we see here is that among the files that are getting added h top is with is isn't the list Now at this point if we choose we can do an oyster admin upgrade Which allows us to boot into the new into the new operating system with h top installed So that was pulled. Let's talk about push Push is very similar to pull with a twist So again, we go back to her example with three embedded Linux devices that are oyster enabled with their own oyster repo With their OTA clients they go out to some public oyster server that gets populated by this mega-corp thing But now the difference is that the developers have access to this other entity called an old OTA campaign system and Now what are the advantages with an OTA campaign system? Well, you get that fine-grained control over which devices get specific updates because that system You can push metadata to the specific devices and within that metadata is an individual oyster commit You can do regional based upgrades and you can carefully control the rollout To limit exposure in case of bad OTAs Also, you can closely monitor the progress of your update campaigns Disadvantages you get more There's more upfront cost to develop such an OTA campaign How but but there are many off-the-shelf solutions such as to rise and quarter Quick tutorial on this So again a bit of preparation work now will add the nano package on in our inner yachto build Push to the server just so that there's a difference and there'll be something we can pull So again on the Raspberry Pi the commands oxtree pull pocky remember that's so that's the remote name And then we give it a specific commit so previously we use the summary file now We're using a specific commit that we Retried via an out-of-band method which would be that OTA campaign system. So now after this We use oxtree diff And we use our current current ref and this commit and then if we can see that nano is among the binaries that are getting downloaded At this point if we so choose we can deploy and boot out of this new commit So offline Again, we have the three embedded Linux devices, but this time there's no internet connection So at this point our options are to roll a truck and get a service person out to our devices the advantages here it allows for air gap systems and improves it in theory approves ability to Ensure that if that the firmware is only used by approved individuals provided that you trust them, of course and Disadvantages are you can't push from her over the internet obviously and Physical access is required but not always possible Think of a buoy in the middle of the ocean and it can be very expensive to visit customer sites Quick detour on this for offline mode We'll begin with some bit of preparation as we've done before we'll add the jq package on to our yachto build and we'll push to the server So there are two modes for offline. The first is using pull local So on the build machine will copy the oxtree repository directory onto the USB drive and then insert the USB drive into the Raspberry Pi Then we'll use the oxtree pull local path to our mounted USB drive in this case It's slash mount slash USB slash oxtree repo and then the second argument is is the ref So the the commits are pulled and now we can say oxtree show raspberry pi 4 We see that the commit is 0 7 eb Which has just been pulled if we diff the previous commit with the with with the current ref We see that jq is among the the binaries that are getting pulled and lastly we can deploy if we wish and boot out of this new system But now there's also static deltas So let's talk about that So for static deltas we generate a static delta file on the build machine and then copy down to the raspberry pi How do we do that? We use the oxtree static delta generate with the ref name in this case raspberry pi 4 we give it a from and a to and Then we give it a from from commit to to a specific commit and then we give it the file name We we look at the file size So in this case is called jq underscore delta and it's only 245 kilobytes So relatively small we copy the file onto the raspberry pi and we apply it So the command is oxtree static delta apply offline and then a path to the static delta file We ask it to show us the commit and we see this new commit. It's the same as before so from this point forward It's all the same the difference is well, let me finish this So we do the diff we see the jq is among the binaries that will get downloaded if we boot out of this out of this deployment So now the difference between a delta and the poll local from before is for the delta We're dealing with just one file one binary delta file Whereas previously we're dealing with an oyster repository with with metadata As far as which to use its use case dependent so it But but they'll achieve the same thing Okay, so now lastly the distributed option So again, we have the three embed links devices that are oyster enabled, but now they're on a local area network So it's peer-to-peer and they use MDNs So they can find each other on the local area network and they use DNSSD for service discovery So they can figure out which repos are being served and who has what refs They do this using collection IDs their collection IDs are universally unique identifiers relying on reverse domain name handles an example is a classroom of devices where all devices get updated as soon as a newer device appears on the network as In my opinion, this is the most interesting distribution scheme. However, I don't fully have this working yet, so I don't have a tutorial to share I'm sorry However, here are two links that go into more details on the distributed option in Austria. I Encourage you to watch them So now let's talk about getting started with an oyster enabled Raspberry Pi for build using the octa project For the sake of time, I won't be able to go line by line. However, I will provide a high-level overview So we'll start by setting up the repo tool This tool is responsible for bringing the source and and installing it onto your local machine We'll set up Yachto for a Raspberry Pi for build We'll use the meta-updater from the advanced telematics get repo using the dump fill branch Then we'll do a repo sync This is what it should look like once the meditate once the source is pulled. I apologize. I realize you can't see this in the back apologize But it comes out better in the video. I'm sure So while doing this work, I realized that there was a boot issue And the boot issue is that it couldn't find a configuration node This is an error in in new boot but not a problem because the issue was was was straightforward and I generated a patch and threw it up on bit.ly You can apply this patch like so and then at this point you can continue So we kick off our Yachto build by sourcing the ENV setup.sh using Raspberry Pi 4 as your first argument Then in our local.conf we'll set the Ostr update summary equals one variable So that every time Yachto builds generates a commit the summary file is updated accordingly And then lastly, we'll we'll kick off our our Yachto build using bit-bake core image minimal When the build is done, you should see something similar to this picture here So let's talk briefly about the newly generated Ostr repository that that was just minted by Yachto So if you're familiar with Yachto, you'll recognize this this directory structure build temp deploy images Raspberry Pi 4 Ostr repo so in this directory if we use the Ostr refs command we'll see that it uses the Raspberry Pi 4 ref If we use log we see the newly generated commit f621 and if we briefly look at the structure of this repository excluding the objects directory because then otherwise it would just be massive screenshot What we get is a folder structure where we have a config that describes the type of repository Raspberry Pi 4 file under the refs head directory that that indicates the commit that is the ref currently points at and a mirrors and a remote directory very similar to to get because Ostr is get inspired If we cat the config File we'll see that it's a type archive z2 Meaning that this is an Ostr repository that can be served over the network so clients can pull from it And lastly if we cat refs heads Raspberry Pi 4, we see the commit that this ref is currently pointing at So now at this point you want to put your Raspberry Pi The way we do that is that we use the flash image script that's inside the meta-updater Raspberry Pi 4 layer The first argument is the device of your SD card So in my case when I put that into into my laptop it came up as SDA On on yours. It might be slightly different the second parameter is the path to your wick file So here's an example of what it looks like when I when I ran it on my machine So once your SD card is complete you plop it into your your Raspberry Pi you allow it to boot Here's what you should see OTA enabled Linux 1.0 The login is root no password very secure So let's talk about pushing to an Ostr server So now When you're doing your initial experimentation Ostr I recommend that you use the same machine to host your build software Along with the Ostr repository itself But eventually you'll want to adopt a more formal process dev staging and prod So more more than likely you'll have different machines for for for each day each deployment process The problem here is that Ostr doesn't natively support pushing to a remote server However, the Ostr community has written a tool to address this gap So we'll be using the db Nicholson github repo for Ostr push Let me illustrate how Ostr push works. So we have a development machine and Development machine that you can imagine is what's on your desk and then an Ostr server Maybe some machine on aws or somewhere So when Ostr push is started it starts a local hdp server providing the contents of the local Ostr repo on the development machine The development machine then connects the Ostr server via ssh and tunnels the hdp server through the hdp server port through the ssh connection The Ostr receive command is executed on the Ostr server with the url and tunneled hdp server The Ostr receive then creates a temporary remote using the url and pulls the wraps from the development machine At this point the commit was pulled and the Ostr server is up to date So I've provided the the uh, how to configure server and your development machine. However For a lack of time, I'll just provide a high-level overview So on the server we'll provide we'll install the Ostr push project Using the db nickelson git repo. We'll initialize an Ostr repository Then we'll install patchy 2 or nginx or whatever web server you prefer so we can serve up that Ostr repository Then we'll we'll configure the the dot config slash ostr config file. I've provided some reasonable defaults in this diagram here Oh, and by the way, I've installed this on the boon 2204. I just So as far as the development machine Uh, very similar idea. We install the Ostr push git repo Using the db nickelson github repo We'll create a little script so we can define all the variables that are necessary At this point you can set up passwordless ssh login. This is optional But uh, I would recommend because it makes life easier And then as a final test to make sure that the integration is done properly We we can we can run ostr receive dash dash version and if we get the version back then you've performed the steps correctly So now at this point, let's try running the ostr push command from the development server So we run the script and the script shows us the commit that we're trying to push And it performs a push On the ostr server We can use ostr dash ostr space refs to show us the ref that was just pushed We use ostr log raspberry pi 4 to show us the commit that was just sent This commit matches the commit that was sent from the development machine We can also use ostr summary dash v to see the summary file And we see that the summary file is also in agreement with the commit that's on the server and came from the development machine As well as the time that the update was was sent If we go to the server, we can see that the commits are being served over over htp So now let's uh quickly cover command line ostr command line basics So you're given an ostr enable device. You want to see if it has ostr Well, first of all, I would I would check to see if it even has the ostr command after that You check ostr admin status if you see the following output It's a very very good sign because there requires a lot of machinery to get this far So ostr admin status shows you the commit of the currently deployed uh current deployment You see the index number so you can see if that commit was deployed more than once You see the operating system version if you're only using one os then then this will just always be the same name I mean the version is configurable And the ref spec which is a combination of your os tree remote and and the ref which in this case is raspberry pi 4 And finally we can use os tree log the commit to verify uh these details Oh, okay, so we use os tree log. It shows us the commit But more importantly it has the content checksum So this is a really cool feature that os tree has because os tree supports content promotion So if you have a dev staging in prod So you want to take a commit that starts on dev And and after testing you want to promote it to to the next stage You cherry pick that commit, but then your commit changes. That's not good for for traceability So os tree provides you a content checksum, which is a representation of the file system itself excluding the metadata So even though the commit will change as you cherry pick them the content checksum stays the same That's really nice for traceability And then finally the date version and content id is configurable and you can I encourage you to make this as descriptive as you want As far as pulling specific commits from the os tree server as we discussed before the command is os tree pull We provided some remote name that we've defined in uh in this case. It's test server colon along with the commit that you're trying to pull We'll see the the number of metadata files that were pulled Along with the content objects or the files that that is trying to pull The size of those files and how long it took to do the download to do a download As far as deploying the commit And uh, so you you you do os tree admin deploy After a reboot you'll boot out of this new commit. Um, and so you'll be in this new file system So a word on cleanup so over time the client's os tree repository grows and can eventually fill up the file system I've seen this firsthand It's recommended to periodically run os tree admin cleanup which removes commits that are not associated with deployments So a question can changes be made to os tree up to to slash user So slash user is read only by design It follows the user merge design pattern and it looks something like this diagram to the right So instead of having physical slash bin slash s s bin Folders those folders live under slash user and instead what we have are sim links. So Slash bin is a similar to user bin slash s bin is a is a similar to to slash user s bin Now the advantage is here is that we that the the file system is consolidated under one directory allowing allowing Allowing it to be easier to make that one directory read only as opposed to making a bunch of directories are only uh, so This results in the majority of the file system being read only Um, I if you would like further details into why user mergers used I I recommend this open susa Talk called user merge and beyond But now how do we develop in the system you ask so os tree provides os tree unlock command And what this does is it applies an overlay over slash user allowing for modifications This overlay looks like this diagram to to to the lower right where the lower is your read only file system That is the pristine version and we don't want to touch that Then we apply our changes in the recorded in the middle layer And then what you see is is the top which is the combination of the read only plus the modifications So This this unfortunately, this is removed after a reboot, but it is nice for quick tests Personally, what I use is the os tree adam in dash dash hotfix So again, it applies the overlay over slash user However, it's persistent across reboots allowing for for application development But keep in mind that after deploying a new commit the overlay overlay fs is thrown away So what about slash etc So can we change slash etc technically yes, but here be dragons So os tree uses the three way merge scheme and my experience is a big source of frustration for os tree beginners The tldr here is that if a file and slash etc slash foobar is locally modified and later slash etc slash foobars Updated in a newer build your locally modified version of slash etc slash foobars kept yikes However, the new the the new read only default version of foobar is stored in slash user slash etc So os tree provides a command called os tree config diff Which inspect slash etc and informs us what has been updated modified and deleted relative to the current commit So if slash etc slash foobar shows up as modified all you have to do is cp Slash user slash etc foobar into slash etc if you run os tree could os tree admin config diff It'll show up as no changes. So now now you're good For an explanation of why we're using the scheme Uh, I've provided a link to a blog post from calling walters And at the very end. I think he describes it very uh succinctly He says if you modify config file in any way that wins So Another question does os tree protect against corruption due to power loss? And the short answer is yes The long answer is please watch this drew mosley's Webinar are implementing power safe atomic up Over the year updates and here's a link. I fortunately don't have time to go over this But I promise you it's a very very good webinar and I know someone will ask so I figured I drop it here so Let's talk about booting into an os tree commit So how do we boot into an os tree commit? This diagram I tend to lose a lot of people once I go into this diagram, but I'm going to try very hard to explain this So we have a on on the left. We have the storage side and on on on the right. We have the runtime side So on the left think of that as the physical sys route I'd like you to think of that as an emmc or zsd card On the right you have the virtual sys route think of that as the login And so when you log in you type ls and you see something that that is that shows you a typical file system layout So back to the physical sys route At the top level all we have is slash boot and slash os tree So this is very unlike anything that conforms to the file system hierarchy So in slash boot slash loader. We have a unv dot text This file describes what kernel what it didn't run my fast and what kernel args the boot loader Should use in order to to uh to boot the kernel properly Um along with that We also have the the uh device tree the netrum fs and and the kernel under slash boot slash os tree slash uh os os commit So the os commit uh is also special here because os tree does not put the boot artifacts as part is the file system It keeps it separately for for for the reason that um It we don't want to we don't want to change those artifacts unless they've actually been changed. So so they're kept separately Then we have the the uh the the uh The hard link farm. So what this is is a this this is what you'll eventually charude into It's a representation of your file system with hard links that go back to that that link back to your os tree repository Then we have var var is shared across the different deployments and that'll get mapped onto your virtual sys root the the uh The root of the physical sys root gets gets bind mounted onto slash sys root So we are able to access the os tree repository when we're when we're logged in in our virtual uh sys root environment So how did we do this? How did we get from the physical sys root to the virtual sys root? So here we dive into the net remf s So the net remf s helps us with os trees chicken and egg problem The kernel cannot start the net process until it switches into the deployment And it cannot know what deployment to use until it mounts the file system And to mount it it needs to know which deployment to use and so on and so on So on boot up the boot loader you boot in this case Instructs the kernel to start the net remf s which contains a tiny file system With an init script whose job it is to parse the kernel command line to figure out which deployment to use So that's the get os tree sys root function The init remf s mounts the physical sys root and again think emmc sd card onto slash sys root The init process performs the switch root using slash sys root From this point forward slash s been init or even system d continues with the deployment as its new root You got all that right? So lastly yachter integration So for yachter integration Fortunately the meta-updater Layer does a lot of the heavy lifting for us It built it among other things it builds the target and the native os tree binaries So the native binary uh lives on your development machine and it's responsible for creating that commit That will later get pulled by the clients the build target version of os tree is what gets installed onto the root of s that gets installed in your devices So after after building the net remf s, uh, sorry that is uh, let's see Okay, so one of the bb classes that are in the meta-updater layer Image types os tree bb class has a section that performs the os tree commit So as far as the commit it points the local os tree repository As well as the yachter root of s And the branch that's used so this whole time we've been using the raspberry raspberry pi for ref It will likely be different in your project And then lastly we can describe the subject in the body and and I encourage you to make this as descriptive as you like And then lastly we uh, we we check if the up os tree update summary variable is set Um, I I always like to keep this set so that the summary file is updated Uh, as soon as the commit is generated. We want the summary file to reflect that new commit so Some some more words on yachter integration So we need to prepare the physical sys root, which is that the physical representation that goes on your eamon crsd card We need to prepare the deployment sys root, which is that hardlink farm and it's located in slash os tree slash deploy slash state root slash deploy slash commit We'll want to make the boot litter and init remf s work together to boot the deployment This involves setting up the env you env dot text so that you boot nodes which deployment to pass on the kernel command line So you'll you'll first need to make sure to control mutable state in your system So what this means is that operating system and application binaries under user must always be read only An application runtime data under slash far should be read, right? Initial setup of files and directories in slash far can be created using systemd temp files during first boot And templated config files can live in slash user, but must be copied to slash far during initial boot apps should use the copying slash far Fortunately, the first three steps are already done by the meta-updater layer So here's some final thoughts and recommendations So history is powerful and flexible update technology I would recommend that you periodically monitor the size of your os tree repo Otherwise, it will eventually use up all your disk space. So os tree admin cleanup is your friend Periodically monitor the amount of memory used Especially if your root if s is larger than your available memory This may cause you to anger the out of memory killer at times. I've also seen this for sand Also use os tree admit unlock instead of just remounting slash user as read write whenever possible to avoid your file system and Whether to use pull push offline schemes is use case dependent If you forget all else, please just remember this Mutability is the root of all evil. Choose immutable file systems systems whenever possible hear my references And I'll take any questions I have two questions. It seems the first one. It seems like that for the full system update There isn't the support for dm verity And the second question also is there support for a b updates? So for the dm verity case, I actually Can't really comment on that as I don't have experience with dm verity But as far as the a b updates what I what I will say is So I was on a project for a long time where it was just one partition One partition that had the whole os tree repository Now I've seen another project where they're very concerned about potentially, you know, what happens during a partition corruption Where you have Two partitions with two os tree repositories that keep each other up to date that can work Do I recommend that firsthand? I'm not sure. Uh, but can you can you combine it with a b updates? Absolutely anyone else And thank you for that question. By the way, can you comment on os tree apply live and its limitations? Sorry, can you os tree apply live? So if I want to apply The commit without rebooting the host right, uh, so so Rebooting sorry applying the commit without rebooting. Uh, that's That's mandatory. So what I have seen projects do because That doesn't quite work for everyone is to separate the applications using using some sort of container technology uh with that then of course they could they could update their applications and Restart them at will but as far as not doing the reboot for os tree. I Can't think of a way to do that So does that answer your question? Well, mostly in a desktop use case like if I have to install some applications like if I install mvmecli then I have to use Apply live if I don't want to do a reboot. I just wondering what limitations There are when when I use the apply live command Oh, sorry is apply live like a desktop rpm os tree It could be implemented in rpm os tree not os tree. Yeah, yeah, because I'm not I'm not really familiar with os tree on the desktop I'm primarily focused on the embedded Linux side I imagine if you ask the desktop folks, they probably have some wizardry, but And embedded I haven't seen how to do that. So Any other questions? Yes. Oh, sorry. Let's let's wait for the microphone. Sorry Why the cleanup is not automatic? Um So that's a really good question actually and I would say that the reason why the cleanup is not automatic is because you probably don't want it to be automatic in use cases where um, you've pulled a commit and Let's see your network goes down Or maybe you don't want to keep pulling that same commit over and over again You would probably prefer to just keep it in your repository assuming you have sufficient space Um, so I think that's why it's not automatic to give you that flexibility so that And okay an example is for example, you you download commit a And then you switch over to commit b And then later you can you switch to commit c But for some reason you really would like to go to commit a but you've rather not downloaded So if it's in your repo Then you save yourself that pulled over the network If it was automatic by the time you get to see a is gone so Personally, I I like that it's not automatic because then it lets me decide what I'd rather do as far as the deployments when you By default all street keeps two deployments the current one and the previous one that is automatic So in other words, you deploy to commit a Then you deploy to commit b Then you deploy to commit c a is gone But the deployment is gone the commit is still there So that if you want to deploy to commit a you can do that But if let's say it's a very small embedded device. You don't have a lot of space You really can't afford to have a bunch of commits around then. Yes, keep running Austria admin cleanup As much as you want Does that answer your question? Thanks. Okay. Thanks. Yes, sir Uh building on that previous question sure, uh, are the Commits small when there's not a huge amount of change from the previous one Hmm So that depends on your project If you're making lots of changes and configuration files, then sure it could be it could be small but I've been on projects where You had one gateway that was hosting an image for an android device And this could be many gigabytes worth for one file So It really depends on the project So now if you have two versions of that same entered version Now your your repo explodes. So you probably can only afford two or three different versions of that Because that image is is humongous But like I said, if it's just configuration files a couple little images And you have a mmc with a couple of gigabytes, then yeah, you could probably Probably keep keep a lot of it Around but it really depends on your on your case Does that answer your question? Thanks. Any other questions? Okay. Thank you everyone