 Quick reminder that we have a grand prize giving and quiz and something happening at 430 in this room So please come along. There's a chance to win some really cool prizes and Gonna move swiftly on to the next talk and I would like to welcome Will Woods to the stage Hello, can everybody hear me? Is this working? Yes? Cool. I didn't know they're gonna give me the big room I didn't think anybody wanted to know about system upgrades Because I sure don't no wait. That's fine So yes, this is a system upgrades past present and future. I'm Will Woods on the installer team at Red Hat I have the dubious honor of being the guy who has written every upgrade tool So everything about upgrades that we have done other than what it used to be in the Red Hat Linux days is my fault and I am sorry So the one slide version of this talk is here at least the current state of things RPM upgrades are kind of terrible They've always been bad, but we've they've been getting better and better and better And the current new thing that just landed them for the last release is DNF system the DNF system upgrade plug-in And it's really the most sensible way we have of handling them So how do system upgrades work? There's two. I mean basically there's two styles of upgrade that you can do in the Linux world offline or live Offline is where the upgrade process runs from outside of your system This is how in the olden days when you would upgrade Red Hat Linux You would put the new anaconda CD that you got from cheap bites calm in your CD-ROM drive And you'd load that in and reboot into the installer until it to upgrade your system Pre-upgrade and fed up did variants on this live upgrades are Running inside a running system. So you're upgrading that sort of out from underneath yourself This is what apt-get dist upgrade does. This is what young based upgrades do There are pros and cons to both advantages or pros and cons of both approaches Offline is cool because you can do it without the network. This is important for secure facilities and things like that And the installer media can handle both installs and upgrades if you've got the upgrade tool on the on the media And you can download one thing and upgrade a whole bunch of systems with it, which is nice And plus this system is offline. This is important if you want to do things like backing it up and restore And so you can restore it later or a migrate file system You really don't want to migrate from like XE 3 to XE 4 while your system is running. It's it's probably messy On the downside, it's a larger download than necessary at least for the old way of doing things You have to download an entire CD or DVD to do the upgrade Media is really slow and once you're finished, you know, you've just installed from the things you have to install of the updates after you Finish with the upgrade and the really big problem with the upgrades in the old red hat Linux days Was it was difficult to correctly identify the system to be upgraded which you don't think about when it's just your system But when people do things like put the you you have a system That's you know in a rack and the rack is attached to a big box with a thousand discs in it And you boot the installer and the installer looks at Every one of those 1,000 discs to try and find a version of red hat enterprise Linux to upgrade That takes a while it usually picks the wrong one because there's usually 17 of them in there And if there were a lot of bugs about that Anyway, live system upgrades people love live system upgrades. It's so nice You can watch it in progress because it's just on your regular system. You only download the things that you needed for the upgrade It's one less reboot. You get to apply updates Can't really migrate your file systems like I was saying and when you upgrade things that are running like when you install new libraries behind a running program things get really weird and Really hard to debug because if you try to debug the problem You're you don't know whether you're gonna be looking at the old binary or the new the binary or the old library or the New library and people don't do upgrades all that often So every time people will do live upgrades. I'm like, yeah, it mostly went fine I mean one thing was really weird, but I'm sure it's fine and they just fix it afterward And we never know what happened. We never know how to fix it. And so it's always a little unreliable Sometimes it's a little unreliable and sometimes it's massively unreliable It's not a good idea in general. It's unpredictable and sometimes catastrophically bad So yes in the olden times the anaconda upgrade was the way this is up until I think Fedora 7 Was it was it was it still for Dora core at that point? I don't know anyway up until about for Dora 7 you install you upgraded using anaconda. Sorry Yeah, but I don't remember where pre-upgrade appeared his but anyway anyway before there was pre-upgrade You just upgraded using anaconda, which is to say use the installer media You just all you go to is both download all five CDs and burn all of them Or you boot the net insta iso and let it download the packages and hope it downloads everything Okay, because if it fails to download something like one of the mirrors goes down your upgrade just fails and then you get to Fix it. So they suck There's no updates. It's really bad finding the root partition burnt media is Slow and so unreliable You know, there's certain brands of CD that won't that you can burn and then they won't be readable by the CDR that's in this system like but if you get a different brand of CD, it'll really just fine like Optical media is incredibly unreliable and it's magical that it works at all So a huge portion of the bugs that we would get about upgrades were just basically Burn a new CD and try again and that fixed like 50% of the bugs we got The and then the rest were things like oh You should have used a better mirror. Sorry that your system is now host Sorry, it's broken. You get to keep all the pieces though So that's not great So eventually and I was in I was in QA at this point and I was testing upgrade So I was hitting all of these things and I was like, this is really awful Surely I don't want to burn another CD and like just stacks and stacks of CDs that I was burning for every release It was it was expensive and time-consuming and I don't like optical media anymore because of it. I have like trauma So we came up with a pre-upgrade which was a weird hybrid sort of thing that would Do some of the work for you it downloads the install and the boot and the install images like it downloads the the installer Runtime image that was in the CD and just down shoves that into your boot directory And it downloads all the packages that you need beforehand With the updates which was nice and so it just set up your system to boot instead of next time it booted instead of booting into The normal system it would boot into the installer which was sitting right there in your boot partition and reboot and just do the upgrade This was nice because you didn't have to like click through and have it find your desk for you Um Well, wait. Yes, you don't have to have it find your disk sort of Yeah, you get good things It also kind of sucks Because you need to save boot images somewhere that the boot loader can handle which was always slash boot Which meant this is why boot the boot partition kept getting bigger and bigger and bigger through firms for it I don't know 17 till or 7 till 17 or something like that And it kept not being big enough and then you couldn't do upgrades and then we're like Well, we'll just make it bigger and then everyone's like, but mine's not big enough and then sorry. I don't know There's still no guarantee that the boot images will know how to mount your disks if you have like a custom Well, you have a custom controller. Let's say for your disks. You have something that we don't support You won't be able to boot the installer because our kernel doesn't have that It also made the installer even more complicated than it already was We had to handle yet another special case of we're doing not only the special case of doing an upgrade But the special case of now we're doing an upgrade. That's also somewhat magical and the packages are all on this target system Which you can find here probably unless the disks have moved around good luck It also turns out that changing the boot loader is kind of unreliable There's like one thing that does it well and that's when you install a new kernel package It adds a new entry that's safe and well tested other things can get a little funky and even that every once in a while It does not work. So then there was fed up and fed up was An attempt to improve that situation by having a much smaller image that just handled upgrades We're not going to use the installer at all so this meant we got to pull all of the upgrade code out of the installer and throw it away and Instead put it into this tiny little tool that all it did was upgrades, which is a much cleaner way of handling that problem It was a much smaller image You You add we could add it just like any other image We use the new kernel package tool, which is what happens when you upgrade your kernel package and so we would download the new systems kernel and That image and boot that so we're starting our install process with the brand new kernel and the brand new SELanx policy and brand new tools and all of that which is a good idea or I thought it was a good idea So then and we we boot in we boot that and then we let your existing system set up the discs like it normally does So we can't predict what sort of weird and crazy things you have done with your disc Which is one of the problems with pre-upgrade when you're outside the system And you're looking at it and trying to figure out like how how do you have slash var mounted? You could look in Etsy FS tab, but now there's things like mount units Or maybe you're mounting slash var in I don't know in a nit script somewhere an RC dot local type of thing You could do all sorts of crazy things and people do all kinds of crazy things and it's very hard from the outside to look into a system and Tell exactly how it wants to set up its discs. So one of the things that fit it decided was Let's not do that. Let's let the system set up its own discs and then we'll just run the upgrade afterward Which is a solid idea or so I thought It almost didn't suck So the upgrade dot image is the 20 beg it's like 97 percent the same as the regular in it Rd. It's just also got the upgrade code in it. But because of reasons We couldn't have it be just one unified image I think it pushed things over the limit on certain PPC systems to have everything in there So it was it turned out this way and that was problem not a big problem Small problem. The generic in it already can't always set up your system correctly isn't that is a more important thing? It turns out that there's some stuff that's in your system that when you build a need Rd When you're you know when you install a kernel it looks at your system and sets up an Rd that will Set your system up again It's using some stuff from your system to do that in theory. It could do that with Things in your boot arguments And so all we need were the same boot arguments and we could set up your system the same way if we have the same in theory Boot arguments and the same Drivers then we can set up your system the same way you do that turns out not to work There's a lot of things that require like a config file that it crams into a need Rd that we didn't know about MD Adam is one of these things Sometimes LVM does this stuff various disc things that we don't necessarily support but third parties handle We won't have that in our need Rd And so and we get a whole mess of hey my system doesn't set up bugs and So that's a problem. You could work around it and gradually we were figuring these things out You know when a whole bunch of people are like my MD Adam My MD rate array does not work. We figured out. Oh, we need that MD raid All right, we pull in MD Adam.com and then there was another oh my keyboard doesn't work So I can't unlock my desk. Oh, well We'll pull in the the console.com for whatever you use these days to do that And we kept having to put more and more stuff from your system in the in Rd for you And it turned out we were reinventing a whole bunch of Drake it but like differently and not well Which is not cool It also as it turns out if you switch from the unit Rd, which is new to your system Which isn't you know you're running We have new system D here and then old system D here and then we go back to the new system D to run your upgrade That sounds Like maybe you know, it's not a good idea. It's a very bad idea as it turns out and It would break system D in various ways because when when you do the switch From one route to the next route system D serializes its state and crams it into the new System D and this works fine if you are going from an old version to a new version They support upgrades as well. They should Doing it backwards where you're sending old stuff. Well, no, I got I got that backwards Yeah, sending having an older version send things to a newer version. Yes The newer version understands the older version of the serialization format It does not work the other way where you're sending brand new stuff that the old one does not understand It gets very strange So it would break system D in various incredibly hard to understand ways that I would spend several months every Fedora release debugging for What was it two or three releases that we did fed up? I think yeah, sorry that happened I it seemed like a really good idea and then and then I spent like Two months of every release Chasing very deep bugs inside of system D that would turn out to be like Things that would crash in the middle of other things. I ended up sending up patches upstream for a libessie linux and Which is weird to send things to the NSA, but that's fun and System D and a couple of other things and this kept having to happen and they're really hard to understand and finally I started asking for help from Leonard and the system D team and they were and they said wait, what are you doing exactly and I'm like, oh, no, it's fine It's totally reasonable We we start a new version of system D and then we have it, you know Go to an old version of system D and then and no they kind of like no We're not we're not doing that. That's a terrible idea. Don't do that and So There's other problems. I mean, it was also Python 2 only and it used young and not DNF And we kind of wanted to use DNF and it didn't always beat your system correctly, but the system D thing is really the killer Holy unsupported Leonard did not like it and he was like, well, you know, we do have a facility in system D for doing updates Like the system updates thing when you boot Why don't you just use that and on the one hand? I'm like, I don't does that does that actually handle the problem, but on the other hand now it's Leonard's problem So that's obviously the right answer So DNF system upgrade, this is my attempt to basically make upgrades. It's somebody else's problem entirely So it's a plug-in for DNF It basically just does the equivalent of DNF upgrade dash dash release for 23 and then offline the offline is the part that wasn't implemented in system D Or in a DNF yet. So I that's like half of what the plug-in does is handle that System D does that offline part for us DNF does all of the downloading and the depth solving and all of that and everything is Pretty fine It's Python 3. It's a DNF plug-in. It doesn't it uses your old kernel in in RD, which is as it turns out works Just fine There's rarely so much of a difference in the kernel that we really need that to be able to run the upgrade There's nothing that's Generally, there's nothing that's going to run during the upgrade that will require a newer kernel Yeah, and of course, it's you know on good hub. It's on all of your systems if you're using If you're using fedora The one thing that you lose in that is the possibility the part where you're running outside completely outside of the system So that you can do file system migrations and things like that So well, how do we handle that? We've already done that before when we had to do the user move thing There's a draket module called convert FS and convert FS runs basically the first time you boot Something that has the module and it looks at your system and it says oh your system is set up the old way I'm going to convert it to the new way. It does a migration and then it reboots So the migration happens instead of happening during the upgrade or before the upgrade It happens the first boot after the upgrade, but it still happens. It's still totally possible to do that So we haven't really lost anything it almost doesn't suck There are three main kinds of problems that we see with this style of upgrade. There's DNF bugs where like DNF doesn't necessarily or has been doing a little bit weird things with Unicode and so if the plugins did something else with Unicode and Python and Unicode and you know how that and Explosions everywhere and there's some limitations to DNF plugins the reason you have to do dash dash release ver is That's that's a DNF argument There's no way for a plug-in to override that from inside the plug-in It has to be done come from outside the plug-in and so there's various things that have to be done They can't be done by the plug-in that you have to pass in from the outside And that's something that could be fixed in DNF if it was worth it, but that particular use case I don't know most of what's the problem with upgrades is the same thing that's always been a problem with upgrades It's everything with rpm And I don't mean to pick on rpm here It's just it's not a system upgrade is not really what rpm was designed for rpm rpm is very very good at the case It's mostly designed for which is when you have a running system and you want to install a small amount of updates It does this fairly quickly. It does this very securely. It's easy to get them. It's easy to deal with them This is what it was for for the system administrators to install updates When you're doing the entire system all at once it does not scale well It's an insane amount of disk IO to do this I think we end up hashing every file on your file system twice Maybe three times because we have to check that it's the right one beforehand And then we do and then you do the install and you check again To make sure it installed correctly, and then you delete the old one and so we we MD5 your sum your entire file system twice during an upgrade Sorry And so the disc requirements are also a pain you have to download the entire Set of things you need before you can start the upgrade and you have to unpack them as you go So there's some overhead there that we don't know about and then there's the case that the scriptlets are unpredictable There are quite a few bugs where we we have to guess how much disk space you actually need to do the upgrade because we really have No way of knowing Because we cannot predict how much disk space a scriptlet will use it's just not predictable could be nothing could be 20 meg Good packages will do things like have ghost files that take up approximately the amount of space They expect to use but the kernel is the only one. I know if it does this So we have no way of knowing how much disk a scriptlet is going to use I've I lied to the to the user and tell them you know it requires RPM says it requires X amount. I pad that a bit and if you don't have enough I tell you that it's not gonna work That still doesn't work sometimes people still run out of disk in the middle of the upgrade and it's very bad You also have no idea how long it'll take you have no idea what whether it's doing things when it's in a scriptlet It's it's a mess And so upgrades are not updates is the other main problem here The user expectation of what's going to happen After an upgrade is very different you expect large changes and you expect there to be new code So you you're will you're accepting some risk in doing a major version system upgrade and You do expect it to take a long time, but you're because you're accepting some risk You're also a little nervous about it So you want to know what's going on with it a little more than you do with a regular system update updates are usually pretty quick And usually you can just walk away and get a cup of coffee and you get come back It's done if you're doing a system upgrade you don't know how many coffees you need it might be a whole meal You know maybe it's a I don't know so if you come back and it still looks like it's working so there's the macro scale of It's going to be a long time and you need progress to see how how many meals you can eat before it's done But you also when you come back to check on it We need some micro scale progress to be like wait is it still doing stuff? And if there isn't something spinning on screen or some assurance that the system is doing some sort of thing people get nervous so like a I'd say 20% of the bugs we get about system upgrades are people who got nervous and reset their system in the middle of an upgrade And then they're like my system won't boot and that's and I you know Look at the bug for what can you send me your logs? And then it turns out. Oh you you reset in the middle of the upgrade and they're like no no it crashed So I had to power cycle it. I'm like no it didn't honey. It's fine You Bless your heart. Oh, yes. Yeah, there's it. Yeah, there was another bug where it attempts to blank the screen for you And there's a kernel bug where if you're if you're trying to blank a text console It doesn't quite and they're scrolling going on it doesn't quite do it, right? And you end up with like letters kind of crawling up the screen and it looks really weird All you got to do is touch any key on the keyboard and the screen will you know go back to normal But this made people really nervous and they're like it's obviously Crashing right now and fire is about to start shooting out of my computer and they power cycle it and then they ruin to their system So you have to like The experience of doing a major version system upgrade is a risky operation and people are nervous about it understandably And we have to do better at letting them know that you know everything is okay And here's we can't tell you how long it's gonna take but we can tell you how far we are into the process So this is one of the things that DNF system the DNF plug-in doesn't quite handle and obviously there's no good, but this is this is the part where we're getting towards Towards the what else could we be doing where it may turn into a discussion sort of thing because I have no plans to continue to Maintain the plug-in myself It's it's like honestly 300 lines of code and that's a DNF plug-in So somebody else could definitely handle it, but there's a lot of other stuff that needs to be done around it But at the heart of the matter it's a large RPM transaction like this is I've been doing I've been responsible for upgrades for a very very long time and the major lesson that I have learned is it's really just a big RPM transaction So there you go. That's the entire thing So there's some lessons to be learned about it, but mostly with the lessons I learned where don't be too clever And just let the existing tools handle it because they're the best at it So the really this is just the last bits and pieces we need to make the system upgrade experience like tolerable for users And it's mostly tolerable at this point, especially compared to the old days, but it's still a little Nerve-racking So yeah, we could be doing better on the UI to sort of let you know that this isn't a regular system update It's an actual system upgrade. It could look different We you know have a different pulse thing going on a wider progress bar unfortunately Plymouth only keeps state It keeps the keeps track of the percentage of completion as an integer So you literally cannot have more than a hundred steps from start to finish And so if it's you know gonna take three hours, then that's a hundred and eighty minutes So you're only gonna update once every one point eight minutes So you need something better than the progress bar because the progress bar can only tick once every couple of minutes Or we need to fix Plymouth. It's a you know your color your choice um We could be getting better feedback from rpm or scriptlets Unfortunately, there isn't a good way to make or I have not come up with a good way to get feedback from Scriptlets themselves. We don't have any sort of standard for asking for or providing Progress from rpm scriptlets. Sometimes they output stuff. Mostly. They don't mostly they're silent Which is a shame because it means that nobody has any idea what they're doing. I think That dnf maybe has a callback for when a scriptlet starts rpm does Last I checked dnf was not handling it. So rpm tells us when it starts and stops a scriptlet That's the best we can do dnf does not pay attention to those so that could be better Obviously it would be nice if we were doing backups before and after that's not easy To do automatically for the user until we get everybody on voter fs, but then we have like 17 different problems So I'm not going to recommend that But it would be good if if it was If you have a system that's capable of doing a snapshot do a snapshot beforehand and then if it if everything goes Totally wrong restore at the end of it And we also don't notify the user of any problems. There isn't really a good facility in Fedora that I know of for presenting the user with a Here's what happened last time you booted screen if if you have failures in updates Like just a regular system update you get a notification the next time you log in Which we could use that facility I suppose but we aren't and You know that doesn't help for people who are doing like remote upgrades and things like that So until there's like a general next time the system administrator is somewhere where I can say hello to him Tell him this facility I We don't have a good answer for how to tell the user when there were problems or even if even warnings about things like we did not We installed this thing but not this thing We're also depth-solving funniest with DNF that are being shaken out, but mostly Mostly the problem there is not DNF, but our PMs the the way that we construct the distribution means that if Somebody messes up one of their dependencies then DNF which will which refuses one of the major differences between for those of you don't know between young and DNF and For that matter RPM by itself in DNF is it DNF refuses? absolutely to break dependencies on your system which From a which seems like a very good idea But upgrades traditionally break dependencies on your system and they break them hard like on purpose Willy-nilly all over the place because mostly we assume that what we're getting when we do upgrades is a well manicured depth-solved set of things and we don't really need to worry about whether or not what your system has and What we're doing here are gonna be friends with each other because we're as I say It's a different operation than an update with an update your system is gonna say 99% the same Except you're gonna put this one little eight. No stay alive You're gonna put this one little thing Into the system so you really do need to make sure that everything you're not breaking anything else on the system But when you're dropping an entire bucket of things onto the system and replacing everything you want to make sure that this chunk is consistent, but the three or four percent of things that are outside of that chunk Maybe you don't need necessarily to double-check that was that was the theory in the past That's not how it works now Exactly. Yes, the comment for those who didn't hear it was that With especially with third-party packages. We can't guarantee their dependency stuff is up to date with what we're installing on your system Maybe they haven't rebuilt for Fedora 28 yet or whatever you're installing so We can just sort of assume that you want Fedora 28 and it's our and because you're doing a major version upgrade You know and have accepted the risk that some things will break so When we install afterward you're going to have some things missing And you're fine with that because you know that eventually the third parties will get around to updating their thing and you'll update and then It'll work again. That's usually how people go in with their sort of gut feeling about the expectations of the system after the upgrade But DNF won't do that right, right, yeah, there this problem does exist even within Fedora and We don't have a really good answer for what to do about it yet We have some stuff where you know, we want people to obsolete packages that are obsolete Which seems a little bit silly that you would force you to remove a package just because we're not using it anymore Right exactly Right so yeah, there are different policies that we could use you could say alright DNF should remove things that aren't going to be used and I think that is one of the public either allow a racing I think is the switch you can use that will do that And then there's or you can use the dash dash best I think is the I refuse to install anything unless it's the latest switch So DNF does provide some switches you can flip to get different behavior out of the upgrade But there are some things that aren't quite handled yet. And so we need to look into those So there then there'd be less or fewer problems I don't know I think Honza's in the room so we could talk to him about it, but I Think that the that DNF's goal of not breaking your Dependencies on your system is probably the right thing to do 99% of the time And whether or not it's the right thing to do during a major version system upgrade is still up for debate. I Don't have an answer for that. So Maybe DNF will do that. Maybe they won't I don't know if it's a good idea or not The obviously the correct answer is well, everybody should fix their RPMs That has never happened I'm not gonna hold my breath for that so Until we have some better answers for that what usually happens is that you try to upgrade and DNF tells you you can't And usually the right thing to do there is just wait a day and the broken upgrade great Or the broken dependencies for whatever package it was will have been fixed the updates will be pushed out And then they'll work, but we don't tell the user that very well So they just know that they wanted to upgrade and they can't and that's it So if we told them maybe well, you can't upgrade because Usually when there's broken dependencies that means you need to somebody somebody messed up wait a day We're sorry it'll be fixed, but we don't have any way of telling the user that Either one of those would work I do feel like telling the user wait a day because we'll fix it is probably better than letting them then telling them Here's a way you could break your system if that's what you really want to do They should probably have both options, but I don't want that to be the first That's about it. There's also the Crazy go nuts research ideas about what we could be doing for upgrades if we weren't doing RPM Or if there was other things going on There's a million ways we could be doing upgrades But they're well outside the scope of this discussion because we're doing fedora and rel things and those are RPM based If we had other if we had other things we could talk about other ideas of how to do upgrades That would be that would solve some of the problems that we're talking about here But that's a much longer talk This is really the end of my Discussion on the matter if there's any questions at all Go ahead So I right now I think we only officially support one. I think there's Yeah Imminently you will be able to do leap upgrades as I you just used to call them So you can go from 21 to 23. There's no there's no real reason you couldn't do like three or four Technically, there's no real reason but the the further apart they get the less likely it is that The packager will have thought that you're going to do that for a leap Actually, this is a thing I wanted to say to anybody in the room who is a packager is that anytime you're messing with your package and its Scriptlets keep in mind that during a major version system upgrade The previous version of the package might be one might be anyone from the previous two releases So keep that in mind as you're touching your packages because that's half of what goes wrong during upgrades And so if you want to improve the experience for the user, we have to keep in mind that they're going to do system upgrades There's nothing in the um Fedora guidelines that I know of that require you to think about this because there's nothing really that's required to do it And there unfortunately as far as I know is not a way to detect that you're doing a major version system upgrade It shouldn't necessarily matter, but it does so That's a thing we could add if it was necessary Yes Yeah, I think we switched that I think the original DNF system upgrade plug-in did a up and actually you're right My slide says it does an upgrade it actually does a distro sync operation rather than an upgrade There is a switch you can use to tell it to do the opposite of distro sync I actually don't remember what it's called right now, but it's in the man page. So wait, is there a man page? Yeah, there's a man page Sorry All right, any other questions about upgrades at all All right. Oh, sorry Which one? Be more specific about our PM's problems. I don't know if any of those are actively being worked on You could talk to folks about it, but I'm not aware of any work about those things unfortunately RPM being as Ancient and wise as it is it's a little hard to get new things into it Yeah Yeah, yeah, we started talking about that what like 10 years ago. I mean It's I'm not saying it's not possible I'm just saying it's not like easy and I absolutely understand that the guys who are working on RPM are hesitant to make bold and brash changes, so It's not their fault that RPM is what it is. Oh interesting All right. Well, if you want to know more about RPM come go to the talk about new features of an RPM any other questions All right. Thank you. Oh wait one more. Um, I don't I don't know I think something like atomic is probably the future of how we do these sorts of things But I don't like to be speculating about the future because you know like William Gibson never figured out that there would be cell phones So like there's like Blade Runner. There's no cell phone. So I'm not gonna bother because I know I'll be wrong But my guess would be that something like Atomic would be at least it would be a much more painless way to do all of these things for some of them One less. Oh, that was a different example. I mean, you know, I'm talking the Neuromancer trilogy There's no cell phones and and Blade Runner. There's no cell phones. And yeah, I don't know and I've been meaning to find Richard do you want to answer this question for me? Oh cool. Cool. Anything else? All right. Thank you all very much You