 Hi everyone, welcome to modernize your applications during Fedora upgrades. My name is Vincent Svinstar, my Fedora account system name is Ivelissimo and I'm a community member since 2012, working for Red Hat as a senior software developer. So what are we going to talk about? What is Leap? How can it help Fedora? An overview over Leap and would like to do some live coding of actors to show you how it works. So what is Leap? Leap is a framework for application and OS modernization. What does that mean? Means updating, upgrading the system. In that matter also upgrading your applications, updating configurations, migrating them to new styles of migrations or transforming your data. Isn't that what DNF or RPM is doing? Yes. But also no. Because there are a lot of things you can't do. For example, as I said, transforming data is very, very dangerous to do in a RPM transaction and we shouldn't do that. And there are a lot of things you can't do in RPM transactions, therefore mainly forbidden by the RPM packaging guidelines. So what Leap can do is it allows customization and extension of the upgrade process. It utilizes DNF's RPM capabilities and allows additional modifications around it to be packaged. So what does it mean like modifications? Well, you can modify the transactions. You can add packages to the transaction, you can remove packages from the transaction, you can replace a package with another one and things like that. So how can that help Fedora? Well, you can make optional things available to users during the upgrade, allows you to add the ability to ask the user's questions during the upgrade process. It allows you things to do during the upgrade which you should not or must not be doing during RPM transactions. It allows the DNF upgrade transaction and now I will give some examples of what I had ideas for what we could have done or could do with Fedora and this kind of upgrade process. Now that said, these are hypothetical and all the examples are not implemented. So idea number one, ask the user to switch to the new bootloader specification automatically during the upgrade. As maybe some of you know that we introduced the bootloader specification which gives a unified way of describing bootloader entries for the kernel and with all the configuration options which is basically the same across different types of bootloaders including GRUB2 and I think it's called Zippo or something like this. This is how this could look like. During the upgrade the user could be asked like, do you want to migrate to this new or do you want to convert your bootloader entries to the new configuration format? Another thing what we can do is to detect for example Python 2 application in scrim and subscribe to Python 2 channels and with channels I mean modules like to module stream. It's like an example how this could look like, Python 2 module stream is like the title and then it's like we detected Python 2 scripts on your system, the new version of Fedora gives you the opportunity to Python 2.7 module stream, how would you like to continue? And then you can switch to it, do nothing, it means like default behavior would be and abort the upgrade. Another idea would be detecting Python 2 application scripts and warn the user about this continued support. Very similar message and then it's like ability for example to show what scripts are affected and it would print a list of scripts it found. In this example this has been actually implemented in a way that it tries to compile the Python 2 or the Python modules with Python 3, just compile them and try to see if it fails. If it fails it would give you this message and give you all the scripts that failed. Well there's a PyCompile module, the question was like what I mean with compile and Python has a PyCompile module which I can run from the command line and just pass it a script and it basically it will do the syntax check so if something would not parse it would like immediately fail. Of course this is not like 100% but this was just as an example for the first, for demo purposes. Another idea is like for example to detect that there's a Postgres 9 installation like it was in Fedora 27, there's I think 9.6 if I'm right and we would ask the user if they would like to subscribe to the 9.86 module stream or if they want to upgrade to the latest and greatest version instead. This might actually be something that is very useful for people who are a bit scared about transforming already during the upgrade their database especially if it is big to a new version. It might actually break their application if suddenly they come up with a new version or things like that and that's where modularity can actually be great and we would be able to transform the upgrade transaction not to upgrade it to version 10 but instead would use the modularity version 9.6. So another example of the output thing is like the options are like a switch to 9.extreme upgrade to latest, upgrade and so on. Another idea, ask the user if they would like to apply newly implemented service defaults or stay with old ones. If you want to change service defaults in Fedora and someone has like changed them there might be a reason for that and I'm not sure what the RPM transaction does but we can give them the ability when we detect that someone modified them that they can keep their old settings in some way and instead of getting some RPM new or RPM old files in the end and well there's like just some ideas what we can do. Another idea and that's like also a little bit in conjunction with the title of the talk. We can allow third-party applications creators to easily hook into the upgrade process. Right now this is very difficult because they might want to do something like enable repositories before and or change some or need to do some steps before they can do the upgrade so we will give them the ability to upgrade the applications during the upgrade at remove update, at remove or update the repositories for the new major version and things like that and they have the ability to do this after the download, they can add maybe packages somewhere, do their own download in the interface, they can do a lot of changes to the or influence the RPM upgrade in a way that does not, it's not possible basically right now. Well and I have prepared a demo for this what we do here is like we start the leap upgrade like it shows first of all that is a Fedora 28 machine and I recorded it today and I will run I will show you it's up-to-date with the latest packages and everything and then I will execute the leap upgrade which should be happening any second so and there you can see the questions which I was showing you in the SDA ideas how this would look like in the process. Please beware the log messages you see are currently during your development present but they will be removed and it will be more sensible messages should be printed during the upgrade but it helps a lot during your development to see if actually your stuff get executed or not. Basically this is what you would see in a verbose mode. We would enable the logs that you can see them and if you need even debug logs you would have additionally to specify the minus minus debug option so right now it would like it edits to the DNF call it added actually an additional repository with my repository for the for the leap RPMs so they can actually get upgraded because like if they wouldn't be there it would break also the transaction so it's actually sped up I hope no it's the right one it feels like now it is not the right one sorry the long version hold on okay now okay so we're back where we were so now I can see the cursor blinking really fast if you sit close enough so now you see I sped this up like I think 60 times or something so because it's basically what it does it just like downloads the packages and reboots then it starts running in a in a drum disk and will resume the upgrade and this now you can see that here and then DNF will start running again but this time in the offline mode with using in the cached packages only and he's going it's going to apply the transaction which also of course is never that's that fast wishful thinking well and after that again and basically you will end up in the fedora 29 system and unlike with the DNF system upgrade you don't have to run like two commands I think it is a bit more convenient I don't really understand why vision like that's made but well so here's the reboot into the new system and we will end up in a row height basically like you can already see 29 row height basically I will do the same thing I did before with the OS release that's it so back to this a proposal for this I will try to publish off the flock not immediately but like sometime after flock it's not available yet and I expected time frame when we could do something like this make available for for fedora is like probably around version 31 or 32 I don't know this is really hard to say because it will also need a lot of work from our contributions from a lot of community members because we can't know all the things that can be done and it would be really great to get also some feedback on that what community members have ideas about like what could we do on top of it so an overview overview over leap itself so leap has the notion of workflows which means describes the whole upgrade process what we have seen here that's called in place upgrade and that's one of these workflows and each of these workflows have phases and you I will visualize this a moment later and each of these phases has three stages phases basically give you the ability to order your your execution of your modules or actors how we call them at a specific time during the workflow so if you say like I need to run before the download and I you can do that if you want to run in the internet run this at some point for example after the application the upgrade transaction has been executed you can do that and you can even do in the first run and a lot of other things so it's too much to display we have like a lot of phases and if this is necessary we can even add more actors are basically the individual modules that can actually do some action that means they're the scripts that you can write to extend the capability of the upgrade messages are sent between actors that is a way of providing data from one actor to another that means like if you in the beginning of the workflow you start scanning for something you have one actor and later in the internet run disk you want to apply the data or will use the data to transform it you have would write a second actor that consumes that message and these messages contain a definable model which is basically the you can define that way the payload of the message models are assigned to topics which is at the moment purely it's it's not really used at the moment but the idea is that you can start searching for data when you're debugging stuff based on this and you can subscribe can review locks or anything easier like this because you can search for a specific topic what is in there it could be like something like networking system information storage information that these were would be topics we wouldn't put there and they could be also like for third parties they could put their own topic there is to find their own messages easier tax I used to designate actors for a certain phase and because we have a notion of also repositories where all of this data which you've seen here all of these items are in simply in there and can have multiple purposes so to define where an actor belongs to and into which phase into which workflow and even to which stage that is defined by the tax and yeah additionally we have for libraries they are private and shared libraries libraries are Python modules that can be used by the actors and can be shared across them that is when they are defined on the repository level or can be used like to make the code of the actor better testable because like actors have tests abilities which were like the actors themselves not but we have built-in test support that you can run the full actor or and or just parts of it and that's like would be then used from the libraries additionally you can bundle tools and files so if you have static files which you want to read some data from or if you want to bundle some tools like because Python is maybe not fast enough so you want to have a statically compile compile whatever see application or something like that you could use that as well so what I said workflow is basically a big box of of phases and each of these phases basically contains the actors since the order of the actors is actually defined by the messages it consumes or produces actors that produce messages are usually coming first and actors that consume messages come later and it depends on what they consume and what they produce it's it's a little bit tricky to say but it's basically topical topology sort so I guess this four people who know computer science they should know what it means yeah and with the phases I said like there are stages I put didn't put them here because it would make it so small basically you have the ability to say like okay my my actor would be executed too late in the order I wanted to be execute before the other actors in the face so you have the ability to say like I put this in the face before in the before stage of the face by default they go to the main phase and if you explicitly say that it should be before then it goes to before if you say explicitly go should go after it will be executed after that that way you can ensure that it is executed at around the same time in this face but you can influence the the order that is for example very important if you want to influence another actor with them custom actor you add like for example administrator is not happy about what an actor does for example it removes a file whatever which he shouldn't so it could the act he could write two actors once that backs up the file before and moves it somewhere else and afterwards like moves his back so the actor has no chance to modify it of course this is like a random example I just made up but I think there will be use cases where this is really useful so models are basically definable data structures like this we have a bunch of fields and very similar to what is done by yeah SQL Alchemy if you know Python SQL Alchemy allows you to define data models for the database we have support for numbers that means we can say integers of course like means it allows both we have support for Booleans datetime objects strings in arms lists and we allow to embed other other models however we don't support maps or dictionaries because we don't want the freeform support because that forfeits the whole idea of defining the model if you do that then please just dump it to a JSON string and like be done with it on top of that the data is validated so if usually there should be not a problem but like it could be you due to the dynamicity of Python that someone puts the wrong type in there so it would actually raise an exception if you would try to put the wrong data type the repositories is as I said it's like a big back or cloud of of different types I put there now three but basically there there are a lot of them and these repositories you can actually be linked to each other that they have a relationship in currently in the in the demo which I showed you like they were actually three repositories there was a common repository which hardly contains any more anything at the moment because it's supposed to be used for other workflow cases then there's the upgrade case upgrade repository which contains basically everything that is general apply applicable to to multiple operating systems and then the third one is like the we call it the offline upgrade right now but this is basically well rel or Fedora based systems these these dependencies are actually quite important because if you would like to write your own repository for testing like a new actor you can actually link one of them and can use all the information which is provided by them otherwise if you create a new one a new repository you have no models available nothing so you would have to start with everything from scratch the linking gives you the ability to avoid to have to copy anything or to rewrite anything so this this is the whole idea of this so what I said before with the tags they actually are extracted from these from these repositories it scans all the actors in there and sees what tags are there and by that it knows to which of the phases in the workflow it belongs to so this is you see here for example the green one it doesn't belong anywhere because there's no no green tag like like that so this is a simplification of the way how the actors are selected additionally we have like this bundling support what I said there's you can have batch scripts binaries whatever the the path to it is automatically injected to your of the actors to the actors path environment variable that means like you don't have to actually figure out where are you and what are you need what do you need to do you just specify the name and boom you can just use it files you can ask their special api calls to tell you either where are the folders that have files or the second thing is you actually can search for him if you know the exact name then you can search for it by libraries that's like as I said shared and bundled libraries are shared and private libraries are available so like I already covered that before so the next thing this is our mascot snackter we call him because like it's a python actor tool so snake actor right snackter the the tool is used for creating templates of actors models workflows it creates full boilerplates of tags and topics it can discover in the the repository what do you have there and display them nicely it can export it as JSON and it's used to link the repositories and and resolve them it's used to run actors for testing purposes it's used to run workflows for testing purposes of course if you try to run an in-place upgrade a workflow that will not go well because that is it needs root it needs a root right it needs a lot of things and that might actually blow up into your face so don't try it um yeah cool empty slide no so python is the first class citizen that means like we do everything mainly in python however all the things yes right now but it doesn't have to be if you need to do something in a different language well bundle these these things in a different language and just use them and like for example for like if someone really wants to use bash we might actually start if there are like enough people who who really have a need for it we might actually start providing you some some libraries that or like which you can use like which you can include in your scripts and and then basically send messages as well and and things like this write it directly from there or request the files the same way like it was if you use the bash scripts i mean basically you will still have to write that python actor but we might actually start creating some kind of boilerplate that will allow you to say i i wanted i wanted to call a certain bash script in my in my tools folder and that will automatically start it and we would just like process it so the last thing i wanted to show you how it actually looks like to write actors because it might be a little bit how do you say that intimidating so for that reason i wanted to show it how it looks like let's hope that works so um this is a premade repository it has some tags topics and workflows as i said before you can use the snag talk tool to discover things so if you want to know what's there okay this is really a bit too small that's too big so you see like it will show you what tags you have it shows you what demo topics you have what topics you have what workflows you have defined and in this case the workflow is very simple it's also in python defined it has only one face and it's just for the demo purpose because it's like otherwise it would blow your mind like if you would see the in-place upgrade it is just too long um so for this task for this live coding i was saying like i will write an actor that produces a message and it consumes it and prints it so and then execute at the end the workflow and in the meantime i can also show you how it looks like when they run so snector uh you write there first of all the new model and you uh want to use the the tags you don't have to specify the tags um sorry tags topic you don't have to specify the topic immediately it's just um gives you the ability to avoid having to write it yourself later in there in the produced thing so right now the the boilerplate created looks like this like and uh since we want to send a message we should be a string right so and we make it required and that you write your model you're done um next thing we want to do is like we will want to write the actor so your snector a new actor tag is will define like that we want to have it in the flock face tag which is flock face and we want to have it in the flock workflow so let me say workflow tag and this is like a flock message producer so we got the new uh the boilerplate here there's everything um okay in this case it we also want to produce uh the the flock message so from models import flock message um and we produce this so so flock message so one one thing to say if you would want to use a library for this you would actually just use from leap actors uh from leap library sorry libraries actor import and that is your private ones and if it is like a shared one it would be here like import common so that basically this is how it looks like and if there's nothing magic to it uh from the usage it's just that's for every actor which is actually executed in a own process uh it's a forked child of the main process um it gets actually injected all the private uh libraries that are available and uh so that it always has the same path no matter how the actor is called or anything so producing the message is actually quite simple it's like produce um and uh you just like say what instantiate the the uh model and say message like hello flock 2018 right so now uh we can use snector run uh flock message producer uh print output and it will show you that it actually created a message here um if you don't say print output it will actually not do anything it will just like um show you the um normal output so there's nothing um what you could see from it you could have added a log message which i could show you as well this is uh relatively simple self log info uh sent message and then if you if you run it then you actually can see it uh in here you see and you see that this was your actor who is a flock message producer and uh you can actually use the logs to um filter it for you actor for example um that's that's how let's go on to the um to the produce ah one more thing if you well i will show you the later so the new uh the the consuming actor which will print it um let's do the same thing flock um face tag tag flock uh work workflow tag and um consumes flock message it's like uh flock message consumer actors uh yeah of course so this time i i did not forget to add this this consume since automatically uh adds it here it also automatically imports it and uh for the ones who are very uh keen about the order of imports it's even in alphabetical order um yes messages are are not like uh it's not a like a queue or anything it messaged it and you have there's no way of influencing it so uh they cannot be removed or anything they are there and that's it once they're there they can be consumed by anyone who who knows their name basically and um i mean basically the consuming part or the producing part you you're not restricted to one uh thing uh or self-consume thing um they pass actually consume works a little bit different like the produce like you pass the type and with this it will just do the lookup but you're actually able to say like flock other message or whatever you have a different different one you can get consume both of them but since you uh this it returns a generator so you would have to check what type it is so usually you don't want that but if you if you find with it for example they have both the same structure just different names and you don't care why not you know so i mean that's the the beauty of python in this case that it allows you to do these things so for message in um in self-consume mess self-consume flock message and now you can actually just well let's use the logger um message message and it's like let's do it like this oh we can do it properly with um logging styles so it doesn't do it if the style is not enabled anyway so that's all you need to do to do it uh to to uh produce it now if you would want to run the actor you could do again snack there run uh well actually it doesn't matter if it is uppercase or lowercase i could say uh i can write it like this as well if you see that it actually doesn't do anything because it didn't get the message so before um the other actor we had we can actually um yeah what was it safe okay so it's safe output if you do that it will be locally stored in your in your repository in in the in the database there there's a there's a it's a database actually um which you can review if you want that there's just containing all the data um and you can now consume it in your uh in your other actor so now you will see that they're actually it printed here the message which was forwarded so and um now since we had the workflow you can use snack there workflow run flock and tada does the same thing so much for the demo um are there any other questions so please well right now well how it would be the question is if if uh we would use this in fedora if you would have a repository that we can use someone else already wrote the idea is that the um that the actors are actually present all the time and that if something is not available like for example not installed and it would not affect it therefore should be actually uh multiple sets of actors normally for example there should be in the beginning there should be something that scans for this stuff and figures out there's nothing and it would not produce a message later another actor gets executed which would consume a message but the message is not present so it wouldn't do anything and that way it is uh the idea of of how this should work is like this well your question like i mean basically and the availability to get to the availability um well there would be basically installed um but the default set of like what we support in fedora uh would be in should be installed basically the moment you install the leap tool um who wants to add uh want to use third party ones while you can just include them into etc leap repos d as a sim link the repository and they will get uh for the upgrade they get injected into that they get automatically loaded anything what is in the repos d folder gets actually injected that works without the linking actually but most of the times they use the linking as well because like they would refer to other things from there to consume to be actually using the tags to be injected into that whole workflow yeah without that wouldn't work okay that's okay the question is like basically sharing actors between different databases for example the example was that one developer is maintaining my rear db the other one is postgres for my rear db they are implemented and the postgres developer would like to reuse a bunch of them um in that case i would actually wonder if that would not be a use case for shared libraries um that's the first thing that might actually solve your problem already if you share them on repository level that um then they are like usable by anyone that's the first thing um the second thing is like if necessary yes you can use them this is just like you basically produce the same messages and it would do something but you need to be careful um generic actors might be a sign of that it is actually supposed to be a library because it's if it is generic code that is more a library than actually an actor an actor should be specialized to specific tasks any other questions well to be quite honest i i'm a still a little bit um back and forth with the idea where to oh sorry where should live basically the actors like should they live with the with the product or which which is affected by it or should they live basically in a global repository for all all the things like let's say there's a upstream project which manage maintains them um for for fedora um to be quite honest i don't really know right now what is the best approach here because um initially i would have uh one like one global one only i once place where to put them um and that is because until you have everyone on the same page of like how should be actors written and how should basically you would need to set up something like uh the packaging guidelines you would have to set up something similar and uh this needs to be undergoing reviews so at this because um otherwise it's in danger in the upgrade process you might actually do something which is dangerous and you should need someone who looks over it a bit that until you basically understand how to do it and uh anything that is in fedora i think it would be beneficial to have one global place to like what is by default fedora should be one global place is it's like something external like copper or whatever they should bundle their repositories like with their rpm and have it specialized and it's like something like what what like for example in a in a rel scenario where we products it's called they are like based on rel and we have like for example ref and they would be upgraded to want to upgrade together with with the major version of rel to the new version of rel they should provide for example their own repositories very specific has absolutely nothing to do with the base operating system and then it should be bundled and that's like falling under the section of three third-party application developers does it answer the question any other question but it's fine do you want uh do you want uh want to see that oh wait a minute what did you see once more oh okay okay uh it is actually um yeah the question is like what's the magic behind the the library support um well in in the libraries we have basically um we scan the repository um we figure out what files are there and the moment you start an act that we see like oh there are libraries okay i was like there's there's there are files in there so in that moment we since the actors are started in their own process um but we use multiprocessing for it and fork basically at that moment before we load anything and uh what we do is we dynamically import this and uh insert them into the module dynamically on runtime so that you can do that and and it has support for packages and and modules so if you use a normal python package that works perfectly fine um the only thing is like if you would want to use third-party stuff like let's say esco alchemy or whatever things like this um there's like one one restriction on on actors you cannot globally import them and it is simply because we do interest uh we we load the actors quickly and if you're having a missing dependency it will blow up and uh it will make everything fail but we can ignore this eventually but like right now it would everything fail with it um because we don't we want them to to load successfully it is considered like a broken actor which is a problem because we don't know why it is broken so um the uh functionality there is also because of testing if someone forgets like we want to have some make files that specify the the uh dependencies what you need or or some other way we are not really 100% sure how to go with the dependencies of this most likely we will have to use something else than a make file because we need to eventually eject it into the spec files um so you would need to provide that for the test as well so the test tries to load the actor and will fail so wow it is it's um it's complicated so i i mean like you should look at it then you will see why uh if you're really deeply interested in it but in general it i mean the the the way how this uh the libraries are injected is relatively simple from the idea it's just dynamic import with reassignment of the module to uh to the to the place and afterwards it's also environment variables get injected and it is basically running in a context manager and cleans up everything afterwards so anything else no that was about that was about modernization what does it mean so update update can be it's not in context of fedora upgrades in this case it is more what does modernization mean it's like you bring the software up to date or you bring the data to the up-to-date version i just used update slash upgrade because to me quite honestly this is ambiguous you know this it's there's no clear line what is an upgrade or what is an update at least to me not of us so no no no yeah well actually actually sorry for interrupting you're actually um thinking about it i mean like there would be no problem using to create the workflow for updates as well update in this case maybe i want to see the same message as i would see it during the upgrade for example actually thinking about it what you're just saying like i mean basically the point is that um no summarizing that what you just said um basically the the the question is if uh you could um in the basically have same messages about like what we had here for the upgrades also in the update scenarios um so if someone would turn on a module and uh and make it is available we could detect that of course and it could suggest that he he does that um on the contrary the questions it happens sometimes in fedora that some major version upgrades during the lifecycle happened but it's very discouraged from what i know so i don't think that is really a use case um i mean that particular example i can imagine that there might be some things that you want want to do maybe this might be interesting to system administrators um to add notifications for example it's like something happened something was updated something wasn't updated things like that so we kind of lost track of time okay yeah i don't know i i don't know how long it is actually going thank you