 So, I guess we can start. So, the last talk of the day, I guess everyone is tired and yet one more distribution on RISC-5. So, we're going to talk about Fedor and RISC-5, David Abdukmanov, Independent Software Engineer. So, that means that I'm not employed by Red Hat, I'm not employed by Sci-Fi. So, I'm leading with Fedor RISC-5. So, for the last years, basically it was a full-time endeavor. So, before we go into the talk, let's do a few questions. So, do you have people who are running Fedora or used to run Fedora? Can you raise your hands? Nice, very nice. Okay, do you have people who used to run or still run Red Hat Enterprise or CentOS? Very nice. Do you have people who do not know what RISC-5? Okay, that sounds good. So, we're going to look into basically three parts. So, the first part, again, how we bootstrap a new distribution, a new brand-new architecture. I guess you can kind of go very fast on that, because if you watch the Debian talk, or if you haven't, please go and watch the Debian talk. There's very nice and detailed descriptions of what the Debian did, what is the RISC-5, and the whole history. Basically, 94 plus percent of Fedog directly applies to Fedora and probably other distributions also. So that, let's start talking about how you build. So, Fedora and basically every distribution, it's natively built, so if you don't do a cross compilation, which probably is what you do when you're using an open embedded or build route. So, most software is going to be written for the native design, so though it intended to be cross-compiled, so if you want to do a cross compilation, for new architecture, you need to probably port it, you probably need to have custom environments, scripting to do a cross compilation, that takes all the time, and then there are some pieces of software which try to cross-compile and even attempt to run itself or some utilities, and you're probably gonna need to hack that software to get it cross-compiled for the new architecture. And then finally, you know, you're probably not gonna have a very strong heart with the beginning, and you're probably gonna use simulation. So, at this point, for at least for the RISC-5, the QM is actually a strong target, you can even run on full-length cores, that's very nice, so it's very good to build the whole Fedora stack and probably the distributions. So, it's a bit chicken-egg problem, so again, Fedora is built on top of the previous version of Fedora, so if you don't have that for the RISC-5, so we need to get to that, and the way to do that is basically build a very minimal route file system, it's not gonna look like Fedora, it's gonna be very hacked, but basically, it needs to have a tool chain, the minimal bits, some form of a kernel with some features, and it needs to have Fedora packaging tools. And then you can start putting in the sources and trying building their PMs. So, basically, the picture looks like that, you do it in stages. So, again, you build a cross-tool chain, then you take the tar balls and you build some form of a very minimal route file, which you can build, which you can boot natively or under emulation, and again, it has the RPM tool, so you can start building in RPMs, so you're gonna rebuild it. In reality, when you think stage one, stage two, you think it might be independent, it's not exactly the case, sometimes you need to go back to the previous stage, enable more features, so it's kinda going back and forward sometimes to get all the features done so you can build a large set of RPMs, and then for most RPMs, you actually assemble the new route FS and you keep building on the new one and you keep redoing that. So, we have done all of that basically two and a half years ago. We did some of that, again, then we finally got the final agility version, so the ABI is locked, nothing is gonna be changed, so it's finally we can build the last distribution. The last step, which was done basically almost a year ago, was moving to a fedorant's construction bit, so that is the Kaji infrastructure. So that means I'm talking to the Kaji hub and web, I'm submitting the packages, we have a lot of the QEMU builders and we have the boards, we are running KajiD, so it takes one of the package, it creates a pristine, stage route environment exactly for that build, it builds the package, sends it back to the Kaji, Kaji does internal repositories, distro repositories, simple disk images and stuff like that. So the first bootstrap, I believe, was done, so Richard started the whole project and he's sitting right there. That was in 2016 August, I believe, based on the blog post when he said that he has something which runs our PM build and I think he announced that on Fedora mailing list and very quickly me and Stefan who is sitting right there joined this effort, so we have been working on this for a very long time. In 2016, I think you already had 5,000 packages, so that's like one quarter of Fedora at that point. But yeah, let's move forward. So Fedora is, of course, its upstream first policy, so that means if you wanna build a RISC-5, all the changes we do, we have to go upstream and then we have to rebuild the whole thing. So we have to do not only release engineering, but for structure stuff dealing a lot of all supporting. So a lot of things happened, the GDC happened, building the team that happens, they had to wait for a kernel, G-Lypsy was the last part. When that happened, we started going like mad to get it done. So we also had second and the third bootstrap and the third one was the final one. Once we have done that, we actually move to the Codian for structure. So that's, again, being a Codian, that means we're building the package in the same way that the Fedora official is doing. But this alternative architecture, so it's not using the official infrastructure, we're using a separate infrastructure. So that's a very quick overview of how we got to this point where we can actually boot something. So let's maybe look at how does it look in real life? One thing is to build something, get the console maybe, maybe it looks like a Fedora, like a distro, but what can you do to how does it look in real life? I think so the first time Fedora booted was in sci-fi offices in California. I think that was at the end of the March and last year. So that's first time Fedora 28. It's not exactly Fedora 28 because what we did was we built, I think, 24 on top when 25, 26, 28. So it's kind of a hybrid something, which kind is 28 at that point. So that's a board, sci-fi and leash boards is FPGA. And again, that's a screenshot from the display. Later, I think a month after, we started getting a board. So the sci-fi actually were donating the boards for various projects. So Richard got one, he got also another one from crowdfunding, so we got two boards. DJ from Red Hat got another one for Gillip C, which he also used for Fedora and also using for kernel stuff. So this is probably the most expensive way you can attach a CD to the system. That's probably five or $6,000 just to attach a CD. The other boards that Richard was running, we're running NBD, and we do not use the SD cards because those are crappy on the board. And then Atish from Western Digital came in and said, and wanted to try something, and he had the micro-semi board, so that means you can actually get the USBs, you can have the SSDs, you can get the graphical cards. And I started working on the GNOME desktop, I think, when I visited sci-fi offices. So it was like six months past since that time. And I think it's a total tragedy, it's not gonna work, it's definitely not gonna work. And surprisingly, it did work. The only thing that the struggle was to figure out what you needed to enable on the kernel to get the mouse and the keyboard. And that took like a week to figure out. Other than that, it did work. So this is the first picture of Atish actually booting in sci-fi with a micro-semi. And that's GNOME desktop, the same stuff that's running on this laptop, basically. And also Atish did more demos in various conference. So we have Fedora 29 running an AS demo, here's a media server demo. So it's basically not exactly the toy, so you don't need to go to the console. If you have a hardware, you can actually run real stuff on it. So basically, you're not limited to something. So if you have a goal, if you wanna do something on the board, just figure it out and then just try to get it running. It might work out the box, like it happened with the GNOME stuff, might not, but then come to a channel, tweet me, whatever, and you can figure out what's actually wrong. Another thing, so Bellard, the same guy who actually invented QMU, he wrote an email, and he has another project called JS Linux. And if you go to a website, you actually can boot Fedora 29 in the command line. You can also boot in X11. And by the way, the GNOME desktop, that's running Veylon. But you also can do X11 in a browser. You don't need to install anything. The performance might be not exactly what you expect, but it actually works. And also we have a tiny move emulation. That's also the thing you can actually boot Fedora in the graphical user interface. Not exactly at the current versions, things are a bit changed. We need to get the adjustment, for example, the BBL, and the kernel itself is now separate. And also you cannot specify the integer D for the tiny move, but that's at some point supposed to be happening. So let's go in the current state. What I did and what's probably is very wrong, especially if you're Fedora user, you probably not wanna hear what I'm gonna tell in the next slides. So I would build farm. So again, we have three boards, three physical targets, a couple X8664 nodes for running Kaji, Kaji Hub, Kajiras, the databases, various system D stuff to schedule. We also have some backup solutions for running Seth and Restick to ensure that if by accident the server dies, we can recover and we had to use that just after a Christmas. So originally all our builders, the QMU stuff was running on the Richard server, private servers. We had only like eight, I believe. Yeah. And then we moved to a GCT compiling farm. So Facebook donated loads of servers and we have eight of 16 core servers and based on my assessment, we have about 64 and we can add another 30 something QMU targets. And the whole infrastructure at this point, all the QMUs, that's fully managed by LibDirt. So not running long QMU commands or anything like that. And also everything you build, at least the disk images at this point and the repositories are now fully mirrored to official Fedora infrastructure. So if this dies, you can take disk images, you can set up a new Kaji, you can import the whole stuff and just kickstart when you want. Yeah, and that's the URLs. And we have two streams. So I'm still building 29 and Fedora Rohite. It's basically like a rolling release and we're also discussing potentially doing centralized aid, which is a bit more complicated, a lot more complicated at this point. Statistics. So I don't exactly like the numbers because numbers constantly changing. But according to the Kaji, we had 24,000 successful builds and about 3,500 failed ones. The best week in terms of builds, I did four and a half thousand successful builds per week. But again, I could do more. But what you don't do, I'm gonna be more slides about that, you don't run tests. And I do an other stuff. Basically, when we are running only on those eight machines, I had some hacks to ensure that you have loads of successful builds and you don't attempt something which is not successful and optimized for the speed. And if you do a comparison for Fedora 30, which is a raw hide. So the first line is directly from our Kaji. The last two lines are from the official Kaji infrastructure. So we have more total packages at this point. So Fedora removed some, we still didn't catch up to that point. Total build, if you look at the ARCs and this one is actually smaller, but when they look at a dozen of a package, apparently 29 and F30, we have packages added, which we never built but there are plenty of them for some reason. And then again, RPMs, I just asked for directly for the architecture and how much we have on RPMs. And then also including all the debug info and debug sources in general. So the package is, if you have to think of package as a source RPM and that source RPM produces artifacts which can be multiple of RPMs. For example, TextLive, that's delivering you like 6,000 of RPMs. A certain root package delivers a bit more than a hundred RPMs. And again, some of these packages might be in general exclusive architecture or something or we might produce some sub-packages or not based on architecture. So we count in exactly like, I have like a hundred percent, 95% for each architecture is different number. So that's why I don't like exactly when statistics and numbers. We have lots of problems. Loads of them. We basically try, we are running on more than 10 year old machine which was donated. I mean, the whole infrastructure is fully donated. We don't, there's no big company kind of like pushing our money and you know, to get it done. And we try, we are running basically everything on one single machine. At some point, we also even run the QMU instances on the same, our main server which does everything. Also our domain ended up in HTTP secure or something, something. Yeah, a preload list. And once it ends in that, basically it says if you go to HTTP site, you automatically go to HTTPS and you cannot, normal users cannot go to HTTPS because of that self-signed certificate which we use to log into our Cajun infrastructure. So the people complain that they cannot log in and get the disk images from our Cajun. So we are going to change the domain. So that's going to be Fedora.Ris5.ROX. So this is going to be very simple. And we still have the boards and QMU failing. If I basically every day have like two to three QMUs failing, that's basically CPU stalls. I don't know why exactly. The boards fail maybe, I don't know, once. So in two months and three months, very rarely, but we still happen. And the whole infrastructure requires a lot of time and maintenance. So the more you do that, the less you can actually do on the porting which is truly interesting part. So what's missing? Okay, so at this point, we did not sign any RPMs. That might be scary for some people. We could do that, but that's probably has a big impact on IOs. And again, the server is not exactly handling the current situation. So I'm not doing that, but we could do that. You don't have a body. Basically, body moves RPMs for different tags in the Cajun. So for example, you build a package which lands in one tag, but someone needs to sign it. So you need to move it for different tags before it lands in the final repository. We don't need it. At this stage, you don't need it. You don't need it. You're using a PNG. PNG is something which assembles the repositories and the disk images and everything else. We just use the basic Kaji stuff and does exactly the same stuff. And we don't do defaults to do our disk images, but we probably could do that. I don't see actually why not. We don't do modularity support. I don't think a lot of people are using it at this point, but if we want to do send to us, that's probably a requirement. Modularity basically means that you have a base OS, for example, and then if you want to have a different PHP version, you can just select a different stream and that stream is independent from the distribution life cycle. We are gonna do a similar thing with this RME7. We don't support a bootloader specification and it's basically because we're gonna end up using UBOOT at some point, which doesn't do that. You need to use grep2 for that and we're at patches for grep2, but there's no, still missing parts in the kernel for doing EFI. And also if you need to update the kernel, you can do DNF upgrade and everything is gonna install but it's not actually gonna run it if you reboot it because at this point, we're using a BBL and the BBL comes as embedded kernel, so you need to power down, copy a new BBL and reboot the machine. And again, globally, we don't run any tests. Again, we have a lot more QMUs, so it means I can now start to sacrifice them and I can then start running the tests and I can do other stuff which I didn't do before because I needed to optimize it for actually for throughput. Custom bits. We don't use KajiShadow. Basically at this stage, if you're supposed to use a KajiShadow, what it does, it just follows the official Kaji. If someone puts a build, it automatically goes to our infrastructure. It ran for half an hour and I killed it because it tagged something from Fedora 12 and I wasn't happy because it's so fat. So I have my own custom stuff but it probably needs to be rewritten to do a correct, more correct thing. Again, we don't do git to source package. I do it on the x86 machine because it takes just two minutes to do that. It takes one hour, two hours. If we do on the QMU, that's like expensive thing. Also, my script attempts to figure out if you have all the dependencies so we can build it. We want to have more successful builds, not just some builds which fail in two days and because it didn't have something. Yeah, and we have, so all the source code for Fedora sits in what you call the diskit, which basically are lots of git repositories and they have an overlay. So there are some patches which I'm not yet happy. I'm not in the Fedora. So I'm supposed to go to Fedora plus we're just in general some ports. So I need to work on getting some of that in the Fedora. Probably some of that sending to upstream and some of that never gonna land anywhere because the official stuff is gonna be different. We do four separate disk images. If you don't know what you want, you probably want to go to this developer. That's one big image. It still fits in the Z card. Yeah, it has everything. It's a Kaja builder. It has all the tools for 8 p.m.s. It has X11 minimal stuff. You just wanna do a Flexbox, I3, Awesome, all the Emacs, Vim, NeoVim, the Bugas, all the tools I can figure out to debug L stuff, hex editors, compilers, working with disk images. It has a lot of stuff. If you get an image, the smaller image, and you do it enough, make hash. If your silicon is very weak, you might need to wait some time, 10, 15, 20 minutes. If you need to do that a lot of times, you don't want to do that. If you're a developer and you want to use those disk images for your CI or something, if something is missing, but it's not like 200 megabytes or something, you can probably pull it in a developer image and you just get it out of the box. You need it, you want to test it, download the image, you have your packages in it, and that's it. And I also had to change the scheduling because, again, we have problems with IOs and doing images are a very costly thing. So this is basically where you can get the images. The old one, stage four, that we did before going to Kaji, is still very, you can boot them and I know that some people are still using it. You can directly grab something for Kaji. Most are untested, so there might be some issues because it's automatic. And for the root file system, you need to use VDA1 because there is one partition. I cannot actually not, I have to make one partition required the way it's built. But if you use the new ones, I kind of select and test, so I'm going to boot them a few times, I'm going to do some, I'm going to enable ac Linux, I'm going to do QMU, Libvert basically. I know that they work to some extent. The targets we actually support, so again, it's a QMU, Libvert QMU, a tiny move, but again, that needs a bit of modifications and you can also run it on the boards, and again, you saw the pictures from ATUS, presentations and some people are running it, but it doesn't run upstream kernel. A year passed, it still doesn't support the hardware. I don't know how many drivers I'm missing. Yeah, a few. Yeah, so I guess a few more months also needed to get that in, hopefully. But I guess that's 2019 material, it's going to happen. So, and you don't have, for the latest images, we don't have kernel builds for that. So if you're running on the board, come to our ac channel and then ask for as explicitly, probably ATUS has a tree or something that you can pull and run. A few, maybe annoying things about risk five in general. So the same thing that happened is a 64 bit arm, make sure that your config scripts are up to date. The last time we had modified what, just several months ago, basically six months ago. Risk five only does a what size of Tomics. That means there's a hack basically in a GCC. If you use a minus P thread, then there's a lip spec that's going to say, as needed, link with the Tomics, and then you don't have any issues, but majority of the case is going to be minus LP thread, which works on most common architectures, but not on all architectures. Because that's minus P thread, I don't know, some architectures like minus, it's actually P threads, I believe. So yeah, I think there's a bug report, and I think there's an idea to inline those atomic calls, but no one has time to implement that. And I guess there are only two people who might do that. It would be Jim and Palmer. Another thing that we recently found when they debugged a D language, so D front and final end in the GCC, but we had some problems was dynamic sections has to be read only, but that's not the case. As far as I know, as far as the Jim comment, that's a direct copy path from a MIPS, and does not apply to risk five, even if a comment says it applies. Yeah, so hopefully you're going to send the patch to Geolipsy. Yeah, and another thing is you might think it's slash, slash, slash, lip 64, it's not. Even if we are not multi-lib, Fedora risk five is not a multi-lib, but the way the natural risk five works, it's going to work as a multi-lib. So in reality, what the linker's going to expect is lip 64 slash and ABI. We have sim links, so the old way is still working, so it's somewhat compatible, but in general, on risk five, it's like a different situation. And there's a load of plans, so we have a new main server, so it's slightly better hardware. So Polymer finally sent pull request to Kernel, so 5.0 RCT got my audit support, so that's in the kernel, it's working. The user space is posted, but not yet merged. I have a second patch, I know that it works, but it's not yet in the kernel, probably maybe in the next version you could manage to get it done. The D-lang works in the one in the GCC, but I still work in upstream patches, so again, I need to send a lot of changes from our dis-git overlay to either upstream or to Fedora dis-git. If you still don't have add as support in the GCC, so that's probably going to be a next thing to add, and then look into a Haskell compiler and a Free Pascal, enable the tests, enable building from GIT source packages, improving the shadow codger functionality, dump in BBL and go and use OpenSBI in the U-boot, maybe XT Linux, and also Alistair recently joined the Fedora's five channel, and he asked to do more testing for QMU, so also looking into building QMU daily nightly builds for CentOS and the Fedora on the Intel architectures on also in the RIS 5, because there is a TTG backend for RIS 5, so basically you can emulate Intel on RIS 5, I think, and we could maybe do a demo in three minutes. I have no DC, so this is my, so from the way to use, I suggest use LidWirt, because I think it's much more easier than actually running a QMU. This is command from the Vicky, so we're gonna copy it, and so I'm not, I don't have anything running, I don't have any VMs, but I do have images downloaded, so I'm gonna create a VM, so that should be running already. Yes, I probably cannot. Okay, so we connected to the console. For some reason, the latest BBL takes some time before it boots, but it's booting. So let's hope it boots fast. I have no idea which point it is. Nope. Oh, that's nice. So another interesting thing why it's still booting, what I would suggest to use, Sierra log file, so again, our QMUs are dying, usually the CPU stalls, but other bugs, I highly suggest enable that, because if something happens and you cannot log into a system versus Sierra log, and you're probably gonna find the kernel crash or something, so you can send. So I highly suggest you to run that, and it's still booting. No, I'm probably not interested in that. Okay, so it is five, nice. Actually, I think there's one bug in the kernel, so MMUSV48, the kernel doesn't support that. I did on the mailing list, but no one replied. Yeah, it goes from DTB. DTB says it just prints the one. So what we're gonna do, we're gonna have time. So this is a D-language. I'm not a D-language user very much, but it's still the bugger thing. And maybe one thing that we can change is go back to Intel. There's, if get the vp move to something better than a console, is now we should have at least colors. Yes, that looks nicer. So, you see, so we can compile it. So that's in the GCC9, again, we have a D front-end, which almost works by default. It's not yet enabled in upstream, but you have all the patches when you understand what's happening, so we're gonna figure out before the GCC9 is released. So we're gonna compile a word count, which basically is WC command in the D-language. It's gonna take a bit, a few seconds, I guess, like 20 seconds or something. Yeah, so that's a new language. Nice, so what we can do, so again, that seems correct, I guess. And that's also correct, 20 lines, 48 words. Yeah, it works, so that's a de-application running on Fedora 30 raw height on RISC5, the GCC9, which I think no one yet has. Also, there was a talk from Debin, Debin said that the GDP doesn't work in the Linux, so let's do the same with the GDP. So backtrace, this always make a mistake, how many? Yeah, so you can disassemble, you can do registers. Yes, so what happens with Fedora raw height, which again is the rolling, you get the latest kernel, the kernels are built basically a few times a week. You're in between the RISCs, you get the GDP bumps every week, every two weeks, same as the GCC and a few other bits. So it's, I mean, if you want to have a bleeding edge, it's in the Fedora raw height. If you don't want to wait for release a few months, it's in here, Gilep C is also bumped every, at least a week, the B weekly, and it's a kitchen sink. So the developer image is a kitchen sink. And if something is missing and you need it for demos or development because you want to depend on that, again, go to Fedora's five channel or something, tell us what's wrong, and you're probably going to put it in. It's basically, again, it's a one big integration for the upstream. And that, I think, yeah. So these are the companies that basically are the most important. So StoneQlity does our main infrastructure. Facebook gave us a servers, Open Source Lab in Oregon, I believe is hosting a hardware. GCC Compile Farm is managing run hardware. Sci-Fi did the boards and delivered to the developers. And our latest and greatest major new server is also going to be hosted by Academic Computing Club in Umea University. So hopefully that new server is going to handle the load better. And again, if you have any questions or you're missing something or something is not working on the test something, whatever questions you have, it's five or not, you can go to Fedora's five, you're going to find people from Sci-Fi, there's a Polymer, there's Jim, there's a DJ from Gilep C, there's Alistair now, we have Debian guys, we have Parabolic new Linux, we have Open Madriva guys sitting, so they have a lot of guys from all the places, they got almost 40 guys. And if you have a question with a risk five, you're most likely going to get an answer in that channel. So that's it.