 Two, one, and away we go. Okay. Thank you all very much for coming along. My name's Jeremy Bennett. I work for a company called Embercosm. You may well know us as a compiler company. More recently, we've started to move into the development of operating system software for embedded systems, which is our specialty. I'm joined in the audience by my colleague, Mark Corbin, who heads up that initiative. This is the first time we've actually dipped into free BSD. We're more Linux, FreeRTOS space. This is me with a project manager's hat talking about our experience. So this is the platform we're looking at. It's a quad 64-bit risk five processor, no floating point, it's an integer platform, and sitting alongside, it's got a 32-bit integer risk five platform, providing some general housekeeping functions. This is proprietary hardware. The hardware is not yet available. So we've started work with the high five unleashed board, which is a quad, well, it's actually a five core risk five board that you can get from sci-fi corporation. And there's its schematic, it has essentially four cores, the U54s, and it's got a housekeeping core, the E51, then in this case it's a 64-bit core. It's also got floating point on the core, which we're not going to use. So that's our hardware for prototyping this work, and before we even get to hardware, we're using QEMU, which supports risk five, so we've got a soft environment, and that's quite good, the high five boards cost about $1,000 each, they're scarcer than Hen's teeth, so we've got one of them, and Mark there operates with full anti-static risk bands when he's within 25 meters of it. So this is only a short talk, I'm going to whistle through things, we'll take questions at the end. Broadly, if you want to build free BSD, you build your user world, and from that create a RAM disk, you then build the kernel, and tell it where to find the RAM disk, so it knows where to jump to, and then in the risk five world, we use the Berkeley bootloader wrapper, which adds in the device tree, and the result is you have an image to load, and you can then put that on your SD card, plug it into your high five unleashed, and in principle it all works. And to build that, we have a compiler tool chain, Clang LLVM, which is the default compiler usually for free BSD, typically sitting on GNU binutils and the GNU debugger, and if you're deeply embedded, you really want it to use it as a remote debugger running GDB server, and you need at the very least C libraries, probably C++, but certainly C libraries, those come from free BSD, and we'll see why there are some issues around that in a minute. And we took this project in incremental stages. The first thing was to make sure we can reproduce what the suppliers do with the board, so if you get the high five unleashed, it comes with a GNU Linux system on it, and built with GCC. And the first thing is to make sure you can rebuild that and do that, that's nothing to do with free BSD, that's just checking that what you've got on the board works, and there's a certain amount of who are just getting that to work, but we found fairly straightforward in the first day, we were able to get Linux built from source and put back on the board, and it still behaved like when it arrived with us. The next stage then is to try to bring up free BSD, but we did that with GCC, and the reason for that is that the LLVM for risk five 64 bit is very, very new, and we didn't want to be fighting compiler issues while we're trying to bring up the kernel. And so we first of all started with GCC, and once we've got the kernel working, at least on QNU, then we could switch over to LLVM and check it all works. And so that's the loop we did for 64 bit risk five, and then when we're going around it all again, for 32 bits, so we got 32 bit risk five LLVM and so forth. So what could possibly go wrong? Free BSD has already been ported to the 64 bit risk five. Thank you to Ruslan Bukin, who kicked off the project at Cambridge University and others who've since contributed. And Clang LLVM is the free BSD default chain, so that ought to be fine, and thank you to the people who've worked on that for risk five. But what could possibly go wrong? Well, like all things, you develop something, you document it, Ruslan did a good job, and then someone picks that up and finds all the things you didn't quite write in the documentation. The documentation, as you'd expect for something that's developing, works with head, but quite often head isn't gonna work straight away. So actually, if you're gonna start from the first sketch, start from a stable point. There were some other things. It validated the users against the build machine, and our build machine and the actual machine we're building for are different, and that didn't work, and it used the machine user-based database rather than the target database to build the file image. And when those were corrected, and those are fairly trivial things, Fires Up runs multi-core free BSD on QAMU, and that was early stage, and early win, we'd got free BSD running on risk five on QAMU. But then we put it on the high-five unleashed board, and it didn't work. And the reason is those five cores, the CPU zero is this funny controller core, the E51 and not the U54s. And that starts up the system, then goes and masks itself out and goes away. So we had to modify the free BSD system to not assume it was running on CPU zero. And with that fixed, up the high-five unleashed board came and it booted single core. So now we've got our prototype system working, but then we ran into problems, and this is where we're still working on, multi-core boot doesn't work. When we try and bring up four cores, it usually doesn't work, except sometimes it does. Sometimes it boots and it works. And the reason is there are four cores on the high-five unleashed, and the order in which they decide to come up is random, and it's only if they come up in one particular order does it seem to work. That seems to be best if you've left the machine switched off for a while, and then it starts. So we need to, we're in the process of working on an abstraction layer to try and map the CPUs to come up in a consistent order, at least as far as the operating system sees them. And as far as we've done the debugging, and this is where it becomes clear, this is the first time we've tried to do free BSD, this seems to be a problem with the enabling of interrupts, and it's connection when the interrupts come live, that's when things go horribly wrong. That's a work in progress. I had hoped that I would be announcing it's all working by the time we got here today, but we haven't quite got there yet. But we've still got a prototype hardware working on a single core, and that's quite good for a lot of things. But the actual bigger thing was, and why we originally got involved with this project, because our reputation is a compiler company, for Clang LLVM, and we do GCC as well, we do both of them. And the problem we had is that we were asked to start on this project at the end of October, which was when the first patches for Clang LLVM for 64-bit RISC-5 were actually published. So we were trying to not just work with a Clang that wasn't ready, it was brand new. And we actually had to submit seven patches just to get a Clang that would actually build free BSD. A PC relative addressing wasn't working, position independent code wasn't working, thread local storage wasn't working. So that got us to the stage where it could build. Since then, my colleague Lewis Reville, who's actually sitting in the RISC-5 dev room at the moment, who's not here because this relates to RISC-5 as well, has submitted another nine patches. So we've had to put 16 patches up there. We think there's one bug left. Well, there's one thing we think that's not working. We hope it's just one bug. But actually, this has been a great driver and thank you to the customer who's paid for this work because these open source projects often rely on people actually paying us to do the work. 64-bit RISC-5 LLVM has actually progressed incredibly fast. So we got it all compiling. The compiler seemed to be pretty much good and it still would not run. And the reason came down to compiler RT. Now, for those of you who don't know, compiler RT is the low-level emulation library for LLVM. So if you don't have hardware floating point, you do it in emulation and you do it in compiler RT. And RISC-5 is quite clever. If you've got a machine that doesn't have hardware floating point, what it generates is hardware floating point instructions and they cause a trap which goes into compiler RT and then you emulate them. The problem was that one of the compiler RT programs actually doing the emulation had hardware floating point instructions in it, so it promptly trapped and called itself again. And shortly after this blew up, once we'd realized what the problem was, it was fairly easy to fix, okay? And when we get that there, then FreeBSD runs quite well. You can fire it up, it boots up and it all works nicely. But after about 10 minutes or so, it tends to lock up. And it's a kernel issue. There's something with the compiler and the kernel doesn't work because you can build the kernel with GCC and everything else with LLVM and it all works just fine, okay? So we think there's an LLVM compiler bug. Possibly it's a kernel bug that LLVM is technically correct, but only LLVM exposes. So we're still trying to track that down. Three to five days more work, we think. Now those awkward compiler bugs can't take a bit of tracking. So we've got to fix that, okay? So far so good. We've got a compiler, it more or less works with LLVM but we've got a bug to fix but everything else outside the kernel works with LLVM and it runs, okay? One core on a particular hardware board. But we're not in the business of hey, it sort of works. We're actually into serious professional development. That's what we're paid for. So we want to test and CUA is the FreeBSD test system. And one thing we found with CUA it really doesn't build cross-compile build. It expects to be built native. So we had to create a compiler tool chain to run on the actual QEMU. So we had to build it so we can cross-compile the compiler tool chain and then we had to run that GCC compiler on QEMU emulating RISC-5 to build CUA. Now CUA is quite big. The RISC-5 emulation is quite slow and so we had to leave it several days. That binary is now built. It is many copies. It is locked away in a vault in one copy so we don't lose that binary. We do not want to have to build it again. But the great thing is we now have CUA running and we can do nightly testing. And it takes about five and a half hours in emulation on a 20-core Xeon server. But that doesn't matter because we're asleep at that time. It can go away warming up the machine room and we can nightly test. And that's good. There are bugs to fix but broadly FreeBSD is working. Okay? So what else could go wrong? And the answer is debug. So at the moment we have to debug by putting GDB on the target and GDB works fine on FreeBSD. But that does presume on a deeply embedded target. One, we have a fairly fully featured FreeBSD and secondly that we have the sort of console where I can sit there and it'll put up something I can browse on and so forth. And for a deeply embedded system particularly in a secure application which this is you tend to not want to have more on there than you actually need. And that's why for embedded systems you have a program called GDB server. And GDB server runs on Linux and it runs on LinkSOS and it runs on one or two other things. And it used to run on FreeBSD. But no one's run GDB server on FreeBSD for a long time and when you run GDB server then that small little program which doesn't need much operating system resource runs on your thing and you connect via a serial line or a TCPIP to GDB proper running on a workstation somewhere else. And that's the standard way to debug an embedded system. And the code's been taken out. So actually we're in the process of rebuilding GDB server, reconstructing it for GDB. It will be a new, better GDB server because it will reuse all the FreeBSD from native GDB. And there's two parts to that. One is to get it working for x86 and then we will port that to risk five. That's a few weeks work and because it's not critical to bring up this project which is on quite a tight time scale it won't happen till later in the year. We do need it eventually. We can live with native GDB for the short term. So that's the last thing that we had to fix. So where are we? The answer is we can run embedded FreeBSD for RV64 built with LLVM. There's a reference implementation for the high five unleashed. But it has some limitations. It only works on a single core on the high five unleashed and we're still trying to nail that down. And if you're an expert or want to help us we're very happy we've got money we can hire people to do that. So for people who need to be paid to do stuff we understand that it's a perfectly reasonable requirement we can do that. So come and talk to me. And it's unstable with LLVM and we should knock that compiler bug out in the next week or two. Okay. I would like to have been able to say that the reference implementation will be available there. I'm afraid I've been traveling and it's sitting there and I haven't actually got it onto GitHub where people can help us. It will appear in the next day or two. If you need it please shout at me because that will be a motivation to make sure we don't forget to do that. Okay. So where are we with FreeBSD for 32 bit risk five? There are a number of issues here. One is that we can't pick up someone else's work who's done the hard bits for us. So we're actually, Marx has been plowing through making sure all the bits are needed to run on the 32 bit risk five. And because of the bug with LLVM we thought we'd go with GCC. And GCC doesn't work because 32 bit GCC doesn't have an int one, two, eight type. And that's rather baked into compiler RT and GCC doesn't believe it should be there. And GCC unlike LLVM doesn't have quite such a fluid view of what types are. We're working around, do we make that conditional inside compiler RT or do we just say let's go with LLVM and hope we fix the bugs soon? And the other thing is we don't actually have a suitable RV32 hardware platform. We've got a software platform obviously in QEMU and we're not quite sure. If anyone's got a good suggestion of a 32 bit risk five platform that has an MMU on it, that'll be good. So that's work in progress, watch this space, but we will bring free BSD up for 32 bit risk five. And so that is my talk, it's a whistle stop tour. Questions? Yes. I recently ran into issues on AMD64, which is the most supported platform like record dance. Yeah. Full kernel dance were practically unusable for me. Did you have issues like that with panics and cordons? Could you return with GDB? So the answer is we certainly have panics. Have you been looking at them with GDB mark? Then it was only an exercise just to make sure it would build that way first. So we didn't really do any extended testing. So the question was, can we, for the record, is, can we use GDB to look at core dumps? Because there's been a problem with that on AMD. And the answer is, we haven't had to use that yet, so we don't know if there is a problem. Other questions? Yes. So the question is, do we see commercial interest in a 32 bit version of risk five? And the answer is yes, but we don't know if there is a problem. So the question is, do we see commercial interest in a 32 bit version of risk five? And the answer is yes, because we're being paid to do this work. And this product, and the product we're going, and it's very similar to many other risk five architectures, which is where your core engine is a set of risk five 64 bits. And sitting alongside, you have a 32 bit sometimes engine doing housekeeping. It may be doing slow speed networking. It may be doing time as watchdogs and so forth like that. And that's a setup, for example, used in the low risk project. And commercially, you're probably using free BSD because you're concerned about security, network performance. And if you're going to have free BSD on your main cores, you don't want a different operating system on your housekeeping core. So you want free BSD everywhere. So since that's quite a common architecture setup, I would expect that to be a requirement. And actually, though there has been a focus on the risk five foundation on 64 bit operating systems. The feedback we get from customers is if you're embedded, actually there's still a lot of people want to use 32 bit there. Any more questions? Yes. Is there no equivalent to GDB server in the LLVM? Good question. The question is, is there an equivalent to GDB server in the LLVM project? The equivalent to GDB and LLVM is LLDB. And we get asked about it the whole time. There are things that LLDB does quite well like native debugging on x86 and to some extent on ARM for Apple Core. You can do remote debugging with LLDB. We keep on looking at it because we'd really like to be able to do it and offer it to people. My feeling still is that it's quite a long way behind. On the whole GDB servers come in all sorts of flavors. GDB talks a very simple serial protocol to say, read memory, write memory, start a program, stock a program, read registers, write registers. And that's basically a GDB server sits there processing those. The GDB server application is a Linux application that's been ported to Linksoft and used to run on FreeBSD that provides that functionality. There are lots of other GDB servers. OpenOCD is an example for embedded systems. We have our own one for embedded systems when you're running bare metal or with real-time operating systems. So I think it boils down to there isn't really but Apple sort of have one that does that sort of thing and there are other ones out there. But it doesn't really help us. The solution is to get GDB server running and it should actually compile with LLVM in the end anyway. And we're out of time. Thank you.