 So after the presentation, you know, go off, have fun, I will do a real demo. So, you know, enter a jail, build a package by hand. The presentation will be more me doing this a lot and showing slides. And then at the end of the slides, a few questions. And then we'll do, I'll actually use a real emulated jail, build PKG or some other package that doesn't have a lot of dependencies, and one that I know builds. And take your question, yeah, PKG, it'll build. It has to build, otherwise nothing would work. It would be bad. But I just kind of want to show, for ARM, this will work, for MIPS 32 this will work, and for MIPS 64 this will work. If there are people who are interested in PowerPC on FreeBSD or Spark 64, the QEMU emulator needs a lot of help, and you can come talk to me about that after the presentation. But there's some very good things we can do with this emulation to keep alive architectures that aren't x86, that aren't Intel. All right, go ahead and get started. Good afternoon and thank you for all coming. The last presentation slot of the day, always fun. I'll be presenting today a status on cross-building in a way for architectures that are not x86. And specifically I'll be talking about ARM v6, MIPS 32, and MIPS 64. We'll be using kernel side code, user land code, QEMU, Pudrier, PKG, and we'll get into a little bit of demonstrations of how this works as we go along. The kernel components, I'm going to, I want to kind of show some of the code that's involved in the kernel to actually intercept execution, interpret the elf header, and redirect execution to an emulator. It's very, it's very cool. The user land components is the tool binmichctl, is the tool you use to configure the redirection. You input a certain care elf header and an emulator, give it a name, and then that instructs the kernel what you want executed when the user tries to run a MIPS 32 binary on your AMD64 system. The emulator I'm using is QEMU. A lot of work has been done on the BSD user mode. The free BSD user mode, the net BSD and open BSD user mode tools are not, they're not in a working state that I'm aware of. The final user land component that I'll be using is, as of eight hours ago, Pudrier developed. It's a package you can get online now. Using NullFS and Cherute, along with Pudrier, you can trivially create jails that allow you to Cherute into an ARM v6 or MIPS 32 environment on your machine. If you don't want to build packages but you want to test your own code for whatever reason, this is going to be a technique that you can do right now on free BSD head. Any project of this complexity took a lot of people. I have been the sheepherder of gathering code from other people, committing it, pushing it upstream, and getting a testable version of everything. The original implementation and idea was coded out by Stacy Song. Juergen Locke, the current free BSD maintainer of QEMU, has tens of patches to QEMU to get it to the point where it's stable. Ed Mass, the free BSD foundation, helped in several icky situations with QEMU to get it stabilized. Peter Wem walked me through compiling one piece of assembly and then decoding it to figure out what the signal trampoline address was that I needed because it was hard and that was very useful. Alexander Kabyev sent in numerous QEMU patches. Adrian Chad didn't help me because I needed to learn how to do this myself, but he guided me to the right piece of code to modify in CISCURN. Deist punished me for all this with a ports commit bit, thank you. Dimitri, vital to getting clang stabilized to where we could do this gruesome, unbelievably confusing way of building things. Andrew Turner is currently working on GCC and other ports and linking libraries. Michael fixed my SQL for users on ARMv6, which is going to be vital. Brian has been helping me with powder keg itself. Warner Losh actually made a tool chain of AMD64 binaries that outputs ARMv6 so that I could put that into the jail so that CC is not emulated and it runs much faster. Ben LePore has been decoding assembly instructions and helping me with clang. And Brooks Davis inspired me to do this spring at AsiaBSDCon when he pointed me at all this stuff. And we got the project off the ground. So kernel components, the first thing, there is a kernel module that's built called ImageActBinMish. It is a kernel module that hooks in to the execution layer of the FreeBSD kernel and allows you to execute any emulator, any other tool you want. Through the user land tool, you can completely break your machine in a way that is new and exciting. You can make it completely and totally unusable by anyone outside of yourself. Because you can say, if somebody executes this type of binary, run it through an emulator, which can be a shell or a script or anything you like. It doesn't have to be QE. So it's fun. The module itself looks at your argv sub zero, looks there, and then pushes everything out and then drops in your emulator at the front of it before execution. That's really the magic. It's manipulating your execution string, pushing it out, and then adding in QEMU in my case. Now, I have not made this human readable for a reason because it's not. It has a man page. You can look at it. It works off the following syntax, and I just kind of want to go into it because it's important. In this example, I have a MIPS 32 and an RMV6 execution. You will run this as root on your machine. You execute this. You give it, you say you're going to add a redirection. You're going to call it some name, call it RMV6, call it Bob. We'll call it Bob and it will be good. You say, I want this redirector named RMV6 to use this program, QEMU, if a program is executed that has this elf header. That's actually the elf header for an RMV6 binary in FreeBSD. Match on the following bits in that header, enable it, and you're off and running. Hopefully, you've installed QEMU by this point. This is the mechanical bits that actually get you to the point of, okay, now that the kernel is actually configured, you can say, give me a list of the things that are installed. This will tell you what will be done. If I add a redirection, here is actually how you can see if that redirection is active. QEMU is the emulator that I am using. If you have an emulator for an architecture that you want to use, it should just work. I don't know of a reason why it won't and I'm more than happy to test other emulators. Right now in ports, Juergen has created the static user version of QEMU that's the one that we're using to build packages. It's the one I recommend because it builds a statically linked binary that can be copied around anywhere in your file system and we will be copying it around anywhere in your file system very soon. Emulation is hard to be able to do. You have to really be prepared to learn how a computer works if you don't, which I didn't. Now I definitely don't know how computers work. The main thing for FreeBSD, NetBSD, OpenBSD is having QEMU understand read-write operations on the different IOCTLs, what the syscall numbers are, what they mean, and how you actually execute. That's most of the changes that we've been making have been, oh, we have this type of file system property. We need an IOCTL for that. We emulate that and return some value. A lot of sys controls will return AMD64 values to a 32-bit environment so they have to be shrunk otherwise bad things happen. Those are the kinds of problems that we run into. Pudriere develop is the easiest way that I know of to build a jail for doing this on your system. If you want to use make and you want to build your own jail, good luck. Have fun. You'll figure it out. It'll be a great journey. Pudriere develop already knows how to do all of this. So if you want to figure out how to do this by yourself, it's a great journey but we've made Pudriere know how to do it. It knows how to configure the kernel. It knows if QEMU is installed, it will copy the right binary to the right jail. So you don't end up with a Spark64 emulator in your MIPS32 jail. Pudriere will keep a backup of your jail so that when you break it and you will, you have a way to go back. And please, use ZFS so you can just roll back and be done and move on. It's much easier. You need more RAM, okay, you do. I run it on my laptop, it works. The Pudriere syntax in Pudriere develop will be something like this. If you're familiar with the syntax of how to build packages now, it should look very familiar except that you're going to be passing in an architecture target, target arch syntax into your command line. The dash x will build you a tool chain to help accelerate the builds because it will be native to your host as far as binary is concerned. But then it will output the architecture that you want so it will be a little bit of cross building. It makes it much faster. And then of course check out your ports. So these are three little, these slides are available. I sent these off to Olivier, they should be up already. These are the commands that I run to create the jails. So if you want to do this, you can do this right now on your FreeBSD11 desktop. When you execute these commands, Pudriere will give you these file systems and then you're going to mount the appropriate devfs and nullfs mounts into your jails because you have to have those when you cheroot. You can literally just cheroot in like you would any other jail and you're off. You have a tool chain, you have all the things. Yeah, but there you go. Pudriere dash s? Yeah. Okay. And done. And then you're in. You don't have to do anything else. Do what he says. Pudriere dash s. So this gives you the, I mean as you can see I'm on the same machine in the FreeBSD cluster. I'm AMD64 and now I'm ARM and I can start building and doing stuff. So the bulk demo is done real briefly. For those of you who weren't here, we just finished a build so my timing was off a little bit. We built on this run using Pudriere and the jails and QEMU and all of these tools. We built 5,000 new packages. We were unable, we failed to build 364 because they're broken. And we skipped 4,000, 4,600 because probably GCC and Java. Let's see. Let's take a look. What is, what is the worst offender? Java, you come fix it because you don't have a good start. Indeed. So now we know and now we know these will not build on ARM. We don't have a GCC tool chain, libvpx doesn't build. We can see what packages depend on that. So now you know the scope of the work. You know how much work there is to do to get what you need working. Obviously the Linux emulators won't build. And then the failure ports also give you interesting information. And so we've been working through this list in various places. Some of these are build failures. Some of these are QEMU failures, like core dump. Probably not the code itself, probably QEMU needs to be adjusted. Things like linker error, that's probably the port. And so we've been guessing and fixing and changing and getting ports into a place where we can actually say, yes, you can run FreeBSD with packages on your ARM machine. Maybe. So that was an Nginx server on the back end. The man page for Poudrier has the configuration, or is it in share examples? It's in share examples. So that web front end is available to you if you install Poudrier. If you have never installed it, it's really useful. The core dumps and the question marks and the bad C++ code, those errors are probably QEMU. Probably. And that's where the sharp edges are. That's where the interesting things are that need to be fixed in the, need to be fixed in. So you can actually charude in, type make, and it will fetch the packages, and it will build them, and it will do stuff. We'll do this in a moment. I have failed you all, and I'm sorry. I have not written this down good. There is not a good explanation of this. The binmichctl command, I don't know how you would figure that out if you didn't go and look at the FreeBSD wiki and copy and paste from there. If you understand the L format, oh, well, of course. That's the L format. That's what it is. So we need better documentation for this. I need to spend time and update the man pages. And I would like to add shortcuts to the command, binmichctl, to where you just say binmichctl, give me an arm thing. And you don't have to enter in all the rest of it. It would just do the right thing. So for the future, what's going to happen soonish, I have been promised clang for MIPS, over and over and over again. Clang for MIPS will help a lot. Soon we'll be having ARM64 support in QEMU and FreeBSD, which means we'll be building packages for that. I will be adding some debug handling for IOCTL operations that are not supported in QEMU. I stole some code from John Baldwin of the FreeBSD project, and I'll be putting that into the debug version of QEMU so that you can get useful error output. And I'll show the unuseful error output here in a moment when we do the demo. So is there anything before I do a demo, what would you folks like to see with this? Is there anything that is more interesting than something else, or have I skipped something? Or do you need a coffee? Should it be cooler in here? Should we have music? And beers. Oh, god. Seriously, 445? And there's not a case of beer here? Anyway, are there any questions at this point? Yes. No? Yes, so what'll work on 10 right now? Everything? So on 10, you can do everything except dash x. And if you install PudrierDevel now, you will have one thing I have not MFC'd from current. So I added some features to the image activator code that's generic that I have not MFC'd to 10. I had no intention of MFC'ing it. You can build some packages. But you will see interesting failures when you try to emulate a shell script. Because you need to emulate bin sh. But you've already put QEMU in front of your shell script. But then you don't do that because it is a shell script and it fails. I think you will see that. I'll show you the code in a little bit. So you'll be the most thing in that case? It fails. It just dies. Yeah? So you will be the most thing? Well, it works in a special case that yes, you can. The failure case will be if you try to use my fancy AMD64 tool chain. Because then it's how does it work? Yeah, hold on, I have to get it correct. Because there's a lot of layers here. So the AMD64 tool chain will call make from the AMD64 tool chain, which will end up calling into 32-bit ARM V6 shell scripts. Sometimes it will do it in such a way that only the shell script image activator will fire and not the bin Mish activator will fire and it will fail. And you will be like, why did this fail? And so that is some code that I've made a change in current. And we'll look at it. I have a diff and commits and all kinds of things. But we'll go into it. And you can do some of this on 10.1. So if you have 10.1 installed or 10.0 installed right now, I think you can get away with some of this. What do you type make? It's fine. It's fine. Any more questions? Alan? No, I'm baptized. OK. They're right there. So don't call him. Don't call Baptiste. Don't call Port Manager. You can yell at me. And we'll do some bugzilla stuff and whatnot. What's that? I get a tutorial because it's, look, my documentation sucks. I'm sorry. I actually do feel bad. This slides is the best documentation that exists. You're welcome. Questions? So far? OK. Please. Pudrier-Devel, yes. It will. And we will do that. I will show you at doing it here in a moment. But yes. Pudrier-Devel, as of this morning, does all of the things that are needed to get started. And then, yes. If you install Pudrier-Devel, it will add. When you go to build Pudrier-Devel to install it, it'll actually ask you if you want QEMU. It'll ask you if you want to install the QEMU port. It's an optional dependency. And that's what it's there for. When you execute bin-mich-ctl, it will load the image activator for you. So just executing bin-mich-ctl activates, loads the appropriate module for you. I'll do it on my laptop. It'll be great. Because I can't connect back to the US right now. It's very slow. Questions? Really? OK. So we're going to leave this machine. Maybe. OK. We've left. All right, so I already have a jail built, because I cheated. Because I think most of you know what a build world looks like. I can do it if you want me to. But I think we all want to leave and get beer. I've also cheated, and I put those into my rc.local, because I don't want to ever have to remember how to type this. If somebody would like a copy of this to turn it into documentation. So this allows me to emulate MIPS 64, ARMv6, MIPS 32, and work on the development of PowerPC 32, PowerPC 64, and Spark 64. So those are all. So the fact that I execute these commands at host startup means I already have the correct module loaded. So I don't even have to think about that anymore. The interesting code I want to show you before I start building. OK. So Stacy wrote this code. The interesting bits, just as kind of what's going on here. When you add an entry, there's a whole bunch of data structures being set up. When you search and find, this is the way that it's going about doing it. When, is this the execution part? A little bit farther down. OK. So the kernel executes these functions here. So the kernel is going to execute the registered handler, which is, so on module load, it's going to load the initialization routine. And then when it exits, this is the actual mechanism that the kernel uses to register arbitrary functions and execution through sysinit and sysunit. The actual image, the image activator, this is a common data structure in FreeBSD. You can redefine your own image activator to do whatever you want. It's done through this mechanism. The macro exec set tells current exec, do this when this type of file is executed. And in our case, we're going to execute imageactbinmish and do all kinds of interesting stuff. This looks like the main, yeah. So in here, we look for the actual, does this binary match something we know about? If it does, we're going to mess with argv sub 0. And that happens down here. The shell script stuff I was talking about is up in here. There's actually code that looks at the first character of the file. If it's going to execute a shell script, we have to behave differently than if you're executing a binary. We save off the argv sub 0, or we exit and we don't know what this is. Is that legit? Can you read? Not even a little? Or do you think 14? I like to show code, because code is easier to understand. This code, all it does is take the execution string that the kernel is about to fire off, make it longer, move the actual execution string down, and put in your emulator into the execution string. That's its entire job. And this is the code that does it. Let's see if I can do this without typing my root password for everyone to see. Super secret. So right now, I have that. So I will not do it the Baptiste way. I'll do it. So we do this. So now all I've done is I've mounted a dev tree. I've mounted the ports tree in. And I'm just going to cheroot in. So I'm here on my laptop, now in an ARM box. We really need to change that, because I'm lazy. Let's see if we can get some messy IOCTL information on it. Oh, I'm still building 137. Good god. What's that? You are outdated. Outdated. I'm old. That's bad. Now, what is this thing doing behind the scenes? Yes, what have been? That was good. That's a good combination to leave a column as well. Oh, I'm sure. I don't remember that. You guys don't. Notice the QEMU is not running. Oh, there it is. Oh, it's gone. Because I'm using an AMD64 tool chain. And the QEMU is firing off to run shell scripts, configure, anything else? What else would PKG be doing? So things are happening. Oh, GZip, yeah. Let's see. So we've been kind of considering some way of putting Perl and Python in here to do the right thing. Maybe Shell would be a good idea. Well, let's see. Right now. So these are some variables I learned about. Did you know you can do that? Of course you did. You're the one who told me, I think. I think, did you? Was it Warner? So Pudrier with the dash x does this to the make.conf in the jail. And this is saying that when somebody needs a compiler, use this one. It's overriding user bin CC. Because if we look at versus the back audience. So this is what I was talking about. User bin CC is still there. It's ready to be used. It's clang. But it's going to be fully emulated. It's going to be very slow. What we have now in the base system is a new target called, if you make a native x build with the correct target, target arch, you'll actually get a compiler tool chain that you can move into the jail. And it'll do the right thing. And we can use this to speed up builds a lot, along with flex and sed. All of these are common tool chain components that are going to be run over and over and over again. So we went from three weeks down to one week for a full build. And that was primarily because of that. What would you like to see? You have me here. Or you can leave. No, LibreOffice. Get out of here, madman. Someday. So yes? I was going to ask a follow up. No. No. Well, some of it was. A lot of emulation for us is shrinking values. We have to truncate values, reverse values. So it's very similar to I3D. Because it's ARM32. It's very similar. But in some cases, there is special code in QEMU to handle changing hw.fizmem. So it doesn't get reported to in here as 128 gig. It gets reported as maxmem. So we cheat. So some of those are true. But some are just copy, paste. No. It's standalone. So it will break when we change things in the kernel. It'll be great. What is a standalone port that I would not build very often? The small one. You think? I'm trying to think if I actually, oh god, I've actually built this for some reason. So if I, RM, user, ports, dist files, yeah, I do. No, I want to delete the dist file. I absolutely do. Yeah, I want to show what I mean. Thank you. See, I'm just a new ports committer. He's my mentor, so he has to tell me these. See? Why? The first we cleaned the dist file, the second we cleaned it. I already removed the dist file. OK, let's see if this does what I want it to do. Where's my internet, man? It could be read on the call. No, that's fine. It's fine. What's that? What are we saying? You need to resolve them. In here? I already put it in there, don't I? That's why you do. I'm trying to remember how old this is. All right, fine. Fine. Fine. I'm going to be difficult for the sake of being difficult. Oh, there's not two ends in it? You know, I'd be done already if I just used jail-ass. I don't know. Ah, there we go. That's what I wanted to see. These are the sharp edges. So good, I don't actually have to fail anymore. So my next goal is to turn this error into something that is useful. And you can go see what needs to be implemented. And I can generate an error. So I mean, these don't mean anything, right? Yes, so all I did was, in the jail, do if config. And all of this stuff comes out. So these are a lot of the errors that are causing build problems. This is an ongoing maintenance task. So when you add new syscalls and new ioctals, remember, you're causing me more work. Anyway, any questions? Anything you guys would like to see? Do you want to leave? Because we can go. No? Thank you, everyone. That's good. Thank you.