 how's that is that coming is that coming now we're is working now we are going to hear third talk in this room but just a little reminder I hope you are tweeting don't forget there will be a grand finale at 4.30 in the room D105 with special prizes we have some posters up for grabs if you're interested with this let me pass the floor to Jim Perrin who's going to talk about arm thank you so my name is Jim Perrin I've been with the Centos project now for a little over a decade and in that time I've taken a few different roles with the project most recently I've been working on handling arm builds for the data center I apologize if I sound a little hoarse I am you guys are lucky this isn't a talk happening through interpretive dance it was a bit of a rough night you don't want to see that so for about the last year almost exactly a year to the day I got my first arm 64 box a bit hush hush here sneak this home from foster work on it see what you can do with it see what you can make happen and so we begin the initial build of Centos for 64-bit arm hardware we started with the 7.0 release and plagiarized quite heavily from some of Fedora's patch sets thank you Peter but the majority of the 70 code built there were a few issues that we ran into most of them were easily solvable a few patches here and there to pull in that all worked fairly well and then I got a call from a few of the the red hat guys saying what are you doing wait until 7-1 why we can't tell you that just wait until 7-1 okay fine the beta is already out it shouldn't be too much longer so I still continue to do a lot of the build but I kept it kind of private do we have any not is there anybody here who's not red hat in the room who's not in the family okay so while this was going on I still continue to build but I was kind of keeping my eyes on the package sets in the package lists for 7-1 the the sources for well 7-1 drop and miraculously hey there's a rel so port for it we have full-on rel sources and a lot of the packages or patches for arm looks suspiciously like the things that I've been you know ripping apart from Fedora and back porting into the rel patches so it the other direction it it it built a lot easier the second time around but we had issues with I can't say we had issues the vendors that we were working with seem to be having issues mostly focusing around distribution they were trying to get a lot of the the red hat stuff out there they wanted to be able to give it to customers they wanted to be able to get that the testing out there and they were struggling with being able struggling with being able to to do some of this so with what we have we offer essentially a more easily distributable method for pushing a lot of the same things what I'm trying to get to is a place where we have full parity with x86 64 architecture on arm for data centers I want it to be just bog simple and easy to use there should not need to be a different set of instructions or installation methods or anything else the hardware shouldn't the installation for the software should not be the differentiator the hardware should I want the chips to compute on their own merit if the low-power benefit for arm or more cores for arm makes a difference great if you want power great go for that but I want the chips to compete not necessarily the software stack and who in here is familiar with a pane of you boo other than Peter so that's one of the things that initially we have avoided dealing with my focus initially was on arm in data centers we've got a pretty good handle on that now so these days we're focusing a little more as as more 64-bit arm consumer boards come out we're getting more requests from the community yes keep box thank you the pine 64 things like that as these boards are coming out we're getting requests for the 64-bit build of arm on these devices but I've been focused on UEFI and server boot standard specs that the different things and it's nice to not have to deal with vendor specific things when you can just say this is the standard this is how we're doing it please come play now there's a threat but it's not all fun and games I've been playing both sides of the fence it's kind of what I do the vendors that we've been working with have also been trying to get additional patches in I don't have to worry about SLAs we're community so for some of the hardware vendors we're able to move a little faster we're able to pull in additional patches that either the fedora folks may look at a little I said additional patches what what how you interpret that is entirely up to you some of the patches so one of the vendors and I'm I'm not going to name names because as someone so kindly reminded me there is video and there is a microphone and I got I don't want a lawsuit one of the vendors when I told them that we would accept patches and push faster kernel versions or at least faster patched versions of the standard sentos kernel they said great and they shipped me one bulk I want to say it was like a nineteen thousand line patch for the kernel there were multiple patches one of them was around nineteen and it was like oh yeah that whole ACP I think we don't need that we're kicking that out we're going back to device tree we're doing this we're doing no in no way will I accept this patch just full stop absolutely not I don't claim to be a kernel developer let me ship this to a few other people but this doesn't smell right to me I mailed it to a few of the other folks who screamed and then started laughing so I felt I was right in kicking that back but that was just one instance we have a number of other there are four other vendors that are working very well with us and for the most part they are just pushing slightly newer versions or added support features for the hardware that we've already got so some of the additional newer patches for Mustang to support IPMI things like that that's great bring it on and so what we're doing is we're running a bit of dual stack development we have the traditional sentos focused build on arm and then each month we're putting out limited feature release improvements that are mostly focused around the kernel but the occasional tool set patch and so we've been doing that now for roughly six months and we've had a lot of community uptake for it in handling the server world with that we are I'm completely ignoring my slides here so yeah essentially what we're doing is distributing the base distributing updates extras we have a few in that I haven't thrown a lot into extras mostly because a large chunk of that should be coming from Apple so I've been putting on my other hat leaning on the Apple release engineering folks that the Fedora release engineering folks and saying hey can we please get Apple rebuilt for arm and they keep telling us no we're focused on other things like power and your magical requirements for functioning hardware that might be supported you know I'm not sure we're doing the talk dynamic correctly is the speaker supposed to troll the audience or is it the other way around I'm okay mm-hmm so the other thing that we have is an experimental repository Peter you're probably gonna want earmuffs for the next little bit of this we have been so for that the 7.1.1503 release that was primarily focused around Docker builds the initial go-lang bootstrap to get 1.5 support for AR-64 built was unique and special in all sorts of interesting ways little details like hey maybe the page size shouldn't you know might maybe want to match what everything else is doing on the system so we got go-lang support built in we got Docker support built in it it works it is mostly I said mostly it is mostly operational as long as we're not getting unique with all sorts of fun Docker things so that's where we're putting the experimental repo community the other half of it is we've got a chunk of the community user space right now that is pushing very hard for certain compatibility layers and this is where I say you're gonna want your earmuffs because ILP 32 is a thing and who's familiar with this so for whatever reason there is a large chunk of the arm community who would like to have backwards compatibility for arm v5 arm v6 applications a few other oddball things that may have been built against old hardware they want to just run this natively on AR-64 they equate it to hey I can run you know I 386 or I 686 applications on an x86 64 it is not remotely the same thing when it comes to arm they are aware of this this is an ongoing discussion right now this may land an experimental it may not my requirement has been there has to be a good faith effort to get things upstreamed that stands a fair chances of success and right now that last bit is what is killing the ILP 32 movement there there have been patches pushed upstream to both the kernel and glibc folks who have roundly slapped it around and said no so until that changes and until there's a patch set that looks like it might actually maybe get committed I have a really hard time saying yes we can pull that in even in an experimental state so no that is absolutely not something that I would push out for consumption other than an experimental thing and even that is questionable it it is an ongoing community discussion they have been asking about it for asking for it on the mailing lists what we're working on short to medium term right now is actually adding in AR-64 arm 64 hardware to CBS the community builders that Brian Stinson was talking about earlier but the infrastructure that Fabian was bringing up earlier we're plumbing hardware into this and into CI so that we can validate and make sure that everything we're doing on the arm side is exactly as it should be on the x86 64 side again we're trying to go for parity we have recently gotten boxes stood up in the data center that actually got network connectivity that was a miracle but it happened it's connected it actually works and so we're going to be starting the community is is Niels DeVos hiding in the room he was in here earlier I don't know if he's snuck out he's not okay so we have the the Gloucester builds coming online as our first proof of concept and then the RDO folks have been roundly tossed under the bus so that we can get open stack going on AR-64 very soon as well yeah it work I have it running it works I just need to do it community it has to be out in the open not just in my private build system at home where it's sketchy so essentially we have the CI infrastructure coming up we have all of the the server space that's needed now we're getting more into the community boards we've kind of waived the mission accomplished banner for the server stuff and until the hardware becomes more generally available for server hardware there isn't a whole lot more that we can push once we can turn the community builds over for the various projects that the Gloucesters the seps the open stacks once those communities are able to make their builds public the only real market left for us to push for is the community stuff but all of the community hardware is ignoring UEFI they're doing the same old thing they've always done they're doing the same traditional approach because UBoot is a thing and as much as I hate it it works for their use case for what they're doing it is cheap you so the high key has a special brand of hate in my heart right now I have a build set up for the high key it works you can go get it right now building the disk layout table that the P table dot bin is a bit of a pain and if I take the if I take the P table that they ship which is admittedly for their their Debian based they're a bunch of based distribution it works but they have their UEFI directory set as slash boot not slash boot slash EFI like Fedora and other same distributions have so it becomes an interesting dynamic in finding the right EFI files and putting the image together correctly because the installer wants to do one thing and their disk layout wants to do something different it also gets really particular with the fast boot update to move the image from one to the other it's exceptionally temperamental about the size of the image so even if you're well under the onboard disk space it gets finicky about you know what I don't need to write this whole file I'm just gonna truncate that last 100 megs because you didn't really need that did you I'm not shocked we had discussed this this is hopefully a saving grace where Fedora and Sentos are going to be collaborating possibly commiserating and drinking heavily how many of you went to the beer event last night excellent so the patch set that Peter's talking about allows you to shim some of the UEFI features into a soon-to-be upstream you boot and hopefully fake enough of the funk that we can get things booted in working normally neither one of us has tested this yet we're both pinning high hopes on it and we'll probably be sobbing and crying and you know gnashing of teeth wave it around getting out and trying to make that thing in the 24 time frame and so that I don't actually have to drink heavily up and deal with anaconda and install a process and various other bits and pieces I'll be playing with the this is part of the whole keeping it all in the family thing so in this for the arms base right now Fedora and Sentos are playing reasonably nicely together and collaborating fairly well for the majority of it it seems to be going well we have so I have the pine 64 it arrived about two days before I came here so I I haven't had a chance to play with it other than to notice hey that's bigger than all of the other arm sockets but you know for $15 I'm not going to care so yeah basically the only the only thing that I haven't mentioned here that's in slides that I'm really sort of passionate about is in addition to the installer and pixie support and everything else on the server hardware we're doing developer disk images which we essentially have the user land support with the kernel bits ripped out and some brief instructions for the different vendors to drop in their own kernel bits you say tomato I said you know it's it yes some of it I would desperately love it if the arm folks would stop doing one-off kernels and forking them on GitHub and claiming that that's upstream because you can go look at the code and they're one monolithic just giant patch set I might be a little bitter on some of this does anybody have any questions feel free to interrupt me Peter already has been the whole time it's fine it's not a problem yes the they are creating all new issues we've always said it needs to be upstream we can't maintain a hundred different kernels for this and that's been very much our principle cost for Dora quite a bit in the armspace in the early days it's now weighed out because we support about two or three hundred individual devices out of the box and it amazes me every week when someone comes along with this weird and wanting full device I've heard of and I've gone oh god what do they want me to do with it and they just got actually now it just works it's like wow if you look at the CBA stream for the arm take a look very recently who's who watches the the kernel CVs and the whole key ring buffer thing that happened what was that two two three weeks ago Fedora had updates for this because they do kernel updates we had updates for this because we do kernel updates most of the Ubuntu and other images don't there the update structure just is not there they will throw an image out they wave their hands they say look it's supported it runs and then it's abandoned the user space it isn't usually a problem user space marches right along the kernel and the tooling around the kernel the occasional glibc gcc that's where things get interesting that's a very polite way of saying it we see that very frequently not to pick on individual folks but the hard kernel guy specifically that's the one where I run into it a lot they'll do one hey we forked this kernel it you know 310 or 4 2 or whatever here it is come get it and then there's just no movement on it from that point on I'm not opposed to it as long as it gets you know patches and is maintained but it's that second part it's the patches and maintained that tends to be a problem so it the kernel side and the Ubuntu side they they tend to wave hands and say mission accomplished a lot faster than we do we have higher standards and when I say we I mean you know Ralph Fedora sent us if I tell you this hardware is going to work I'm putting out updates for it I'm not going to expose you to CVE's the only exception to that is the developer disk image where I load the gun and say here if you would like to shoot yourself you may I'm not going to any other questions that's a very good question I don't know they don't tell me these things it is still a it's beta developer tech preview something the way that it's working right now is if you're a real customer you get you know access to the real betas except this one this one you have to go talk to your account rep and get added into the super secret squirrel thing and then you get it which is an added step that why anyway not for me to judge not for me to say I don't care I'm under the impression I have been told but again we we're not in any way related to the business unit so I could be wrong feel free to correct been doing a great job so far I love taking cheap shots at Peter it's fantastic red hat is not pushing the updates they because it's the preview they drop the release this is what you're working with it's expected that it's not a production thing we are pushing updates for it every time there is a user space or a kernel update or CV that comes out we're patching the kernel outside on our own but the user space gets the same updates that everything else does so for us it is it is a fully it may not be that may have changed so more questions I have 10 more minutes to kill and I've completely ignored my slides so what what do you guys have hit me with arm questions KB the builder is available now for the stuff that is already built for the SIGs in Koji their option is essentially bump the release version because Koji gets particular about nvr and so if they the next iteration of their build where they have a slightly newer nvr string then they'll be able to tag in the arm builds and move on with life if they want to build an equivalent version now because say the Gluster folks don't want to run different versions of 3.7 which is an entirely sane stance and I wouldn't fault them for that then they would have to do a scratch build and we would have to work with them in the interim to make that scratch build an actual real honest-to-god thing if they want to tack a zero one on the end of all their builds then great we can go do that now the CI hardware should be coming into the data center in the next month to month and a half at that point I'm kind of at the mercy of the data center guys to get it plumbed in and networked and I can throw estimates out for timeframe all day long I have no way to back that up the the the testing can work on virtual machine we can get that set up but it still has to have a network to talk to the outside world so the community can actually get to it and that has been the the majority of the roadblock I believe so the hardware installation the the managing for the data center type stuff he's entirely outside of my purview I usually just throw stuff at Brian and say hey make this happen and and he goes off and talks to people and screams and yells and stuff either happens or doesn't so I yell at him he yells at other people it's a nice little cyclical thing so anything else does anybody have other arm 64 boards other than Geekbox or pine 64 that they would like to see supported is there anything else that anybody is aware of other than the high key so the question being do the SIGs need to care about individual hardware differences and the answer is it depends if they have kernel modules or if they are relying specifically on something some specific function of the hardware in that instance then maybe if it's entirely a user space thing like Gluster or something along those lines the user space works just fine and they the SIGs won't even notice that they just trigger a build it builds they move on with life everybody installs their software and they're happy never pieces that all the cheap devices actually don't shit with so I guess I guess but that's part of the distribution side of the test. Do you have something to make Gluster scared? No, no basically one of the things I've done with the 8th spec is that standardize a lot of stuff and there is a dot 8.1 and an 8.2 spec which adds extra functionality in but it's not different than different generations of the Intel processor where they have MMAs and then SSE and then SSE 2, 3, 4, 4, 1 and so on. Most of the newer feature specs that are coming in and I'm not sure what is public and what is not so I'll be very vague and hand-wavy. A lot of it is a know-op if it's not supported so for some of the new things if it's not a supported feature it just won't happen and the software will continue on as if it doesn't maybe it's a little slower you may not get the acceleration that you would expect but it's not going to outright break. CRC32 is a wonderful thing. Even down to if we had a SIG that wanted to change up some of the Anaconda bits, if Anaconda runs then it won't make a difference because that would be userland enough that it shouldn't match. It's just getting to the Anaconda bits with, you know, hey does it support Pixie? Do we support this bootloader? Do we have this type of memory register actually available or did we save three cents by not putting it there on the board which is the case frequently? Any other questions? No? So the question is what happens when Red Hat releases their official thing? We already have the source dump of the kernel. We get that now as Red Hat delivers the sources into git.centos with all the other distribution code. We build that and the initial release kernel because that's the newest kernel at the time that goes into that initial release. So we have a roughly equivalent kernel there. What we're doing for these monthly builds is we're taking that kernel and adding in a few vendor fixes or updated patch sets on top of that. That's always an additive process. When the new RELSA kernel drops, those patches, whichever vendor has supplied them, is on the hook for porting those if they haven't already been merged in by Red Hat through some other means. So if the initial build was, I want to say a 318 or 319 kernel and the Zen folks gave us a kernel patch to enable Zen support, this last kernel drop for the 7.15.11 build was a 4.2 based kernel. So we had to get a new patch set from Zen that covered the differences between 318, 319, whatever it was in 4.2. If that kernel gets rebased again to something further on down the road, then we walk that process forward. So we're starting from what the REL kernel source is the same as we always have in the past. We're just adding a few additional things in there as the community need for that arises. So we are still compatible. We're just also walking forward a bit more. And in some cases, if you go from 318 to 4.2, in that example, it could be that that patch set that the vendor provided is now upstream and the patch can just be got all together. Right. With our patch requirement to the vendors of there has to be a good faith effort in getting this upstream, a lot of it is really just a timing thing. The vendors may have missed the window for 4.1 and so they're scrambling to try and get into 4.2, 4.3, 4.5, whatever. And it's been accepted. It just missed the merge window. So hey, it's been accepted. We can see it. I can go on LKML and find this. I'm fine pulling that in ahead of time because I know it's already in the upstream. I know it's going to work its way down. And once we hit that point, I can drop that patch off my list. I don't need to worry about it anymore and we move on and cycle through. Yes, I don't care. Honestly, so Microsoft has been very beneficial in helping to drive some of the server standards. I honestly haven't looked in depth at what they're doing on a lot of the user boards, but a large chunk of the Microsoft code requires ACPI. So because they need ACPI for their distribution, it's been beneficial in leaning on the vendors to push ACPI further into the standard. But beyond that, whatever Microsoft's doing, I haven't touched Windows in 3 years and I consider that a badge of honor. I have no intention of shifting that. So, all right. Thank you all very much. Thank you.