 Hi folks, I'm Steve, I work for ARM and I'm a Debian developer, I know it's calling me to Lenovo. Lucky over here is much the same, and is hopefully going to talk as well. What is he always saying? Or maybe not. I just want to do a quick update on the state of the multiple ARM ports in Debian. Where we are, and what's coming soon. And also I want to hear what you guys have to say, and whatever questions you might have. Sorry about this, short delay. Quick agenda, maybe breathe from through the three ports. The build is the whole, we're using discussion. That's where you guys all join in as well please. Somebody take notes in Gobi? Neil, you seem to be good at this. My usual thing is for any session that I'm running, we can get Gobi notes. I will send out summaries to, I will blog about it on the planet, and I will send a summary out to a public mailing list later. I try to do this every year because I think it really helps so everyone's aware of the discussion. So, RML is the oldest of the existing ARM ports, first released with Lenny, which is quite a while ago. Softloat HVI targeting ARM V14 and higher, still needed for some older hardware. The most common hardware by far that is still using RML is stuff like the plug computer, and a lot of the nav boxes that people have bought. The old QNAS and a few other things. It's still supported upstream. There's still a lot of work going on in the kernel to support some of these older devices, but probably not for much longer. I don't know how long we should continue to support this. I spoke to Ben, one of our kernel maintenance about this earlier. He would love to see RML go away because it does cause him a lot of hassle. I'm going to ask the question, how long should we continue to support RML? How many people here have something that leads RML? There are a lot of users. You know, I might as well be able to say something on that. Oh, sure. Yes. How many people, if we said today, we're not going to release RML with stretch would cry? But I mean, I'm the one... Stand up and talk. I mean, I did a lot of the... Yeah. I did a lot of support for all the teams. The code was in all of those. Yeah. And there are a lot of users to that. So I think that we stretch out to even one more release. Okay. I think maybe sub-arches are the things you should be thinking about. Whether we release with the QNAS or IXP4XX or the other one. I'm just going to have to go up the button. Oh, yeah. Sure. I mean, the alternative, and again, sort of, you've been about this earlier, could we possibly get the people who care about this to support RML for an LTS, or Jesse, or maybe stretch? And then for those people who care with a reduced package set, could still continue to run this. I mean, for most people... Martin, what users are you supporting? It's basically NAS boxes, isn't it? Yeah. So for those people, we don't need any graphics. Right. We don't need anything beyond, to be honest, kernel and network servers. Yeah. And the same for the Freedombox stuff and all. Yeah, fine. Of course, yeah, Freedombox is an RBL device as well. Well, some of them are. Some of them are, yes. For example, Freedombox. Yes. So things that overwhelm me cook with. Is that right? Yeah. Yeah. So that's something I think we should be considering, because to be honest, otherwise, we're continuing to expend a vast amount of effort. Computer effort, fine, we don't care about it. We don't build hardware for these. But for the people who have to actually spend real human time doing this, unless it's important, let's make it easier for those folks. Oh yeah. Is that mostly kernel size? Because the kernel size is what keeps on biting. Right. But that's not something we cook with. It depends on the device as I understand it. Yeah. So I mean, we can talk to Ben to see if there's any time we can drop. Sure. If I'm correctly informed, IXP will definitely not be in the next release. If I remember correctly, Ben definitely said he will not support it for another release. Sure. Yeah, I think you've dropped him. Yeah. So I understand Ben being a medicine post. He's had to do a lot of this work. He doesn't particularly care. He doesn't have any of this hardware. Obviously, we can all take that to use that as an argument. But equally, he's been doing it regardless for a number of years. So we need to have the send Ben record boxes. So my feeling is that the ones that people are actually using are not the ones with massively restricted flash. Yeah. And so start to phase those out. Yeah. Yeah. I mean, we've had this discussion every year for the last few years and we worked out that, OK, the white answer for people who care about those restricted ones is white to trivial boot load you can put in flash. And I've seen no effort at all going into it. Because to be honest, the people who might care are not the people who probably have the technical knowledge to do it. All my ones have big enough flash. Yeah, well, yes. Fine. I wonder why. So this is, so are we in agreement that we may not have Armiella stretch, or at least if it is, it will be in a much reduced form. I mean, subarches, it sounds like a good plan, but we've been taught about it for years and we've never really done it. Yeah. We only have it. I'm talking about some of the existing ones. Yeah. I mean, apart from the kernel, is there any reason we can't have it in stretch? I mean, it still supports it up. The tool chain support's still working. Is it often not sure that threatening's taken away just yet? No. The tool chain support is never going to go away because there are still a lot of devices out there. Are we still selling on 70M hearts, for example? Yeah. It's just, I also understand that Debian is the only distro that sensibly supports the V5 devices for a lot of these people. It's the only distro that's ever really done in bunch. I think they're actually still selling that Armie 5 Kirchwood base and mask devices on the market. So I don't think we're dropping things to much. Kirchwood is actually being retained in the network. I mean, they're actively improving it. So we definitely need to have this as a discussion with a wider audience. Until there's a mechanism for going to part of an architecture in parts that we pretty much have to keep it going. So, well, I guess we can turn off, you know, if the open office stops working in Armie L for some ridiculous reason and nobody can be asked to fix it, we can just disable it at that architecture. And, you know, a lot of the bigger desktop stuff is just, it's not useful on the typical Armie L device. We know that. So, yeah, we can selectively just kill things if nobody wants to work on them. Except for software and platforms. Yeah, that's it, yeah. OHS is the default 32-bit ARM platform these days. Thankfully, we got almost all of this supposed to agree to this a few years ago. Again, I'm pouting what I said last year and the year before and whatever about this. With the RMP kernels, the two of them, one with LPA and one without, basically, it should be capable of supporting not on any reasonable R and V7 platform that is available in the market. So, long as we keep on getting updated DTPs and drivers, they should just work. They didn't go to the usual question on what's about V40, V5. And that's the only cause it's some extra aggravation because things break for V40 and they wouldn't work for V5 because nobody else in the world built it. There is that too, to be honest. We're probably already accidentally building something for V5 already and nobody's complaining. So, I think now we probably could just say we don't care at all. Does anybody here have an ARM V40 device that would break it if our real chair moved? I don't know, music. Does anyone have all that they care about? Yeah. Don't let them ask when anybody applied power to one of the other more than one person. That's what he comes down to. It's a discussion. I mean, we've been having the beginnings of this discussion for the last few years. We'll go on to it. So, I'll make sure... To be honest, there isn't a huge amount to say. There are still, and this hoifers me, people producing new ARM V7 CPUs that don't work with this. It's hilarious. We found one in Lenovo a couple of weeks back. I'm not going to name it, but someone who's shipped... who's just shipped literally in the last few months a new ARM V7 platform without any floating point at all and then said, oh, of course we want Debian and Ubuntu and everything to work on this. And they got quite offended when we left that. This is very widely supported. Modulo, some kernel stuff that we're not going to have booting. This will work on a Pi 2, the Huawei food, the BeagleBone Black, obviously, the QB trucks. There's all of the one board devices. I'm not going to list even remotely all the devices that support almost any current sensible ARM devices going to be a V7 or higher and will work here. And then we have ARM64, which last time we spoke about this and only just got into the archive. Super new and shiny and really, really cool. It's working. It's boring. Not quite. It's working well. There's still bits of the archive that don't work on ARM64. Again, anyone in Matt Doug's talk this morning will have seen he's still organising a contest to encourage people to help porting to ARM64 and even also optimising for ARM64. There's still bits and pieces of the language support. Those pesky languages which have their own runtime within themselves. So you need to be a guru in how the language works and in any new architecture that you need to port it to. Really, really hard. Anybody a fan of SBCL or C-List who can do with a list of implementation that worked? Hands up. I know what you're saying. That's pretty good, right? Exactly. So the obvious big ones. There's a few more things that actually use this. The obvious big ones are worked on because, commercially, they're very important. So we have a, I believe, a fully functional Java port from the Red Hat work done. We have people in Lenovo work done and it's all great. We have... JavaScript support is basically done. However, there are still any number of problems with JavaScript in Debian, like Node.js I think still is not yet working on ARM64. So Node.js does actually have the right mid-8 which is not in Debian. That's slapped up. Yeah. But if there is plenty of scope if you want to get involved and help, please do. You can get rich queued off. So maybe, if you ask it nicely, we'll buy you beer or something. We might even find you hardware if you're really keen. Talking of hardware, yes. Just a short remark. Rain and I are among maintainers of the free Flaskal compiler. It's... Support upstream is already there but not in the stable release. So the moment that they release a new version, we will be working on getting that finished off. Yeah, I think there's a few languages upstream have done, but... Because of the release cycle, it's not percolated, but I expect those to arrive soon. Yeah. There's a monocore, but it's stuck behind commercial doors, which is very annoying. They've been done, but nobody wants to pay for it so they don't want to release it until they can get somebody to... to give them money. So hopefully to... So the big question with this, with R64, and it's not, it's almost, almost something we can answer is, so when can we get older devices? Lenovo has the 96 boards initiative so there are a few devices available. I'll be brutally honest, not going to get slapped with it. I don't think they're very useful yet. They will be. Awesome. Regarding the 96 boards initiative, is there any board in there that actually has upstream current support properly working? Because I have not found any when looking. Not yet, not yet. This space. They're coming real soon now. There is a board that's on a desk burning their mind which has that but I can't see anyone like that. There's a whole... There's a slew of those, I mean to be honest, applied micro or selling the Mustang... So the HG is the name of this... HG is one, wasn't it? Yes. It's now available in a number of form factors. You can actually buy servers based on it right here, right now. Yeah, Gigabyte is selling it. Gigabyte is selling it. Gigabyte is selling it. Asus are selling it. You know, well-known, fairly well-known PC component manufacturers are going to be selling these right now if you want to go and find them. You'll be able to find them in normal retail but if you have the right contacts you should be able to get some. And there's a certain one over there which... Yes, but various other people are working on other things. Certain people may even have an R64 powered laptop here right here today. What is that? Do you want to give us a quick look? What? Is there a boot? No. No, apparently there isn't a boot. It's still probably much underdeveloped. I would say it's underdeveloped. It's underdeveloped. We've seen it booted already. It's very boring. It looks like a desktop. This is the point when you know it all works. To be honest, it all just works and there's nothing to actually worry about. Plug it into the projector then. Where? We'll try it. What the hell is going on? He's going for it. I can't turn it on because everyone around me will complain. There's a workstation form factor for it. There would be space in the data center on the four screens. I have four ports. I have four ports in Sunday Mail but I'd be happy to give up. Yes, no. So we have three ports. One is becoming available. There are commodity things that are going to be available but hopefully if we make enough noise and convince the vendors, it might go mainstream. That's what we're hoping for. We have an on64 laptop project here. I'm involved in another one with a bunch of people here that we bogged down in last year struggling to get SOC vendors to talk to us. That's the issue. The SOC vendors who are struggling to ship deathboards yet aren't interested in talking to you. That's a painful fact of life. The nice thing about on64 is it's been designed properly as a real system from the get-go. It's much more tightly constrained than onV7 in terms of the things that we're going to be seeing. So it should work with a single kernel and a DTB. ACPI is a part of the spec if you're going to do an on64-based server. It's all going into the kernel. It's all basically mainline. Modulo bugs, they should all just be working now. On64 is the other platform where our installer will UEFI boot and just work. It looks just like a PC. It just isn't. It's a decent hardware. We still have a stack of deathboards in on. In fact, I was just talking to one of my co-workers today. We had a maintenance weekend there over the weekend which wasn't scheduled to lose power to this whack what apparently happened. These things sometimes do. So I had to ask my co-worker to go in and just push power buttons on a bunch of machines. We've got a load of donated ModoXP. Which are on V7. They were on RHF. We still have one remaining FreeSkill IMX53 which before we used to use the building were really nice stable machines but oh god were they slow. We still have one as a portabox so people can test. It has neon, the Marvel ones don't. So if people need to make sure that they actually verify what code they're running it will give it going as long as the hardware's alive. We have several different types of R64 machine. We have a couple of Juno's that were being loaned to us by ARM. The ones that basically built most of the archive was to get into GSE. They're still going strong. If anything they've been getting better with time. With newer firmware they now just work with UFI and everything. We have been donated an APM box and Mustang is also in the Wacka arm. I have an AMD Seattle on my desk but unfortunately as an early web it's never going to be supported by a mainline kernel which is why it's still on my desk. It's a really useful build box but it's no use because the kernel on it is shocking the old one and really not something you'd ever want to connect to a network. There's issues though we can see the end is getting there we would really like to actually replace the deathboards. DSA definitely, definitely wants us to replace the deathboards because deathboards are crap for stability and if you do have power outages they don't come back. As we just found this weekend the nice thing with ARM v8 and the ARM 64 support as and when real hardware becomes available hopefully the nice people who are selling these things can donate some. Then we can get we can use stuff hosted we can get things going and the big ARM 64 boxes that are coming up really fast they will compete quite readily core for core with an AMD 64 box maybe not be quite as fast but they'll use significantly less power is the kind of the point if we have real computers we're not just running on tiny deathboards anymore. Steve, when we get ARM 64 donations over there are we going to return the ARM v7 supports and use students on the ARM 64 to do the ARM 64? I was going to ask that question It's a very good question I was going to come to honest. It kind of goes to the how many cores do you have? Sure. Once we get to ARM 64 of course it will run ARM v7 quite happily. This is also where ARM vL becomes problematic. Some of the software that we run on ARM vL cannot actually run sanely on directly on the hardware on the ARM 64 beyond the 8 machines. The newer machines v7 and up have really good same primitives for barriers and locking the older v5 or v4 t especially doesn't It's an incentive for moving to v5 because you have to use vx on v4 t and you don't have to do any v5 more than they still get to use anyway. Barriers are a big issue. What we've actually found is already I found this I posted that we have a cost distro list for the ARM people in all of the different load distros. What we found was we had some software in the archive built for ARM HF but while it's v7 and it's supported it was built using some really bad old config that meant it was using really really old barriers and locking primitives the moment you tried to run it on an ARM v8 it just went bleh and fell over right there. We've done some digging through the archive to see what's left it will fix itself over time we hope but it does mean we can't sensibly build v4t old stuff using an ARM v8 server there is work ongoing by ARM and others to add kernel exception handling for some of those so you can provide a really really slow path to catch it and make your software not crash but it'll be about a thousand times slower than it should be so it's not really an option if you're going to be building a tested software that way. Well actually I think the kernel is already doing these kind of traps with your ARM v7 and ARM v8 SWD construction shut up. And it's a thousand times slower on a processor than on a processor. And you don't use your lock and lock. To be honest the nice thing is the mobile boxes are actually themselves really fast they're quad core clocked up well they've got 4 gigs of RAM again these might be dev boards but they're actually really nice dev boards they're almost servers of mobile. I think we've convinced some people to sell servers based on the same chip set and it basically works okay. What's the problem fanning? Yes. So that's all I wanted to really present the problem it's not pretty five minutes already what else do people want to talk about? I'd look up some scientific software the unit tests time out and get killed on architectures I was just wondering what the best practice is just to skip the unit tests or just not to build at all. To be honest if the unit tests are showing problems they run fine on real hardware they just run out of time they get killed after 150 minutes or something on the buildy page. Just email us at maintainers and we'll set up the time out longer for your package. Or it might be showing that there is a real problem. Do we know how long they run for? I don't know if it tells me. If it's something that's showing locking it might be showing there's been a deadlock and that's why you're not seeing any output. Sometimes I've done myself before now actually go and modify the software just to put a print eff in every minute of something just to show it's still alive we'll actually also fix this. And then stick it on a photo box and watch and wait for it to finish and see how long it actually takes. I said definitely definitely I said unit tests to awesome because they do find these things it's quite rare that if something has an output for two and a half hours that normally is an indication of something wrong or your unit test is just not that well with. Yeah so a bit of output is effective yeah all aspects of the delay if we know that it doesn't work possibly there's underlying optimization from this thing so for example the Doe's suite uses a camel which architectures where there's a faster camel optimized version it takes five minutes or 20 minutes of something on a box it takes five or six hours on 64 I think because it's not optimized. Yeah so that sometimes. The other thing to do is ask on the internet which is a software there's loads of this on there if we think it's interesting or we think it's useful you might find one and have a look as well. Equally a very old find it's certainly not worse having it for RME L. RME R1 from HF so all we can say is that it's not RME L. It depends on what the use case is. What do people want to run on? Are there realists who can go run on? In my case probably never in particular. There is a lot of scientific stuff but it's probably never been run on an old RME L device because there will be any perverse things too. On the other hand Debian exists to have this stuff built so people can try it. It depends on what you do. Some people might actually buy in the big heavy where it's really great just with a small dataset that does work on the old RME devices or whatever. You never know. Use of freedom, it's cool. I was wondering there are devices that have a smaller use space what are there some lower bounds that you would require to ship at ETP and so on. I think mainly in terms of shipping at ETP if it goes and makes my kernel we'll ship it. duty means a tiny they cost us nothing. What are the kernels requirements or just a couple of batteries they actually require? I don't know. I think as long as it correctly describes a piece of hardware that is real I don't know if they reject things. They don't want to renovate themselves I don't think so. A lot of the things for the kernel it's important. Yeah. If it's a if you've got any particular concerns again ask the kernel folks ask the Devinong team we'll help if we can. Is there anything else that people would like to know? Very similar topic I'm playing around with PolyMix, PolyMix, all we know is 20 ports and you can have them on the NCD display and you have to add to the device tree it's a simple frame buffer device. And to set up the simple frame buffer you have to adjust to U-boot I'd like to see an as easy as possible way to get that in Devinong. At the moment it's really fine chasing kernel works great on the device so U-boot you're going to get the amount. Your kernel? We talk after this one. Yeah. Casting knows a lot about U-boot and things. Yes. I'm on the 20 ports of PolyX too also but I noticed that I'm trying to create bootable microsd's with some image writing software and I failed to find exactly where to put and how to put the U-boot in but I didn't find it in the wiki and I didn't yet get to go through the Devin installer and find out what exactly it's doing. Okay. You're trying to pre-write the U-boot environment. Yes. Or don't write one. I didn't write one and it didn't boot and that's what was in the wiki and I need one and I grabbed one from somewhere. It should boot because you would include the compiled in default environment. So even if the SD card does not contain a valid environment there's the default one compiled into the binary. So once the binary is written there will always be a valid default environment for it. And it should boot what flash kernel is going to write into the first partition. That's the fact. It didn't work for me. Which body do you use? The A10 on Jesse. Do you have it with you? No. But I can bring it tomorrow. Okay, bring it tomorrow. The point is I have tested that on the A20 but not on the A10 because I don't have one. Okay. We'll talk afterwards what I should bring. We'll try and problem-solve if you know it. I didn't get you. We'll try and problem-solve if you know it. I've kind of said everything I want to just to give an update of where we are. When you do development with Qemio and you switch to real hardware what can you expect from a that was not covered by Qemio or otherwise when should I start on real hardware? Depends on what you're trying to do. If you're just generating what's a normal user land software you probably won't notice much difference apart from performance. Pressing fix with Qemio now. Qemio user. I used to start quite hard. Qemio user or Qemio system? Static. Static. Static. Okay, so threading may change quite a bit. Like it might work or not work. It's not multi-threaded at all. Yeah, they're working on it. Here's three. Qemio user. Qemio user. What central cross compilers would you like to run on your ARM machines for other targets? X86-64. Anything else? X86-64? Okay. Just wondering what demand there is from part of the other way. X86-64. A little bit, but not much. Between ARMs? ARM64-2, ARM-HF? ARM64-2, ARM-EL? Yeah. You might need it for the build for one day. Yeah. How's 64-1, ARM-HF? It's just true. It can be. Much like ARM-X86-4-5-3-6. Under the native-builder for a native-builder. You might want to cross-build. Yes. One thing I should mention that is coming is hopefully it's going to be on HF maybe, but definitely ARM64-5 images. Neil and I have been mostly Neil and I have been watching the people working on quite a lot of set up for this using the Anda Bootstrap. It will create images however you configure them and it also knows how to run QMU to be able to build cross-images, essentially. We're hoping to be making announcements and actually getting those out officially in the near future. So, for those people who have an ARM64 desktop I appreciate that's not many yet but you'll have to be able to plug in a USB stick and it should just work with a live image just like any X-X86 would. But again, with ARM-V7 ARM-HF the variability in the devices is going to be a big impact on whether you're actually going to have any kind of live image could be really good. I expect it'll work on some of that kubi truck because you're going to install a kubi truck anyway. A lot of other devices you're likely to get some kind of problem or you'll have to do a lot of other sort of specialized setup. You'll have to fight with your boots or that kind of thing. One thing that you mentioned on the mailing list before the conference that would there be interest for having a Lava Lab for testing these Mario's ARM ports that Vivian supports because our support is quite as advanced when we do it. And then because we're all volunteers we might forget about it for months and it might break down. So the discovery when you actually tried the newly released yesterday debut installed that in fact it doesn't work at all. Of course the case that we get some mails after a new debut release that actually got broke things down because no one had the specific part. So there's a couple of things with that. Yes, I'd like to do Lava.dev.net. I'd actually like to be able to do all that kind of stuff. There are issues with testing bootloaders. We need some hardware that we've tried three times now and we just can't get reliable, stable scalable hardware to do best e-marks. If you look back on our blog you'll have got all the history of why that doesn't work. So that is blocking any kind of testing of a great system including the bootloader. We can test everything from using the bootloader there with commands and loading and running the kernel and running the rest of the image. We can't actually test a new build of Ian's new boot loader. We can't we can't just say oh here's a new version of you booting and sticking on the device and test automatically. That can't happen yet. Yeah. Testing all the other stuff is still usable. Exactly. So my upstream work on that is taking up all my time. So it's a whole question of how we're it's an interest in doing CI on kernel boot and I sort of have a lot of this kind of stuff that I've been doing to help some of the people in the world who are using it and let's see what we can actually set up. It would be absolutely lovely once we get more devices up and running. If our devices don't make it, whatever we can plug them all into a Lava instance and then we can do regular testing. Linaro is more than happy to allow Lava access for anybody in WMW who wants to do arm testing particularly if you're testing stuff that is out of the main flow of the busy devices in the lab. So there are particular devices that are more restricted and they tend to get busy but there's a range of ARM v7 and some ARM v8 devices available in Linaro and you can have permission to submit to those and actually do some tests using those. These are one, these are two but however that you know it's likely to be one or more of them available in a Lava instance ready and waiting, being maintained properly and then the kind of hardware that most people here will actually be thinking of testing is likely, mostly idle waiting for jobs to be submitted. Most of the jobs that go on through on stuff that I do know really the rest of the people in London don't have real access to them. Now of course if you are struggling for porting access there will be seven porting boxes we've got two different two different, yeah we have two different ARM v8 porting boxes available within four Debian users if you're looking for something to help porting something even outside of Debian Linaro also has a slew of different machines available deliberately to help software authors port their software with a shout because with the rest of the team we can actually give you a hacking session on that box there's nothing to do with that with hacking sessions so anything else I do have some freebies they're not particularly special but I do have some freebies you see the the nice little sticker on Powering Debian on Powering Debian developer stickers I have a load of these up the front here am I just broken video no excellent so if anybody would like some please just ask they're free to give away please if you if you know anybody else who might be interested by all means take some with you unless there's anything else I think we're only just about done for the day so Neil and Wookie and I there's a whole slew of other Armand Linaro people around during the week if there's anything else you want to talk about feel free to try and find us I can't promise we'll be readily findable if we're in talks and stuff obviously but we're happy to talk about Armand Debian and stuff any time about what the hell your partition world looks like oh you spoke to me about you were here yesterday okay cool so thank you hash Debian Arm we're always looking for more help we're probably the most active of the ports of the ports in Debian aside from XAXA which is really important but still there is always more work to do I think that's a sad part of life so cheers thanks for watching