 Hello everybody, thanks for coming. So my name is Will Shepherd. I'm an embedded software engineer. Who'd have thought? With embedded bits, and now we're techie-o. I've been doing this line of work for about four or five years now, and this is my first presentation, so go easy. I'm based in Bristol, in England. I was going to say United Kingdom, but that phrase sounds silly these days. There are many technical presentations here, and this one's not going to be quite as in-depth technical. I've been to a few embedded Linux conferences, and seen a lot of people talk about some very interesting things, being kernel maintainers, contributing to open source, and as me, literally stuck in 2009 with an old kernel on my desk working away, not really feeling huge amounts of connections to that. But I think coming has really helped me kind of challenge some assumptions, and most importantly, whether you do have to be stuck in 2009 or not. So what does this mean? Well, I had another couple of ideas for a title, which I'll just get working at some point. Great. OK, first presentation, and Windows has given up. Anyway, the other titles were Fun with Backports, and the other one was Elegant Compromises. But this is really a story about three things. Standard format, something I did, why I did it, and what I might do if it came back around. So what did I do? Well, we had a project delivered to us. Had a five-year-old kernel on it, on the BSP, and as a result, we had a lot of back porting to do. And so why did we do that? Well, this was it. We felt that it was self-evident that it was too much work to port to a new kernel. And there was no current board support in any of the new kernel for that. It was all provided to us by the BSP. So we just felt, in order to hit the ground running, we shouldn't pursue that avenue. So see, what I might do if it came back around. Well, this is where it gets really interesting, because all the work that we did to do all the back porting, if you add it all up, actually amounts to quite a bit of work. And if you compare that to taking a brand new kernel, straight out of the tree, straight out of the mainline, and giving it a go, I actually think that my original assumption, and the assumption that we had in the office and the assumption the customer had of that being way too much work, is actually comparable. And another thing was, what I'd love to do is really next time around, try and be more confident when talking to customers, specifically, about this in the first place. Because one of the most important things when a customer, when you've got a new contract, and you're talking to them about what you can deliver and what they want, and they're telling you about the feature set on the BSP, they want all of these things done. And the idea that you can just say, well, hang on, all of these things you want done straight away, you want to hit the ground running. How about we spend a week just thinking if we're going to have a new kernel, just investigating whether it's worth it, maybe another month doing a full board port. And at the time, it just didn't enter my mind that that was something that we could do. So what I'm going to do is ask you to bear with me just one second when I try and get these slides back. OK, fine. Up and down has forsaken me, but clicking is OK. Thank you, Windows. Right. So 2009, was that a fond look back? Well, actually, it was a whole host of disasters according to Wikipedia. So maybe that doesn't bode well for that particular year, not to mention the fact that James Cameron's film Avatar was released, which is kind of a catastrophe in itself. But Linux version 2.6.29 was released and found its way into a BSP for a board on my desk in 2014. So 2.6.29 in 2014, why did that happen? Why doesn't everybody just have the latest kernel? So that was a five-year-old kernel at the time. So maybe there was some sort of internal delay in the release of that BSP. That's more of a speculation. I think what's infinitely more likely is that getting kind of last year's product probably is financially interesting. Perhaps there was leverage of existing business to business relationships. Again, something that's very common. Or perhaps there was some other technical reason. That's very kind. Thank you as to why that happened. OK. Yeah, so as I said, at the time, we just had no idea about the possibility of going to a new kernel. I put here chicken. I know, right? But is it really that I was just too scared? Are there actually? I mean, obviously, it's quite nice to just want to have a new kernel. When you come here and you hear about Device Tree and it's fantastic and clever new things that get added to the kernel. And it's kind of one of those things that customers will go if they are dealing with software engineers. And so these guys just want shiny stuff for no real business reason. But it's kind of nice to actually go through it and think about, actually, it's not just that. There are pretty good business reasons. And I should point out that we haven't, this project that I've been working on, we haven't posted it to the new kernel. We did stick with the old kernel. So what I'm really talking now is about the challenges that happened at that point. I mean, maybe next year, having ported it to the new kernel, because of all these reasons, I'll come back and say, I wish I was still in 2009. Who knows? So let's just imagine, apologies to the clash here, let's just imagine that I did ask myself this question back in the day. New board, five-year-old, straight away. Should we stay or should we go? So should we keep the existing kernel or upgrade to the new one and just go straight into a proper board port? So as the clash said, if I go, there will be trouble. If I upgrade to a new kernel, there will be trouble. Jostrum are very insightful. So yeah, first one, how much effort is it going to take? Again, as I said, if you've got a new project, it's quite difficult, as you'll see, to know exactly what all the pitfalls are of just going, OK, well, let's just go for this. Why not? We've been doing this for five years. We go to the Invented Linux conference. We see how easy these guys will seem to make it sound. Let's give it a go. But it turns out it's actually quite difficult to get very accurate measurements on this. And especially when the customers just ask you to hit the ground running, they've got their dates, they've got their time to market. It's not looking good, really, so far, is it? Again, the customer relationship, I mean, what are the limits on their expectations of what we're actually going to provide? I mean, would it seem a bit kind of strange for us to say, well, actually, why don't we just take it? Is it a bit of a curveball for them? Will all the board features advertise still be supported? I mean, obviously, they've spent time getting the right price for their board. But also, it's very critical that all of the hardware blocks on the SOC work. And they can at least have some peace of mind that the supplied BSP, as awful as it might be, should support all of those features. And so they should be able to get to the product that they want and get it delivered. So closed source kernel object binaries, version conflicts. One of the learning points of this was that, actually, that's not quite as much of a problem as I thought it was, specifically in our case. But of course, if you have got some closed source kernel objects that are specifically called kernel functions that have been obsolete, you know, I think that's probably quite a sticking point. In our case, what we actually had was some closed source objects. And it wasn't until we actually dug out NM and just run a very simple NM command that we found that it was all just local data. And in actual fact, what they were were a way of mapping things onto the SOC registers internally, because there must have been some NDA on the particular peripherals that they were running. This was the video accelerator, the hardware resizer, all that sort of stuff. So I'm not really too sure why they do it. Some of the things they do is just beyond explanation, as far as I'm concerned. So is it a vanilla kernel, to what extent? Well, this is a real interesting one, actually, because just because it says 2.6.29 inside the make file and something about herrings and fish doesn't mean that it's 2.6.29 and something about herrings and fish. It could be anything. What I call it is Frankenstein kernel, because for whatever reason, they decided to not only modify, you know, obviously they provided some platform code, but they also decided that they were going to go around modifying core memory allocation functions, which was just fantastic. I mean, you know, it's really nice of them to do that. Their DMA and their Ubi implementation somehow didn't like each other. And they decided that all memory allocations had to be physically contiguous, which is wonderful in a system with a limited amount of RAM, because there's nothing the kernel likes more than providing physically contiguous memory constantly for everything. So yeah, so that was a bit annoying. There are other things I'll look at, but suffice to say, that gives you a kind of general feel for this OEM. Do we need a tool chain, too? Is the tool chain the thing that's holding us back? Is it tied in with the kernel? And of course, the level of existing support in the mainline, for which, you know, I mean, obviously, for the board itself, there was none. But obviously, for some of the other bits and pieces, there may have been. And doing an investigation is quite useful there. OK, so benefits of a new kernel. Well, I mean, new features. I know that that kind of sounds a little bit kind of nebulous, really. Especially if a customer really wants to know exactly what that is, it's not just the kind of geeky shininess. It's, yes, this is going to do this for me. You know, it's going to get more CPU cycles. What are we going to do? You know, are we going to get more IO throughput? What do you mean by new features? But as it turns out, part of this and part of the learning process is recognizing what those new features are. And an actual fact, there's a lot to learn by, as I call it, going shopping through commits and just kind of looking at the commits that are ahead of where you were and going, oh, it sounds interesting. I'll just get cherry picked. Oh, that's nice. OK. So yeah, better performance. I guess that's kind of an implicit one. But again, for a customer potentially spending a bunch of money getting you to do this, again, they need to know exactly what that is. And of course, knowing the differences in the different iterations of the kernel really helps. And this whole process actually, directly helps knowing that. Better support for new peripherals is just absolutely the biggest one for me. Because customers, being semi-technical as they are, will say, OK, well, we had this kind of awful while one loop on some bare metal. And we'd like to move up in the world and go to Linux. And it says it's got USB. So I'm just going to put in all of these brand new USB peripherals, the latest 4G and the latest Wi-Fi. And that'll work, right? Yeah, sure. And of course, if you're using an older kernel, as we, of course, found, despite the fact that the USB conduit works, obviously the driver endpoints may not work at all. And of course, short-term loss, long-term gain, I think is really the perfect way of kind of categorizing that. So again, just to the last time, I'm sorry, Joe, but if I stay, there will be double. He really was against keeping the kernel as it is. But let's see if there's any truth to that. Can we support modern external peripherals? Well, as I said earlier on, I mean, this is really the key to my presenting this in the first place is that the assumption that you've got two things here. One, the BSP people are saying, yes, the borders it is. We provide you with the features, the frame rate, all of the things that we've said in the specification. But you're now talking about providing these extra devices. And we're not about to just list devices from all over the world. I mean, 4G, for instance, I mean, they have so many different ways of talking to the modem. You've got serial, you've got some other ones. Their names escape me. But yes, I mean, the next time this comes up, it's going to be the first thing I say to the customer. What are you going to do about external, you know, what are your dependencies on external peripherals that go beyond what the BSP specifically says it can do? So security concerns. My boss Andy here was telling me, actually, that there's quite a lot of work done to make sure that there's backportable patches for security issues that are found in later kernels to get back. So to some extent, that might be OK. But if you remember what I said about the Frankenstein kernel, you try and apply these patches. It may just fail. You know, I got to this kind of anxiety state whenever I type get cherry pick enter. And you're just sort of waiting for it to say, all merged fine. And then if it doesn't, you're sort of shivering a little bit. So toolchain, you see libc version. I mean, yeah, it's old. It's old because it was built for that particular kernel. And in actual fact, there was no native Linux threads, which we didn't find out straight away. And it turned out that our system, our user stack, was heavily dependent on threads. And so being able to recognize that and change over was actually quite useful. Again, if you stay, yeah, I mean, you've got short-term gain, long-term loss. Because based on the idea that you're going to need functionality that you don't have. In actual fact, before I got on the plane, a colleague said to me that he was doing some crypto stuff because we decided that we were going to provide DM crypt and provide containers that had to open on Windows, which is pretty onerous. But given that, I mean, we got right to the end. Everything built. It's nice to get a whole bunch of open-source packages all building nicely. And there was no CPU used. There's no CPU left, which was a bit annoying. And then we wanted to change something else. And we found that that particular part of the crypto didn't come in until 3.15, something like that. So again, immediately, we started talking about, have we reached that magical point yet, where we get to actually port this to the new kernel? Because we're doing so much work to avoid doing that. I mean, I guess part of us, if we are doing more work, maybe we should just keep quiet and go, yeah, yeah, sure. So the benefits of keeping the old kernel, well, you can hit the ground straight away running, as I said. And in actual fact, because there's always a focus in Linux to make sure that user space doesn't break, you can actually get user space going. And if you do upgrade to the kernel, you can do that separately. And hopefully it will still work properly. Yeah, so two steps forward, one step back. I mean, this is, again, kind of really characterizing what this is about, how much work is going forward, how much back porting you're going to do. It's very difficult to tell, even now, actually, even now, because we haven't taken that step. And this is what I'd really like to do is compare this to a time where we actually have taken the step and actually really maybe look back on the time sheets and on the kind of work that we've done, and maybe on the Git history even, and actually have a look and maybe stand back aghast at how much time we spent trying to make it work with the old kernel. So can we make forward port elements compatible with new kernels to avoid wasted work? Yeah, I mean, this is the other thing. Love hearing about device tree, never used it. And I mean, if you're going to do more platform code to 2.629, and then you want to do it again for the device tree, again, it's a retrograde step because you'll have to do it again. And obviously, some of the platform code can be forward ported to the new kernel, but it just kind of feels annoying to have to knowingly do something that will have to be undone and done again. So what are some back port examples that we did? So the ASOCK codec. This was a thing whereby we'd had it provided in a big kind of blob, and it was pretty unpleasant. And we had to pick it apart. So that you have the various different modular parts of it so that you can have a codec. You have the part that deals with I2S, and then you have the part that talks to Alsa and all that sort of stuff. So how did I go about doing this? Well, what I did was gitclonekernel.org, have a look at what's going on in the slash sound directory, and really just pick elements from that, and picking and choosing git cherry pick. We had to do some more platform code as well. But it's very interesting because at this point, if I'd put a new kernel on there, I wouldn't have had an opportunity to look through all of this. So again, there's another kind of silver lining here where you're learning a lot about these things. And not only that, but when you look through the git history, you can actually see where people have gone, oh, actually, you know, that algorithm's not perfect, is it? Let's do this, and the code does this, the code does that, and it's just very interesting as a side point for your own kind of personal development to see this happening in sort of, well, it's not real-time, 2009, but as it was, if you like. Yeah, so that took quite some time. I think I looked at the git history, and that was a good month, at the very least, if it wasn't more, and there's bits to come back to you and invite you, it can be a bit annoying. So, the other one, SDIO Wi-Fi driver. So this was just, this was fantastic. Well, it was, it was great that we had the Linux Backports Project. I don't know whether you guys, I mean, you probably all do know about this, but I didn't at the time. And, you know, it's obviously incredibly useful having the Backports Project around, which is really just something that you can clone. It's make menu config, as all great things are, and you can decide which drivers you want. And, you know, one of the things is, there's a lot of code in there, a lot of macros that are testing for kernel versions, and you have to be a little bit careful that you're getting exactly what you think you're getting, because actually what you could be doing is you see the Wi-Fi chip that you want. In my instance, it was the WL127X, which gives me shivers saying that word, actually, at the moment. But I mean, some parts of it would work, some parts of it kind of needed a massage, and some, you know, but because 2629, I think it was a little bit too early, and again, the Frankenstein element was rearing its ugly head again, and it was kind of like, oh, I was expecting that to be there, I didn't expect, you know, contiguous physical memory there, what's happening? So, again, so there's two things. What's happening there? Two things, one, we need to Backport, because that's our choice. We wouldn't have needed to do this if that were the case. Yes, there's a bit of platform code, but also because of the Frankenstein kernel, it doubled the problem because even the Backport wouldn't cleanly apply. So again, you know, you've got more work. Really sounds like I'm trying to talk myself out of work here. Maybe this whole thing should be the other way around. So the other one really was just browsing, as I call it, which has turned out to be actually really interesting because if you can't go straight to the new kernel, like I said, what you can do is you can take where you are and just look what happened after that. And it turns out that our system is also heavily dependent on IO throughput on the MMC card. You know, we're constantly filling up the page cache and we'd like it efficiently delivered onto the SD card. And it turns out, I think somewhere at 2632, something like that, they got rid of PDFlush and they bought in something else that was more efficient. And so again, there's an option there to try and claw back something without it being too onerous and without there being too much work too. But again, as I say, because of the Frankenstein kernel, you get cherry pick and there are often problems. Yeah, so as I say, there can be problems because of kernel modifications. So I did mention surprising and unexpected benefits and I've kind of talked a little bit about that. Yeah, tour of a kernel. I mean, I don't regret any of this work and perhaps if we had straightaway done a port up to the brand new kernel, I really wouldn't have had a chance to look at a lot of the subsystems that I did manage to look at as a result of needing to get them to work both with these peripherals that I was talking about and also trying to right some of the wrongs that this OEM had done. It was quite funny because he'd always put his name in the code and I always used to say, thanks, man. You know, you've really given me a lot of work. I'm really glad that you're able to provide these suboptimal solutions. And yeah, you can see how far the kernels come. It's kind of interesting to learn about the kernel this way because when I come here, I'm always learning about cutting, super cutting edge, containers, virtualization. Yeah, yeah, yeah, you know, and I don't use any of this but you know, when you're doing this, you actually can see some time ago where maybe slightly more simpler in a simpler world, in a simpler world where things were actually just being kind of worked out. And it's very interesting for someone like me that's kind of new to the industry to see how that happened. I mean, you could, if you wanted to, like if you were gonna read through Hansard, for instance, read through all of the commits and just see how it's progressing but seeing it break and fixing it with commits, with your own code really is a real silver lining to this actually and it's really helped me, absolutely. One thing that I missed out earlier on before I just get onto the conclusions here is that we were able to recompile the tool chain which is, I must say, it's a really low-hanging fruit and it's a great thing to do. BuildDrew is one of my favorite things ever. Again, make menu config for the win. And you know, you can just clone that from Git and you can get to a level that was around about this sort of era. I mean, it's kind of like a push and shove, really. It's kind of, well, I want UC LibC to have NTPL support but I also want it to be able to support the 26229 headers and actually, surprisingly, that kind of works really well. There was a bit of munging around with compiling U-boot and making recognizing CPU versions and all that sort of stuff but again, if I had another project that had a very out-of-date kernel, there you go. You've got another kind of avenue for immediate effect. What's the tool chain? Have we got NTPL? Because if you have got heavily threaded user stack, it does make a difference. So where are we now? Well, I briefly said about the crypto troubles before I got on the plane which was just funny, really, because it kind of really just framed this whole thing, really, and it would be really fascinating to see if we do manage to convince the customer to start a separate project and investigating actually going up to the new kernel and it would be really nice to kind of provide some upstream support. I mean, we're always looking to try and get more into providing upstream and doing contributions back to the community. You know, coming here really gives you a sense that, you know, that's what kind of this whole thing is about and it's really nice to try and move towards doing that. I mean, unfortunately, the platform that we've got some obsolete it but, you know, it's still good to have to hope. So, yeah, so that's pretty much where we are. I really hope that's clicked with some people. I hope that some other people are in this horrible position of having to deal with old kernels and things like that. Just quickly, as a show of hands, does anyone have to deal with, you know, four or five year old kernels? And so keep your hands up if you kept that old kernel and just went ahead anyway and keep your hand up if you regret that. Okay, all right, good. Okay, all right, well, look, that's probably a bit quick. It's my first presentation and I had to deal with Windows although it worked out all right at the end. But has anyone got any questions? There's a lot of hands up earlier so there must be some questions, surely. I don't think I came across the situation that necessarily warranted that. I was kind of hoping that there'd be a way of refactoring a lot of the board code to get up to the device tree but again, I've not had a huge amount of experience in that. But yeah, with respect to those closed source kernel objects, I think we'd have to work a way out of doing that but they did provide some source which managed to link to that. And as I said, it was only local data. But if you were in a situation where you had closed source that actually had links to kernel structures, functions that were absolutely, yes, that would be, you'd have to think about wrappers and I'm not really too sure what you'd do in that sort of situation but yeah, I hope that's answered your question to some extent. Anybody else, way ahead? Yes, absolutely, absolutely. And if that BSP feature set that they've sold the BSP on is, and you're within that, that's it. It's just that I do think that there tends to be a lot of assumption about this implicit support of modern external peripherals. I mean, obviously, you're right, it's case by case basis. I guess a lot of the work that I do does, is a little bit multimedia based, a little bit more general purpose, a little bit more expected to kind of work all over the world and in different situations. But absolutely right. I mean, there's no point in doing it for the sake of it and I think that this is kind of the hearts and minds issue where customers might think, oh, you know, these guys just want shininess for no good reason. And maybe in your case that would have been, that they would have been precisely right. But I think that if you only spent three or four days using backpours and I'm certainly not saying don't use backpours, it's an excellent, very excellent resource as you found out. But if you're in a situation where you're doing so much of it, you know, it just begs the question, really. Anybody else? Well, no, no, I mean, the conversation didn't go and this is precisely the reason I wanted to do this presentation because I'd like to not only change hearts and minds of other people, customers, but challenge my own assumptions about what can be done, what should be done and, you know, really try and, well, also try and focus on seeing if we can do the thing that the next project's kind of good at, which is, as they were saying earlier, that it's been proven over and over again that contributions actually have a net benefit effect. And this idea that it's difficult to see that sometimes is the thing that I'd like to try and get over. And obviously in this case it's very interesting to see that that's the case. Anybody else? Yes. So which kernel would I choose to go up to? I guess that's something I've not actually thought a huge amount about. I guess I would have just chosen the latest ones right out of the box. If I wanted to avoid device tree because I wanted to leverage or refactor the existing code as easy as I could, maybe that would be a factor. Again, no, that's something, well, again, that's going to be part of an investigation, isn't it? Whether actually it would be better to choose a particular version. One would assume that the highest version, well, I would, I'd assume the highest version was the obvious choice to make, but absolutely there are other concerns, like if you want to minimize the amount of porting you're doing. Yes? Yeah, well, ask me next time I come round with the lecture, the corollary of this that says, yeah, we did the new kernel and I really wish I was back in 2009. Like I said, so no, I don't know. I think customers tend to be a bit skittish about this sort of stuff and I think that, again, this is more about psychology than it is about technical or maybe it's both and it's kind of, it's an interesting one and I'm interested to see how this develops and how I think about it developing and when we get a new project, it'll be interesting to see actually how we go about discussing the initial pitch and what we say. So yeah, still in flux, really. It's in a bit of a twilight zone. Anybody else? Well, look, thanks very much for listening. I'm really glad it's, oh, there's somebody, sorry, sorry. Sorry, terrible eyesight. Well, yeah, so UC Lib C was the limiting factor for having to use Linux threads, which was the precursor to NTPTL. So yeah, it was. That changed in 2011, so I had to get a kind of a build route that had a post 2011 UC Lib C. In actual fact, I think that we could have got away with just going full blown, G-Lib C and not actually need to use UC. I mean, we've got quite a bit of RAM. A lot of it's used for video data. You can see the NDA wolves approaching when I talk about this stuff, but yeah. Well, fascinating, right? So is your assumption that compiling a new two chains just added the question? Yeah, so when you've got less control over it, that's definitely, yeah, no, I know you're paying on that. I do know you're paying on that, but if you manage to... Well, what I did, I did a bit of spike work. I tend to go a bit unilateral sometimes and go, come on, I can do this. And I did a bit of spike work, come to a meeting and go, hey, I did this and it actually worked out really well. And we've got rid of million PIDs and now we've just got PIDs, we've got our TIDs. And that annoying manager thread was driving me insane. I don't know if you've seen, I don't know if you've ever worked with Linux threads. Well, maybe you have, but every thread, multi-threaded application has this random other thread that is just this P thread management thing, which looks after signals and all sorts of stuff. So again, another learning experience about another obsolete piece of software, so fantastic, and my brain's just full of really useful stuff. Any other questions? Well, again, thanks very much for coming and listening to me rabble on.