 Hello, my name is... What weird name over there when they listen to Neo. And if you know me, if you've ever deal with the Bigel Bone Black or the Bigel Bone, which is a really low cost open source hardware board by the BigelBord.org, and I'm here to talk about what it means to actually buy into the whole device tree thing and try to migrate from a platform device port file to a purely device tree architecture. As you see, we're going to talk about the world story, which is exactly what it was. It wasn't exactly easy because the big problem with the device tree is that it assumes that everyone has already everything ready to migrate to device tree. That means all the bindings are there, all the drivers support device tree. And as we found out, that's not what happens in the real world. And that led to many problems, but we managed to make it work. So that's what I learned out of this whole experience. So yeah, it's going to look pretty horrible at first, but eventually you can get it done and it will help you and you get something out of your troubles. So at first this was a bird of feather session, so I would expect you to participate and if you can share your stories. So how many of you here has done any driver development? Can you raise your hands? Pretty good. How many have done a platform device board file based platform development? Okay, how many have done a device tree? Okay, just a single guy. Okay, more. Okay, so I think we have a lot to talk about. So in the beginning we had the original Bigelbomb. It was named just Bigelbomb then, it wasn't named Bigelbomb White because there wasn't any Bigelbomb Black then. It's a very inexpensive Cortex A8 board. The latest release based on a normal board file platform was on a 3.2 kernel, but as you can see it was pretty heavily patched. It had almost 15,000 patches on top of the vanilla version 3.2 kernel. So as you can imagine that's pretty hard to mainline. It just wasn't going to happen. The thing is it worked. People sold it and lots of people used it on OEM products. There was a really big decision to actually try to support everything that Bigelbomb White did on the Bigelbomb Black and that led to most of the problems with the Black. So if you know about how a board file based platform works, usually you have a board file and on that board file you define everything that has to do with your platform. So you declare all your platform devices, you register them and then you support them. The problem with the board file of the Bigelbomb was that it wasn't just for the Bigelbomb, it was for all the AM335X based boards of TI and it was pretty huge, something like 4000 lines. Obviously not on mainline and it supported besides Bigelbomb quite a lot of number of TI boards. So when we started doing the Bigelbomb Black support, we took a look at it and said, okay, this is going to be really hard. So we said, oh, that device tree thing is the new way to support ARM boards, so let's do that so that it's easier to get it in the mainline. So we sat around to do that. So what was the biggest problem of the original Bigelbomb support was that the Bigelbomb has a number, it's capable of stackable expansion boards which are identified by a square apron and you read that and you figure out what kind of board you have and you configure the drivers accordingly so that they work. So this is a fragment of the board file there and you could imagine that this was going on for a few hundred lines even more just saying, okay, if you have this board, then I have to do this thing but the thing is you can stack those boards so expansion boards could conflict and then you had to do some really complex logic about what to enable, what to disable and in the end it was just very hard to add more expansion boards and on top of that, if this board was sold to hobbyists and all these guys want to do hardware with it, they want to create new capes and they want to do something new with it so they had, in order to support it, they would have to hack the board file and do the same thing for their own board, making sure that nothing else was affected and everything else worked so that they could send us the patch and then we could support their board out of the box but using the board file just wasn't going to happen, it was just too complicated so let's start about what happened and Linus decided, okay, no more board files, we need to do something different I don't know if you have read that thread where Linus just exploded and said, okay, you should really get your stuff together and you should do something differently so at that point in time the PowerPC device 3 support was completed and a lot of PowerPC refugees were in the ARM kernel community and they pushed for having device 3 as the solution to all the problems that the ARM kernel had so that's what started the ball rolling and that meant no more new board files so the thing is ARM is really complex and if you have no board files, how would you go out and support every different board that you would create and this is what device 3 is supposed to be it's just a structure that describes hardware and really that's all that it is, it's nothing more, nothing less but the other stuff that's happening now in the community is just more self-inflicted problems by the community so it's not a problem of the technology, it's just that the way that we set out and implement the technology and the way that we did things led to many problems it was hard to get anything in the main line since everything had to be perfect because the hardware doesn't change so the binding in theory shouldn't change but that's not the case in the real world the hardware might not change but that's exactly the same thing as saying that the driver should be perfect just by writing it the first time you shouldn't make any changes to the kernel driver but we know that's not what happens so that's what should happen with the device 3 bindings, at least in our experience so I don't know if you've followed the device 3 for dummy session and you know how device 3 works so you have a source file DTS let's use device 3 compiler to compile it into a DPD device 3 blob and then you load the kernel along with the device 3 blob and when the kernel boots it will unflatten the tree and iterate over the device node to create the platform devices and that's it that means that in theory you could have a kernel that would support any kind of arm board well, this is a very distant target for us right now so it is possible to support similar boards which is something that we do with the big board and the big bone but it's really hard to do that on any arbitrary arm board at least now because there's still work to be done there so what's the difference when you use device 3 is that you don't use platform data so what happens is when you have a platform device the configuration of that device and all the things that you need to set up for it to work was defined in a platform data structure which was passed to the driver so that it knows how to configure the driver but this now is gone so on your platform device probe method when you check your platform data pointer you see that it's null so you have to use open firmware to iterate over the device 3 node and pick up the properties that configure your driver so another big change is there's no more board or architecture specific drivers well, or they shouldn't be so anything that you've used that was specific for a specific board or for a specific architecture should be a generic driver and you should use that as much as possible instead of custom drivers for example on OMAP there was a custom pin control driver OMAP MOOCs driver actually that allowed various things to be configured even on runtime like pinmoxes and all that but this is a duplicate functionality of the pin control driver so that had to be removed and the pin maxing had to be configured by device 3 now so many of the old customers of the big bone white had big problems there because they based their design on using that driver and that driver just went away so they were pretty unhappy about that and the final thing about device 3 is that it's just pure data it doesn't do anything related to callbacks so you just define data and then your driver should act upon it there's no callbacks to your platform, specific platform stuff if you need to do callbacks you need to use some kind of device notification mechanism which is not easy to retrofit on existing platform data driver but up to now we just talked about the stuff that's just standard for every ARM board when you go out and switch to device 3 but there's something really special about the big bone and those are the capes so the capes can be really really simple just be PCB that connects directly to some pins in the SoC packets and you just add a minimal amount of hardware so that you make it do something that the SoC already supports so you could have I don't know an audio cape you could have a CNC cape you can have whatever you think you can do by using the capabilities of the SoC so the target of this board was hardware hackers and hardware hackers doesn't mean that they are kernel hackers there was no way that just a hardware guy can figure out what's going on in the kernel and patch it and configure it escape to work without considerable effort so what we had to do we had to enable it we had to enable people to do work without recombining the kernel and just by combining a device 3 overlay which will come up to that they could support whatever hardware they need to without modifying the standard angstrom image that we used so another complication is that big bone black has extra hardware on it which is a non-board EMMC and that's the AMI framer and those use the same pins of the capes that the big bone white used and they were available before so there had to be a mechanism so when you plug in a cape that used the pins of the HDMI framer for example the HDMI framer should be disabled and then you could use the pin that you want to overwrite that functionality and the same thing with the EMMC it mostly worked but there was needed different hardware division so that the HDMI framer could work reliably some resistors had to be adjusted in value another thing that we really want to do is make that all automatic so if you design your cape and you programmed your eProm correctly that identifies the cape as a specific cape which there is support for it it should just work, you should just plug it in and it should work, it should load the drivers that you need and you should have it working and you pretty much done that and you need the cape manager to do that the cape manager is just a specific big bone driver that handles the reading of the eProms the reading of the baseboard eProms so that it knows if it's on a black or a white and load the HDMI and the EMMC driver and then it probes the i2C bus where the eProms of the capes are and then figures out which device you overlaid to load loads it and then you're done, it's working after that due to that this is probably the most advanced usage of device in a shipping product right now so now that's the easy part so now we're going to talk about the hard parts the hard parts is the driver problem so I don't know if you know how silicon vendors work after they ship a product and they release the initial version of working kernel 3 they pretty much don't care about it they will only care about what's in that chip and what the drivers do if they use the same hardware IP in a more updated silicon version that's the only way that it makes sense for the silicon vendor but to keep on working on the same SOC because there's no more money to be made there all chips are not that... they don't have much profit there so when we started the Bigel Bone we were under the impression that device 3 support is there we just need to take the drivers included in the kernel make some minor patches and it would work but that wasn't the case we found out that the significant amount of drivers had no device 3 bindings at all so we had the case where we had a device 3 only board but half of the drivers didn't work because they didn't have device 3 bindings so if you just care about the board booting and having a serial port and ethernet but that's not how you can sell product you need to have almost every hardware IP in the device support it and support it right out of the box and we also found out that some of the drivers didn't have working bindings so that's different so the way that driver development happens it happens on many different kernel trees or subsystem trees so you start with the mainline tree that's your base and then you start pulling patches from whatever other tree you need to do to pick up patches so that it works but even after that it doesn't mean that they work together so you had for example USB it's pretty much... was really hard to get it working because right at the moment that we started working on it the whole... there was this idea that you should use something called USB 5 drivers but right at this moment there wasn't anything available so they were just in the middle of rewriting everything so we had to do whatever we had to make it work a little bit then we have cases where the device tree bindings that were there didn't make sense all the time so for example we have a camera cape so that camera cape needs a specific clock to be brought out to the connector and then being fed to the camera chip but no one thought about this use case when they created the bindings so you had the case where you had to support something but no one thought about that something needed something from the clock driver to be configured so we had to go there and fix that as well I don't know also about what GPMC is General Purpose Memory Controller but that's the interface where you connect memory map interfaces like satirgrams and Norflash the way that the driver is written is like you have a GPMC driver and then you have various drivers that you connect only specific parts that the driver knows about so let's say you have the GPMC driver and you can only connect nonflash and you can only connect satirgram or something like that but that's not the right way to do it in device tree the right way to do it is to have a GPMC subsystem and then have a driver of that GPMC subsystem export it to the specific board driver but it can use it to configure the GPMC interface for whatever it needs so we didn't manage to fix that but we know that this is pretty bad so we need to figure a way to fix it in the kernel so a lot of the bindings that are there only make sense for a very limited set of evaluation boards that the developers of the bindings had at that point in time but a lot of these things have to be reworked in more general purpose mechanisms and the final thing is you had to hunt for all the bits about the drivers that you need support all over the place so you had the main land tree you had TI's internal kernel trees you had some bits that we had to pull from the DRM kernel tree we had to pull some bits from the USB developer tree so it was just all over the place and I think that's the problem of the device tree bindings acceptance because the developers submit the drivers and they end up in the special kernel trees of its subsystem and then when the subsystem maintainer tries to push it to the main line kernel some device tree guy would just say ok this binding is not good, this binding is not good and just bindings stay on those trees for a few months which is pretty bad when you have to ship something and of course there is something called booting mainline what vendor usually means when it says I'm booting on mainline means I have a serial port and I can boot over in ETRD technically that's correct but that's not what the customer expects when you sell a board to a customer that says I'm running mainline 3.12 or something they expect all their hardware to work they don't like saying ok I can boot with this kernel but I don't have audio, I don't have video, I don't have USB which is now the case I think so there's this Cuts22 which means the vendors cannot deal with turn which is constant changing in the kernel without any visible progress being made and visible progress means I finally get my driver into the mainline 3 and I don't have to keep after it but the amount of time it takes for drivers to be developed and the bindings to be developed and submitted and accepted it's close to 3 months at least it could be even more and by that time the kernel has advanced so you might have to do more changes and eventually you will converge and getting it accepted but that might mean almost a year has passed without any visible change to the management the management would say ok you said that you would mainline the driver but where is it and this keeps going on and on so there's no monetary incentive for the vendor to keep on doing that for a long time so everyone will have its own vendor tree, evil vendor tree and when a customer comes to the vendor he would just say ok use my tree but that tree is too far away from the mainline and afterwards the customer needs to do a lot of work to get to switch to the mainline and the customers really do want to switch to the mainline because that's where every new thing lands and you can use whatever new peripheral is there but you cannot easily do that on a vendor tree because you need to do all the back porting work or you can ask the vendor to do it for you but if you're not a really big company the vendor won't care but if we're talking about hobbyists I'm pretty sure that TI won't care about doing that work just for the hobbyist market and the final point there is that even huge companies have limited development resources so from a financial perspective it doesn't make sense for the vendor to keep his developers but making changes just to get something in the mainline he mostly cares about having them used in the new parts so he can sell more and make more money and that's it but on the other hand the community doesn't really like all kernel versions so who here has bought a development board and found out that its kernel was 6 years back so did you like it? Yes The thing is at the point in time that you bought the board and the amount of resources that the vendor needs or you need to get everything updated it's just too much and most people just stay with that vendor kernel forever even though they hate it they cannot do anything about it because all the driver support is there but since we're talking about just open source guys a lot of distribution just they wouldn't accept your board to be supported if it's not mainline I think Fedora is one of them and you know a big point of the Bigelbond was that you can use whatever distribution you want so while it was shipping with Unxtrum there is a port for Ubuntu but we couldn't get Fedora to include that in the regular builds just because it wasn't a mainline kernel and if you do anything if you do your own board and you want it supported by some kind of open source distribution most of the distribution wouldn't care as long as your board is on mainline and of course you understand that it's a good thing for you if you have your board supported on any big name distribution because it means less work for you and you can just point your customers download this Fedora version and it will work on your board and of course the community is an infinite so if you have to support 15,000 patches and keep them around basically no one is going to do it for you you have to pay them for it and finally it's just not fun you know if you do stuff for fun this is definitely not fun it's something for people that are getting paid to do it so what's the main point about device tree and why people are whining about it I think device tree technically doesn't have any big problems but it can be solved the technical problem is that there is no checking of the bindings so if you make a typo in your device tree you won't find out only after you try to put the kernel and it won't boot but that's pretty minor really it's something that you can work on the problem is that it was solved something different than what it is it was like something that would immediately solve the ARM kernel problems which is there's great many diversity in the ARM community there are so many different boards and it's really hard to support it device tree does help that it doesn't mean that it will immediately solve all your problems you have to do your you have to do considerable work to do it so the problem with the device tree is not really technical the transition was handled pretty poorly I don't know if anyone around here used to be a PowerPC developer so do you remember the transition from ArchPPC to ArchPPC that was handled pretty well but that's not something that happened on ARM instead of having an old ArchTree where stuff would continue to be supported the same way that it was I would try to move immediately to the new stuff and that doesn't really work it's just so hard and then we have some maintainer hostility because you do know that the ARM kernel maintainer doesn't use the device tree so the whole thing about moving to device tree is a few years old but we still have the main kernel maintainer they don't move to device tree they still work on their own platform and board based stuff so who's gonna we need to put our best people on that thing so that they can find the bugs and report them and get them fixed so if you're not using the stuff that you're telling other people to use how are you gonna fix it and finally it was really easy to do board support with board files because no one cared no one really cared what went in there you could do whatever you wanted in your own board file it would just pass a minimal review and you could do everything there but that's not the case with the device tree first of all there's no place where you can stick your driver or stick your stuff without anyone caring and secondly the device tree format is readable so someone doesn't have to be a kernel developer to figure out that this is not right so you can see bindings that are not really right but without knowing anything about the board it just doesn't feel right and oops what did we get by working with device tree well we lost a few months obviously but it actually works it's not something that is impossible to do and for our target market which was hardware hackers they can do all that stuff that I say here they can create a cape and they don't have to recompile the kernel for it they don't have to modify the kernel they don't really care about what the driver does they just can configure it by setting properties and the device tree overlay the device tree's syntax is a little bit weird but you can read it without really being a software engineer and we kind of did what the whole point of device tree was to support different boards with the same kernel image and we do the image that we see actually works on both the white and the black and all the peripherals work on both without making any changes and that's the last kernel that we used 8.3.13 and we only have about 750 patches on top of that which is much less than 15,000 so let's talk about it now so many patches outstanding for the big bone support most of those patches are driver patches they will end up in the main line when the subsystem kernels are pulled into the main line kernel so we had to pull those patches in so that the peripherals work but we don't really keep them around just for our own board they are TI patches that will eventually end up in there SPI up on the big bone black so the big bone black 3.12 or C with 60 patches gets you most of the way How many of those patches revert prior patches? Not for ourselves Ok The problem we talked about device 3 are the present work with the PowerPC? PowerPC is considerably easier because it's not as complicated as ARM ARM they do exist but when device 3 was done for PowerPC we didn't have this idea of device 3 as an ABI or that device 3 only describes hardware but we didn't really use it when we created it it was really more free form I didn't see anyone whining so much about device 3 on PowerPC we just did it it wasn't this much of a problem or are we planning to unify firmware API or whatever it is because today's device 3 maybe someone has come up with a great idea there are at least 3 versions but we know it's device 3 it's ACPI and it's also something called FEX but I'm not sure I'm going to use that it's a Chinese thing were you in Dave Hart's presentation about the Mino board? you know the answer probably it's going to be a unified firmware property access method you could either have an open firmware backend or an ACPI backend this is general Is that all going upstream? I'm going to repost them next week the thing is the device 3 overlay support will probably be easier to get in than the cape manager but I know that there are people that actually use that so it seems to work but we just need to push it the syntax might change but it's possible that other cape manager vendors can use the cape manager concept as a whole there's absolutely nothing big or bone specific so if you had a platform with the same I2C IEPROM access methods you could easily do that on whatever ARM platform or PowerPC platform whatever you want there's nothing big or bone specific but you could get another possibly simplified board file well device 3 is doing the work of the board files you cannot do what cape manager does with board files because it's completely dynamic at the moment that the board boots you don't even know what's connected but a firmware u-boot in our case doesn't even know what's connected on the cape connectors so it's a kernel that has to probe it considerably after the board init method has been called well in board generic it's a more general way to do the board files we can talk about that later yeah we did and I think there's more capable of answering that So I was born then bootloaders are evil everything and everything in the kernel we cannot replace bootloaders basically because they work with monthly hardware and then 2 years later at the bar street and we said oh we have this new board with even more things and they say move everything to the bootloader so we tried moving it to the bootloader but then you had to match your bootloader but you wake your kernel and that's how you work you don't have to match your kernel well but that doesn't work it's just you expect that everything will be perfect right from the start and that's not going to happen isn't it? nope well I can do it, if you can do it that's okay yes Well Well there's a big difference about your audience if your audiences are professionals they will be trained on working on your system that's okay maybe you can educate them to do that properly but if your audience is people that buy a development board for 50 bucks it's impossible to have them understand how to do that on your boot but you have considerable Linux and you boot experience so you end up with brick boards I'm saying for example the device tree before passing it to the kernel based on what's detected in the problem that's not something you leave with the customers that's something usual the customers make sure they have to do a bootloader and the kernel is booted but the thing is that means that everything that the customer is going to use is provided by you but one case is that the customer might want to create new boards new hardware In this case you can handle the case where somebody has a breadboard and they're plugging wires into the header they can have a device tree overlay for that breadboard and they actually can remove that device tree overlay make some changes, reload it without having to reboot at all We had it on a vehicle board and we had RMAs for people being unable to figure out how to edit a textile So that doesn't work out Russell King, the maintainer he cannot figure out how to refresh you on all his boards So even Russell King cannot figure out how to refresh a bootloader 90% of his users won't be able to and that is the problem we're facing If you ship with a bootloader and you do it 2 years ago people will be using that till the board dies and that's why I'm moving it to the bootloader if we can get everything working perfectly In one go for all future hardware then yes but the reality back seems different Well I think, I don't know if you follow the latest device tree thread and ARM Linux kernel Well there's a big master and Nicolas Peter said that you cannot hide complexity so if you have a really complex system eventually you have to do complex stuff to figure it out and make it work but we have a choice either we split things up and have some complexity in the firmware or instead of having a complicated firmware have a single firmware and then do everything in the kernel and I would prefer to do it in the kernel because I can fix the kernel I'm pretty sure I won't be able to fix a broken firmware easily or at all Ok there are people that Ok but I'm pretty sure people won't even sit with either uboot or barebox they might use their own bootloader which really isn't like barebox at all It's nothing, just boots, that's it The problem with the bootloader basically is that it's a boot time and we have very very many people tinkering on the big bones during runtime That's the biggest problem and whenever you put everything into one giant ttb block that gets parsed once they have to reboot over and over and over and over again and basically just judging from the support questions on booting things we see in hash deal and hash tanda once they get the system running keep them or make them keep it running never shut down again that's basically the bottom line And just on that we uncovered pretty a lot of bugs in the way that platform device removal works because up to now really it's impossible to remove a platform device after it's loaded but we found out that when we removed an overlay and we unregistered the platform device almost 90% of the drivers don't work correctly So that's another thing that we need to fix Ok you sir Ok so the overlay is a mechanism to augment running kernel's device tree with something else So let's say let's just forget about the big bone and everything and you have a kernel that's device tree and you connect something that brings out a new uR and you want this uR to be visible to Linux and applications You can load an overlay that just says insert this uR driver node in the running kernel's tree and apply it and it will be exactly the same as if you had that uR device node as part of the booting dTB and it will work exactly the same Yes We know because some both altera and xylings are thinking about it but they are thinking about it it doesn't mean that they are going to do it You mean like a very low to dT converter They will do it I'm pretty sure that's how they will do it because they have their tools and they generate So Is anyone trying to convert from platform device and board files to device tree right now or thinking about doing it? Okay Maybe it's easier at that point Okay