 My name is Bartosz Golaszewski and this is going to be about a concept that I would like to introduce or rather reintroduce to the Linux kernel that is the early platform devices and drivers and yes, I work for Bay Libre. We are a French consultancy working in the embedded Linux field. We're based in Nice. We're around 30 engineers at the office and a couple guys remotely worldwide. We do all kinds of projects. We work on upstream Linux kernel development and maintenance. We help our clients ship consumer electronics products from hardware design to software development. We are driving the kernel CI project and personally I've been in this field for nine last years so far. I contribute to the Linux kernel and to various user space projects and I created and maintain the GPIOD which is the user space part of the GPIO framework in the kernel. For the last two years I've been on and off working on supporting an older family of chips for Texas Instruments that is the DaVinci family of SOCs. The thing with DaVinci is that the upstream support for DaVinci dates back to the time when we didn't have a lot of nice frameworks from before in adoption of device tree and also we now still have support with board files and some boards and some SOCs are supported in device tree so this is all a bit complicated but we're moving slowly towards making the support for DaVinci more modern, more in line with contemporary standards in the Linux kernel and quite recently DaVinci got a new shiny common clock framework driver and during the work on this driver it turned out that we could use something like the early platform devices that I'm going to be talking about and the thing is that before that we didn't notice that it would be useful because it was all hard coded in Arch Mac DaVinci directory and now it's an actual driver and it turned out that we may need to change some things in the core driver model. So I would like it to be an actual discussion so feel free to raise your hands. We have mics around and let's have questions. So if you work on embedded systems you are guaranteed to have come across a concept called platform drivers. So platform drivers are basically drivers for devices that are non-discoverable, non-hutplugable so you have to define them somewhere and the concept behind the platform drivers is that you separate the driver code from the device definition and then you abstract it so that the definition and the resources for devices can come from many different sources. So nowadays it's mostly device tree for architectures such as ARM64, ACPI, I don't know much about ACPI I just know that it is there and for all the architectures like DaVinci it's the board files and either you use the legacy platform data or generic device properties which are better because they can be used the same way that device tree properties are. And then resources such as register ranges, interrupts, some higher level resources such as GPIOs or files in case of network devices are defined in let's say device tree or board files and then they are abstracted and fed through various frameworks to platform drivers and yeah so platform drivers are not discoverable on any bus so they have a virtual bus just called a platform bus and for the purpose of our talk the most important thing is that is when these drivers get instantiated, the devices get instantiated, when do they get created at the earliest. So in case of device tree we have an arch init call which is defined in drivers slash base somewhere it's called off-o-f platform default populate init. So as an arch init call it comes after the core init call and post-core init call everyone knows what init calls are so these are these callbacks that are called during the system boot in certain order and yes so in case of device tree we have this arch init call so this comes after all the frameworks the different frameworks for device drivers are initiated. The same pretty much the same goes for board files so this example comes this example comes from ARM but I think it's the same for most other architectures so we have another arch init call that is called from the init machine callback this is the init machine callback is the one that is in the machine description struct in legacy board files and in ACPI I don't know I assume it must be the same or similar but if you know ACPI let me know because I know how it's done in this case but there is a problem so sometimes you need devices to be initiated earlier than that earlier than arch init call and what do we do then so in this case so for now like the common consensus is that we should use this mechanism of declare macros so the mechanism behind this macro works like this you declare a table in your in memory you name it somehow and you put devices into a device sorry the structures into it and each structure contains a pointer to a compatible string in your device well yeah so this is OF so this is specific for device tree but let's go through it you define the device compatible string in this structure you assign it a function callback like a function address this is going to be your callback that is going to be called and then the architecture will go through the device notes earlier and scan for the for the compatible strings that you define and call these callbacks so the problem with that is this this is nothing to do with actual device drivers so these are just birth functions that get and device note structure and and do something about it so you don't get to use all the all the fancy api's that you have for device drivers no device resource management this is not really a big concern usually because these are not devices that will be removed but you don't get this nice abstraction layer for resources for for register ranges you you can't use the device specific logging and yes so the three main families of devices classes of devices that usually make use of these macros are clock source most clock source drivers and clock events certain clocks so certain clocks like when you have a clock controller certain clocks control that certain clocks need to be enabled earlier in the in the boot process I'm going to show it on an example of the idea of inchy and certain interrupt chips interrupt controllers so like the generic interrupt controller from arm needs to be initiated much earlier than than for instance GPIO expanders which which in a like which export interrupts which can you know be regular platform devices and so during during the work on the on the Da Vinci clock driver we came to a point where the clock has been implemented as a platform driver the clock controller driver was has been implemented as a platform driver and it turned out that one of the so sees doesn't boot with this clock controller with this clock driver and it just turned out that one on like on this specific so see we need one of the clocks for the clock source so this can you see this commit gap so this forced us to to introduce a not so nice work around which basically allows the driver to not care about the struct device so we need to check you know at it if deaths if that check if the device is null and act accordingly if it's not null then do something else and it also forced like this is a single user but it actually made the entire code change its way so we no longer can use the DevM functions and and all that and I started looking at it so because I I didn't really like this commit and I started looking at what what is already there and can be used to to to come back to a more unified code in the driver and I it turned out that someone already had tried to implement something like early platform devices and drivers so this this was back in 2009 so so not not so long after after TI after the support for DaVinci was introduced in the kernel so this was based on early param so basically it was invoked by architecture calling the power silly params functions function and this was question so this was not very convenient to use because you needed to deal with with classes of device you needed to use certain strings that would describe a class of devices and then this this class string would be passed to this early param the partially params function and post a bit complicated and also the thing is that the problem with this one this solution was that it made it seem as if it was a part of the of the driver model but in reality this these early drivers only used the same structures so strike device track platform driver and so forth but they these devices never really became a part of the driver model so they were like separate entities just reusing the same structures and yeah so this is very specific to the SH architecture and blackfin but this is now removed the the only user outside of arch SH is SH mobile in arm so and I actually have a patch series pending on on the on the LKML that actually moves all this code into arch SH but I just don't get any response from from a safe maintenance except that there wasn't any any resistance to to this part of my work so I know how to get it merged yeah so I started thinking about actually so initially I tried to reuse this one I sub into the series that adds that cleans this framework a bit at support for device tree but this was rejected mostly on the grounds that I needed to make changes like I needed to somehow mark the early devices in the device tree and since this is nothing to do with the hardware description it was rejected and rightfully so so I started thinking about a new solution and the idea was and the idea was to have something that that would be much simpler that would be a part of the driver model so I would like to have early devices that would actually become regular platform devices once all the older ones the entire driver model is initiated and I also wanted to make it much simpler so without the the the burden of having to declare some early param classes and all that and for this particular case when we have both board files and device tree I wanted to support both so I came up with a solution where we are two new data structures they're called early platform device and early platform driver which is by the way the same that that the previous implementation used except that in my series we first we see we first move this old code into RHSH and this is the the new implementation so users will never have to deal with early platform devices directly they will just define them but actually we don't use we don't add any new fields or any new fields the structure that the user will be concerned about and for the early platform driver we only add a single callback that's called early probe and this is because basically we have we have to choose between the two we either have a single probe function but users will need to know whether they are being probed early or not or or normally and I think that cleaner solution is to have a separate early probe callback and then do the regular probe afterwards so the this is the only change we just have the early platform driver within with one additional callback and the general idea is that we have the architecture code calls the early platform starts function and then we register a post call in it call so once the the frameworks are initiated we call a function called early platform finalize and this what it does is that it seamlessly converts all platform driver or early platform drivers to regular platform drivers all early platform devices to regular platform devices and once this is done all subsequent calls to any functions related to early platform will actually become regular platform functions so if you have like early platform device register it will after after this this post calling it call is executed it becomes regular platform device register and what it does so the advantage of this is that we now have we now have access to all these APIs that we use with with regular devices of course still if you probe a device very early you need to pay attention to what you're doing because some or many frameworks many many many things may not be initiated and available yet but in general this allows for code unification which I'm gonna show you so yeah let's go through an example so this should be my brand yeah so this is the branch that that was last yeah with the series that was last submitted to the list so with the implementation itself I also included a an example driver it's I think yeah so this is just a dummy driver that shows how to use this new API it just has a an example data structure so this is how you do it you define your own your off-device ID just as you would with with a normal platform driver and then you register an early platform driver and just as I as I told before there's an early probe and then regular assignment to the to the platform driver structure and yeah so we have two probe functions the early probe function and the normal probe function so we get to use all these APIs that we would normally normally use so we can use the device specific logging we can set driver data we can get platform resource so we get to actually the resources from device tree and then board files are actually getting they get they get converted to the standard form you don't have to parse it either manually or using the the OF underscore family of functions that deal with with device notes yeah and then you have the normal probe which actually can get access to the same the very same structure that has been that has been assigned as driver data in here the driver just shows how we can retrieve driver data in that we are signed an early probe in normal probe and as far as device registration goes it's quite simple so so this is the how it looks in device tree all this device is this is very simple so we just have the reg property but this this is how we would do it so very very simple and then in board files yeah this is just a dummy resource and then the registration of the of the of the early device so as you can see we register an early platform device but in reality we just we just use the platform driver we don't use any any of the private fields of the structure any questions so far okay so I got some feedback like this this was the series the series that I last submitted and I got some feedback so repairing said I skimmed through just just a question so why do you need to wrap the platform driver into a new early platform driver why don't you just put the early probe inside the platform driver so why what's the purpose of wrapping the platform driver into a new yeah so yeah maybe maybe I should have described the architecture so the idea behind this early platform drivers is that we contain a very small framework for early platform drivers separate like that is separate from the from the whole platform driver model and then the very core platform the very core driver model of the Linux kernel because they are not some some things may not be initiated yet so we just initialize so we want want to have something separate that is initialized from the start so it's very simple you simply have two lists one holds the early drivers structures and the second holds the early early platform devices and once the either the driver gets registered or a platform device gets registered they are matched and if you find that driver that matches this device you do the early probe and yeah and and and you can you can run your code and that you need an early probe otherwise if you don't match it they stay in the list and then you're gonna match it at a later time once a new driver has been added and then what it what it also grants you is that it supports probe defer so unlike of declare when when you have to actually pay attention to the order in which these devices get initialized in this case you can simply return from even from early probe you can return E probe defer and there's a device that that has been marked as deferred is going to be probed at a later time once other resources are available so this works like this is a minimalistic version of the of the real platform device mobile this is how it works and and this is why why why we wrap it in a set of separate structure so we don't want to reuse any fields from from real struck device or struck platform device we just want to have something separate that after our post-eat call callback is called no longer matters so just to give you an idea for instance the the previous early platform driver implementation actually reused to link the devices it reused the dev res link so the the the the struct list node that was used to that that is normally used to link the dev res structures has been reused to link the devices and then at at some point it was clear to this the dev res link was cleared but this was very confusing because if you try to use the any any dev res function in from your early probe in the previous the limitation would break everything so this is why I I wrap it in a separate structure I don't I just don't want to reuse anything from from from from the from the actual structure this is something so so during the review Rob Herring pointed out the same thing like why don't you reuse this why don't you reuse this and I I told him the same thing like I don't want to reuse it on purpose because it's confusing if you if you try to have another question so if you you said that it supports the third probing yeah so suppose your describe you describe your clock events device correctly in DT so it has a module clock and it's part of power domain and it's deferred when will it be retried because you need a clock event well drive quite early so yeah it will not work but let's just say that yes I tried to make it not not only for the for the for my use case where I don't need it but for for general use case and this is I don't have a use case in mind we're in early device probes in in doing an early device probing you would need a probe defer but it works in that it's this driver is marked as deferred and then once another device is probed another early device is probed or another early device drive I think this is not needed but I also also do it if another device early device driver is registered it's going to be retried like reproped and so this works pretty much the same as normal prob deferral is just that it works on the on the early platform structures yeah so you have still have to make sure that the clock driver is initialized first because probably the clock the timer needs a clock oh yeah control me the clock and if the clock driver needs interrupt to then your screwed you can only get as far you know with so in this case yeah without without the clock source driver you're not gonna get far yeah so let me ask the most stupid question ever why is the normal driver model available so late and what prevents from bringing it earlier because after all these just data structures that you allocate in memory with pointers and you can already do that at the early stage so there must be a fundamental reason why it's available so late and and why adding another like layer is needed to have it earlier rather than having the real thing earlier it's a very good question I also asked myself the same question I just kind of assumed that someone someone did it on purpose I didn't investigate it but maybe maybe we should yeah I was kind of afraid of asking this question on the mailing list yeah without a timer your system may boot up fine if you don't use m-sleep so what m-sleep m-sleep yeah so because MD late uses the delay loop so that works fine but m-sleep relies on working interrupts and a timer yeah and actually I've seen cases where everything worked although the the timer was initialized but as soon as you have some driver that actually uses m-sleep you have a problem how is m-sleep related to just probing things and not not not only m-sleep if you do anything that actually relies on the I mean it could be if you need m-sleep it could be like a resource that you need to have a sleep-based thing and so you could do a probe defer and wait for that that thing to be available then because probably if you need m-sleep you're not one of the early driver and you could like consider that as a resource and defer that probe and get reprobed once that sleeping with a real sleep becomes available right so in theory I theory the driver model is just data structures linked to each other so there should be nothing that prevents that as soon as you have like came a lot available pretty much you can you can do it right yeah so usually like I haven't seen a case where you would need to initiate a device without slab available so even even the CLK declare or the whole of declare is called after came out is available yes so I think there's yeah back back to the deferred probe from early platform drivers as soon as you've allowed a defer aren't you acknowledging that there's no need for that to be an early driver because it's not going to get instantiated until really much you've done that so I started thinking that maybe yeah as I said I personally didn't have a use case for that it's just that I sort of think that maybe you have an early platform driver that depends on another early platform drivers or driver I haven't seen such use case maybe there is the deferred until you get to late in it calls so what you're not going to deal with your deferred probes until you get to late in it calls so that second early drivers never got to get yeah but no no this this is not the same deferred probe okay this is a separate mechanism in the early platform implementation maybe I'm gonna show you that makes sense yeah just just answer that's good like maybe we do have a use case for that because we need a we need the PSC clock in Da Vinci for the clock source driver and they are they both should be early platform devices right because right now the clock sources in Mac Da Vinci it's like hard code it's not a real driver but once we converted to a real driver unless we do have this early platform in place we would have to use timer timer declare so I think we do have a use case for that actually so basically what we do let's say we have two like for example of Da Vinci we need two early platform devices one is going to be the PSC clock and one is going to be the clock source driver and then let's say that one of them the clock source driver gets eerily probed but it the PSC driver is not yet probed so it differs the probe we probe the other one and then we can come back to the clock source driver is it that does that doesn't make sense yeah there's a question I believe you also need the the potentially need the deferred probing to assure that an interrupt controller early platform device has been initialized before your timer because if you use the same struct then you don't have the currently hard coded ordering of the various OF declare options did you follow no no I good loss sorry I can repeat that when you're using an early platform device for a timer yeah then you will likely have an interrupt input into that one yeah and for that you need to have the interrupt yeah also have its early platform driver yes initialized and that may require them to have the deferred ordering at the early level implemented to assure an ordering of those two types of devices or did you still have any particular hard coded ordering of which types of early platform device yes so the thing with Da Vinci with this specific platform is that a lot of things are still hard coded in in arch slash Mac Da Vinci thing so this like you know we have these board files and even in case of device tree you still have some some quirks in the in the board files so this is not yet in the state that it should be and I hope we will move towards it but yeah this is how it is right now so yeah you're right so basically the idea behind the early platform devices implementation that I did is that it's a minimalistic version of the real platform driver model and so when you when you have dependencies between early devices you need probe deferral or you need to track you know dependencies in the device tree but that's that that is not happening any time soon so yeah I'm speaking of a lot of ignorance so if I say anything stupid just tell me it's stupid but it sounds like you're getting bigger and bigger and bigger and you want to replicate I'm overseeing the case but eventually you want to replicate the entire device model just earlier once you get all these people who keep in their special cases and they say oh but that would be great all but that would be great all but that would be great so it sounds like what you just acknowledged a few minutes ago is you just assumed that you needed this this tool you didn't look at what the real use case was whether it really is required right now or really which use cases demand this functionality the point I'm headed toward is if you don't really constrain it and corral it into the smallest possible piece it will naturally want to grow that's just the way our software is and people find new ways to use it all of a sudden it becomes this big bloated functionality so if you don't understand what the real use case is and don't have a way to constrain it we're just doomed to it becoming a really large monster I think yes so end of philosophical statement so just for the record like my implementation like the core the the core in drivers slash platform is 300 lines of code something like this even less for now may be there there there needs to be some some locking in there this is not perfected but for now it's around 300 lines of code and I have I'm under the impression that if it was it was widely adopted it could save a lot more code that could be removed so for instance we have a family of functions that deal with devices that deal with for instance device properties like the device properties and deal with with with struck device in general and then we have like for for many of these functions we have something that re-implements them but actually deals with struck device node and not really device and this is only used in those callbacks we call from OF declare and I think that is if we moved all all the all the drive all the users of OF declare code to early platform drivers we could remove all this code because it would no longer be like relevant so yeah I think the Frank the I think the use case is fairly well understood because it's you I elated bars you elated that it's just CLK OF declare RQ chip OF declare and what is it timer or time or whatever the name clock event to OF declare so just it's just three things that that today are using early and we need the timer and for the timer we need clock and interest and that's that's that's what we need so it's yeah I'm just wearing well bounded I'm just hearing people wanting to add layer and layer and layer when we don't really need layer and layer and layer so if we do understand where it's needed and limited to that that's that's great so I like what you're saying also like one more thing I'm now thinking about what we said earlier that why do we have device the device model initiated so late so as far as if I remember correctly we use a kernel thread for deferred probing maybe this is the reason why it's so late and we cannot use this kernel thread earlier yeah I really do you know what the what the reason was right we don't actually do that deferred probing processing until really late in the boot so we would have the threads at that point you're just putting the deferred probe onto a list early so that's not a big deal right I think a lot of it is related to PC legacy but like you had a interrupt controller early available you had a some timer available without needing to know from a clock controller what the exact clock frequency was and actually what Thomas said that I think you can merge early devices and normal platform devices if all threads operations and M sleep like that would return minus probe deferred and then another thing was that you would initialize and if you if you use it like that then the system would probably start working correctly with but there will be lots of probe deferrals because you you don't have timers and that and then the second thing to improve that they've improved the efficiency of that would be to take into account relations between consumers and providers which you already have in DT so right now the probe defer it would not be used if the system would already know that as a P handle to a clock controller so and the clock control is not there so it it doesn't make sense to probe the device yet so right now it probes the device anyway and then it returns probe defer so I'm I'm thinking that if I propose to make the driver model available earlier as early as I do with early platform devices there would be much more resistance to it from from the maintainers than to to this new implementation like just just just my thoughts so also are you sure that memory allocation is available at the time of yes yes because my example works and it and it uses assist yeah okay no you actually have this function is slap available or something and I called it and make sure make sure that that it's available yeah it's available for all of the clear callbacks so this is exactly what I encountered like like last week I sent the patch for I mix and do it exactly in the same situation because the clocks so it was a off declare and then Rob I think told me that hey you need to make that platform tried that and it didn't boot up so I'm guessing if you're doing that that would help us too so there are a lot of use cases that might benefit I've been actually actually CC'd on two other patch series which encountered the same problem so it's it's common up obviously so this is this is what the what the DaVinci driver does now we have two entry points we have one entry point with it which is the CLK off declare and another entry point to the driver code that is the probe function so this is not very clean because you have to make sure that both cases work if you start the older if you start supporting older as you see it's like the Cortex a9 where then you don't have that and then you need early blocks so yeah like you said the board files legacy systems they all basically implement those those those very basic features by hand it's these are not drivers these are just poking some registers enabling some clock sources and this works like this so yeah so when I send the series I got nice I got some nice feedback from repairing set I skimmed through this and it doesn't look horrible and I think this is this is very positive right so you said you had a deferred that was after core right I'm sorry which one of the slides said something like you would there it is you have a post-core in it yeah and you were gonna do a deferred after what this what this does is like we start we start the early platform driver framework very early and then once we reach the post-core in it calls we convert all early platform drivers into regular platform drivers and they become the part a part of the driver system of the driver model so later everything works like before so we make I in my series I make made some changes to the core driver model where if a device has already been populated it's not populated again so the core driver is aware of early platform drivers and make sure that it doesn't you know allocate a new structure and just everything stays in the same so if we use some dev res functions they will be available after this conversion but you can't use them before yeah yes you can so you go into early probe you allocate some resource and then once you get once you do the conversion and get back to this the same driver in normal in regular probe you get this resource back it still exists which is not the case with the previous implementation where it wouldn't work because you wouldn't it would never become a part of the driver model and it also never doesn't work because of the reusing the the diverse node and yeah so just these were these are the questions that I would like to ask doing this both so what should we or should I proceed with implementing this like what should I should I respond this series and then try again and is this implementation so you can you can look it up it's I think if you Google this one so this introduce support for early platform drivers you should find the series is the series any good is this the right way to to do it and or should we do something completely different and so I think this like if this would become while the adopted it's it's something that should that would save us a lot of code because we wouldn't have to have separate entry points for many drivers and there's a question over there if you have early probe and then probe why the question is why do we need late probe or regular probe if you have early probe what is difference what is probed the functions are the same and they have the same signature it's just that one of them is called early on and the second is called after the driver model is available so why don't read off the second one so what why don't why don't remove the oh yeah because we have drivers like the the one that the clock driver for Da Vinci which initiates some parts of the of the driver of like certain clocks it needs to initiate them early and then the rest can be initiated late or any any any such use case where you have some part of the code that needs to be available like executed very early on and then the rest the normal code is executed later because you cannot do you may not be able to do everything in early probe because some other frameworks may not be initiated yet so you just do your very basic stuff you I know you enable the clock source and then something something something more like on top on top of it you do it is it well defined what we can know what we can after because it's it's not very well defined no yeah because it depends on on the order of initialization they were yeah yeah yeah yeah so you cannot and sleep in the early platform drive like in the early probe but you can in your normal probe there's a mic so if you put the early probe inside platform driver and keep them in the same list right and then whichever driver has a early probe function so it's not now you call that and then later on you go with the probe if it if it exists why isn't that again why is this wrong approach keeping them together in a single function no in the single structure so that would be because I don't want to spam the regular I don't want to enlarge the struct platform device for everybody I just want to like because you know platform devices are used by most systems I don't want to make this structure bigger by eight bytes for everybody if it's only a small part of users that's gonna that's gonna need it okay it's the same for the for the early platform driver structure which also you know has some some private fields that that that are used by my implementation but I don't want to add it to struck device because it's already huge and we don't need more or that more fields in it so okay this was the idea also I don't want to read as I said I don't want to reuse anything from from these structures yeah so so the question is should I repeat this series because I I know it no longer applies to to the current kernel that there were some changes to the core driver code and curiously what's what's DevM functions did you use in your early probe used a mallet or I use the DevM KK mallet but this is this was just an example in my dummy driver this is one of the things so this is not the biggest concern because usually these drivers never get removed but still you may want to use the DevM functions for your higher level higher level code like the later code but you need to allocate some structure in the early probe so you just use it to unify the code to have the same entry point let's say yeah I say you need the K mallet garden IO remap and some of those things but I was just wondering how many how many did you try what did you yes I tried to find something that broke or you just you're just making a dummy and say well kind of well you know if you want to use DevM GP IO get something this will obviously not work yeah but I don't think this is the case for anyone so usually it's the the resource the register maps interrupts maybe and yeah and memory allocation because we're dealing with every very simple code usually it's not something it will not you know you don't need files in this in this case so yeah I use a couple the simple functions yeah and maybe that's enough maybe that's all you you know maybe it's like yeah you can use these but obviously you're anything more than that's not an early probe anyway so so we have any more questions or you want to say something okay so I think I think we're on time so thank you very much for coming and I'm gonna be here for for two more days so if you want to chat just stop me on the corner in the corridor and thank you