 Hi, my name is Marti Bolivar. I work for Nordic Semiconductor for one more week. And I'm here to tell you about system device tree support in Zephyr. So right now I want to give you a quick overview of what we're going to talk about. The first thing that I'll mention is that the work is currently paused since I'm transitioning jobs. But it will be picked up back again. So this is not just still work, but I do want to let you know that just right up front. I'll tell you a little bit about myself, recap how device tree fits into Zephyr. And more importantly, talk about why the way that we're using device tree in Zephyr is not scaling anymore to meet the needs of today's hardware really and tomorrow's hardware quite certainly. Then we'll talk a little bit about evolving specification called system device tree that we think can help solve the problem that we've run into with device tree in Zephyr and in other places. Talk a little bit about the contributions that have been completed already to add system device tree support to Zephyr and outline the next steps. So Nordic will continue this work, but the details are TBD. At the end of the presentation, I'm going to give you a reference to a GitHub issue, which is sort of the umbrella covering all of the work that's ongoing in upstream Zephyr to meet the needs of the heterogeneous SOCs that we're increasingly seeing. And so I will encourage you to follow that if this is something that you're interested in. So about myself, I co-maintain the device tree subsystem in Zephyr. And I've been working on system device tree support for a while and I have contributed a bit to the system device tree specification itself. To recap device tree, okay, so if you're here, you're at this effort developers conference. I'm assuming that you've used Zephyr and if you use Zephyr for better or worse, it's a little bit inescapable at this point you've used device tree. But here's a quick recap just in case. So this is a picture from our documentation about the configuration phase that happens in CMake, right? And that's way too small for you to read, but this is all device tree. So not only does device tree sort of happen very early in the build, it then goes on to influence just about everything else that happens in the build, right? It is where all the SOC hardware is described, where it's boot time configuration lives. It's, you know, including the things like the system clock, very core kernel objects. It influences which drivers are selected by default in the device model. Then within each driver, it decides basically which devices get instantiated and it will handle a variety of application-specific hardware description and configuration if that is what you choose and quite a lot more. So this is sort of a central abstraction and it's done a lot of work for us. It's continuing to do a lot of work for us, but it has hit a scalability limit. Let's talk about that. So multi-core AMP, and by AMP, I mean asymmetric multi-processing where we've got a variety of different cores in a single SOC, maybe with different architectures. Maybe some of them are in an SMP configuration, but they have to talk to something else over there, over shared memory with some inter-processor interrupts. This is not as well supported, kind of in general, in Zephyr, as we would like. And in terms of sort of one of the dominant architectures, multi-core ARM V8 MSOCs with trust on support were sort of the last straw in terms of our ability to work around it within the bounds of the existing device tree specification in particular, but sort of generally in the Zephyr hardware model in general. So why? What are some examples? So memory addressing is a pain because in device tree, a node that represents a hardware peripheral has a single address, but in ARM V8 M, the peripheral addresses will vary by the security state of the SOC. So there's no way to write a single data structure that actually captures all of the addressing. We have therefore a big problem, especially when there are multiple cores on the SOC, because then you've got some static memory allocations which happen in device tree that have to be duplicated across all of the different builds that you're doing for any core that is involved in your system that has to talk to the other cores that are in your system. So this is obviously very cumbersome, it's very error prone, it has stopped scaling. The IPC resources are kind of a pain, right? Because if you wanna allocate this SPI peripheral over here to this core and that you are over there to that core or that you are over there to TFM, but this you are over here to your main application, that's yet another big configuration layer that you're not really able to do in terms of having a single data structure that can encompass everything that you need to define for the whole system. And of course, this is only going to get worse, right? The general industry trend is that we're incorporating more and more cores in just bigger and bigger AMP systems in a single SOC. So if we don't rise to meet this challenge, then it's gonna be a continuing problem for us as maintainers and kind of more importantly for anybody that is using Zephyr. So this is why we have been looking for new technologies that can try to address this problem. And I wanna tell you about one in particular called system device tree that pertains to how we think we can do this at the device tree layer. A little bit about system device tree. So it is a backwards compatible extension to the device tree specification. So device tree is a standard language, it has a specification, and system device tree is a separate specification that is a backwards compatible extension to it. It, like the regular device tree specification, is developed in the open. In a very nice set of circumstances, the people who develop the system device tree specification are in constant and regular contact with the people who develop and maintain the standard device tree specification. And of course, the people who develop and maintain the standard device tree specification also happen to be the people that maintain device tree within Linux. So the fact that this is sort of an existing standard that is hosted by the device tree organization with a lot of cross-pollination between all of the people that we rely on to keep device tree alive and well was very interesting. And if you download the slides for this talk, you'll get to see that the specification itself is hosted on GitHub, and you'll be able to read more about it there. So I'm not gonna obviously read the whole spec to you, but I do wanna talk a little bit about some of the areas in particular that system device tree seems like a great fit for solving the problems that I outlined earlier. To do this, we're gonna have like a little vis-a-vis, you know, comparing contrast table. So on the left, we'll have a standard device tree feature, and then on the right, there will be the name of the system device tree feature that sort of addresses a problem with the standard device tree feature. Let's start with cores. So in standard device tree, we have a node slash CPUs, and slash CPUs has, well, okay, all the CPUs that should be relevant to the system. And it's assumed that the software that's running on the client program that is consuming the device tree has access to everything in the device tree and that all of the cores that are relevant are under slash CPUs. Well, that just doesn't scale to the AMP world, right? Because different cores are gonna see different things in terms of what their memory map is, what they have access to, et cetera, et cetera. So in system device tree, there's a concept called a CPU cluster. And CPU clusters, you can basically think of this as a generalization of slash CPUs. So slash CPUs is basically kind of a degenerate case, if you like, of a CPU cluster. And you can have multiple CPU clusters within your system device tree, but they're all independent, and they can each individually contain cores of arbitrary architectures. So I can have one CPU cluster over here with a risk five core and another CPU cluster over here with an ARM core, and they don't necessarily share any other resources, but we can fit them in a single data structure. So that allows us to kind of model the processors in an AMP SOC in a nice way without having to do all of the kind of workarounds that we have in Zephyr with lots of pound includes that branch out from which board you're using. Next up, let's talk about peripherals. So if you've looked at a standard device tree, you've probably seen this slash SOC node that tends to contain all of the peripherals on the SOC, all your spies, all your UARTs, what have you. Now you may have noticed actually that slash SOC has a compatible. It's not magic. The compatible is magic. The compatible for slash SOC is simple was. And as you know, compatible for a device tree property sort of says what it represents. And simple bus is actually a very special compatible because it has a meaning that is defined in the device tree specification. And that meaning is anything inside of here is visible to the CPUs with the trivial address mapping by default and you can configure that with some other stuff. So the problem with that is of course now we have a bunch of different cores and they don't all see the same peripherals. So what can we do? So in system device tree, they have a concept called an indirect bus. And in exactly the same way that you could view a CPU cluster as a strict generalization of slash CPUs, you can kind of view indirect bus as a generalization of a simple bus. So anything under the indirect bus still has a way to translate addresses from the contents of the simple bus or the indirect bus node to the root address space or the address space of the root node. So you can know what the addresses are of the things inside of the indirect bus but you do not have any assumption that anything can see those things. They are sort of invisible by default. And that's useful because we can map them in to CPU clusters via some properties, right? So you can say here are, here's my risk five cluster, here's my arm cluster, here are some indirect buses. This arm core can see these two but not that one. This risk five core can see these two and maybe they both can see this one but these ones are independent. So that is extremely useful because it allows us to explicitly the model of the hardware that is particular to a CPU cluster but also shared between CPU clusters. Again, within a single data structure without having all these sort of town include workarounds that we have relied on up until now. Next, let's talk about this notion of the client program which I kind of briefly mentioned earlier. If you read the standard device tree specification it'll talk about a client program and this is just the thing that consumes the device tree, right? And the is important, right? Because there is one thing that is assumed to consume the standard device tree. And of course that's not really a valid assumption anymore in an AMP world because you have a bunch of different firmware images that are executing at various execution levels maybe in lockstep on one core or maybe it's a bootloader that hands off control or maybe these are things that are concurrently executing. And since there's no model for that you run into a lot of problems like I was alluding to earlier in terms of defining even where your images get linked if there's a shared flash, right? You have to sort of duplicate this information and it's a hassle. And so what system device tree provides as a feature is this concept called an execution domain. And so an execution domain sort of explicitly modeled the different firmware images, the software images it could be a hypervisor or whatever that are executing on your SOC. Now that's great because you can restrict execution domains to just run on a subset of the CPUs within a CPU cluster. So you can define as many domains as you want and then you can say each of them runs on this cluster but not only on this cluster but just on these CPUs within that cluster. So that's useful if for example you want to partition an SMP cluster into this core is running Zephyr and these cores are running Linux. Finally, you can also explicitly restrict the peripherals which should be accessible to the execution domain, right? Earlier we had an example where we want TSM to have this UART and the application to have this other UART. So it's not just the case that if we have a CPU cluster which principle can view this indirect bus that we necessarily want the execution domains that are running on the CPUs within that cluster to have access to all of the peripherals within the indirect bus. So the concept of an execution domain allows you to also explicitly say you run on these CPUs in the CPU cluster and you see these things but not anything else. So that's lovely because it again gives us a sort of single data structure to define our peripheral access, our hardware map and also what SRAM, RAM and Flash each execution domain runs on and has access to you. So the last thing that I wanna talk about is this notion of how many data structures are there. So it is really nice that a standard device tree is a single data structure and like I mentioned earlier, a standard device tree is super useful and therefore we have built a lot on it, right? It's served us well for many years. Those of you who were here at Anastas Talk, you know, device tree was introduced several years ago and it's scaled quite a bit and it works really well and we don't wanna break that. So another sort of useful feature of system device tree is that you do have a single data structure, the system device tree, but you may be wondering, okay, how am I gonna access the data in the system device tree? Am I gonna have to rewrite all of my boot code that looks at the device tree or rewrite all the crazy macrobatics that you have in your driver to instantiate your devices? And the answer is no because the system device tree specification authors sort of designed it specifically so that this individual data structure can be post-processed into separate standard device trees and then you get to reuse the existing client programs that you already have put together without any changes. It kinda looks like this, right? So you've got a single system device tree and then you sort of restrict the view of the world to what's available in a single execution domain, which like I talked about, then gives you a view of the world that is restricted to a particular CPU cluster, is restricted the particular CPUs within the CPU cluster that the domain should run on and memory and peripherals and all that good stuff. And then you get a standard device tree that looks at exactly what you want that domain to be able to look at. So it's a lot less error prone, but you also have a single point of truth on the left. Sounds great, all right. How, what are we gonna do? And the answer is just build. So there's a link to the docs if you wanna download the slides, you can look at it if you haven't already, but SysBuild is already in upstream Zephyr and it's been part of our work upstream to see how we can try to address this increasingly heterogeneous world. So SysBuild is our meta build thing, right? Any project that gets big enough has a meta build thing that builds all the real things and SysBuild is ours if you haven't used it. So it's sort of a meta build system. It sits on top of other build systems. So SysBuild sits on top and then it spawns additional build systems that are regular Zephyr build systems or whatever arbitrary build system you want, it's not restricted to Zephyr images. Now in the case that SysBuild is spawning a Zephyr build system, that build system will have its own device tree. And so this becomes a natural point to integrate system device tree within Zephyr. And this is kind of the architecture that we've been moving towards. So we want, and I'll sort of show on the top left, there's a system device tree for the SOC where we will define Zephyr clusters and buses, right? Then there's a board system device tree underneath there which can define board specific hardware but also perhaps a default set of execution domains, things like that that are specific to your product for example. And then that gets fed along with some system device tree overlays into this new tooling that we've been prototyping and out you get on the right in purple the individual Zephyr build systems with their individual device trees. So I don't have time to get into all the details but here is a way you can download the slides and sort of look at what we envision going where in each of these files. Let's talk about contributions. I'll start with the specification. You have a specification link earlier in the slides. One of the first things that we did was convert it to the Sphinx format. So that's the same format that is used by the standard device tree spec and it allowed us to move from a collection of markdown files to a single document that reads like the device tree spec. And it doesn't just, it's not just also a PDF, right? Another thing that we did is we reworked the style and content of the system DT spec to match the device tree spec. So if you're used to reading the standard device tree specification and you know how to look at a standard binding up there you'll see the tables in exactly the same format. So it should be very familiar to you now and you should be able to work with it in the exact same way. We've also added a variety of examples. We've fleshed out all the tables of properties. And at this point I wanna thank Stefano Stavolini and Bruce Ashfield from AMD Xilinx who are the sort of core maintainers of this for all the reviews and clarifications as we've gone along. So at this point that's kind of where we are in spec land where it's something that we think other people can look at and read and it should read in a familiar way. And in terms of the implementation we've been working on this RFC, this is closed now. Like I said, since I'm moving on to another opportunity but we do have a system device tree demo where we have implemented our own sort of system device tree tooling within Zephyr and gotten it booting on different cores in various AMP systems. What we have in the RFC is sort of a key move board because that's easy to evaluate but internally at Nordic we've all supported this to a variety of our own AMP SOCs. This was done without SysBuild integration because we weren't quite ready to do the SysBuild integration but it's taught us a lot about how we can proceed further and it also actually revealed a few issues within the system DT spec that we think need to be cleared up before we're gonna merge something like this into the mainline Zephyr. That's where we are. I'll talk about next steps now. So the next steps are the first thing is we're gonna cut system DT spec version 0.9 is what we have now. Right now you have to build the spec yourself and so there's a bit of release infrastructure that has to be built out in order to make a spec and then we have to sort of finalize the specification. There are some issues in the same repository where the spec is hosted so it's all on GitHub. There's a variety of GitHub issues that have been filed against the spec and once those are closed then we can call it 1.0 and it'll be ready to go. Earlier I mentioned that there was sort of umbrella issue that this is all part of. So if you're interested in following the effort further you can download the slides and this issue better support for multicore and PSOCs. It captures a wide ranging set of changes that we're working on upstream. So this isn't just advisory, we're also looking at reorganizing the main K config tree so that it's friendlier to use within SysBuild. We're looking at changing the way that we model boards so that it's better captures the fact that we have these increasingly heterogeneous systems and if you load this you'll get links to a variety of sub issues that capture the sort of day-to-day development. You can follow those issues, you can subscribe to them for more updates and if you're kind of interested in getting involved I would encourage you to ask in the build system channel on the Zephyr Discord. So this is short talk, I try to keep my talks short so that there's time for conference questions. Please go ahead if you have one. Yeah. So the question was if I'm someone who wants to adopt SysBuild and I have a target that has cores with different architectures is that supportive? And the short answer is yes because SysBuild is responsible for spawning separate Zephyr build systems. So each of those gets its own selection of tool chain, et cetera, so that you can sort of do whatever you want. We've learned a lot actually, this is work that many people at Nordic have been working on for a variety of years and our first attempt tried to do everything within the main Zephyr build system but you run into limitations within CMake itself that there's only one tool chain and so that's one of the main reasons that SysBuild sort of has to exist is so that you can spawn these sub build systems. Thank you, good question. Any other questions? Yes. So the question is, is there any link between system, device tree and boards with multiple SOCs? Yeah, so this is also a very interesting question. My short answer is that the system in system, device tree is the same system in SOC. So no. That's kind of why, as I mentioned earlier, there's an umbrella issue which also covers some changes to the board model and the generalizations that we have to do the board model transcend the generalizations that we have to do to device tree because those of you who have sort of struggled with this type of hardware already will know that Zephyr's board model, like when we call something a board, it's really just like an individual target within the board. And within, you know, now that we've got these sort of complicated SOCs and maybe a single PCB with multiple complicated SOCs, the board model itself has hit scalability limits. So that's kind of interrelated but separate work. Other, yep. Do I know if trusted firmware or OPTI is looking into using device tree? I don't, I'm sorry. I'm not active in that community. Does Linux have the same scalability issues with this? So, you know, I'm not a kernel maintainer. So this is me speaking outside of the bounds of my expertise, but my understanding is yes. And that's one of the reasons why the system DT maintainers got involved. So at AMD Xilinx, my understanding is that they're doing a lot of work, you know, with like Zen running or puppeteering a little bit of Linux and they want to be able to have something that can sort of work in both domains. Okay, so I'm not sure exactly what you're asking, but you're asking like what the purpose of having like a Linux development environment? Okay, yeah, I'm not gonna talk about line endings. That's not relevant to my talk, I'm sorry. Yeah, that's okay. I have my own opinions and I won't tell them to you so that the people who disagree with me won't yell at me after I'm done talking. Any other questions? Thank you. Okay, great, thanks. So let me repeat that. So for the people who are virtual and in the recording, so the spec is up on openamp.readthedocs.org right now, the system DT spec is. Sorry, openamp.readthedocs.io, correction, thank you. Okay, yeah, so there was a webinar in December which has more information on using this with Zen and Linux within the openamp community, so not this effort community. Yes, sir. How does this build relate to something like Yachto? Yeah, so, which also orchestrates build systems. Yeah, so I said that every project that gets big enough has its own meta build thing and SysBuild is ours. Yachto is another one, so I sort of see them as they are doing similar things in different domains. I don't know that I would wanna run SysBuild under Yachto, for example. That wouldn't necessarily make sense to me. I remember a talk from a couple of ZDSs ago or maybe it was another conference where people were spawning the regular Zephyr build system under Yachto as part of a recipe that filled out a larger system. Okay, I think I can probably answer this, although yeah, it's also not exactly system device tree, but is SysBuild capable of signing images in multiple different ways? Short answer, I think, is that it ultimately has to be. I don't know that it can now, but I do think that we sort of have to solve the problem of user-plugable stitching scripts if we're gonna make this work. Any other questions about system device tree in particular? I'm happy, you'll see me at the, around the conference, my name is Marti Bolivar. If you recognize me, you can come ask me about line innings or anything else you want. Yes, yeah, so the question is if I have a device tree overlay that has to be applied to multiple samples or I can generalize maybe a little bit to say multiple applications, multiple Zephyr applications, how do I do this? And I think that, yeah, one way I think in the future is that if you can express it as a system device tree overlay, then it goes into your whole system device tree and from there it goes into the right places in all of your regular device trees. So we don't have system device tree today. So what I will say is now we have snippets, that was a feature that was added for version 3.4 and so you can look at the snippets documentation upstream and that's a way to put a device tree overlay in, you can do more things with snippets but one of the things you can do with snippets is put a device tree overlay in one place and then ask for it in multiple places. Okay, how are we doing on time, Keith? You're muted? Yeah, we've still got about 12 minutes. 12 minutes, okay, great. Great, Q&A. So the question is if we, so at the hardware level we have ridges between buses, right? Do we have something like that in system device tree, Demi? That's a good question. I feel like device tree is such a simplification of the hardware bus structure in general, right? Like we put everything under slash SSC regardless of how many layers of buses we really have. So one thing that I haven't really looked into and maybe we should talk to Bruce after this talk is I think Stefano's here, actually Bruce is not here, is there is a concept called a bus firewall and so maybe that's something within system DT that could help. Bruce is here as well. Bruce is here as well? Okay. In the back? Yeah, that's a great question. So the question is if I've got like ARM Cortex-A running Linux and I've got an ARM Cortex-M running Zephyr and I wanna put the hardware description for both of those in a single, for example, system device tree, can I do that? And the big obstacle to that today as the person who asked the question alluded to is that we don't really share bindings with Linux. So Zephyr has its own device tree bindings which are sort of our schemas for what goes in the nodes and since Linux and Zephyr have separate device tree bindings, often there is no way to write a single device tree that'll work on both Zephyr and Linux. And that's, I guess sort of speaking with my device tree maintainer hat on, that's not really for any good technical reason. That's really for historical reasons. And it's a known issue. And one of the things that we would like to do in the long term is move the Zephyr bindings over to DT schema so that we, not only do we have our own bindings, we have our own bindings language. So we have a bigger tooling problem to solve before we could use the same bindings. We would actually have to be able to even use the same bindings language that Linux is using. And that's a big lift. I've talked about it a bit with Kumar who's my co-maintainer. I think everyone wants that and help would be welcome because I'm not sure actually that we're gonna have the resources to do it ourselves. Go ahead, Briggs. So could we in system device tree have sort of potentially separate indirect bus nodes, one which contains the nodes in the Zephyr bindings and another which contains the nodes in the Linux bindings and then the regular split into standard device trees would solve the problem. Sort of. The one way that that would not work is if we had the same compatible but a different binding, for example. Back with your hat on. Yeah, so as a policy, the question is as a policy decision we wanna do things in Zephyr with device tree that Linux doesn't do. So one of the ways that we do things that Linux doesn't do is we might have a property that begins with Zephyr comma that will then contain some software configuration. Since device tree is one of the main configuration planes for Zephyr and we don't have a user space telling us what to do in the same way that you do in Linux, Zephyr's user space is very different. You know, that's not something that is done in Linux, right? And so we want the ability to sort of have our own bindings in order to continue to do that which we've decided that we want to do and is that gonna be a major obstacle? I think probably yes. I don't know that it is insurmountable but I do agree that it's an issue and we're gonna, we would need some way to sort of keep the Zephyr changes on top but a compatible subset that would be enough to be used for nodes that are in common between an A and an M, for example. Yes. Is there some support for dynamically modifying the system device tree, for example, to migrate something between cores? Is that the question? Yeah, so in Zephyr we are actually not going to use this because our build system infrastructure is really sort of not set up to use this but the system device tree specification, if you looked at the link, it was actually under a repository called Lopper. It was github.com slash device tree org slash Lopper. And the reason why is that Lopper is the standard tool for manipulating system device trees. It's a Python application that is designed to be a data-driven way to do anything you want, make arbitrary transformations to a system device tree. So if the rewriting can be done in some sort of user space that can run Python, then yes. And then otherwise you might have to make your own tooling but the design of the specification itself does not preclude that and there is an implementation of a tool that does exactly that as a proof of concept that that can be done. The question is, how does this transition play out in the long term? If I have an existing standard device tree that represents my system, am I going to be able to continue to use it? In the short term, the answer is yes. And in the long term, if I have my way, the answer is yes. But that's kind of a policy decision that has to be handled at the project level. We know we need a new model for boards. And so that's going to be introduced alongside the existing model for boards because we don't wanna break anybody, especially out of true users. Then in the long term, I think that boards will probably be transitioned to the new board model once it exists. And so the question has come up. Do we want every device tree to be a system device tree or do we wanna drag everybody into a system device tree or do we wanna have a world where only the AMPSSEs whose vendors want to support it can buy into a system device tree and everybody else who wants to use it the existing way can continue to do so. There is some disagreement among the upstream maintainers about what the best course of action is. And my vote is let's continue to support both but I don't make the decisions, right? It's a big community. And I would definitely encourage you if you have a perspective as a user, we really value that. And examples and use cases really do help drive decisions. So if you don't feel like you can contribute technically, contributing as a user with what you need is a form of technical contribution and it's incredibly valuable. So please chime in. Any other questions? Awesome, great questions. Chris, you know me, you're my friend. Yeah. Can we have mixed endianness in a system device tree? So I guess that means like within the property value, you would want mixed endian? That's an interesting question. Within Zephyr, the answer is that it doesn't matter again. Well, no, it doesn't matter because it does sort of matter. Yeah, because you put a 64 bit integer in two cells. No, that's not, as far as I know, that's not contemplated and we just sort of, because it's meant to be backwards compatible. Yeah, you have to do it in exactly the same way as you do today. Okay. Okay, I think I'm gonna call it there. Again, everybody's welcome to talk to me. Appreciate your time.