 Okay, I have to turn everywhere so I might as well get started Was ever anyone else at the bar last night because I I felt I was talking over so many voices and I think today My my voice is a little bit rough, but I had a really good time. So hope everyone else had a good time Yeah, so I'm Chris Fried And I'm here to talk about the POSIX roadmap for Zephyr LTS version 3 and actually if I'm not mistaken we have our LTS 3 Maintainer in the front row. I'm not going to point names or point fingers name names, but maybe we'll chat later at some point These are some of the things that I've been doing it's like my kind of word cloud I've been doing stuff like this for a long time You know going way back to like open embedded Jun2 For a long time essentially But within the Zephyr community I'm a maintainer of the POSIX API I actually have to write these things down because I'm forgetting them all the time, but FPGA maintainer Driver maintainer, I should say hash utilities thrift Collaborator on C++, C, GPIO, IEEE 802.15.4 Texas instruments platforms Not yet part of networking, but I have done a bunch of networking stuff Also because META is a member of the Zephyr project I am lucky enough to have one of the seats on the technical steering committee so if you guys ever have You know issues that you need to escalate then I'll be there and I'll be able to vote yes or no And here we go If you're at the keynote this morning, I just wanted to point out my manager Mario right here He did a great talk on some of the things that we do at META With Zephyr, which is amazing because we haven't actually been able to talk about that publicly until recently so Of course MSVP, that's our first in-house ASIC developed at META and of course like why would we need a video transcoder? We have over 2 billion Global daily active users and it's still increasing in terms of monthly. It's a third of the world's population So people love CAD videos. I'm just saying that you know we've got to keep up so This is a link to our actual published documents about that describing that from engineering and then of course We have a high demand for this sort of thing so Sort of readjusting once again In terms of numbers we have four billion views of videos per day on Facebook, so that's a lot and You know How does that affect our system as well? That's a huge amount of power consumption. You can imagine transcoding different resolutions that sort of thing Storage I mean obviously the better Trans sorry better codex give you better storage Metrics so and of course performance. We don't want our CPUs just you know mindlessly burning through cycles for no reason So let's use an ASIC because that's how well we accelerate things in the infrastructure world So yeah with our ASIC We've achieved 9x faster throughput for H.264 which is great 50% 50x faster throughput for VP9 encodes and then of course Six times better performance for high-quality video on demand big thing for me because I I ride my bike I Bit of a friend of the environment, but to me this means a lot and power consumption is down significantly Which I think is a big deal Obviously we have a next generation coming in and that'll support the next video codec called AB1 additionally we have our AI inference accelerator ASIC version one So of course, you know, we still get the same Load of users, but why do we use AI in the first place? So basically just your news feed relevant ads, you know like ranking that sort of thing you can imagine so the reason that we need to accelerate this in terms of hardware is because again like power storage and efficiency so These models and I should have mentioned to you that At the time Facebook, but now meta We released a paper way back when that was not I'm not talking about last week But many years ago and it was actually one of the I would say seminal publications about Artificial intelligence, so I wish I had a link for you, but I just learned about this last last week in a meeting so But apparently we've influenced a lot in terms of AI So in terms of numbers Compared to, you know, the popular GPU of today, I'm not gonna name names again but we're getting two times the efficiency in terms of power consumption And of course, we want to enable this for our users. So we've open source pytorch. There is pytorch 1.0 Now there's pytorch 2.0 Which is I believe a Linux foundation project as well and If you want to read some more technical details aside from what Mario delivered in his keynote this morning I you can check out the ISCA paper that was last week and I'm sorry. I'm gonna have a lot of links in my slides So please download the slides if you want to get a direct link to the information So what are we talking about today an overview? Goals for LTS version 3 and then I'll give you a bit of a status update and then what's next? This is the next slide is potentially like a little bit boring You know, it all started way back with Bell Labs, of course, right? C and Unix were very closely tied together and of course once Bell Labs was able to prove that this was portable via the C programming language It just kind of caught on like wildfire. So we had licensees left to right in the center I'm not gonna read all of them, but I'm gonna point out Alexander's gonna hate me for this but this little guy right here in 1991, of course First free BST, I'll mention free BST too, but yeah, there were a lot of really great ones These are the two that I considered my personal favorite in terms of desktop operating systems based on Linux or Unix, I should say Then of course we had before we had a POSIX we had the single UNIX specification, which would have been 1996 approximately And then that later became POSIX. So I trivially standard 103 1003.1 Originally recently released in 1988 and of course talking about standards, you know, there's as POSIX has evolved so as C and C++ and so on and so forth so You know, I'm no like math expert But if I look back on this and I did a little bit of numbering and stuff like that To me that indicates that this year POSIX has turned 35 years old. So if anyone's to guess over under over under Over yeah, I'm over. I'm over. I'm old in that And also I'm no math expert, but that also means this year Bella as you next turns 50, which is pretty amazing In my opinion Get back to that in a second, but in this is I should specify this is since the official Public release of it Now let's look at Zephyr I'm not giving a really great overview of the history of Zephyr No, I'm just gonna do an overview of the history of POSIX and Zephyr Really quickly if I can so It's looking for my mouse It's somewhere and I can't find it. Oh, here it is great So first 2015 was kind of the original first commit if you will for for Zephyr and its current incantation The sockets API was introduced by Yucca Rasperson who is still involved in the project He's no longer a network maintainer, but takes a very active role. Really great person to talk to Of course the socket API is BSD socket so until introduced the file system API in 2016 K-Pole similar to the polls 2017 Again in 2017 we got more official sockets API support and I was via Paul Sokolowski Who is incidentally the previous POSIX maintainer prior to me? And then we had p-thread support added by Andy Ross in 2017 Lenaro we added get adder info again by Paul Sokolowski Otacon added the POSIX arch native POSIX support. That was Alberto Escalar Pietras 2018 POSIX talks Intel POSIX timers Intel MQ Intel p-thread key p-thread once again Intel Mutex Intel all 2018 so you can see it's kind of exploding things kind of caught on Finally The the POSIX API configurable. They're like, yeah, we have a POSIX API. Maybe we should put a cake and think so for that 2018 so things were started to really escalate Select etc etc Event FD technically not POSIX, but kind of fits into the same basket. So that was added 2019 I added socket pair and I'll get into the details of this a bit more but POSIX thread support not technically compliant because we don't automatically allocate our thread stacks I made a couple attempts and it's been several years I'm gonna touch on that a bit later But other things, you know, there's some additional You know additions over time Rerate of event FD, which is really useful for us internally p-threads etc etc etc. I Just really wanted to produce technical debt. So that was one of the things that I tried to get done You might ask yourself why why POSIX? right Well, basically just like portability A lot of people believe that POSIX is specific to UNIX and that's actually why it ends in IX Which is wrong, right? It just stands for portable operating system interface And if I had chocolates to throw at a people I would be quizzing you but I'm not gonna do that next time So The reason we like portability is because if you have to incorporate a third-party library into your code You want it to just work you want to adhere to a standard? You don't want any unexpected surprises essentially And additionally, it's a mature API that means there's a lot of developers who are familiar program POSIX APIs and That means your hierarchy pool is bigger reduce time to market is another reason Yeah, so I mean why else well nowadays Linux at least but other offering systems like Q-Nex are powering cars I'm just guessing one billion, but you know, it's just a wild guess We actually have automotive grade Linux happening at this event too. So I'm sure there's an expert that knows that exact number That's not me We're powering two billion desktop Laptop and this is a I know this looks like a printer, but I really wanted this to indicate print servers So that's also pretty old probably like over 35 years old like who remembers print servers probably a few Interesting connection there, but Lastly 16 billion mobile devices that's cell phones and tablets now just Putting it out there. I'm not trying to start a war or anything, but that's about 71% Android. Just just stating a fact You know the wrist is iOS. There's another 1% Who would that 1% be I guess like Windows? Oh blackberry blackberry was the other one. I should have remembered that Should remember that faster anyways, so Onward What are our goals for LTS v3 with POSIX? Originally if you checked out the RFC that this was based on and I did a talk in January at the architecture meeting You would have seen Quite a mishmash of tickets and references to odd things and that was basically because we were just collecting a list of Areas that where people thought we needed to do some work. So high level goals We just want to be able to maintain it better We want to improve the interface for applications and the implementation And we want to improve portability as I said, so that's really just it Now In terms of the interface, there's one particular area that's Actually, I'll get into that later But yeah, so maintainability one of the things that I worked on personally was Getting rid of the structure details ie implementation detail for things like p thread mutex p thread condition variables p thread itself all sorts of things and what we did is instead of Giving a struct to users that they would then declare within their stack or whatnot. Maybe statically We just gave them an integer and then we had an array of Internal things that we could keep track of you know create destroyer that sort of thing And that actually gives us pretty fine control over the number of resources that we're using Of course The synchronization primitive for me that was a big deal So internally if you'd ever looked at the implementation of p threads or mutexes or condition variables within the Zephyr project It was kind of like dogfruiting itself. So it would have like a That's not this diagram But if you looked at p threads it would like p threads was implemented with p thread mutexes rather than kernel mutexes And to me that was a layering violation. So it increased the cost associated with maintainability Yes, it was dogfruiting itself And of course we have kind of things existing where they're not supposed to so we have lib os fd table Which is this really great structure if you're familiar with Linux kernel development. You've got your file operations re-array I octal map, etc. Etc. So that's something that we're trying to kind of fork out into its own library and The reason for that is because if you check out this diagram, you know, it's this is my best artwork here It doesn't get any better than this But we had a actually a cyclic dependency between POSIX and network. So The way to eliminate that is you move that off into a common dependency So for lack of a better word I'm just calling that ZVFS because it's like a virtual file system for files But it's also for file descriptors and you can kind of implement you what you want And if you need to extend the file descriptor table, it's really easy like I mean really easy I'm hoping that one of my metamates here can actually implement that soon because it's a pretty interesting and challenging project And of course, this is the biggest one in my opinion the fact that The the POSIX architecture in Zephyr is mutually exclusive with the POSIX API That seems really counterintuitive You got to think like well if I can't use POSIX on POSIX then how am I going to test POSIX? You know, like I have to use Kimi or I have to run it on a board or whatever It seems kind of counterintuitive, but I was really pleasantly surprised to learn that Alberto who's now at Nordic is Actively working on that and I think we'll have that done very very soon Get some details about that later So in terms of the interface, of course Here we have our typical layer diagram if if you're wondering what these acronyms mean then You're probably not alone PSC 5-1 does anybody know what that means PSC PSC PSC. Oh Yes There was one. Okay, so it sends for POSIX embedded profile 5-1 is what they call the minimal specification 5-2 has some additional features such as the file system and so on and so forth BSE SOC is the same thing but basically We want to we I think one of the goals and I'm specifically going to say I'd like to achieve PSC 5-1 for LTS V3 I know we have BSE sockets already and we already have file system But I'd rather do a little bit and hit that target and then maybe we can do a little bit extra. We'll see how it goes So just for informational purposes LTS V3 is likely early 2024. So It's on the wiki So one thing that happened recently was as well We include we created this Sorry, I have this backwards on my slide. It's not POSIX Zephyr uni standard H. It's Zephyr POSIX uni standard H So we prefixed all of our includes with Zephyr And then of course if you wanted to include the POSIX thing then you'd have to say not only POSIX But also Zephyr POSIX so and so forth So to me that was an annoyance and that was basically like a showstopper for anybody who wanted to use a third-party library And didn't want to modify it So that needed to be changed Of course one thing that we don't do as well is what I refer to as Activate POSIX features such as you know headers type definitions and that sort of thing using the feature test macros such as POSIX timers over here But this is actually the way that all the existing POSIX toolchain editors are configured so new live You know New live minimal profile or whatever it is a picolip see you see lipsy like they're all basically the same in terms of following that sort of approach One of the issues is I feel is a bit contentious, but We probably do need to address because We will be supporting third-party tool chains that probably don't support POSIX They're not going to provide us the declarations that we need so that leaves it to Zephyr to actually supply those headers Which is what do you do in that case? That means you have to copy them which is kind of an annoyance But that's the option that we have right now for lack of a better run So here we are gonna jump into a deep dive right here. Actually This document I'll provide all update the slides with the link to it But it's from the Austin group meeting and this obviously this isn't quite the eye chart so I apologize, but We've got basically PSE 5 1 in the left 5 2 and the second left and so on and so forth. There's five three five four And then I also covers desktop profiles if you can squint really really tally you'll probably see that The shell is a requirement for the embedded profile of POSIX compliant shell. I'm just thinking Are we ever gonna get to that? I don't know why this was included in that profile, but In any case, I think we're probably gonna exclude that from our our work Onto interface. Oh, sorry, that's backwards my bad So just going I'm gonna try and run through these things as fast as possible, but These are the features Feature macros or option of arms that are required in order to achieve PSE 5 1 So single process that's sysconf. You name sig id set sig del set sig mt set. Oh, that's signal Yeah wrong slides So we've got single process out of that we probably just need sysconfig or sysconf Which is really handy for static configuration of your application can be done at compile time And you name which is all just kind of informational We don't really need an environment in Zephyr like we don't need to store environment variables. So that just seems unnecessary POSIX signals you might think well signals are only process to process But that's actually not the case you can send a signal to a specific thread and you can receive only specific signals For a specific thread. So this is something we'd like to support as well And for the most part, this is actually not that difficult Obviously, there's gonna be some exceptions like alarm pause raise whatever we're not gonna kill the holes that for Artos because of a signal that we've received. So probably not gonna implement that part because we're not supporting multiple processes Thread base you might think well, this is practically done. It is almost practically done But some of the things that we're missing Even though we never spawn another process is at fork like we're never gonna spawn another process But the implementation of that even if it's not linked to Would help us to achieve that sort of compatibility Additional barrier attributes also trivial to implement Get p-shared set p-shared This is kind of interesting but slightly less trivial to implement Cleanup push clean up pop a little bit complicated I think we'd have to statically allocate resources for that so involve the k-config P-thread kills how we send a signal to another threat And then sigmask is how we ignore certain signals for a threat Cancel state test cancel already talking with Keith Packard who's the Pico Lib C maintainer about implementing that Clock selection. This is actually a really important topic because Everybody wants nano sleep. Everybody wants really high resolution timers And that's just not for the application. It's for things like radio I don't know if we have Florian here in the room, but he's implementing T-S-C-H for 8 or 15-4 That's like a direct use of them But basically any highly reliable system has to have pretty reliable clocking Things like set clock get clock you can do that for multiple clocks So we've got the real-time clock, which is like your calendar date and time and then of course you've got monotonic Which obviously probably wouldn't want to update, but that's actually pretty important for for You know getting your Accurate nanosecond accurate time And Let's see so in terms of the p-thread con to attr That just means that if you're waiting for a specific mutex or something like that I want to use this specific clock. So I want to use monotonic Which would be the normal case I would think Posit shared memory objects. Why would I need this? Why would I need to do M-Map? On a microcontroller system It's mainly for compatibility, but I was actually discussing the other day this has a Interesting application for GPIO because right now if you look at the time it takes to write a GPO in Zephyr, it's like milliseconds because you're going through a syscall interface. We need like fast fast fast fast GPIO reads and writes Shared memory open shared memory unlink again like why would I use this? I don't know Sharing memory via file descriptor it could potentially have a use CPU time so this means having CPU specific clocks so measuring very fine with firefine accuracy how many cycles you spend in the thread That could be used for statistics, I think that a lot of of the tracing programs that are out there will likely need to have Accurate time reporting and today if you look at a Linux system Most of this is done for you by the libc and a lot of this is how they work so So again POSIX timers we cover this a little bit But right now the only thing we're missing is getting the resolution of the clocks that we support but Kind of hand-in-hand with that is like how do we know out of the 18 timers that we have on our system? Which one we want to use for our real-time clock? Which one do we want to use for our monotonic timer? And that sort of thing So the proposal that I have right now is adding device tree chosen Nodes and so you're thinking well if we want POSIX to work on every system out there What do we do? Do we make these device tree nodes mandatory? And that's actually kind of the irony of it. We can't make device tree chosen nodes mandatory They have to be opt-in so it will probably involve a bit of Legwork with all the different vendors and manufacturers, but I think it's something that we can get done pretty easily Again, I mentioned this earlier, but time-circuit nice channel hopping for 8 to 15 for Florian who's kind of working on that RFC has a very strong desire to do this now Would we do this at the POSIX layer? No, why because we want POSIX to be a very very thin wrapper around the Zephyr core features so again, just like Kernel threading is we want to make the timers just kind of a layer within Zephyr that we already use so that would imply things like K clock monotonic K clock real-time And then just adding the constants for the POSIX subsystem. So how are we doing? since becoming maintainer After six months, I got nine out of the 16 tasks in my RFC complete. So yeah Again a link to the 2023 January slides from the architecture meeting. So yay Since becoming maintainer after 10 months, I improved event FD read and event FD rate by a factor of 10 and Eliminated the deadlock scenario that was causing a lot of problems even in thrift. So like if you were using thrift a while ago, sorry Yeah, there was only just one minor bug and actually Martin in the asterisk he he submitted the fix for that very shortly after So that would be in 341 for those who are curious Again here we are Sorry for the eye chart, but this is just kind of your typical Z test output This is for the p-thread pressure test. So you can run it today on various architectures and on hardware after 11 months I Increased the performance of p-thread create and p-thread join back-to-back in a stress test by an almost infinite amount It's because previously it crashed so you can actually measure it Yeah, so that's that's That was fun. It was fun to work on it took me a couple of revs But we got it running and it required some help by Andy Ross and in the process. We discovered some Sub-problems within the kernel threads themselves. Oh This is I don't know why I have this here, but oh Oh, yeah, sorry. This is the pineapple poo emoji again, sir And if you look at these numbers, how does this how does this make sense like so it's been a year It's since a year ago. I became maintainer and why did why did these numbers look so bad now? It's like 17 out of 54 tasks. How does that work? According to the RBC, I mean you can click on it right now anybody you can check it out, but It wasn't actually You know a problem or anything We just broke down the complex tasks into smaller more manageable ones and that's actually a really good thing Because now we're at about 70% completion for everything if we take all those pretty trivial functions and and you know complete them so I'm gonna get some help I'm hoping so The next thing Dynamic thread stacks if you're at the thread a thrift talk yesterday. I made an important announcement, but I've been working on p threads for a while. I mentioned that in the timeline Literally yesterday morning. I got an email because I've been collaborating with Intel a little bit on this and Flavio who's our security guy Email me and said the test which is all green. That means we have POSIX threads running with automatic stack allocation. That means it's done according to the POSIX specification so To me, that's a big deal I'm gonna say it's a big deal and I'm raising my hands so I hope everyone else is excited as I am But I've spent a lot of time working on this like four years I did it as part of a contract a long time ago and ever since joining that it's been just like here my spare time Here and there but to me this is a big deal. It means p threads. I so see 11 threads a C++ 11 threads standard mutex standard condition variable counting some for binary some for all those things lock guards all the really cool things You can do a C++ So if you have lots of RAM and you want to compile your C++ application, then you're good to go Then it opens the door for other languages such as Rust or Python or Lua You know like Lua is a kind of a cool little language that you can put on microcontroller Same with Zigg, which was demonstrated last year actually So how are we doing this we're doing this with a separate kernel level API called kthread stack allocate and kthread stack free Now you may be asking. Well, what if I'm like in safety critical, you know, I can't do any sort of mallicking or You're gonna be safe. You're gonna be fine And it's sorry that you weren't safety critical to you, but you know like everyone's been there. So This same API will cover both heap based allocations and pool based allocations. So that means it will Underneath the pool based allocator. It'll just be a static Array of threads stacks that you can think you can you can pick from So you'll always have a maximum number that will be running at any given time and that's the important part Additionally with this there's a configurable for if you want to have both you can actually Choose the order. So if you want to have heap stack allocations first Then you can do that. But if you want to pool stack allocations first, then you can do that And of course you can have none of those so you can still use the statically defined thread stacks and it all works Yeah, so coming soon, I don't want to you know I don't want to you know say it's done because it still requires a bit of integration, but we're pretty much there again Arch POSIX POSIX API. That's another big thing that we want to get out of the way And as I said, Alberto Escalar is working on that he put up an RFC only a couple weeks ago and Managed to get pretty unanimous approval and he's going ahead. He's doing it He's been part of the project for a real long time if if you recall from the history of POSIX and Zephyr, so it's been quite a few years and I remember I'm sorry Alberto I know I've been a thorn in your side, but I've been requesting a lot and you know from the native POSIX system I really appreciate it He's adding to two new boards one's native sim one's native sim 64 As far as I know these will be alongside the existing native POSIX native POSIX 64 So and that's the way we do things we want to have it a stable transition And so on and so forth so so Currently I have zero collaborators and POSIX and I'm just saying like we're hiring I'm not gonna give you any money But like please please help I'd even take a co-maintainer honestly But the nice thing about breaking things down into smaller more manageable tasks is that for new people to Zephyr? this opens up a huge list of Really simple intro bug fixes or enhancements that they can add so Here we are here's a list of the really trivial things from my perspective that we can add and I actually have a RFC available right now for sysconf Other things like you name pretty trivial Flipping bits in a signal set pretty trivial Barrier destroy barrier and that's pretty much no ops Get p-shared set p-shared that basically just means Limit what process can cross this barrier or sorry what thread can cross this barrier to a specific thread So again, it's pretty trivial. It just means an extra field within the barrier Of course guarded by K configs. We don't want to add that if we don't have to Another thing I'd really like to add that's not necessarily the hanging fruit is K config control of all the POSIX feature test math growth So the bigger items that are you know getting us from 70% to 100% Those are require a little bit of extra effort and obviously as the maintainer I'm gonna do that But I'm also enlisting help. So if you're a POSIX guru, please come and talk to me. I need collaborators And yeah, that's basically it. So with that, I'm gonna open up for questions and I think we have five minutes left. Oh Is there a mic? Oh, there's a mic right here. I Think they want to hear you online though. I can repeat the question if you prefer but oh There we go Josh has volunteered himself as the runner Okay, so question one is For somebody who doesn't need this all of this can be compiled out. Absolutely. Yeah, yeah Everything okay, and second question is is there a free Version of this code that you can heavily borrow from while doing that. I That's it. That's a bit of a tough question. So One of the things about Zephyr is that we want everything to be a page to license to possible So that means I mean we can pull in a BSD 3 Or BSD 2 implementation, but then we need to get We need to get TSE approval if we pull that in. I just been informed that we only have one minute So maybe one or two more questions. Oh three minutes. Okay three questions anyone We've got a question in the in the back there. Oh Murray it's as well Does with the dynamic threats text does MP work? Yes, that's right So the dynamic threat sex that was the biggest thing I basically hit a wall and I'd only been able to work on this here and there for months Like it'd be six months between when I had a chance to work on this So what happened is that I did a rev with some feedback from some reviewers and draft And then I broke my own diff, which was really sad And I hadn't touched it for a while. I spoke with the NOS He said like oh Flavio needs this really bad for something we were working on so like oh, okay Get in touch with Flavio two days later. He's like you put an FDF right here. I'm like, ah, that's it So yeah, it's amazing like when you when you step back a bit You get a couple of extra eyes on the problem. You can you know you can get there, but Questionary here's maybe you have already mentioned during your talk But do you use some kind of official? conformity tests There is one it costs money and not monopoly money and not the money that I have on my credit card, so It's something that the Zephyr project might consider if this is something that's valuable enough to our members But I can't you know, you know promise anything, but there is a conformance test suite Yeah, then how do you test to make sure that? Your code actually Okay, sorry. Yeah, there's there's there are tons of POSIX test suites within Zephyr itself. Yeah I thought you meant like compliance actual like certification, but you make your you have your we have our own That's right. We have a pretty mature test suite. Okay. Thank you. Thanks Maybe one extra minute. I Think we're good. So, okay. Thanks everyone