 Okay, good. Thanks. So, yeah, as I did plan to be there in person and not only deliver my presentation, actually, but also meet a couple of people, well, not a couple, actually quite a lot since I've been missing all these events since COVID times, but the case situation changes, and luckily I was able to do that with that way. Anyways, so again, hello everybody, my name is Alexey Brodkin, and today we are going to talk about POSIX in Zephyr Arthos. Hopefully you'll find that presentation interesting of some use for you. So, let me briefly introduce myself. Also, you may understand where I came from. So, I've been involved in the Zephyr Arthos development for more than five years now. Primarily, that's all development happens around our course because I work for Synopsys, which has its own process architecture, and that's what we support. Before diving into Zephyr, I was working on other open source projects like Linux, kernel, various build systems, namely build root, open embedded, that works cursions to UCLA and other things. But these days, I mostly review ARC related code in Zephyr and participate in technical and not very technical discussions about Zephyr. And so that's what I do in Synopsys as well. So, I lead that team of software developers which work on Zephyr, again, mostly focused around ARC processors. And so I think we are quite successful and I may say that support for ARC processors in Zephyr are possibly one of the best, at least on par with other major architectures and much better than something else because we really care and so that's of our interest to be in a very good situation here. And for several years I've been wearing a badge of Synopsys Zephyr ambassador as I've been talking on different conferences, promoting Zephyr both in the open source community and among our customers and even among our internal users in the company. So hopefully it was not given to me for free. Now, what we are going to talk today. We'll start from introduction to the Zephyr, Arthos and POSIX so that we are on the same page and better understand what those substances really are. Then we'll see how those two substances blend into each other. And finally, we'll try to understand what does that really mean for developers of applications on top of Zephyr, Arthos or even in a more generic case, how knowledge about POSIX supporting Zephyr Arthos might help product developers. Now, zooming into the first introductory part, very short on what Zephyr is. I hope that most of you already aware of that and familiar to some extent with that thing because you attended that Zephyr part of that open source conference. And then we'll see what is POSIX actually and what it has to do in context of embedded systems. That slide I shamelessly taken from so-called Zephyr Golden Deck. I think it is maintained primarily by Cade but other people also contribute a lot. That's a reference slide deck which is maintained by Zephyr project as said by multiple people contributing and that gives you a kind of high level overview. And so likely you have seen that already today or previously but still that gives a good pitch on some key benefits of Zephyr Arthos. Here's yet another borrowed slide from the same deck but copied here on purpose, not just as the previous one. Now with it I wanted to highlight the fact that there are a lot of things available in Zephyr Arthos today and more will be added. You may see you have BSPs, device drivers but also complex software stacks like TCP, IP, Bluetooth, Wi-Fi and so much more on top of that. And from one point of view it gives developers so many tools and possibilities but on the other hand, well as some people used to say there is enough rope to shoot yourself in the foot. So with high complexity it's not easy to deal with. And so what we can offer for managing that complexity and I would say okay as software as we as a software developers what we how we solve the problems we just get another layer of abstraction and so which brings us to the POSIX discussion. So a couple of basic facts. So that name POSIX was suggested by Richard Stallman and is nothing else but an acronym for portable operating system interface. Well X came from those Unix days and it was very popular as well which gives us the POSIX. And it was just a standard or attempt to standardize interface between different versions of Unix. And so although it originated originally referred to the IEEE standards 1003.1 the name POSIX more correctly refers to family of related standards because you may have IEEE 1003 and where that's and might be one, two, three, four and so quite a lot of those. And part of ISO which is also sometimes mentioned there comes from that international organization for standardization. Much later POSIX.13 that's exactly that's 1003.13 standard was developed as a subset of POSIX.1 which is supposed to be much more applicable to embedded systems compared to the fully featured elder brother. And we may know that the first version of POSIX.1 standard was developed in 1988 while the most recent version of it is dated 2018 which means it's pretty alive and still being worked on. Another interesting note we are talking about these set of documents and I have more complete list of those basic POSIX standards in the backup section in the end for those who is really curious. So we are talking about tens of thousands pages so that's quite a big standard since it's not easy to implement everything. Only POSIX.1 that's a generic part of the standards of is about 4,000 pages. So you may already see where we are going. Still what is very important for us embedded developers is POSIX.1 standard specifies application programming interface or API or set of APIs at the source level and so that's that means it's about source code portability that's not about binary portability or compatibility and that's fine that's what we need in embedded world. So what are those APIs provided by POSIX.1? You may easily guess some very very very straightforward stuff like P-threads and other things right but apparently also there are some IU functions like open close and all that but on top of that or also with that so you have very basic functionality which is part of C standards or C runtime as well that's exactly what it meant to specify. So now speaking about POSIX in context of embedded systems as I said was what is important in case of embedded software not binary compatibility but source code portability so that you may basically recompile your source for different platform sends reuse your existing code and in that sense when your application implements everything yourself and not relying on external library source services there is definitely not there is no need for POSIX and that's kind of software might be portable though not necessarily and it requires proper approach for to coding things right it is not that easy as some of you may know but with increasingly complex software even for a deep embedded solutions it becomes not very practical practical to develop everything from scratch because well this is very time consuming and so we want to try to reuse existing components and so those created by us for previous projects and those which were made by others we most likely will use C runtime which comes from your tool chain usually but sometimes from your operating system as well driver software stacks abstraction layers and so on and so that basically means portability of our entire project depends on many more components what else you have also associate vendors which may provide you some base support software so it might be BSP or it might be some driver sends very often you may get just one solution like one vendor will provide you BSP for Linux another for three RTOS another for some other RTOS and that's why you either need to have multiple implementations for each of these SOCs of your business logic or you may basically recompile if you have that compatibility layer in place and in other interesting use case which is worth mentioning that's application of your full featured powerful x86 host or ARM host business where you have a lot of things like a valgreen for finding memory leaks and everything Linux perf for proper profiling and then once you're ready with your business logic you just recompiled it for your tiny embedded targets and you don't need to debug there and here on the left I intentionally put all these logos so these are operating systems which at least to some extent support POSIX so you may see if you have proper supports or enough support of POSIX on each of those or at least some of those RTOS and your software uses POSIX or relies on that quite likely you'll be able to move from one RTOS to another which provides you that abstraction of your platform and so you may easily migrate from one platform to another okay so speaking about real-time and embedded application support so as I mentioned this is that i3.e 1003.13 standards and its latest version was published in 2003 and this specifies something called ringtime system profiles from PSE 51 to 54 so as we learned already POSIX standard is huge and initially was developed by big Unix workstations but most likely not everything which was meant to be used on these bigger machines is applicable to embedded systems and with that our minds subset of the POSIX one was created which is called POSIX 13 and it exactly tries to cater to embedded application support. POSIX 13 describes real-time environment profiles so called PSE 5x with some numbers there starting from minimal real-time profile PSE 51 up to the multipurpose real-time PSE 54 in reality there are way more differences compared to what will be depicted now so the simplest PSE 51 profile is meant to be used on very small embedded devices with single processor and some kind of memory like the on-chips RAM for example sensors or control devices which require no user interactions fall into that category still it allows for using p-threads likely mapped on to the underlying RTOS tasks signals and for communication between threads and also relies on C-library obviously for providing compliant basic primitives and we'll talk about C-libraries in a moment then on top of that there is PSE 52 which extends PSE 51 with simple file systems such as RAM disk or flash memory and message queues at the same time there is no additional requirements for underlying hardware only that provides you a little bit richer or software functionality it could be used on the same systems as PSE 51 pictured previously or below then what we have in PSE 53 which differs significantly because it introduces concepts of multiple processes which are basically isolated from which are those execution units which are isolated from each other by hardware and so those we are talking about use of MMU on a single core system or in addition to that we may have S&P system or which stands for symmetric multiprocessor and so it's no longer usable on the same simple hardware as previously discussed profiles also there are finally networking capabilities so we may talk about some kind of IoT device for example some edge device which communicates and talks to its clouds or counterpart and on the top of the list there is PSE 54 profile which is good for very complex devices we may have shells displays for user interaction and we may even have multiple users so that and full fissure storage but that's for kind of high-end embedded devices since in that sense you may even think about embedded Linux so why bother in case of RTOS or something so that was an overview of Zephyr and POSIX so to separate substances now let's try to mix them together and see what we get out of that we'll start from historical overview on to demonstrate some milestones in POSIX development in Zephyr and then we'll briefly touch on the current state of POSIX supporting Zephyr RTOS still I would strongly encourage you guys girls to go or visit another presentation which is going to happen today made by Chris Feed and he'll be talking in much more detail so the current state and much more finer technical details so as you may see now we when we started discussing first submission to POSIX around POSIX landed in Zephyr in 2017 which was only a year and a half since the first release of Zephyr so which means even back in the day there were some demounted interest one thing I'd like to mention in particular so-called native POSIX boards which were also introduced in the very beginning basically that's a lightweight translation layer between Zephyr APIs and POSIX compliant host in our case Linux distro on your development machine and so you may think about that as wine on Linux or WSL Windows subscription for Linux of the first version on Windows if we have time in the end so that presentation I have a backup slides with some more information on that now another noticeable item is CVT web application but we'll discuss that just in a second and what's also I may tell that now 2023 seems to be a year of POSIX in Zephyr again because of current maintainer Chris has a very systematic approach and so he contributes a lot of fixes and improvements as well as he has a nice and detailed roadmap that's that you are more than welcome to contribute as there is a lot of work ahead of us in that area and on the right now you may see a list of people who significantly contributed over time to that part of Zephyr developments these are maintainers which used to be in the project and Chris is now the last one and the active so you see over time a lot was done actually these are not only changes which I have here but these are only milestones I wanted to highlight okay so it was history but what do we have today if we look at the official Zephyr documentation about POSIX support and you may get there by the link in the bottom one you have these slides available you'll see that block diagram in the beginning and you may immediately see those POSIX system profiles which we discussed a couple of four slides ago so you may see PSC 51 52 53 unfortunately that does not really mean that's all of them or at least the simplest PSC 51 are fully supported in Zephyr today and I will remind you that the real real full scope of these aforementioned profiles are rather big so even though there is a conformance test provided by the open group consortium for each of those profiles I doubt there are many implementations of POSIX that are usually that are actually fully compliant and even if those exist probably it's not what you want in your embedded system because it will obviously consume a lot of memory and possibly won't be that efficient and in a sense of Zephyr it's not unique in a sense that we don't strictly follow standard but rather we enable features which are really needed for some particular development still from that diagram you may get some idea as we saw earlier PSC 51 is bare minimum profile and then if we add something like file system to it will it already looks kind of similar to what PSC 52 is meant to be and on top of that we enable networking and think here about PSD sockets which is very nice programming abstraction for data exchange between host or network so then you add MMU and SMP and then it is something very similar to PSC 53 right what we don't mention here is PSC 54 because well that's a little bit too much in my opinion though maybe at some point when we add multi-user capabilities in Zephyr it will be already somewhere there at this point we also don't have complicated file systems I think only FATS and so something reasonably simple as well anyways below that diagram if you again go and visit that page you will see quite a long list of functions already supported and not yet supported and that is actually I would say a more realistic view on posing support in Zephyr though it is not exhaustively complete because it's not only a certain amount of functions but all the peculiarities about their attributes their parameters and some other things which we are going to talk a little bit more on the next slide and so basically what you will see in that list maybe something I tried to capture here so in Zephyr we don't really care at least at this point about compliance with those official things like PSC something but we are more focused again on something which allows you to do some real things to implement real APIs and so that's why what is what might be more important for you go and check which primitives or which functions are already supported rather than just saying okay if a PSC something is supported or not okay so okay so now let's talk a little bit about limitations and differences from traditional posix so even though a lot is already in place for reasonably good posix support in Zephyr RTOS there are still things to consider and improve so first of all Zephyr RTOS has a bunch of its own its very own APIs more or they keep evolving we have some interfaces get obsolete some new interfaces appear and some just change over time because this is very actively developed project but with very few but very few if any at all were meant to be used specifically for implementation of posix functionality maybe now we'll start seeing something like that but before of that we try to basically map this map posix on what was already available and then fight you something that said inevitably there are some translation which happens between posix api and zephyr native apis which means there is a little bit more codes gets used more instructions being generated by the compiler which means you may get a little bit different possibly a little bit bigger memory footprints and more instructions executed in runtime by your cpq which may translate to some performance penalty or overhead but as always it is not that black and white it does not mean that without posix your application or your implementation will be tiny and fast and with posix it will be slow and huge it might be barely visible that's impact of posix on your application and so so as usual before jumping to any conclusion one needs to do proper evaluation profiling do measurements and to see if what you get with posix if it is really still okay for you or there is something which you don't want to have then as in any good piece of embedded software in zephyr we prefer to do a statical allocations of to not be surprised later on by an expected overutilization of precious resources such as memory for example because we typically don't have gigabytes of RAM on embedded devices as opposed to modern laptops smartphones and even smartwatches sometimes and so we want to be and also from a certification standpoint which is also important for zephyr well we don't want to use dynamical locations because then you barely can certify any safety critical system but apparently it is pretty typical to use to just randomly create new threats and other objects in runtime because well it is so convenient for us developers and hide some complexity and apparently people use that widely those we need to enable that functionality in zephyr which didn't exist again with good justification and people is not happy when we start talking about dynamical locations in that kind of embedded software and so this is now work in progress with so we try to do that carefully so that you may enable that dynamical locations only if you really want to and so keep using statical allocations but you may escape the dynamical location so then since posix defines a lot of common and standard things like data types function prototypes and we already saw those sort of things like abs a to i and so other things and see it even defines structure and header names so it's not very obvious how to squeeze posix headers in the existence zephyr codebase because zephyr codebase is already quite mature and quite a quite a quite there are a lot of things there right and so it's not easy to squeeze in yet in other quite a big substance or blend them together because they might be clashes more or as i mentioned initially added functions were introduced in kind of random manner and so their definitions need to be moved in some other places so that we have so that we follow those requirements imposed by posix as well and as we saw just now there are a lot of components specified even by pse 51 and even if we're minoring simple functions and our bigger goal to be compliant so so that the wide range of existing codes could be simply compiled as zephyr application for that we'll need to implement missing functions and again as we discussed posix is not something completely new instead it is it's an attempt to standard ties based standard ties standard ties based on what was already in place and so some of existing functions will will require adjustments as well speaking about seal libraries in context of posix it's a very good separate topic and worth probably a separate session because significant parts of posix relies on what is provided by c runtime by c library and at the same time most of c libraries especially those which are part of our tosses and zephyr for example also has its own minimal c library implementation so they themselves don't provide everything required by posix what's worse some c libraries may have their own definitions of types and functions which clash with posix definitions and in that sense well we have new lip which seems to be the lowest hanging fruit because it was originally written to be compliant with posix and yet there might be some subtle issues in case of c runtime which comes from your proprietary tool chains it's clear that there are lots of complications are waiting for us and some of these things could be easily solved and we did solve that for some of our for our proprietary tool chain also not everything but we just started to work on that but there might be some things which will be very hard to fix oh and things like syscon don't exist in zephyr these days and so that is a very tiny utility which allows to get information about properties of the system so like its software and hardware features and states in runtime so that you may not rely on something which is hard coded during your configuration but you may use obtain the data when you need that and again this is now work in progress and so there is a good discussion going on like how to do that so that we don't do everything in runtime something possibly could be obtained from during build time via some constants so that we don't blow the code and we are efficient still because we are still we are talking about embedded software now one one since we are pretty much done with a lot of theory let's look at how that could be used in something real and so we will discuss two cases one tightly integrates application which will be civet web and then porting and you let's call it random application which relies on postings internal okay so here's an example of how it used to work sometime ago literally once your zephyr build environment is ready just execute two commands the first you may see that's on top west built and path to that example will build you that example and the second command was build minus target run will execute that right and so well it's very nice so once you connect to that to that embedded targets which in my case was just you you may see that's okay there is an ordinary web server which may give you some data right so it is all nice and so now let's look a little bit closer so what is that so that civet web it is a web server which is meant to be used on embedded devices and one might argue that's okay web server is not that's common embeddits not common for embedded systems but I would say definitely there are reasonable use cases one good example which I know from my experience that's an interface for configuration or exploring logs for example so you don't need to have any other way to obtain data from that device or submit some configuration you may even do that in an automated way because you will submit some JSON for example so also why I find that particular example to be a very good thing to discuss here during that presentation because it is actually very complex third-party application and it uses exactly a lot of POSIX API and in fact it became a test vehicle of a kind for getting some of those APIs polished and improved or even implemented in zephyr and as I said it was exactly like that somebody got on each and started to scratch it which gave us quite a good result of much improved zephyr functionality in in POSIX functionality sorry in zephyr and another thing which I have to mention well unfortunately that very nice example got removed from zephyr some time ago because again it was quite a complex third-party project and it required some maintenance because with the involvement of zephyr and new vulnerabilities phones in different parts or different sub-projects used by stevetweb we'd like to see a person who'll be in charge of that and will work on improvements even on zephyr site not only on zephyr site but apparently nobody was really interested and that's why it got removed in zephyr version 3.2 which was two version before now we are in 3.4 but what is good actually there is quite some interest in functionality like that and so there is ongoing discussion and hopefully during google summer of codes replacement will be developed or will start the development and so we'll have something like that but more tight integrated with zephyr and so we'll see hopefully it materializes sometimes soon and so on the in the bottom right corner you may see that's a nice animal this is exactly that seavet or a viviera that's a mammal from sub-tropical and tropical countries such as africans of african asia so that's just a funny fact and now speaking about example which was that's random third-party example which was never integrated with zephyr whatsoever and so for some it might be something pretty well known core mark at least that's previous or older version is kind of industry standard for benchmarking benchmarking of embedded targets but i intentionally decided to to work on core mark for because it utilizes something called MITH which stands for multi-instance test harness and for that multi-instance they actually decided to use p-thread and so that gives you or gave me that nice sample which might be useful and possibly used for benchmarking if prepared and polished in the right way still it will be easily ported on top of zephyr as just that vehicle which provides you different threads moreover frankly my intention was to showcase how s and p system might benefit execution of your project because then with s and p supports will be able to execute those separate instances of core mark on different cores and demonstrate very very nice efficiency scaling because you have like 20 cores and you get the same set of things that you executed 20 times faster so even though that thing does not work or it does not compile as an as an application on top of zephyr out of the box because it was never prepared like that and we need to do some plumbing but if we go and look at the documentation which explains how to integrate freestanding application with zephyr we may actually easily do that because for that what we need to do we just need to introduce a couple of additional files to instruct cmake build system of zephyr what we actually need to build and how to do that and then we add some some configurations options which will be which will accommodate things which which are actually used in that project okay so then let's switch to the next slide and see what we actually need to do i hope you may see on in the projection of my slides something i was told that possibly that background might blend with contents a little bit comments are not that important but i hope otherwise something which is not commented out is visible to you so that's actually quite some or not quite some some boiler plates which is typical for zephyr also these are first three lines so we say which version of cmake we have to use or at least what is the latest so what is the minimal version which we need to use and so then we add really boiler plates and so then project stuff and then we simply start to add our source files which we want to cmake built infrastructure to compile and so you may see just a couple of lines and then on top of that we add just a couple of a couple of substances which are unique to that project because we want to be able to pass some parameters in case of zephyr as with many other embedded systems there is no such i think as a common line arguments because well you have your remote sensor which is doing something on its own so how do you expect to pass a common line parameters so that test was meant to be used so on to be executed on like bigger systems with i o and so so we basically will treat that force providing those arguments and for that we have that definition here and what we have in the additional configurations options exactly what we want to be enabled posix api we enable pthread ipr ipc and we enable clock then i had to disable get off because apparently there is implementation of that in in the test itself so that's why during builds you will see that's a clash of multiple multiple defined symbols and so we disabled that i and i intentionally enabled a new leap stl library because otherwise by default that will be minimal lipsy which is lacking some of the capabilities which are used in that test and also well in the end that's a benchmarking application so we want to compile not by default by size but we do that for speed and so what i said we also add that's a little bit of instrumentation in the test itself so that we may accommodate that's a fact that we don't pass arguments and still pass it somehow okay and then we again with the very simple common line similarly to that web server and here i even squeezed that into one comment we say builds we built in that particular folder and then just say run and it's okay we got it executed and so you see we had four contacts four workers or you may have even more and it will equally well scale on your s&p system and you will see improved execution time actually shorten that because in that case there is no relation between executed units so that's you have 20 of workers or 20 contacts for example and they will be scheduled for five cores and then we will have for four times shorter execution time so that that was the intention actually so i think i'm pretty much done with what i wanted to say in that presentation and here i'd like to just summarize that so one first thing zephyr community has been working on posing support for more than five years and a lot was done throughout these years and so we keep working on that primarily kris is working on that and he's asking for contributions because there is a lot of work to do at times the development was more active at times only some minor changes were introduced but most of the time there was a person or a group of people who was interested in proving a posing support in zephyr and as i said people start scratching their itches and get those nice improvements so we may see that development of posing features is really driven by real needs of those who use that in their hobby projects or even in commercial products what is important for developers for who already use zephyr artos in their project or only thinking about jumping on it is that a lot of posix functionalities already available and a lot of things could be done and that means that porting libraries and applications which were originally created for some posix compliant systems they might be very easily reused here and again we may easily jump from one artos which i mentioned previously to zephyr for example or even probably from linux to zephyr artos and as usual there are pros and cons of using that yet another production layer it helps with the software portability but it may cost you something like penalty and performance in memory footprints though you millage may arise we usually say and so with that i'd like to thank you for all the attention and so we'll be happy to we'll we're happy to answer your questions if you have any and i think we are right on time so we don't have a lot of time for backup material and probably just one or two questions okay i'm done with my slides again thank you all any questions if we have time we have time for probably a quick question or two depending on the length does anyone have okay any questions nope alexa i think everyone in the room is all set but if they had them can you tell them how they could get a hold of you or contact you please so there are a couple of things go or a couple of ways so the simplest one just you may find me as a contributor in zephyr github and contact there or i'm also available on discord channel and see in the end i may share my email or probably links foundation folks may share it and so so you will get it and may contact me that way okay thank you guys thank you all bye