 All right. Thank you all for coming today. How many people flew into Portland or drove more than say 100 miles to get here? Wow. Wow. Portland is my hometown. I was born in a hospital a couple miles north of here. Went to high school a couple miles south of here. Went to the University of Oregon. So I'm a local guy. Thank you all for coming all the way here for this conference. It's been a great conference and here we are finally to the end of it almost. I'm kind of new to using open source software. It's been a few years and this latest project has been the most intensive thing I've ever done with open source. And I think the power above everything to me is the quality. And that's the passion of the people developing open source software just to make sure that it retains that very high quality and it makes it just wonderful to use. A little bit about myself. I started developing software actually about 37 years ago. I've done, I did some CAD cam in the early days of mechanical CAD systems where we drew vectors on a screen before there was pixels. Integrated circuit CAD. I've spent 28 years doing different kinds of embedded development from 3D computer graphics systems to things like live television and video processing for, for example, this projector might have some of my firmware in it. And also 2D and 3D printing. And in my current job I've been there about three and a half years and I'm leading the investigation into Xenomai. So today I'll go over a little bit about 3D printing technology and I'm passing around some 3D parts. Those were printed on the printer that we developed locally here just a few miles south of Portland in Wilsonville is where the 3D systems branch office is that develops this print technology. So there's some parts going around. I'll do an overview of Xenomai to maybe some of you it's just a refresher maybe some of you are new to it. I'll give a typical use case in this case a PCI device driver. And I'll tell you about my practical experience basically taking an existing very large code base that has been out and running software that's out in the field that's been through lots of hardening iterations. That's running on commercial RTOS and taking that code and running it on Xenomai under Linux and then hopefully there'll be some time for questions. I work for 3D systems. It was founded actually over 30 years ago by the inventor of stereolithography. And that's a technology that also uses vectors that takes a laser and draws it in a vector layer by layer by layer building up a plastic part. Later they developed selective laser sintering which is again a vector based layer. And then they started developing I guess it's 20 years ago multi jet printing using an ink jet style print head and this is a pixelated technology. And so like these screens here and everything we're used to everything's pixels and in the 3D world they call them voxels but it's basically 2D planes one after the other. There's lots of different technologies that 3D systems develop stereolithography selective laser sintering the multi jet printing color jet printing digital light processing which is basically stereolithography but instead of a vectoring a laser you use a projector and you project UV light to harden the resin and direct metal printing. Most of you how many people have done 3D printing maybe have a 3D printer probably what you have is the fused deposition modeling FDM type of printer with a head that moves around and has a filament. Actually 3D systems did offer one of those called the cube and we no longer offer that so we actually don't have an FDM printer that we sell today. This is the printer that we developed in Wilsonville it's multi jet printing it uses a print head that has hundreds of jets 880 individual jets firing at something around 40 kilohertz and so these little nanoparticles of material are being laid down in a layer of voxels and layer after layer until you build up the part. We take as our input a 3D model and internal to the printer we geometrically create slices and those slices are those pixelated layers or voxelated layers. So there's parts of that layer that's air parts of it that are the plastic material and parts of it that are the support material that are laid down to support the parts that might overhang. And the support material that we use in our technology is a soft wax and it can just be melted away so that's very convenient for sort of post processing you don't have to break off little bits or do anything. You just heat it up a little bit the wax melts away and you end up with your part. Here's a video that our marketing department put together to talk about this technology a little bit more graphically. So you can see how it's sliced into layers and then we have our print head with the hundreds of individual jets firing something like 35,000 or 40,000 times per second laying down millions and millions of little droplets. Building it up with wax to support the parts that overhang and plastic for the part and there's a lamp that fires to harden the plastic. And I think the rest of this video shows how easy it is to replace the material so just pop it out put in the new material. Each jug is only a few hundred dollars so it's very inexpensive. These printers are intended for like an industrial use typically so very high resolution devices very high resolution parts that are printed. And they can also be used to create castings for metal one of the big uses also dental they're used to produce dental like crowns and things like that. And the power of this 3d printing even though it may take many many hours to complete your part through the 3d printer to mechanical designers. That's way way faster than the maybe multiple week turnaround times they would get to actually get prototypes that they can test for fit and usability and things like that. But you can kind of tell by looking at this imagine all the real time systems that are here and and these systems include motion control of all the different motors, the thermal control. There's things have to be heated up some things have to be cooled down. The curing of the resin is exothermic process so we have to get rid of that heat. We have material delivery delivery so we're sensing the levels and the printhead and and in the bottles and things like that. We have to pump pump it through and open up valves and gates and the printhead itself has to be controlled and we to control the jet fire and the data load. And the before printing calibration things and and also as the data comes in it has to be reorganized because the printhead has an interesting geometry of orifices. And as that as those jets are firing during the motion you need to organize all your pixels so they're in the right place at the right time. It's not just a simple five hour something that where you just push the pixels through you have to reorganize them in real time. Now in this case also while all this real time is stuff is going on we're running this slicer that's taking this big geometric object very potentially very complex geometric object. And geometrically slicing it and figuring out what might be above it doesn't need to support and you know what are the boundaries of it and all that so. That's the slicer is in is very compute intensive it uses lots of RAM it uses lots of floating point. And it pretty much takes over the processor in our current systems. We have a Windows PC basically that's running the slicer that's embedded PC in the in the printer. And then we have another power PC 32 bit controller that runs a commercial our toss that runs all the real time systems. So for my project my technology assessment and this is not for any particular product to go out the door yet but just to see is it possible. Can we run this hard real time controller control on a on the same controller that we're doing the compute bound slicing. And that's a fairly big challenge and we chose Linux with Xenomai because and Xenomai specifically the co kernel version of Xenomai because that provides us this hard real time capability even on a system that's heavily loaded. And you can imagine these print jobs some of the big print jobs might take a day 30 hours. And if 29 hours into it you miss a deadline and you have to abort the print that's not going to make the customer very happy so we really absolutely need hard real time response and it has to be extremely reliable. In our early development we had a secret weapon and that was Philippe Jerome he's the originator of Xenomai as far as I know he's one of the main contributors to it. And he also does consulting so we hired him to kind of get ourselves kick started because personally I I'm not that familiar with open source development and I'm definitely not that familiar with Xenomai. I read some of the documentation but having Philippe along kind of give us a quick kickstart. We gave him all our embedded code and he kind of massaged it into into Linux land for us. He created this auto tools based build environment which is what Xenomai currently uses and so that was the natural thing for him to do for us. We could also do CMake or whatever but it's we use we're using auto tools and he delivered these real time driver model drivers and API functions to access the drivers based on this original code that we gave him. And again what he originally gave us was kind of an example that we used and then we looked at it examined it and then kind of threw it away and started over. And actually this was a great way to get started but anybody here who's interested in doing a Xenomai project you don't have to hire Philippe. You there's great there is pretty good documentation and there's great user community support and when you get the Xenomai download it has like example programs and test programs and things that you can look at as examples. And if you've done kernel development you'll be a lot more comfortable doing it. So to get started you have to make a decision in Xenomai 3 you have the choice mercury or cobalt mercury is single kernel cobalt is the co kernel what most people think of when they hear Xenomai is this co kernel. But if you don't have any hard real time constraints and you just want to use the libraries that Xenomai provides then you can use mercury and you can put that on any kernel and you can have the preempt our key patch and you can then get the benefits of the library some of them like that emulate VX works or emulate PSOS and some of the other libraries like that. If you're hard real time then you want cobalt. You patch the kernel with this thing called I pipe this interrupt pipeline that basically takes over handling the interrupts so that you get that real time response. And then you want to select a Linux distribution that's compatible with whatever patch you're going to use. And also note that the single kernel is only in Xenomai 3 so if you're familiar with Xenomai 2 that was only the co kernel so mercury is new in Xenomai 3. So you start with getting the kernel source and after being to this conference for three days I think everybody knows how to get a kernel from kernel.org or you can in this case I'm running an Ubuntu distribution and I can just grab the source for that distribution. And if I do the first method I can get the version of the kernel that exactly matches the patch that I want to apply. Now I could take a chance and try to apply a patch that's close to the version I have with my distribution and I've heard that may work but I've only tried it the first way and it's worked really well for me so. This is an example when you go to Xenomai.org and you see the different patches that are available you can see the kernel versions here that are for X86 version 4 of the kernel the different kernels that the patch will be applied to. So for example we were using an Ubuntu distribution just to get started we started with 14.04 and then later we moved to 16.04 and so we upgraded to a newer patch when we changed the distributions. To get to Xenomai source you can clone it right from Xenomai.org or they also have tar balls that you can download and then put onto your system. I like cloning the repository with Git you get the updates you can there's a stable branch and you can pull from the stable branch and get the latest updates as you're going along. Or you can even if you use the Git source there's tags and you can check out a tag and just work off the tag also. So Xenomai like I said is using auto tools so if you're familiar if you've done auto tools builds and I think people that do work in Linux probably have a lot of exposure to this. Typically bootstrap and that creates a configure file you run configure then you just say make and make install and then you've got Xenomai is now built but it's not in your kernel yet. You've just got the executables of Xenomai and the libraries of Xenomai built. Then you want to go get the I pipe patch whichever one corresponds to your kernel. They recommend using W get sometimes I'm having trouble with W get and I read somewhere that curl is more reliable and I've had success with curl but occasionally I've had failures trying to use curl to get it and I don't know why but it's a network thing. Either one works generally curl I found is more reliable. Then there's a prepare kernel script and you run that and then you configure the kernel and for me and I haven't done kernel development much before I've done a little bit of driver development in a project where there was already a driver and it just needed modification and I didn't have to configure anything or whatever I just look at the code and see where it needs to be modified and rebuild the kernel and it was pretty easy in this case I had to configure. And when you first come up you'll see something with lots of scary warnings saying oh if you have page migration on that may increase your latency and you might probably want to turn that off and I had to learn how to use the slash search for term in menu config and find what page it's on and turn it off. So if you're familiar with menu config it'll probably be really easy for you. It was a little bit of an adventure for me and but this is that kind of a one time thing you get started you configure your kernel and then you're good to go after that. And and I learned a lot about how to configure a kernel by going through this process so it was all good. And then you just make install like you would normally do with a kernel you once you've done your config you just say make you know install and I usually put a local version in. So you can like put a little some number of characters so when you say you name dash R you can kind of see oh yeah I'm running the special Xenomye version that I wanted to. And then you can then the next thing typically you can do is just to make sure that your kernel is actually patched properly you can run this latency script and you get some nice numbers out of it. It kind of gives you this warning I built with debug and so I have quote unquote high latencies which are like in the forties and fifties so that was pretty pretty much OK with me. And the other thing to keep in mind is that when you build a Xenomye application even if you build it as user Fred you have to do pseudo run your application as super user so to get into the into the Xenomye libraries. So now you've got your kernel up and it's got the co kernel you've got the cobalt core there and then you've got these RTDM real time driver model drivers. And then you have some libraries on the user space side that your application can call into to get to those drivers and. So. The real time driver model this is this is intended to be a unified interface for both users and developers of real time drivers and it's the Xenomye programming interface and it was also it's also used by this real time application interface RTI out of Italy. And when you create a device with RTDM it's in this dev RTDM RTDM structure. There's RTDM functions in the kernel to register IRQ handlers. There's ways to actually register your devices and here's the typical use case for a PCI device so let's say you have. An FPGA somewhere and it's and it's on a PCI bus and so you when you boot up the Linux will go out there and kind of probe the PCI bus and and then you can basically get called back and say hey. Here's this device that matches the vendor and device ID that that you want. And then this is just standard Linux there's no Xenomye in here at all this is just standard PCI driver stuff where you want to enable device and request the regions and. I get the start address of the bar and maybe I'll remap the bar into memory map space and but then once you have all that then you can call this RTDM function and you just say I want this handler to be called whenever this interrupt happens. And that handler is going to be called in this real time domain and. So. This is how you register the device so you can create a class and then you can use these RTDM functions again to set the system class and register the device and this is with an RTDM device. Data structure and then you then. You want to set up the device structure which references the driver structure and that gives you the basically the path to your device that you want to. Open and send write commands and read commands from and all that stuff. So a little more detail this is the RTDM driver structure and you set it up with what what kind of device it is and the kind of the key thing here at least for me is that there are real time and non real time versions of things like IO control and read and write. And so you can your device can act as a real time device but it can also act as a non real time device at the same time the same device. One thing to keep in mind is that if you're going to have a process is going to be accessing the non real time part of it you don't want it to lock. You don't want it to lock the parts of your device that the real time is going to want to access that could be a bad collision there. You can also create a POSIX compatible clock so let's say your device has a hardware clock. You just write these functions to read the ticks convert the ticks to nanoseconds or just read the ticks as nanoseconds or read monotonic in nanoseconds to be able to set the time to a specific value in nanoseconds. Convert nanoseconds to ticks and ticks back to nanoseconds and then ticks to nanoseconds with rounding and program local shot and global shot which is basically saying this is when I want to wake up. So this is the time in nanoseconds when I want to get my interrupt and you this is how you do that you write a function to do that you populate that in a data structure. Called XN clock or Xenomy clock. You pop you just give the pointer so those functions. And then in your interrupt handler you want to call this Xenomy clock tick and that's basically going to when you get when you get a wake up interrupt is going to do what's necessary to wake up any waiting threads or kernel processes. Then you you register it and again this is Xenomy cobalt clock register you give it the XN clock structure and you get back a positive clock ID. And then you can use that clock ID in these basically what look like positive functions the timer create timer set time clock now to sleep just looks just like positive. So that's that's very nice to be able to essentially have a real time clock that's your hardware that you can access like you're used to doing with posix write a user space application. Here's a nice thing once you've got got all this set up then this to me this was kind of like an eye opener I did not realize that in Xenomy that most of my hard real time stuff is going to be happening in user space. I thought all of my hard real time stuff was going to happen in the kernel but that's not the case at all. Very little actually is happening in the kernel all the hard real time stuff that's doing everything is happening in user space in user space threads. So you you create your user space thread and you open your device that you created in this KB case the FPGA control maybe read and write stuff to it to get it to do things. There's also a real time thread create see in our case we create a couple dozen probably 30 maybe even 40 threads that are all doing motion control and thermal control and material delivery and head set up and data control and all these other things. And so with all these threads running we there's real time mutex is there's condition variables which we use as a way to pass information back and forth between the different threads. And when you get it going this is a case what you can see this old dusty printer they gave us we just ripped out its brains and just left the hardware alone. And we put in this x86 PC basically it was just a little motherboard a little x86 motherboard the motherboard cost us 50 bucks we got a processor and some Ram and plugged in the PCI bus to the printer and here we're running three different axes we're running a head maintenance axis Z table of going up and down and then the printhead moving left and right so. That was just an early example of like hey I can actually not only can I control this motor in real time but I can have three of them running at once and then more and more and we kept adding and adding and we just basically took this huge. You know with I don't know decades of development person hours developing the embedded control code for the 3D printer and again this printer is shipping this is a printer that's shipping you can buy today that uses a different kind of controller. And that code that's been hardened by going to customers finding bugs going through QA cycles and release cycles and it's a very well hardened code. This is now running on Linux with ZMI so we were able to print a beautiful looking part. And it was actually a big surprise along the way we ran into some problems and these are kind of problems if you're doing ZMI you may run into yourself. Check the time here I guess we got plenty of time left good. So thread relaxes so. Your threads run primary mode is the real time mode secondary is running just like a Linux any Linux thread and this is really nice actually because if you need to do something in the Linux world you just do it. And the thread becomes the Linux character and it runs in non real time mode and it does its thing. And then when you come back and you access your real time functions boom it pops back into primary mode and it's a real time thing if you want it to always be real time. Don't call those other Linux functions. But when you transition from primary to secondary it's called a relax and sometimes you have a spurious relax is because something happened. That caused it to go into secondary mode but you weren't really expecting it to and these actually are you get notified by ZMI. If you look at the syslog you'll see you know relax on this thread and it's very useful and there's another utility that where you can turn on and get a log of all the relaxes and get trace backs of where the relax happened. And so this is very very useful. An example where this could happen is let's say I'm doing all this code almost all this code is C++ by the way this embedded control code was all developed in C++. So it's kind of handy to use a collection class and then if you start adding things to the collection class and it needs to grow if that collection class happens to be on the Linux heap. Then it's going to make a call into Linux and you're going to relax into secondary mode. There's you can create memory pools that are real time memory pools which is what we do now. But you get these relaxes happening just by saying insert something into a into a map and you're like how'd that relax happen. The other thing is if you memory map a file and then you have a pointer and you're actually accessing the file like you would memory. But the file system is Linux only you don't get the file system in real time mode. So you'll get a relax and sometimes just referencing that will cause a relax. So in our case we had this issue where we had a non volatile parameters we kept them in a file. We memory map that file so we could get to all the parameters like they were in memory but they're kept in a file so they didn't go away when we rebooted. And every time we read from that memory that memory or wrote to that memory we would get a relax they're happening all the time it's like oh we got to fix that. So we made a copy of that memory and so we just access this cache and then we found this really cool thing called the cross domain datagram protocol. It's part of Xenomai and it's where primary mode thread can push data out to a secondary mode thread. And so that was nice when we need to update our file we push it through this XDP socket. We never leave primary mode and the other side never leave secondary mode it's reading from this socket it stays in secondary mode it writes to the file in the background and everybody's happy. And if it's writing to the file and we need to do something now Xenomai takes over and does this thing and we go back to what we were doing. So it's very nice that solved it no more relaxes no more weird latency issues that just fixed everything for us really nice. We had a case where we wanted to do a time stamp of some material like when a bottle was put in and so we wanted to like time stamp when a new bottle was inserted. So we use this get time of day call in the kernel in primary mode and that works great because you're just reading from a location in memory somewhere at the time of day and it works fine. Memory access is no problem but sometimes the OS has to update the time of day and if the OS is updating the time of day you kind of go into a little spin lock until the OS is done. But the problem is I'm in primary mode Linux can't run when I'm spinning and so Linux can't ever update so dead and so we put in this little change to time stamp and everyone said I want the thing we just die and so we when we roll back to change that lock up went away so we kind of knew all right somewhere in these changes. This is problem we were to track track it back to get time of day and a big help by the way was the Xenomai forum it's like an email for my if you use Xenomai sign up for the email form it's extremely useful. And just searching through that forum there was somebody had a similar problem and explain the whole thing of what was going on so I like goes on I see what's happening. There is a real time clock get time call as part of cobalt and there's special type of time called clock host real time so that was our solution it's a different way to get the time where we didn't. And that that's in user space but in a kernel space there's actually an RTDM call that just gives you nanoseconds since the system booted so that's that's pretty useful. So the NK lock is an internal Xenomai lock that kind of protects some of its internal data structures. And so we also had our FPGA could have multiple threads accessing and simultaneously and so we wanted to block out certain references to the FPGA with this RTDM spin lock which is a very very useful kind of lock that lets you. Make sure that you're not clobbering yourself on multiple threads running. And so that's how we protect that resource with this RTDM spin lock is very simple and easy to use and very and it works fantastic. And then while we would have the FPGA resource lock maybe we use the Xenomai select signal to wake wake up a thread that's waiting on us on a select. Down under the cover somewhere it for a moment locks this NK lock internal Xenomai lock to do its thing for select and then it unlocks it and then we come back and we unlock the FPGA. But on the periodic timer when we enter the periodic timer function NK lock is already locked. And so then if we lock the FPGA we get the ABBA locking issue and boom. It just comes to a screeching halt and again there's some nice features debugging features where if you if your thread goes to sleep or whatever it you can like get notified it's hey. You were asleep for more than three seconds why and it'll just like send you a signal. So to avoid it we we could have say that whenever we have the FPGA locked we just don't use these Xenomai or RTDM functions so that we don't ever have that problem of locking NK lock with FPGA locked. Or in the periodic timer function we just don't lock the FPGA and we only do things in the periodic timer that don't require us to lock the resource. Or we could build our own timer interrupt and not use the sort of built in Xenomai timer feature and then we don't have NK lock locked. We chose the second one our periodic timer thing was like a watchdog timer that had you know poked a register and we're the only thing ever poking that watch watchdog register so you don't need to lock the FPGA in there. We had this interesting issue where in this code base that we inherit you know that we brought in this big code base. It has the user space timer capability that's kind of a library function and so it it starts up all these timers and when it gets a timer interrupted basically says every timer that was set to go off before now I'm going to wake up. Even if maybe this isn't my interrupt I don't I don't know that this interrupt was associated with my timer but I know that if the time is T and somebody requested me to wake up before T I'm going to wake up all those things. But we had an issue where you say I want to wake up at time T and it converts that nanoseconds into ticks that rounds it down a little bit so when we finally get woken up the now time is actually before the time when we wanted to wake up. Which is not unreasonable but in our case we wouldn't wake up anybody would say oh the timer came in and nobody's waiting so then we'll wait for the next interrupt which then never will be one and we would just stop and go how come the timers aren't going anymore. That was a pretty easy thing to fix we just said oh when we read the current time we add one hardware tick in nanoseconds to that now time. What's interesting about this what I found really interesting is that on our on our proprietary system it had the same problem it rounded down and everything the same way. But the latency was big enough that that little bit of rounding down never was a problem because by the time we got into that user space function that compared to now and when I wanted to wake up time it was always after. It was always after the latency was big enough that we never had this problem in Xenomai it was very fast so yay for Xenomai for being fast. So let's see we have our entire embedded control code base which is actually all hosted with in Git on our local Git server. It will now run on Linux and Xenomai that embedded control all runs and we can we can do everything we can print and do everything we need to do the whole code base is running. The old controller was a 32 bit big endian machine the new one is a 64 bit little endian machine commercial RTOS and they're actually compatible so that's very nice so we can we can build that we can actually maintain the two code bases. It's one code base we can maintain the code base and the two systems together. So the key thing was to bring in the slicer so we brought in the slicer as this was running on Windows we built it on just as a standard Linux app and we let it rip and it takes over the whole processor. I mean it is loading that sucker down this slicing away as fast as it can and there was never a problem with the real time I mean Xenomai worked extremely well I've run jobs for two day solid huge print jobs there's never a glitch now granted that as my hardware guy tells me who's a little bit nervous about you know an unsupported operating system. It's like well we have three printers running now in the lab and it's like three printers a very small you know sample and you've only run it for a few months and it's like the fact that you had no problems isn't good enough you know but I think it's pretty good it's been an extremely successful. Investigation now we have to get past the oh what are the legal ramifications of open source software what what are the support ramifications what are the how do we fix problems when we're in production all these things but anyway any any questions. Yes that question is that we analyze the cost difference and we have the Windows license I'm not exactly sure but I think it's like around $150 per machine I'm not sure don't quote me on that well actually I think I'm being recorded so I do know I do know that our our commercial art toss costs us more than $100 per printer that we ship more than $100 per printer so I think our total cost just for licensing. Is like $250 or more and this. As if you were at the open source. Sort of licensing talk that was here a little bit before me. It's I don't ever want to say that Linux is free you know we there's a cost to it we want to contribute back to the community but. The bean counters will like it. Yes so the question is why are we running slicing on the same process or they were doing real time control and hey why do we even need real time control because. There's lots of smart motors and other things that don't really need. You know hard latencies and those are good points. The reason to run the slicer on the printer is when it's slicing it's slicing in a way that is very particular for that device so it doesn't just slice up the model. It has to slice it into the exact resolution and pixel dimensions and everything that are for that printer or that printer class. It has to understand how the supports fill in and things like that now we could do that on the host computer that is like that that's driving the printer could do all the slicing on that. But what this provides for our customers is. They have we have a thing called 3D sprint which is our our application that runs on your desktop PC or your laptop. You bring in your model you select the printer you wanted to print to you say go print. And if you want to start another job the exact same job on another printer you just select another printer you say go print. And so they may from one little laptop be controlling 20 printers that are on their shop floor. That they spent millions of dollars for these printers. You know they're now they could have a separate laptop for every printer. And so each printer has a dedicated laptop but this this is something our customers have found to be very useful. And for us it's less of a maintenance headache when we release the 3D sprint application. It doesn't have to know anything about how the slicing works. It just has to send the send the job to the printer and the printer figures out how the slicing works. So that's why we do that. And again in the current application we have a windows slicer and then we have a separate thing that's running the real time control separately so they don't have to interact. And we could keep doing that. Our hardware guy really wants us to keep doing that. So he wants to develop the next real time board because that's fun to develop. But we'll see how it goes. And then as far as do we really need hard real time. That is actually a really good question. And I went to a couple of talks at this conference that made me kind of question how how would this work if we didn't have Xenomai. I suspect that we would get the occasional glitch now. So part of this is that we're running at an existing code base like I said that has a huge legacy history. This code has been developed over a long time with lots of developers. It's very well hardened. And so we like to keep this code base active and it does depend on hard real time. We've tried we actually tried running it just as an application and it didn't even come close to working. So yes. Yeah, go ahead. Yeah, so the question is how many slices ahead do we need to be processing as we're printing. And there's two answers to that one is the slicer generally on most parts will run quite a bit faster than the mechanism can print. So we're getting slicers maybe we're getting two or three slices per printed slice. And so the slicing gets completed well before the prints done in turn on the embedded side. We have chosen because because this kind of gives us a good amount of buffering 16 layers. So we we bring in and we start processing and we have 16 different threads running the processing we need for the data on 16 layers. The throughput is way fast enough to handle a layer per printed layer. But there may be some amount of buffering that's required so that we never get starved. We can't allow to be starved. The question is do we do any kind of on the fly correction for you know errors in the positioning of the of the platform things like that. No, actually we do not. We require our platform to be exceedingly high resolution on its motion. So we have a platform that we can move like one two thousandths of an inch accurately. You know it's very very high resolution on that motion. Yes. Yeah. So basically our generally our deadline or our window biggest latency that we can accept is on the order of 50 microseconds. And generally speaking the preempt our key patch will give us that but it's not 100% guaranteed. And so we could miss a deadline occasionally. There's ways that we can recover from that and a lot of the things like the thermal stuff if we miss a heater control operation. The heater control operation are happening at half line cycles so that's happening at 120 Hertz so that's you know pretty slow anyway. So we don't really need hard real time for that. But the problem is that we we've had the experience where even on the hard real time system on the commercial RTOS if we're not coding the priorities right things like that. We'll still miss a deadline. And when that happens it's not good. We when we were originally a lot of this code was developed for printing solid ink for color printing. So it's a similar print head and technology that's for that. And if you print a bad page. Well you can kind of reprint it or something you know kind of get around that but this this is not I mean if you if you miss print a layer it can leave a bad feature on it and it's not acceptable to the customers so. Yes. So the question is how many engineers how many firmware developers essentially are on our team. And do we require or does you know my require a multi core processor on our team. Well originally it was Philippe was working as a kind of an consultant. It was just me and then eventually we brought in a second guy and the two of us were able to take this extremely large legacy code base and get the whole thing running and making these beautiful prints. So we were very it was really some of the most fun I've had. I mean it was a great little small team and we got an amazing amount of work done in a small amount of time. And it was a ton of fun. And by the way I was going to say that we all are super lucky to do what we do. This is really a great job to have is to work in embedded systems and it's so much fun. As far as the Xenomai Xenomai would work on one core. In fact we thought it was using all the cores of our four core processor. But then when we started looking at the at the affinity it turns out that by default Xenomai has sets the affinity to a randomly selected core. And then as you as you spawn your threads all the spawned threads are on that same core. So we looked and there are 30 threads we're all running on one core. And it was still working fine. So we thought well that's you know maybe we could spread it out but there was no reason to. And then the as far as the slicing application yeah we could set the the slicer to run on the same core as the real time is and there would be no problem. I mean it is running on this it is running on that core as well as others but we could run that slicer on a single core and Xenomai. That's the really great thing about Xenomai it's I mean I'm writing user space code and it looks just like user space code it builds like user space code. I lots of the threads I'm doing are true Linux stuff manipulating files and talking to the network and doing all the things I need to do. And this is a quote unquote a Xenomai thread but it never goes into real time space but it's communicating with my real time threads in a very easy way. It's just a beautiful I loved it. The art we're using Intel i3 and it was running at like three and a half gigahertz didn't need to be fast. We were doing some measurements comparing you know what it was before and after and it was like 10x faster on some of the stuff we were compute intensive stuff even though it was running on only one core. It was still 10x faster than the commercial art so it was like wow we have way too much power here so we could dumb it down a lot. Yes the question is are we looking to move to like an arm processor cheaper hardware etc. And the answer is I'm not allowed to talk about that so you can guess. So personally I really I'm I don't have any problem with running on an I3 I think it's fantastic I love it and these printers are selling for 40k pop so why not you know why why do we need to. But. It's. I can't talk about what's going on in the labs. Another question yeah so no we don't have the question is do we have any other sensors to kind of see how the prints going along and kind of make sure that everything's. You know correct without having a I think this is I'm I think I'm getting the hook. We have an end game to go to which I don't I've never been to but I think it's going to be fun. Thank you all for coming today thank you for your attention and also thank you for your contributions to open source because it's fantastic I love it and I want to keep using it thanks.