 Good afternoon everyone and thank you very much for coming here to my presentation today So I'd like to start off by introducing myself You see my name is Tiejun Chen, but you can come in Tiejun. I'm from VMA China IND ATC Advanced Technology Center. In our team over the past year I'm really mostly working on that LOT internet I was seen and as computing and other Exploration related to LOT, but actually besides them also have some personal exploration like Unicolonel Last year also made several presentation about my Unicolonel exploration For some open source summits hosted by Lin's foundation another thing catch my eye is that Real-time virtualization. I think that is a good candidate to LOT Okay, before I joined VMA, I also worked several companies You might hear some name like when the river system Where I will respond before that when the river links kernel and BSP development and when the river the virtualization technology Like when the river hyperradiol and our power to let the guest OS and also worked at Intel OTC Open source technology center mostly our team were trying to enable that Some new Inter-hydrofeature to open source technology communities like QEM and ZIN and QMU. Okay, so that's something about myself Today, I'm going to talk about that pre-emptive rather bearing links Frankly, I think it's a small story you can imagine, but I think that should be helpful I think when you are sitting here, you should allow links. You probably use a rather bad build for some LOT application I think that the primary RT can contribute to some extent Okay, each time I have to make this sort of statement You know, it's just my personal development my personal exploration So it's not a real map or commitment from VMware Let's go through to the agenda There are a few items here First of all, I'd like to introduce my motivation and the secondly Let's look at that red barry pie hardware quickly and then suddenly I like to talk about a bit that real time And especially for real-time links Firstly, let's review one how and what we have done in terms of that preempt red barry links We also had that a single evaluation The last part is about a road map on my side What we're going to do in the future Okay, motivation. I'd like to make my point from here So in the very beginning I mentioned I'm working on LOTs So LOT basically means that we connect that a variety of device and we exchange data Then we have that data analytics either in our cloud or either at edge side Whatever, so this data processing can come out the real beings value like That's provided that to predict that behavior Then can make our life better and safe and even it can contribute to some cutting edge technologies like motion lane and the block chain You know motion lane need that huge amount of data to train now LOT can come out. That's this huge amount of data And blockchain blockchain can help us protect that sensitive data in some cases like the New vehicle vehicle needs to communicate to another vehicle. We need to this is a sort of that data sharing case We need that block chain, okay Go back to LOT is that Building our intention device and system make up LOT that internet I was seen But the rest of majority center actually our traditional embedded system a red barry patch should be the one of this Typical embedded system. I guess that should be good reference for your LTI on that system design and We know and links had become such a popular open system to that embedded system and now to that LOT devices. I remember that was reported last year and that said so two-thirds of our out gate with What deployed with that links? So you can links can play very well in the LOT case So in my point that links based the red barry patch should be the best way to get started on working on LOT development To build up your LOT knowledge and expanded more that LOT user case But out you needed to consider your time for example Talk about in our car. We have that airbag. We have that brake system. We have that EBS these action associated action should be responded in real time Otherwise, that should be the disaster Even talk about some that gadget like some variable devices We've probably use that to monitor your body feature like heartbeat. If this device cannot connect that information precisely I think it's important less okay, so I Think a new device LT device needs to meet that real-time requirement But the needs cannot do this. So I'd like to integrate the prime links to that the red barry patch links Another reason I'd like to mention here separately is I guess many of you got maybe like me I would like to use that the summer open source a hardware platform or some open source that software ecosystem To build your LOT application, but the often time you want to use the primary links, but they don't support that primary links So you have to build that you have to do that by yourself So it's maybe it's not very difficult, but it's not a convenient So last year I started thinking By the way, we can put some time and effort to enabling that Provide links to this kind of user scenario I want to construct a project that EPLE enabling primal links everywhere So provided that I'm to write better links should be one of the parts Another part is that links kid are already integrated that the primal links to the links kid A link is is that the two kids are founded by a docker cooperate Basically, it can help us build that a minimal that links kernel system We just keep those are necessary components and each component of each system service is a container You can replace each other components with another component as a new requirement I think that the needs a kid should be a good candidate to the LOT So also have some remaining project. I'm working on Now let's look at that Right by pie so to be honest, I'm not from rather path foundation, but I think rather power very successful I guess so you can see that the right by pie in the school for education and Even you can build some DIY device It's very some of my friends that they use that right by a path to replace that That by fair router to build up that smart home like a right by pie is that out to get a way in the home And none of my friends they use that multiple that's right by a path build up So I was at a minimal data center. He deployed that spark and the do that the real-time data analytics and even in some of that commercial usage you also can find the right by a path I read one that article that mentioned Some company night and you see they put the right by a path in their huge Displaying for to cooperate a customer and some of that web hosting provider. They provided right by a path service So now I guess one day you can see at the right by path in the real data center So to me, I think a red bar pie should be a fantastic Now let's look at that hardware If I remember that correctly the first right by path was launched in 2012 and since that Several generations of red bar pie has been released. So now we have that from right by a path zero one and two and three So it's based on different that armor architecture Talk about the latest red bar pie three. I think it's very powerful. It has a full call that Cortex a 53 is based on our movie eight and the Besides that it has that some features it featured the VDAT a way for and the USB and HDMI So I had that one gigabyte to marry. I still like our traditional desktop So basically I think right by path should it's small but affordable. I mean, it's low cost in by the system you can help you Accelerate your IoT development Especially we can put that in the IoT development and I mentioned those example So now let's look at that real time What's real time? I think that's a one good way to end this just look at those existing RTOS real-time operating system look at what they are trying to provide what they have featured you probably can find some common words that keep us like Interrupt and a good context switch or preemption or that latency and jitter I mean the variation and also maybe they said first and I like the point of here that When people talk about the real-time system, there are some common misconceptions the big one is that this thing if we're reacting into Mill seconds or less that can make opera system real time but if we are on take a time like a minutes or several minutes that cannot make a real make opera system real time But that's not true. I think the real-time system should that is a system can reliably take that Seam among the time and each time do the same thing So In terms of this point the links is just a closer to that the software real-time software had a software real-time operating system So it's because it's now designed at that RTOS But we need it so couple years ago. We have these preempty links Actually, it's that set our canal patch Basically, it can help us convert that many links to that fully preempty kernel By hundred percent. It's not hundred percent. It's close to hundred percent What they did like this right LQ basically, you know, they can put the interrupt handler into that Kind of thread contact switch that thread can be scheduled and then they also Trying to get a critical section preempty So what I would so what they did on they are trying to get that the main links to have that a quicker Reaction time and help us bring down that latency to the minimum as very possible Just like that hard time hard real-time extension to our links So compared the prime links to that the main links What has changed? I think after we had that a problem links some feature like that HRT high resolution timer and those interrupts thread and a mule tag They actually had already been merged into that the main line links But it's still gap bigger gap between that primary leaks and the main line links There are two parts action the first you know prior to millions Preempty links the mean it can call them all that risk indication. So it can cause some problem That the level met in the traditional Mainline links you may have to figure this out to make sure a brand links stable It's one of part another part. I think that should be a big one that conversion though Permanent links in the aim of the ad converting that to get that many links preempt Get that a critical section preempty that to mean the you have the Dress that spin lock issue your spin lock cannot be sleepable No, we need to convert that to be sleepable So we need this conversion conversion that the traditional spin log to that sleeping log They still outside that many links Also, um, let's look at that the red bar by links I'm not sure if we want those that that red bar pie hardware platform Actually that that organization that the red bar pie foundation. It's not only Provided that red bar pie hardware platform It also had to develop and maintain one like red bar by links that is specific to that red bar by platform They had such an it's only at the community and has a forum that source code I think you can get from this link. It's the one that I get up It help reject So on beside that sort of a native links code you can find some links distribution links red bar by foundation Provided that rather been it is based on that the bean that's the institute building Specific to read by pie. Besides that you also can find some of existing sort of a sort of party links distributions specific to that right reply You can find a you went to and the windows the IOT call you can find that gentle and open to see But anyway, so these links called all these needs the code all these these distribution They don't provide that the providing that feature So again, if we want to use the prime links, you have to grab that ad patch You have to apply that on your source code You have to try to build that and make and fix some issues to make that work So it's not very convenient So before we and integrated pralink's code, let's look at what's kind of state error So basically and now there are two that get a report to that in term of that to promote the links they hosting that they host a different that means to call code With additional that from a deep patch the first to get a report that hosts the current of RT links development If that one the current development of one particular active version Maybe one day be stopped when that They want to switch to the next version That's the developer motion would move down to the second gate repo. We call that a stable RT version That will be continued. There will be continue be being maintained as that stable RT version If you don't want to use that RT repo, you can use that I patch directly So here you can grab that I to patch You can use that Select appropriate against know you that links the source code Okay, so now let's look at our way how to integrate that into the Pralink's into the red bar links. So when I try to do this I have some discussion with that one of them. I meant to links When the one of that primarily The red bar links maintain the failure. We had some discussion because I have made sure we are the same pity table Make sure this can be pushed into that official branch so Now we have some step to make this happen The first digit is ready. It's just simple like When you want to contribute one of that It have project just focus that and colonize it locally and add that to remote because it can help us That gets the latest on committees simple save good to that rather a pattern The second one the stage one that develop cycle and I discussed with that Philly So I thought that one time I check out that standard branch It's the base of RT branch And then at that moment I grab that the latest RT version patch. I plan that on this is the standard branch but If you during this that develop a window, if that any new RT version patch was Available, we have to replace the original one when you use that a new RT version patch This happened this happened until the stage two During that to maintain a stage. So that means standard branch is released is frozen So step launch. We don't have a base again. So we have to follow this Same policy in term over that RT branch So the big thing is that we don't replace that original that RT version instead We have to figure out that deep that the delta file we apply the delta file out to that standard RT branch and Sometime we have a need to a more that standard branch to make sure RT branch have that a new feature support or new that Commits to that a Raspberry Pi Another thing if you have some bug specific to that a Raspberry Pi But it's related to RT issue. We just apply that on that RT branch So finally, I guess one or two weeks ago. We push this branch to the official so official branch I think you can get that It's that API-4.14.1-RT. It's based on the latest that RT version RT19 if you like that you can try this So during this development cycle, we have we met some issues. The first one is that When you will check out this coming the commissar in that standard branch You find that some commits are really touch that common file like that soft IQ and HRT like this commit They want to convert that one soft IQ commit. I don't know why they want to do it because that's short-legged See nothing said the NASA. I don't know why but anyway So that means each time if we want to replace that new RT version patch you have to replace that again But fortunately, it's not very difficult. So another issue that lock up issue this is because Some of that Raspberry Pi soft code they doesn't exist in the main links. So RT branch have not fixed any issue to that That Raspberry Pi links like this driver USB driver Then that the USB driver the interrupt as ready them they observed that lock up a shoe But fortunately, I think some guys already fix that. So I just pick up that and but with a little bit of change You know in the case your primary RT. We don't need that to disable IQ We can use a local IQ and they still restore this spare. No RT Another issue I guess some guy reported that They observed this warning message that's a disabling IQ on number 59 And I discussed the way that feeling so basically that to mean so in the case of the Raspberry Pi Three device that SPI one and two and you want to one this share the series the similar interrupt line so when these Interrupted handler are straight either. So they call this a problem But we are trying to address the problem. What we could do. So basically I think that one way we can Not force that threading IQ just like our original interrupt handle But that could have a big impact to that real-time performance. So instead of really are recommended He want to that If we want to mask one of these IQ we just a mask all the CPU They are working on this approach Now, let me show that one example how to build that and in my case I'd like to use a cross-companying, but yes, you definitely can build your Raspberry Pi in your Raspberry Pi platform So but if we want to use a cross-companying you can download that from an arrow I use that 64 bit So We now we already did enable that commodity by default. So you if you use that to default to configure fail You can get that enabled You also can make IPM and then install that IPM, but actually this depends on your different distribution Okay, and we have that evaluation in my evaluation environment. I use that Raspberry Pi ps3 model B. I Please install that open to see 64 bit open Swiss is that first official on links to the built-in to support that the Raspberry Pi Typically, we use that cyclic test. It's a It belongs to the belongs to the ag test ag test is that the test suit have us that test different ag kernel feature And especially a cyclic test is the help us to test Kernel latency How is that work and how does it work? Basically that's a cyclic test. I start that Pre-defined numbers Marilyn's thread those marine thread are walking up periodically with that Define the interval by an experiment timer. And then so after that that difference between that pre-programmed and the timer and that Actually effective time Would be a calculate and then you can find that minimum or maximum and average that latency But I recommend that you can use this on batch scrapped because they still based on the cyclic test But it can generate that a diagram It is convenient to see that the data So I test the three different situation the first one I didn't enable Anything it's like that typical that key configuration in term of that desktop So you can see that latency. It's about that three point three dot six second When we put that latency in the context in the context of a desktop It's not a big deal, but it's not good in the case of IOT So next I enable and prevent This can configure have other reduce that colonnades them from the two perspective it was one it adds that More that for option points explicitly and also is if your code they are not executing in that critical section They can get this code to be a preempt like that a low priority test then it come to that When it comes to return from Cisco that is low priority task would be scheduled out that a hyper Already test the camera can run Now you cannot latency is down to that two dot six a millisecond male second It's they reduce a little bit. So last case we enable the permarty is our Completed that's a primary kernel It it further reduce that kind of latency by replacing that spin log to that sleep log Then we can all but not all and make sure a kernel can be preempty Because I'm a low-level code. You still cannot be a preempty You can see that latency is about 150 151 millisecond. So I'm based on our test result. You can see that Primaji right by links really can work it help us reduce the latency from that 3.6 of millisecond to that 151 millisecond but Here's some tips or some issues. So some guys reported that they observed that They observed that some of the accurate thread spikes on CPU to some certain percentage like here You can see that some USB thread spec CPU about 25% But I think there's not a problem in the permarty in the red barrier platform Because I found even I don't enable that Prima-arty you can find that there are a lot of interrupt and that is triggered in the Red barrier pack. So I think these are too much interrupt would have that too much thread because this thread had Had to be handled again again But I want to say here it is really another argument between that I will throughput and the contact switch in some of the links. So Prima-arty links want to get that all Good get that a good context which that mean the interrupt handle cannot be handled immediately sometime That could have a big impact to that I will throughput. So how to balance that? But typically I think we have this way So I recommend that you can isolate the one CPU and then you can dedicate that interrupt to this CPU Like here, I just pass that kind of pyramid like as CPU Equivalent to the three that means the physical CPU of three is isolated from links schedule domain And then I dedicate actually to that the physical CPU three like this you can you can see that So now let's run that in a cyclic test You can see that latency is down to that hundred or fifty that new second. It's reduced about that 30 mil second so Okay, let's go to our last part on the map So this brown official bond just be created the last two weeks go So we have more work to do we needed to test that more platform And we have to run the wrong more tests here as need to run some sort of a sanity test LOPTV and I'm bench and politics to make sure that our edge branch stable and Again, we needed to that support the echo affinity So so far that the red bar links doesn't support that I could finicky But this is a good feature to that if you have want to get that to go to our performance in terms of providing links Another thing about a virtual relation, you know, right by press 3 is the based on top is based on a v on a v8 It has a virtual relation expectation. So you cannot deploy have rather you can run that multiple guest rest So this really another my interest that's real-time virtual nation I think there's a good case to that LTE because what relation can help us address some challenges in the case of LTE like isolation and that's a security issue and that's a consolidation And that's the I tried to do I'm trying to do in the future is that uniques. Now, this is another exploration about the unicolonal I'm not sure I've got to have heard the unicolonal. So you know Is that a special light? Single address cannot image machine major construct by the using lab or Labor OS single dress speech labor OS. So most time is just kind of wrong one process But we have a lot of existing unicolonal, but that doesn't succeed So why is it existing unicolonal have yet to get the larger popularity? I think they are feeling some challenges. So you might unicolonial explosion I try to convert links to that Unicolonal like a unique kind of links to dress is a problem I have that simple POC, but it's over the x86. I want to bring this to that arm It's my present. It's just a small story. So if you have a question For a lot of your test results you were saying you were showing maximum latency that you were getting and it was quite a dramatic drop, but did you also characterize the Latency jitter or you know, how how often you were getting that say 114? Okay, so I mean, I'm not Detested that I'm not test that But I guess you can find some test result before that So my worker is trying to integrate that primal links to that official branch But before that, I think a lot of guys already did some evaluation test based on the rather very pie Links you can find some results. So far, I think that should be good That why we cannot get Pre-emptive You encounter any stability What's the special So what's your question I got your question What's that specific question a specific issue to that From There was a concern that those Critical sections which are now preemptible can potentially hide some race conditions And did you notice any stability issues during your Is Current archie washer, I'm sorry Went down, yeah What I did in my evaluation test I just I see CPU 3 That interrupts Also, oh So what I did in my evaluation test I isolate the physical CPU 3 Then I dedicate that interrupt us be interrupted to the physical seal 3 this can make sure that Your interrupt can be handled in the physical CP immediately without any Infection I mean so but your artistic This space on a second leg test that policy is still trying to aside architect across the for CPU from CPU 0 to CPU 3 It's average that So it doesn't mean that the physical to be three had that to minimal that latency and question So if you're known for the question, thank you. I got all right