 All right, so for our last session of the day, we're living the canal and we'll venture into a build system So please a round of applause for Tim and Ilyas Thank you All right. Thanks a lot So we're gonna be talking about what it really takes to maintain a community BSP layer of pretty high quality one and confidently updating Through a lot of major changes. So I'm Tim Orling. I'm a principal software engineer at console code group And this is Ilyas chargui chargui who's Was supported by meant for med by Medtronic I always like to put the abstract into all of my presentations so that you can go back look at the slides later and know exactly what I submitted Okay So console code group. We are a services company. We specialize in embedded Linux and open source software We really do everything literally everything and Nothing's too hard. I've done some very crazy things including live OTA updates from Windows to Linux in the field We're based in San Jose, California, but we're actually a global team. I've got some of my Bulgarian and Portuguese Friends here in the room Hello everyone, my name is Ilyas. I'm working with Medtronic as principal software engineer Medtronic is known in the healthcare technology solutions. We are based like everywhere I'm attached to the robotic department these robotic departments Provide recently the Hugo robots these robot is for the surgery in the OR operating room These teams in a base in in in the US However, my selfies in in the UK London and then my focus is in the real-time video processing as well as I'm part of the OE 40 open embedded for tigra and So this is a link to our github community So this is where all the layers and all of the open source stuff that we're talking about can be seen We have absolutely no connection to NVIDIA at all except that we are supporting their hardware So they're not necessarily, you know paying for any of this in any way That we do get some early information, but that's about it There's been several previous presentations including one by my colleague Leena Navi There's also one that's really good to look at from Matt Madison himself Which is a good idea of the overview of what meta tigra is and how it was all put together There's a lot more detail about secure boot and Disconcription and things like that that I gave in a talk in Dublin So we're gonna go over that today, but please go back to look at this talk to get a lot more info Also, there was a talk earlier this week about secure boot itself by Northern tech, so please go watch that video to get a much better idea if you're new to secure boot So we're going to talk just briefly about Some jets and hardware that's actually supported by jetpack 5 there are definitely changes here You need to be aware of what they are The main purpose for this talk was talking about embracing change and how to move forward confidently So we'll go over a few details about what has changed Then we're gonna talk about the future just to give you an idea of what to expect that's coming because again There are more changes coming and then finally we'll end up with a call to arms Then in jetpack 5 we are supporting silver hardware mainly Orin series and Xavier series from the orange sides. We are Supporting Ajax Orin. This Ajax Orin is like considered as the height height and hardware the Jetson Orin have a 16 CPU core and 2048 core GPU The median and Jetson is Orin and Enix Orin and Enix have a CPU core and then 1024 core GPU Orin Nano is similar as the the Orin Enix in term of a GPU however or less CPU core and it is considered as the low-end Jetson platform Then the prior platform was the Xavier series. So there's the Ajax Xavier. This was the highest end at the time This has eight arm cores and 512 GPU cores There's also the Jetson Xavier Enix This has six karma cores and our arm cores in 384 GPU cores so you can see quite a bit of a jump into the Orin series So as we said the major reason for this talk really what we're trying to talk about is embracing change and feeling comfortable with that So the first thing we want to talk about is jetpack 5 and in jetpack 5 as I mentioned previously We are supporting the Orin series Orin family as well as the we are keeping supporting the Xavier series however with we drop some other hardware like TX2 series and nano series we moved from kernel linux 4.9 to 5.10 Regarding Ubuntu because Ubuntu is considered as like the base reference file system from the nvidia side then we moved from 1804 TS to 2004 Regarding the arm trusted zone we moved from trusty to opti Bootloader was seaboot Now it is UEFI bootloader As well as regarding CUDA and then accelerator hardware we moved from 10.2 to 11.4 and then we have a working in progress branch we want to bring as well 11.8 with CUDA 11.4 So if you want to follow that progress the first QR code is to that link for the PR The second QR code is what we recommend you go look at to get a better Deeper dive into the new jetpack 5 stuff. So this was a presentation by nvidia We actually use some of this content in in this presentation So if you look right now at this QR code, you should see this page This just gives you an idea of exactly what's in the current release that that nvidia just put out Okay, so a lot of this stuff is you know Some new slightly new platforms came on board and those you know were introduced and things like that The one thing that we're excited about is this is the first time that nvidia has presented to us their OTA mechanism. Okay, it's different than what we're going to talk about in a little bit, but Um, we need help getting this implementation For jetpack 5 into metategra. It's not there yet. Okay um And again, this is you know, that link at the bottom is is where you can keep up with exactly what the latest jetpack release is One of the biggest changes that we had to embrace maybe a lot of you that are in the octo world had to embrace Is when you're moving from dunfell to kirkston or any Release before hannister to you know after hannister the override syntax changed There are other changes such as variable syntax and spdx licensing and things like that But the one that's probably going to bite you or confuse you the most will absolutely be the override syntax So in the old way it was with underscore and now it's with a colon So if you have a machine override like tegra 194 That would go after the colon for an append or prepend or remove Tegra metategra we use overrides a lot If you don't know how overrides work and what they're good at and how powerful powerful they can be Go look at what we're doing with it. Okay, it's really An amazing tool, but it's a little confusing to switch to the new syntax The other thing is that the mixed use of Append plus equals was changed. So plus equals means append this thing with space after the the variable I've already got right Having that happen at a different time than when append actually applies It's just really confusing to people and people were using it wrong not getting the behavior they expected So it now throws a warning Don't use this just use append. Okay, it's just easier There's two commits where all of this happened. So if you want to see The majority of you know, what happened and what kind of changes you might need to be looking for in your own layers Look at these two commits Okay, um, as I was mentioning in in jetpack 5 we we we moved to like we we we We go from like a different like Implementation and then here we will mention like the bootloader from sibu to efi ufi implementation if if anyone interested we have the link there This implementation is based on genocore a dk2. Why we moved to to to ufi in this because of like Trying to avoid any any additional any customization required from the operating system as well as we will we will we want Like going to the standardized side of things regarding secure boot as well as update firmware If anyone needs detail about this implementation as well We have the wiki page from like nvidia have a specific implementation of the edk2 Then please go to to that wiki page As well as i i in previous slides we i mentioned the The changes on the trusted os trusty to opti Opti open portable trusted execution environments This project is open source and then based on the arm trusted technology maintained by uh, Leonardo Leonardo group Here we we mentioned like explicitly if you want to port any trusted application from trusty To opti you need to align with the global platform api And then here as well we mentioned another example regarding the trusty tier Applications, which uses the ipc ipc mechanism To handle a low-level message communication between the trusted application and then client application However opti use rpc remote procedure call Oh, sorry Okay regarding the secure boots implementation in the in the previous like even like jetbag We have a different version and then we have the ga and then out of the ga We have different revision the earlier visions. We are supporting only a signing mechanism Signing mechanism Mean only one one per key per key is like the picasi key public key cryptography But in the the latest Jetpack here i'm mentioning explicitly the linux for tegra releases. We are we are supporting We are addition. We are adding the encryption Like functionality support and then here we have two two kind of key one for signing which is per key a public key program cryptography, sorry and then a secure boot key Regarding regarding enabling the secure boot You you need to go in in like a mod to program the device this like programming more focus on the diffuses burning I really like the the implementation recent implementation. We are using like The diffuses like configuration file xml file which make the things like the burning process much easier Here i'm we are mentioning explicitly like the the difference between the two series or in series and xavia series Or in support pkc with the different kind of algorithm RSA 3k and then elliptic curve digital signing algorithm p p256 and then five to one secure boot key It is 32 bytes. However, the xavia series is supporting pkc with only rsa algorithm 2k and 3k and then regarding sbk is 16 byte If you need any any detail about how you enable secure boots Via the tegra metallier, please go to the link and then we are like defining really well like the the detail It's like super easy one important thing here because I experiment these things with with jetpack 5 Please make sure that the boot security info Is well defined like the the value Here is like well defined because otherwise you will break your device this value You can get like the definition from the Tegra fuses specification And one bit off and your device is bricked and we have all done this all of us have done this Regarding the ufi From the nvidia perspective we have like secure boots ufi secure boots We make we are trying to make the things easier for like if anyone wants to enable this from the tegra metallier With these two two variables Like super easy key and certificate, but in the reality we have three different kind of components like a platform key this platform key signed the Key exchange key and then this key key changed the the database at runtime when you run your device You have secure boots everything okay in the ufi stage The images kernel dtb in a tram Like goes to like a verification mode if the the signature verified Okay, then you have the green light and then your device boots however if the signing is Like failed then you are like you are stuck. So after this you're you're into the kernel, right? So this is only very very early boot. That's all this is protecting. It does nothing about the rest of runtime at all Then here we we we are Mentioning the dis encryption. This encryption is like more to get like an extended Chain of trust then I I mentioned previously secure boots ufi secure boots And then this encryption like here to protect your like your root 5 system. Okay, and then Previously in the previous Prior jetpack we have like the nvlax application Implementation from the trusty, but we are moving to opti One more important things here in the like the The nvidia platform. We have an entity called ekb encryption key blob to generate this blob You need this for kind of inputs OEM key to this is mainly for the audience series key key key encryption key For the xavia series a fixed vector and then same key file For kernel encryption the kernel encryption is not supported yet because the ufi doesn't have the encryption Mechanism to to do to to perform this The same two key file is more for this encryption To perform this this encryption we need trusted application and then client application here. We are mentioning the nvlax serve application Mainly to get the passphrase or or the the password and then you can perform like the disk encryption This application use the disk uid as a unique context because you are You are like giving to opti a unique context and then you get like this unique passphrase Like to perform the disk encryption as well We are you like you have to use as everyone know the creep startup application And then creep startup application use the dm crypt Like kernel module as as backend again one more important thing here Please make sure that you are using the right a fixed vector for the ekb generation This fixed vector need to match what's opti os Define otherwise you have this like application This is in top early like stages before ufi If the json user key pta failed then the disk encryption failed There is no disk encryption you cannot do you cannot perform this and you won't even notice this if you're only doing secure boot And you haven't done disk encryption yet because the default ekb only has zeros in it So you will never come across the fact that it isn't working or that you need the new fv Or a new build your own ekb until you get to the disk encryption stage So everybody's interested in ota updates. So we mentioned that nvidia has their own implementation That's not currently, you know, we're not using their implementation yet But there's several open source implementations that are out there All of these also have commercial support available One that's a little bit special to us is mender because of the fact that we had a fully working jetpack for You an example in the tegra demo distro we could use some help updating that to jetpack 5 right So when you're talking about mender, you've got the meta mender layer Which is where all their core stuff is at and then there's the meta mender tegra layer in the mender Metamender community layers right so by definition its community is community supported Right now that probably only has support support for jetpack 4 and the older devices right Raoq is a similar story Raoq's got the route community layer where there's also meta rock tegra One of my colleagues actually did some contributions in this space And again, that was for the older platforms You may also want to look into software update or os tree There's a lot of different mechanisms out there pick one of these any of these is a great choice Uh, one of the things that was important about this whole thing and one of the things that meta tegra is a great example of Is how to keep up with yachto project. Okay You don't want to be using Somebody's vendor stk that was based on Zeus that's been end of life forever And still claiming that you're secure yada yada yada right you need to be moving forward always So the non lts branches they go end of life in seven months, right? That's not very long time and after that you're not going to get any help from any of the community Supporting you because we've already moved on we've forgotten what was in that Right, so it's a constantly moving target stay agile The best way to do that is keep part of your team or part of your ci Building on and I literally mean this master development branch. Okay, things will break Make sure that that particular pipeline and workflow Doesn't stop everything else from occurring, right? But let it break have somebody on your team go and look at the breakage Figure out what it is Have a branch somewhere where you're fixing those things It's a great opportunity to to contribute to upstream because you're going to catch something that matters to you That we haven't caught yet. All right Now minor breakage bit by bit is a lot easier to deal with than major major technical debt in two years or four years. Okay So i'm not going to talk as much right now about the specific Timeline i'm going to cover that in just a second, but please don't wait four years to switch to the next lts release Switch to the next one immediately as soon as it's available Use the current lts that you're on for that particular product line that might not be getting updates or something You know might not be getting changes, but please move forward So how do you do that the only way you do this is automation and the only way you can do that is through continuous integration So we as a project as a community have the oe for t auto builder So that first link will take you to the instance of that. It's based on build, but The word based there that'll take you to the code for it It's not the same as the octo project auto builder. It's similar concept, but it's different It builds these three layers metategra metategra community and tegra demo distro multiple combinations of things But it only builds the current Currently maintained branches. So right now as of today, we've always got master or development branch It's never going to be stable Never based product on this right, but this is where all the active work is going right now That's the some details about what's supported there The current stable release from the octo project is michael door. So that's also on the same jetpack release The current lts is also on that While we were working on all of this stuff, the prior lts had a different jetpack for version, right? So we have a named branch to let you know that we have the other version available And we try to keep the prior lts up to date, but you know, we can't do everything So now it's time to talk about the future Yeah, for the future we are expecting like jetpack six soon. Hopefully In as use as everyone see here, we have the ga version and then different revision for jetpack five We are expecting as I said jetpack six in q3 the changes here. Maybe it sees us like minimum We are moving from like kernel 5.10 to 5.15 and then as nvidia use ubuntu, we are moving from 20 or 4 to 22 or 4 But we keep we will keep jetpack five up to date like but mainly on the security fixes And we'll mainly do that on the kirkston branch, right as we're moving forward masters only going to be on jetpack six All right, the latest stable release is only going to be on jetpack six One more thing. We will in jetpack six. We will support only or in please make sure like you are keeping forward with the hardware. Otherwise, xavia, for example, will be dropped and that's nvidia's choice, not ours, of course, so a little bit about what are the, what kind of terminology, what thought processes do we have but behind the supported ductile project releases Okay, like every other open source project, we have a limited number of resources. There's only so much we can maintain So we always have to have the development branch. This is the only way we're we're going to stay Agile, right? So that's always going to have the latest jetpack. It's only going to have support for the latest platforms that that jetpack supports But whenever upstream changes the layer compatibility, we're going to change that so you can't count on this being static at all, right? It's just going to be constantly moving We also have the current stable branch So right now that's michael door That's the current jetpack as well, but it's again, it's only the platforms that are supported for that that will end support as soon as we have a new stable release We also support the current LTS that should be on exactly the same jetpack Unless something broke so much that we can only do the new jetpack on development branch, right? It does happen We will also try to keep up with prior jetpack releases by just doing a named branch such as we have for kirkston right now We will try Best effort to keep new updates to the previous LTS branch, but that's you know again going to be best effort basis We just can't do it all So more about the actual supported yakta releases you may or may not be aware of this So this comes straight from the yakta project wiki releases page So right now we're in michael door 4.2 is the stable release It came out in april You can count on this tick tock very very closely to this. It's very rare. We deviate from this very much So the next release will be nambield So the next release will be nambield that comes out in october In april of 2024. We'll have the next LTS release come out So if you want to get a jump on what the LTS is going to look like For your own projects and everything and be ready to pick that LTS up as soon as you can Please start with nambield Immediately like as soon as you can if you can't be on michael door yet, but spend some time between you know october and April so at the end of the year switch your team over to the new LTS release Then we're going to have three stable tick tock releases in between each of those is seven months And then we expect another four-year support starting in 2026 In april, okay Please move to the next LTS release as soon as you can There's a lot of factors including other, you know, sdk's and vendors and everything we understand that But please don't stay on an LTS release for four years Because all you're doing is creating technical debt Don't do it. Okay. It's a really bad idea So a little bit just to kind of give you an idea how to get started with us right now Go and clone tegra demo distro Check out the kirkston branch and we use get sub modules One of the ways of maintaining layers. So that's what we use You want to go ahead and set up the for instance the jetsons aviar nx dev kit And then build the demo image and you're off to the races, right? So this is really a pretty quick process to get going And that dev kit is not terribly expensive The last thing we want to talk about is as a community we need help just like every other community does right So how do you get involved with us? This first link is to the wiki. This is kind of a you know getting started or welcome to us page We're going to do a little bit better of an actual getting started guide, but it's not ready yet And then the other thing is we uh communicate on gitter For geither if anyone wants to to join we have as well some rules the rooms Sorry specific rooms for updates as well as security So that we have we specialize rooms for special, you know restricted conversations It's also a good place to come ask questions and things like that Uh, we really have to thank the people behind us. This is us standing on the shoulders of giants, right? So matt madison did the vast majority of all this work. We couldn't done any of this without him down walks is one of the people keeping the community Aware moving forward communicating and so on and kurt keifer did a lot of Security work and secure boot and so on work that really helped us out But it's really the entire community. It's an amazing community meta tagger is one of the best bsp layers out there to Get an idea of what you can do with What a vendor gives you Uh, so uh, that's it questions. Thank you And please wait for the mic anybody that's ever watched one of these virtual or After the fact it's really annoying and you can't hear the question Hey, I have a question about the um kuda 11 8 support If jetpack 6 drops before you integrate that are you going to backport it or wait for the jetpack 6? You mean jet packs jet packs fix? Support kuda 11.8, right? I just mean if you don't get it merged if you don't get the 11 8 merged into your branch And then 11 and then jet pack 6 drops. Are you going to make sure it's yeah We we see if jet pack 6 bring kuda different version. We need to align but This this is like super new and then and vidya provide this for jesson specifically because there is kuda for the sbs Uh kind of architecture then yeah, we will we will see how we will manage it for jet pack 6 for sure Yes, so as I said this will take you to the pr. So it's actually in progress, but We're also here at the conference. So we're not doing as much You know work as we might be we will try we will try to get it like really soon magic. Yeah So semi related to that. We're working on a customer's product We're currently stuck at jet pack 511. Okay. Okay. Where there's ab update issues which nvidia have acknowledged. Yes on their forums Is it a case over there? You're gonna say we're not going to bother fixing that till we go to jet pack 6 so it's That was why the and that introduces a whole load of other bugs Because they're bound to break other things when they go to 6 I'll try to just be polite and not not comment on bugs But yes, there have definitely been bugs and flaws and things like that that have occurred My other talk was not quite so nice. So go watch that one Um, I mean, do you feel like they're rushing forward? So in in in particular We were very excited about moving to uefi and so on because now we're moving towards modern You know what at what arm system ready and things like that is heading towards right However, the big disappointment was there was absolutely no sign of life of You know Some of the secure boot support and ota support and so on so the very very latest 531 release is the one that had the ota update mechanism from uh nvidia themselves, which gives us I don't know 5 000 lines of bash scripts that we can look at To see something that they have actually proven to themselves works So in case any of our stuff had to change We can you know, we have something to look at right and something that we expect to have been supported um I you know, I won't I'll be blunt like it's been frustrating that that particular thing Took a while and so did secure boot. So i'm also a quite understanding about folks that are struggling With other releases that they're on I all I can say is you really need to direct your Uh I are at at nvidia, but we're we're trying we're in the same boat. We're trying to help you all out with that Uh, yes, um, I'm by myself a small maintainer of a bsp layer as well I'm also facing always with a question How many of these non bsp related tools should I add to my bsp layer? Or it's not better just for architectural purposes for readability purposes to make a new top layer for that stuff So we really try to embrace, you know, the octo project approach, which is that you know A layer is trying to do one thing and do it well, hopefully, right? Um, and so we we try to not add a whole lot of extraneous stuff. However If Five people want five different ota implementations and all of them have work and are demonstrated and they have great documentation There's no reason for us not to go ahead and include examples for all five, right now if in the future Those people that were supporting that drop off the face of the planet And we have nobody else are willing to continue to support that then we're going to drop that feature but um I think there's a fair amount of tools that you would be surprised that we have and that we have actually created ourselves um, so it's I would still say like if you can find it elsewhere if you can point to some other layer that's well maintained or whatever that has that that thing then Be modular and do it that way unless you really really have to have it Um in your own layer. Yeah, I want to add as well regarding the meta tegra is more focused on the bsp But we have meta tegra community which like we are like kind of try to get aligned with the with the fox For example, we are we are like getting like triton server for inference. We have deep stream there We have vpi We have the different things like this is super related to the platform But we are trying to get like the best we can But separate let's meta tegra focus on the bsp But we can bring other stuff in meta updates. We are doing like our best to get the update working And also, you know a lot of nvidia's uh additional Enablement they're doing with containers these days. So That type of stuff is a good candidate to land first in community And maybe if later it becomes so robust and trusted then it moves into core, but you know, who knows? Yeah Okay, thanks We have to give tomas's uh exercise Well, first, uh, big. Thank you for your work on meta tegra. Uh, I wonder with the Change from cboot to the uef uef i bootloader if the flashing procedure also changed I think it was done through tegra flash helper script. Uh, is it very different now? It is not different because we are we are like based on the xml file flash then the executor like doesn't change Is that a matter of flash layout change? And then you replace cboot with uef i and then the whole link like together You we still like support the difference mechanism You are booting the the like kernel in it and then dtb from the boot partition Or the three of them lives in different like partitions This one works fine for secure boot uef i secure boot As I mentioned in the one of the slides encryption is not like implemented yet But we are supporting differently designing and then flashing mechanism doesn't change like it is not like big deal for you And so like the tegra flash script itself like we we wrote that right, but it's based on the Thousands of lines of bash that nvidia wrote right, but like we go we do go look at that To see if something did change and and things definitely have over the past um Unfortunately, I haven't actually worked very much on jetpack 5 so I can't speak to it from my own experience, but Uh, you should be able to count on those tools We're trying to keep them The same experience as pot as much as possible, right and then we will let you know what broke and if it broke It's almost always because something underneath us Change and we have to just we have we have to support what works Yeah, that that's the end of it, right? Yeah, it has to work so Hey, nice nice talk You say before you have a certificate you you create to the certificate which provide for the the nvidia Sorry the certificate for to the encryption you create to the the certificate so I think so it's provided for no No, no, everyone can can create his own like a certificate This one is like for the you mean for the ufi signing, right? Yeah, the pica and then key key and then the the database, right This guy is right. Yeah, you can generate them and then we have in the I don't think so we have it in the wiki, but we have some links to the nvidia Online documentation and then we we will show you how you for now. We are using open ssl But in in the future, maybe if anyone like move to production mod and we will maybe provide Something else to generate this this This certificate so from a product standpoint production standpoint, you should actually be using, you know HSM or tpm if you can you should be using a pki You know server of some kind or you know has your corp vault to be storing your secrets Locally and there's there's a whole lot of best practices that you should be doing all we can really Tell to people that that is generic enough for everyone to be able to use it as kind of the developer workflow But you should absolutely be using your own keys, right? Like don't use our demo keys. Don't use nvidia's keys use your Yeah Thank you. Thank you Any question no question no more questions. Thank you. Thank you very much