 Hello everybody, welcome to accelerated mainline Linux development ahead of SOC availability. That is a mouthful My name is Brian Before again, I really wanted to thank all the open source teams that we have at TI They they really let me do the really cool stuff all day every day including giving this talk Wrong direction For those who haven't met me. My name is Brian Bratloff. I am a part of Texas Instruments Baseport team. I primarily focus a lot on the initial wake-up of our SOCs So like bootloaders Linux, that's kind of my bread and butter day in and day out Pranithe he is also a part of the Linux Baseport team here at TI He's a big Android guy. He maintains our product trees. He's also my manager So he makes sure as I stay organized and focused on what I'm supposed to do He will be the the first person to tell you I chase butterflies constantly Unfortunately, he couldn't be here today. So I'm going to try my absolute hardest not to let this fall off the rails So, yeah Next slide. Yeah, this is please don't use this to get me in trouble. I don't like being in trouble Okay, here we go So if you can remember back into your student days or maybe you're just starting to get into hacking You know you teaching yourself a project would look kind of like this, right? You're given some hardware. You find some hardware. It's really cool. It has absolutely no Linux support on it So great. You get to write your Linux drivers and your software all by yourself you know, maybe there's documentation and Hopefully it'll do something cool at the end You could be like me You could find anyone and something else to write your drivers for you So you don't have to But eventually you'll you'll find someone you'll you'll get some assignment back. You'll you'll start emailing Greg Get stuff merged and you're like, hey, this is kind of cool. This is kind of fun Let's let's see if I can make this into a career. So you start applying You know, you you find you hear back from an SOC manufacturer and they they're really excited to hear about you And want you to come on board and then you quickly realize that that is not the model of how we develop SOCs So the the development flow for an SOC kind of looks like this There's a lot more stages than what we were when we were amateurs, right? The the main difference here is when we were amateurs or students we only got access to the SOC after it was designed This this means we only have a fraction of what we could do when we when we're inside the company, right? So we get to do things like influence the design process but as a as a student you The capabilities the hardware capabilities dictated what you did in software But when we're working with the manufacturer we get access like I know everyone's name in this On all six stages of this so we can all we can use that to our advantage to help it Linux evolve So I know this is kind of a new concept for for some people working outside So I kind of wanted to go walk through the design process and kind of show people how We can design an SOC that comes with Linux support built in from the ground up So the design process is pretty cyclical, right? So the lessons learned from a previous SOC will go into the next generation, right? It's also a very creative process. So even though I lay it out in like a nice circle like this It is you know pieces in From one stage or going into the other stage and lessons learned from here are going there so it's it's a little bit more lively than a simple circle but This so you know, okay stage one. This is the this is where we generate the idea, right? so Well, we'll have support teams, but oops. Sorry support teams business leads They're working with the community constantly. They're working with our customers. They're combing through mailing lists They're they're they're going through forums and they're trying to understand what people are having problems with or what do they want? How and they're trying to find an opportunity that for that for us To to make a new chip So things like did customers have any pain points with you know some component? What could we improve on you know? If customers are having trouble with this component in in our SOC, they're not going to use it So how can we make this easier? The things that you'll typically hear in this stage of a chip's life is like ease of use How can I make this easier for for our customers? So we also have system architects in this in this stage. I call them architects. I don't know if that's their actual name, but They're they're veteran embedded people so they they know the embedded ecosystem inside and out So they're they're typically involved in this stage by checking out new IP, right? So They're making they're working with IP vendors They're they're trying to find new components that we can add into our SOC's and So they'll usually have like demo boards and FPGAs out and they're trying to understand how Linux will behave if they if we integrate This component into our chip The critical part of this design process for me is that step, right? So it speaks to the maintainability of our IPs The level of interest that these IP vendors have in supporting this component Especially if it's like an upstream friendly company They're they're really actively engaged that tells us a lot about how Dedicated the vendor is at supporting this which ultimately goes to the longevity of that component The last thing you would want is, you know Some some component requires some cryptic firmware that you know may get support from two or three more years Only for them to renege and now we've got dead IP in our SOC So this is a critical step in in my belief because it really gives the The the longevity of these chips meaning So once we've kind of got a basic idea of what we're going to do, right? Well, we have IP blocks selected, boot architecture, low power modes, key performance indicators, a Market segment that we're going after We'll start bringing in the larger people More people into into the project, right? So we're thinking about details at this point. How will the how when we integrate all these IPs will it function correctly? This is another critical come part of the design phase really because it means we have our domain experts here They have a working relationship with the subsystem maintainer, right? They know that the subsystem inside and out And so they know what to look for in this design phase when we're looking at all the details of this SOC So this is where we flesh out How these IP blocks are in integrated to right? So we're looking at the clock tree. We're looking at power domains You know estimating resources that we're going to need for this IP block interrupt lines What what are we going to need to make this IP work with the Linux subsystem? The goal here is to have the team trying to catch errors, right? Most of the most of the hardware team, you know, they they work in hardware They don't know the Linux subsystem. So it's simple things like tying the Cortex application cores To our C7 X or our digital signal processors That's a great idea because it saves on a PLL, but you know, that's not great if you want C states So, you know, this is one of the kind of things that we're catching here And so once we have a solid plan, you know, we flesh out all the details We think that like we can execute on this we'll move on to my absolute favorite part Which is actually developing and validating the chip Here our hardware design teams get the okay to start writing all the verlog, you know integrating all the all the components They'll validate each component. I call that kind of unit tests And eventually we'll have enough to start a software development And then we can start validating our software on this component. So how do we do this? So for new IP blocks or our changes in how we're integrating the IP that require a lot of Refactoring to a driver, you know, we we switch an idea around and so now the assumptions for our driver are completely broken So we need to iterate on some code. We can use the lab by for this I it's it's a company ASTC the Australian semi-conductor technology company I really I think of them as QEMU for SOCs, right? It's a software simulation and and because it's a software simulation. We can run this at our desks, which is really really cool It allows us to quickly iterate on on a boot architecture our design So, yeah, VLab is perfect for a boot flows that come from like a chassis change or you know Anything like a big IP change and because it's a simulation We can produce and iterate on our software very very quickly This also means that we can give feedback to our hardware teams. Hey, this isn't working out Can we do this or hey? This is this is really working Let's kind of lean into this and that allows us to quickly change our code again as these updates come Come back from the hardware team. So we're constantly updating this model as we're Developing and so the software and the hardware team are working together at this phase to develop everything that we need to and so So there are also times where we want to be cycle accurate, right VLab is great for simulation, but it's a simulation so to to get some cycle accuracy we use FPGAs and Unfortunately, these aren't something we can put on our desks. I mean these are absolutely massive I wish I could convey the size These these this equipment has billions of gates They're capable of modeling the entire SOC and in because they're the size of the rooms, I mean they're a shared resource and But it allows us to do cool things like We can learn about and understand the performance numbers So the key key performance indicator KPI We can expect from from Lennox so So I work a lot with DDR so things like DDR access latency is very important to me So like how long does it take a transaction to come from the Cortex-a cores to DDR and back? That that is hugely important to me and it's something that we can model in our FPGAs All of it can be calculated, but it's at that point. It's still an estimation Being able to run cyclic test on an RT kernel That is what's gonna give us good performance numbers and we can take that and show that and like hey This is this is what we can expect This is also a great time to find bugs, right? So we're great integrating new IP We have you know, we're exercising drivers in ways that we didn't really think where we're going to and so This is a great time for us to find like your small bugs in our existing IP So fortunately at this point a lot of our existing drivers are already upstream. This means we can send patches out immediately and this greatly reduces our workload that we have once we get into the later phases of waking up a chip So we do start seeing a few bugs and like okay Well, let's send a patch and so that's that's a great way for us to offload a lot of the burden There are limitations though, right? So I've talked about V lab, which is a simulation in FPGAs and so There there are things that have analog components in the SOC think fies So they Unfortunately, they can't be modeled with it or there's more things that can't be modeled within FPGA So those components will have to wait But the entire point of this that that validation phase is To make sure that the software team has confidence in the chip and the software that we have and you know, you know It'll behave in the expected ways or in in ways that satisfy everything that we laid out in the initial idea phase This means that the the the later stages of the Validate stage really will have Linux booting and running For a while nine as we exercise each component There are some variations to how we do this I should probably add that for for simple things like Adding a DSP to an AM62, right? We we want a digital signal processor. It might need a new DDR controller The the cycle is much quicker because it's not as Complicated at least for us. It's not as complicated But there are certain situations you have a complete chassis change a new boot architecture a new boot ROM Where this you know, all these dates will kind of shift around accordingly However, once we're happy with the hardware design, we're confident in the design We're confident that we can support it that we the Linux team can support it We can start sending out the the hardware design out to the manufacturer. So we can start with tape out However We're not done here, right? So We've made the SOC But now we need to start working with the board design teams, right? We use these boards to showcase what you can do. They're they're a template, right? How do you lay out DDR? What kind of PMIC should you use all these? Supporting components are also very critical to a working Linux system So this is where we begin working with our analog counterparts, right? You know there they have PMICs they they need to upstream they've got drivers that they have to do This is also where they can come and ask us and we can go and to work with them to make sure that That the PMIC will work in a Linux friendly way I mean these these PMICs are basically tailor-made to work with our chips Why would you make it hard to support Linux at this point? So we're also we're also working with our embedded engineers our electrical engineers our analog components teams to to really support this chip so that when when you go to make your own Solution you can you can rest assured that these components will have great support though and they'll be great at supporting Linux My computer locked up that's unfortunate Okay, so well you don't get to watch the the red letters move around okay, so Wrap up So once we get the SOC's back We usually get some early samples blind samples that don't have any trimming, right? We can marry those with the boards and that is when wake up officially starts We have a somewhat functioning SOC So we can start testing every single component and this is where the bulk of Ti patches that you see come from right? We're checking out each component It works a lot of this existing IP is already there But there's going to be new components a new DDR controller a new way a new remote core all that kind of stuff But the majority of this has already been taken care of a lot of the previous stages that we've done We we already have Linux running. We know what we're doing and so it's it's basically documentation and device trees at this point And then once we're once we get the majority of our basically all of it Upstreamed it now becomes an easy thing. It's it's support right so people are going to start integrating this They're going to start using A chip in a way that we didn't intend it to so they're going to be more bugs, right? Your subsystems are going to change. There's always going to be support and so At that point we're just maintaining and so as long as we keep this working in upstream. We I think we we say 10 years This is going to be an easy process because everything we're working in the community And so we get the maintenance burden for free here But we can work with our customers as well. Like, you know, we're they're going to find bugs Think a lot of workarounds We can work with them to work on that and to fix that and that feeds right back into our idea, right? Why did our customer have to find this a lot of themselves? Why why did you know all this kind of stuff all feeds back into the design and so the I Guess the number one thing is even though we say we're software developers the The support system is through and through to make a functioning SOC that supports Linux. It's not just software Yeah, and I guess that's kind of everything I had I Really wish I could show you this ending slide But Pranitha and I are pretty active on email. We were also on IRC so Linux Linux TI on Libre Chat. We're always online. We're pretty active So if you have any questions come come ask and thank you guys Yeah, yeah, yeah, let's do that What is most interesting in this process is how do I make sure that your software simulation indeed simulate your software with chemo you need to write Separate code what how do you talk with this wheel up? Yeah, so they they use system C And so that is another concern, you know that there could be a bug in the model and so that's why the Validation step for us is is kind of important Well, you will usually find these bugs But then tracking it down. Is it is it an error in The V lab model or is it an error in our hardware or is it an error in our software? That is that is kind of a harder thing to do Wake up support. Yeah. Oh Yeah, I forgot about that. Thank you Texas Instruments and the Linux Foundation I wouldn't be able to do this without them. So thank you guys and then yeah Here here's all the stuff I What point in the process do you start submitting patches for drivers for those new controllers or peripherals? so one of the the big rules that we have just internally is we want to have at least a rough draft of the Technical reference manual it really gives our maintainers something to look at if they have any questions And so any of the new IP or new controllers that that'll be on the wake-up step after we've gone through taping out But a lot of the existing IP it's like I can think of like an RTC like we've integrated an RTC in the AM 65 It had an Arata workaround and then in in simulation it didn't and so I felt very confident like okay I can send this and fix that real quick So maybe you can also elaborate a little bit also For others on the market how you got to this upstream first process I guess it was a process coming from the classical idea of BSP first So what did you do to convince management about that or did you get feedback on that and things like this? So how could this be a role model for others as well? So I absolutely lucked out when I joined TI when I did because Nishant did most of the heavy lifting sitting behind you for this So we made mistakes a lot of them That was the hard way to learn right and this is kind of a this is why we are making this presentation Because we know that there's a lot of people creating their associates great. Good job We also know that there are Soc vendors who are made the same mistakes that we did years back I come from OMAP and if you remember OMAP 3 Okay, let's not go there, right so So the big thing that we figured out was value for customers Once we started seeing the pain from customers in terms of long long-term sustenance We collected every single feedback that we had Listed them all out and we fed that back into our management chain None of our management chain wants to spend one cent Unless it actually helps business Right the business value is straight if you have a chip that can only be supported for the next two years You don't have a business So what's the best way to do ten years support upstream? It was straight straightforward answer at that point In fact, we got support to work with the IP vendors or some vendors everybody that we work with With the same mentality because we want these products to sustain and stay updated over the years Hi, so what emulator are you using for TI? This is this is one question and the second one it will be like how long does it take to validate the on the emulator like I Believe this is the longest part because this is where you validate the the hardware before take out Okay, so first question What validate or what emulators are we using I will say that there are two companies and if you choose one I will say yes And then for simulation or emulation and how long it takes that can really depend right So like some new chips that I'm really excited about that I cannot talk about Are taking a little bit longer, right? There's just the The number of things that have changed Are are are going to take some time to look at But smaller ones, you know, they'll definitely Like like the am6 to a that I was giving an example for basically we just bolted on a DSP Our C7 X and a bigger DDR controller that that took a substantial a smaller amount six months. Maybe yeah So if you're looking from a boot time perspective, for example, a Linux booted complete full-fledged Rom boot kind of scenario Just to give you kind of a roundabout thing One of the fastest FPGA among the two emulates around an hour 462 for around a minute of real-world time To get to your shell for example, and this is stripped on corner. Yeah, it You find other things to do once you hit the run button What's a turnaround time at the relapse stage? So you say you'll find something some bug in Or something you want to have changed to work better with Linux and how long does it take for a change in a hardware to feed back into? So it's actually really fast So our hard hardware teams have a lot of scripting as well So if we find a bug in the hardware, it's usually within a day or two that they can get something fixed It depends on what the the fix is VLab so basically we'll get updates every day. We'll integrate this new model. We'll download it and then Yeah, and so VLab is as much faster than then emulation, but it's you know 1.5 times is like as slow if that makes sense. I should probably use division. So it's like 33% or whatever that is Thanks. Yeah, right. Thank you all. Yeah, thanks guys