 Awesome. Welcome, everyone. I am Paul Figerberg. I work for Google, email and GitHub there, and today I'll be talking about how Zephyr accelerated a tool that we were building for testing external display support. So here's kind of an outline of what we'll be talking about, the problems that we were facing, what chameleon is to solve those problems, and how Zephyr played a very large role in helping the project be successful. And we'll also talk a little bit about open source. So our problem statement, we have hundreds of Chromebooks. We have thousands of monitors. We need to do external display compatibility testing, and this is a, it's an M by N problem, and both of those numbers are very, very large. We don't have the lab space for all of those monitors. Can you imagine the person whose job it is to go down the aisle of all these monitors and plug in the Chromebook, make sure it works, unplug, go to the next one, plug it in, unplug. Very labor intensive, subject to a lot of human error as well. And so to solve that problem, we created a device called chameleon. So what is chameleon? And the shortest answer is chameleon is a monitor. As far as the Chromebook knows, it is, it's just a monitor. It has HDMI, it has DisplayPort inputs, it supports multiple resolutions and frame rates, but unlike a monitor, we control exactly what chameleon will advertise to the source. And then it's a monitor that you can't actually look at. There's no panel on it. We don't display the video signal on the panel. We capture it to RAM instead so that we can do analysis on that and use that in our lab tests to test the interoperability and to identify graphics bugs and regressions in a scalable and reliable and repeatable way as well. And of course, this is the embedded open source summit, so chameleon is open source. So everybody always likes, I don't know, I like to see the hardware, maybe other people do, but here's the hardware, here's what the hardware looks like. It does, it operates as a standalone device, so ignore the PCIe connector on the bottom there, just pretend it's not there. You connect a DisplayPort and or HDMI from a Chromebook or other device. And there's this absolute unit of an FPGA that handles the video signals. There's also hard silicon ARM cores on there that are running Linux and interfacing with the lab network. And then down here, we have an embedded controller running Zephyr. So a high level block diagram view. We have a DUT, a device under test, and it either has a USB-C interface to HDMI or DisplayPort, one of those dongles. If the DUT has an HDMI or DisplayPort output, we can use that directly. There's a switching matrix to route the video signals. We have a couple different options for decoding. There's an HDMI decoder IC in there, or we can route things into the FPGA for decoding. There's ARM cores in the FPGA running Linux. We've got a daemon accepting RPC calls over the network, and then the embedded controller, or the EC as we call it, is running Zephyr. It's handling power supply sequencing. It handles the video stream routing. We monitor voltages and currents, and it also controls that HDMI decoder. So when I started on the project, the hardware designer had already chosen the embedded controller, and he chose an STM32, an ARM core, the F103 in particular, because it had a lot of IO pins that met the product requirements. So I have to work with that particular chip. Now, Chameleon's embedded controller has the same goals as an embedded controller in a Chromebook or in any other laptop. It's doing all those things that you need to do when the main processor isn't running or isn't running yet. But there's almost no common functionality between the laptop and Chameleon. We don't use USB power distribution. We don't have sleep modes for the main processor. There's no keyboard. There's no touchpad. There's no lid open switch. There's just really no significant intersection of functionality. So it's basically green field development. Now, in the Chrome OS group, we had recently started moving towards using Zephyr for the embedded controller in Chromebooks. So this was something that was already familiar to me. I'm like, hey, let's use Zephyr on the Chameleon as well. And Zephyr had support for the F103 already. So I grabbed a nuclear board off of Digikey. Am I allowed to pitch? It doesn't matter. Anyway, I bought one. I bought one and started experimenting. No worries about the RTOS, the drivers, none of that stuff. Just write your application code and move on. And that was really nice. It made board bring up very quick. So we had inter-processor communication. We needed to communicate between the embedded controller and the larger the application processor. And we've got to have these commands, these queries come down. We've got different requests coming in to switch something, to check on the status of a thing. And the only hardware connection we have is just a UR channel there. So any of you who have done any work on Chrome OS where you've had communication between the main processor and the embedded controller, you'd be familiar with the C commands.h. And it's also in the Linux kernel. So we keep these two files in sync. Being in the kernel brings in some additional restrictions that we have to work with. And so, you know, with all of this, we have messages, we have structures, we have to do byte packing and unpacking and all kinds of different things. I hope you got the ending in this right. All of the problems that you're familiar with in inter-processor communication, once we factor in all of the restrictions that we have because we're in the Linux kernel with these EC commands, it's actually a pretty good solution for what it does. But Chameleon is a new project. It doesn't have to share anything with the embedded controller. Do we need to do this the same way? The answer is no, no we don't. So we're thinking about this and we want to use RPC. And when I hear RPC, the first thing that comes to mind is big hefty system. There's a lot of resources, you've got network stack, you open up a socket, you're listening, all these things come in. It seems like a lot of effort for a little small embedded controller. And at this point, it's that F103, it's Cortex M3. And it seems like it's going to be pretty heavy weight to run that. So we're working with the Pigweed project at this point. And they've got libraries that we can bring in. So the Pigweed project, they describe their work as an open source collection of embedded targeted building blocks. And you basically you bring in only what you need. The libraries are mostly independent. Pigweed, for example, has a logging subsystem, but we use Zephyr's logging subsystem. So we just we don't don't bring that in. We we brought in the RPC library and then a transport layer. And it just it made it very easy. Now we did have to do some work to adapt it to Zephyr's build system with CMake. And we had to provide some kernel primitives, but that was a one time cost. And now it's benefiting anyone, anyone who wants to use Pigweed with Zephyr. And a big shout out to Yuval for the outstanding work that he did to make this happen. If you haven't seen Yuval's talk, it's a virtual talk, you should definitely definitely check that one out. So normally, again, I have preconceptions about RPC. The server is the beefier system, right? All the clients come in and ask one server. But in this case, the EC is actually the server because the Linux system makes the requests. So here we have a get video params call, it gets encoded into a protobuf. We HDLC encode that and send it over the UART. There's calls made into the HDMI driver to get the information about the stream. And then we assemble back the the details into a protobuf for response. And it goes back over the wire. So there's some references to our open source code for where we added the server and the the HDLC connection there. I'll have a bunch of links in here. And if you download the PDF from the conference website, you can follow all those links. So I chopped out a whole bunch of the definition, the RPC and the protobuf definition to make it fit on the slide and not be like this little dinky eye chart. But we have a service for this HDMI decoder. We have our get video params method. There's lots of different fields in the response. Some of it's very useful for debugging. There's things like getting the TMDS clock, getting the pixel clock and custom enumerations for color format, so on and so forth. Again, the entire definition is in our open source repository there. So we had to had to do a little bit of tinkering for fixed size buffers, and then add it into CMake. And it was it was really that simple after the groundwork that that you've all put into that. So how does it work out? You know, the RPC compiler generates a whole bunch of code. And but that's not in your code base. So the compile fails. So you copy stubs into your generated code in from the generated code into your header and your source files. You run it again. If you add new protos, it fails again, you copy things over. This is this is the method that I adopted. There's probably a more. How would we say it? There's a better method to do that. But this is the one that I adopted and it works great. So I was like, okay, and just copy paste copy paste the stubs and fill in the bodies. So we fill in the implementation there, you know, the generated stubs just say, hey, I'm unimplemented. Right down there at the bottom. And so we fill in the implementation say, okay, we're going to get all of the details, we're going to start copying things over and and then return an okay status just to say, okay, here's here's all your information. So and then starting up the RPC server is is also pretty simple, not a lot of code. Pig, we provide some great examples for that. There's some global variables or static to the module, depending on how you want to treat those. You register a service and you run a processing loop forever. And and you're done. That was easy. So I want to talk about supply chain shortages. You know, you remember, I said chameleon used an F 103 past tense used. So early 2021, there the ongoing COVID pandemic. There was a drought in Taiwan. There was an ice storm in Texas, where there was also a semiconductor plant. There was a fire at a renaissance plant in Japan. And there were some other events. And all of these impacted the IC supply chain. And all of a sudden, supplier said, you need to order one to two years ahead. And I'm thinking, well, I would have liked to know that one or two years ago, because I kind of need stuff now. The F 103 just became unavailable. There were no lead times quoted, you know, I would have been happy to have a one year lead time for that. But they were like, Nope, infinity. It's an allocation only. You can find some on the secondary market for $84 a piece for a little, little microcontroller. And that was just that was not acceptable. So ST was awesome. Shout out to the application engineers at ST. They helped us find a chip that was, as I say, almost pin compatible. There were there about four, four pins that we had to change. And it wasn't, it wasn't that bad. And my microphone is slipping. I'm sorry. Got my got my little microphone thing here. And it's fallen off my head. But yeah, so it was a similar cost. It was almost pin compatible replacement. And Zephyr has support for that chip. So this is awesome. Ish, it's awesome. Ish, you know, that's great if we're starting a new project, but we already have a very large body of code at this point. How much, how much are we going to have to throw out and redo? I mean, how many of you have had this where you change a processor and you're like, Oh, crap, Chuck that, Chuck that and rewrite this. And and it, you know, we're kind of looking at that and saying, This is bad. This is very bad. Well, it turns out the answer is almost nothing. It took about two hours to make all of the required changes. And here are the actual changes that we made. And then we have commits there for all of that. So we changed the processor in the config file and the device tree, just like F 103 to an H 7 43 or 753. We got a little sloppy with that and I'll explain in a minute. The clock configuration of PLL settings went to the device tree instead of the config file. So we had to move those over, get a little small speed increase to 96 megahertz, even at 72 megahertz, we weren't really having any issues with processing speed. And I also, I believe we had to upgrade to a newer version of Zephyr at the time to get the support for that. So that was also part of the device tree change there. Update the metadata, we went from 96 kilobytes of RAM, which is actually still pretty large for an embedded system in my experience. Anybody here remember 8051s with 128 bytes or 256 bytes of RAM? 96 seemed huge. And then we went to a megabyte. And it was like, wow. So we update the the CMake file. And it's all good. We're just about there. Except we have to change things in the device tree. Some of the pin names changed spy bus pins had the word master removed from the part of their name as part of an inclusivity effort. And the ADC mapping was just a little bit different. These pins were still ADC pins, but they went to different channels. So if you notice what I've highlighted here pin PC to change from ADC number one channel 12 to ADC three channel zero. And that was actually the biggest part of the effort was just remapping those ADC pins that took most of the two hours. The rest of it was was just simple as a cakewalk. We could get a Nucleo H743 board. So we got a few we did this this basic smoke test on our application. And it works with an asterisk. It's missing all the other chips on our custom hardware, of course, the ADC inputs, the GPIOs aren't connected correctly. But the basic application is running. And we can probe things with a scope. We can make sure the behavior is correct. And it shows us we can move forward, making changes to the hardware. So we rework one of our boards to use the H753. You could get 753 chips and 743 eval boards. This was the the the chaos that was the supply chain at the time. And those chips are almost the same. The 753 has a cryptography accelerator that we're not using in chameleon. So we could use them interchangeably. And that's why we got sloppy with that 743 753 and some of the files there. But as I mentioned, the pinouts almost identical. Few small differences. So lift some pins, do some jumper wires. And then we load our code and we ran our tests. And it worked. And there was much rejoicing. Because this was basically this was a project project level extinction level event for the project that, you know, if we couldn't get the microcontroller and we couldn't change projects dead. And so that was that was just awesome. There was there was a lot of celebration for that. So we built the next rev of the hardware with the new controller, we deployed it in our test lab and with early adopters inside the company for testing as we're continuing to develop features. And we've been running tests in our labs for several months now. And this is an internal test report showing the different tests that we're running. And the I truncated all the board names up there to just the ABC. But you can see some of them are passing great, and some of them are not. But that's that's finding gaps in functionality. That's finding some bugs. And it's been really, it's been really helpful. And it's been an awesome project to work on too. So and now I want to talk about open source, you know, if you can't always get everyone on the open source bandwagon, you know, it's great, but not everybody wants to. And so sometimes we need to include closed source in our project. And we had this with a driver for a particular chip. Now, at first, the chip vendor wanted to keep the code confidential. And of course, we're going to respect that agreement. So we have to have a way for that. Now there is a there is a side note here that recently the vendor decided to open source all of that code. So that is awesome. And and definitely props to them for deciding to do that. But at the time, this was very close source code. So, you know, we're going to bring this code into our open source repository. We're grateful to the vendor for deciding to open source the code. Maybe your vendor isn't willing to do that. So set up your build, there's there's your changes right there to just pull in the code from a second repository, a closed source repository that you have. And my build works great. My build, working for Google. Outside of the company. It's it's broken for everybody else. Did I put this? Yeah, the builds just it's basically broken for the open source community, because they don't have this code. And the code links in to this, you know, just several functions in there. And it's like, okay, now what? So to solve that problem. At the time, we made a new version of the RPC methods in the open source code. And every RPC method for the closed source driver just says unimplemented, instead of calling into the driver. So now everything links, you lose functionality, but you can actually build, which is you know, the goal. Well, that's part of the goal. We'd like you to have all the functionality, which is why again, it's awesome. They decided to open source that code. So we had to do a little extra work in the make file. We're globbing file names. So we wind up with both versions of the RPC methods. And we just basically check, Hey, is the closed source code there? Let's use that in the build. If it's not there, we'll use the unimplemented version of those RPC methods. And it's actually because we're globbing file names, we actually need to remove the one we don't want instead of adding the one we want. But now everyone can build even if they can't access all the functionality. And again, I want to emphasize the vendor decided to open source the code. So this story has the happiest ending of all. But if they didn't open source it, then we could remove that functionality, which is what chameleon did at the time. We could also link in a binary blob if that's a solution. So here's the code. There's the link and there's what's going on. We're running Arch Linux. We've got the source code for the demon. There's some utilities, the embedded controller code there, which runs Zephyr. There's an enclosure. There's a plastic enclosure for use in the lab. And then there's also system verilog sources for the FPGA for everything it's doing except the decoder blocks, the IP blocks licensed from Intel for that. We can't open source that. So I want to give recognition to all the people who made chameleon possible. Many of these people worked full time on chameleon. Others helped with specific portions of the work, or they gave technical guidance in critical areas. So thank you, Shuba, Eddie, Alexandru, Tom, Yuval, Yatzik, Konrad, Pavel, Machi, Piotr, other Piotr, Mielek, Igor Ladek, Ude, Alon, other Ladek, Matt. I could not have done this project without you. You are all awesome. And with that, we have time for questions. Yes. Okay. So for the benefit of the recording, I'll repeat the question. It's a question about the infrastructure on the Linux side for the RPC calls. Was that in user space accessing the UART? Or was that something fancier in the kernel? And the answer is we just did it from user space. Pigweed had libraries on both sides. So the embedded library that's built into our code, and then a client library on the client side that's just running in user space, accessing the UART, sending the HDLC data to the controller and managing all of that communication. So yes, just user space in this case. Other questions? Yeah, can you go back to the block diagram? Sure. I really should have that bookmark. Yes, that one. Yes. So you've got this display dongle right between the the the dot and the chameleon. Yes. Was there any consideration to incorporate power delivery and alternate modes directly into chameleon? So that was not in scope for chameleon because that's got its own set of standards to follow. We can use chameleon to validate a dongle. If you have chameleon as the known good sync for the HDMI or display port, and if you have a Chromebook or other device that you know is good, then you can basically run tests to characterize and say, okay, this dongle works correctly or it doesn't. There was a case, I won't name names, but there was a case of a dongle where it did not handle the hot plug detect correctly in in certain situations. And I mean, honestly, we beat our heads against the wall for a little bit like, why are we doing this wrong? And then we started looking into it more and bust out the oscilloscope. We're like, Oh, it's doing it wrong. So that was that was both frustrating and very, very nice to find that, you know, we were doing things correctly. And this thing was actually the problem. Yes, the question here. Okay, so the question is, is this an internal Google project, or is there a possibility of it becoming commercially available? Currently, it is an internal product. We are working with our development partners on providing chameleons to them. I do not know if it will be available commercially like you can buy it on a website somewhere. But we have worked in the past aside from our development partners with members of the open source community who are particularly active in certain areas to. So this is actually, I should probably say, this is chameleon v3. There was a v2, like eight years ago, that's been going on, but display standards have, you know, moved on since then. And so there are, there's, there's older chameleon hardware out there. And we have provided that to some members of the open source community who are particularly active in those areas. So the question is, does pigweed require C++? Yes, it does. Pigweed is very much a C++. I believe C plus plus 17 library. Fortunately, that did not pose integration challenges for our project. I mean, the chameleon code was all C because I'm kind of old school, I guess. It was just, it was simple. It was easy. And but integrating the, integrating the pigweed and the C++ in there was very, very little effort after Yuval's initial work to just get all the CMake stuff set up. So yes, C++ for pigweed. And I think there was another question closer in the front. Yes. This one. So amongst the issues that we've seen, repeat that question. Oh, yes, repeat the questions. I'm so sorry. Thank you, Keith. So the question is about this test matrix. It shows test failures. And can I share some of the regressions that we've caught with chameleon so far? The answer is in great detail, no, but issues with particular resolutions. We've found some of them where a particular device was not happy going into a certain resolution. 4k60 is, seems to be particularly fussy, because it's the highest bit rate. And so, you know, you need good PCB rounding, you need a good quality dongle, those sorts of things. We have also found some issues with mirroring to an external display and changing the resolution to match that. When you've got a Chromebook with a certain resolution panel, and then you have a higher resolution display and you tell it to mirror, sometimes depending on the state of development, you can get issues where the Chromebook says, yeah, I can't go that high, I give up. And then the resolution gets all wacky. So, yeah, various issues with switching resolutions. Any other questions? All right, I'm going to take that as a no. Thank you, everyone.