 Check check, okay, well, let me get my myself tucked in here Oh, okay, great Okay Okay, great. Thank you Okay, that's your I gotta know what time it is in Tokyo. Yeah, you need to know Okay Hello everyone, welcome to this session of embedded Linux conference And I'm happy to be here with my co-presenter Marta Ribczynska Ribczynska, yep, yep, I think it's better and better every time we practice I get a little bit better at it Anyway, so I want to talk about the nature of this talk is just We periodically take a look at what's going on in the kernel and in the embedded industry And this is not comprehensive. There's no way we could cover everything that's going on And so this is just a few things that Marta and I saw And There are a few extra slides at the end of this deck. I actually prepare a longer version of this Talk that I give at other events So there's an hour and a half version of this talk But we're not going to do that today We've distilled it down to the important points But the slide deck has extra slides if you go past the end and they're already uploaded So just a quick overview what we're going to be doing We'll talk about the Linux kernel technology and then some technology areas a little bit of industry news About the community and some conclusions. So let's start with the Linux kernel I think John Corbett famously said the looking at the Linux kernels schedule is a little bit like looking at a tide chart It's not very exciting because it's so consistent So you can see that it's either nine or ten weeks between kernel versions So I mean you can pretty much set your clock by it We've had about five in the last year. That's pretty normal very very normal We've been at this kind of cadence for many years and so that's good. It's Companies know what to expect So I'm just gonna go through a couple of things. Oh, no, I'm gonna let Marta go through a couple of things with these kernel versions So We have distilled the embedded information important for the embedded developers from different kind of versions starting from 6.4 So in 6.4 there was the slope allocated that got removed new documentation for building a kernel The model license declaration removed for anything that you cannot build as a module and user trace events API that Was merged in that version then six five Has seen Work on parallelization of the boot CPUs if you have a multi CPU architecture there is important to Actually boot all of them and their optimization to To reduce that boot time of everyone Every single one of them x86 only for now Quite likely others will follow a big organization of 42 bit device tree files to more to match other Like armist 64 MIPS and and so on And a trace and path updates Of course all links that we are seeing in the slides They are clickable in the PDF version of this light. So you can download and read everything up 6.6 is It's possible with 6.6 to build a kernel without buffer head support. It has been in the works to replace it with different Different mechanisms and yeah, there's this work. This was this work is happening We have new event FS subsystem in this version to for improved memory efficiency and tracing and we have shadow stacks on Intel platforms only for now and let's get a little bit into this You have probably all heard about attacks true Overwriting the stack pointer and then going somewhere nasty so they are hardware mechanics that allow to prevent that and They have they have been long discussions in the kernel about how to handle that has Actually landed in 6.6 There are patches for arm 64 and work on muscle Also has been started then six seven You can enable disable felt it to a bit emulation on x86 on the command line also to reduce attack surface one architecture remove ea64 More efficient kernel merging of same page and And Part of the real-time work Print case supporting per personal locking six eight deadline servers the first rust driver Refactoring of network code data structures in some use cases up to 40 percent improvement Yet another memory allocator removed the slab one so the slope is the only one that Is remaining Probably all reducing confusion also because all those abbreviations with one letter difference that was complicated for years and There's a new path that are a type profiling and Deadline servers Okay So one of the things that was added in 6.8 is known as deadline servers so this is actually a replacement for real-time throttling and So the what you want to do when you're running in real-time is you want to avoid the situation where The real-time tasks are starving the regular scheduled tasks Because when that happens you can't like break in with SSH or as an admin and fix what's going on So we've always had something called the real-time throttling That was in the kernel, but it required a fixed amount of overhead and We don't like to kind of waste CPU cycles and so they now have a new mechanism. So there's there's a new thread It turns out that there's another scheduling category the deadline scheduling category And it's actually got a higher priority than real-time. So they've got a way this works as a deadline server set up to be started And it's running all the time, but it only activates when it detects That Possible starvation so it's actually not consuming any any cycles And then even when it is active it only takes about 5% of the quanta the CPU quanta So it's actually more efficient. So that's just something shows you that Real-time stuff is continue to be work on The other thing that I thought was really cool in this release was the perf data type profiling So this is a mechanism where you can annotate the perf data On on a perf field basis. So you can actually see these charts saying how much time was spent accessing a particular field in a data structure and this is really really handy to go out and look at Whether or not your data structures are well aligned for cache boundaries, right? So you want the most use the most frequently accessed fields to be all within the same cache line and This is a tool that helps you do that and I don't think I've ever seen this level of granularity in the tracing system I think people have already said that they found some things Where they're able to optimize the data structures in the kernel So this is this is I think a really nice new thing In terms of Linux 6.9, it's not out yet, of course, but we've had the merge window so there's a couple of things the ext2 file system is now deprecated and So that doesn't mean that if you have any ext2 file system on disk you're out of luck Because the ext2 4 can handle the ext2 in backwards compatibility But it does mean that you may be out of think about moving forward I don't think many of us are still using ext2 file systems But you never know if you have like an old design just just be paying attention to this issue there's been some Timer subsystem rework and this kind of blows my mind because the timer subsystem has been around For forever basically and they still find ways to make it more efficient and You think we wouldn't find fundamental Areas of improvement like this, but but we do and this had to do with Matt how Timer expiration expired timers were handled And let's see there was work on reducing network stack contention the kernel energy model can be updated at runtime and then You had there was a vulnerability on Intel Register file sampling and there's a mitigation for that so we continue to see security work in the kernel I want to talk just a little bit about LTS, right? So this is a big issue for us in the embedded space So LTS kernels have been historically supported for six years or they were for a while supported for six years but the new plan is to Only pick one per year and maintain it for two years So the old the LTS support is going to kind of run out and So there are some places in the industry like CIP or Ubuntu LTS where you might be able to pick up some of the patches But upstream is not going to be doing this. There's also some commercial efforts actually around doing LTS support or You could consider switching kernels during your device's lifetime, but we all know we never do that so Let's see. This is just a quick chart showing contributions by embedded Linux companies and You shouldn't read too much into this because companies end up going off and doing different things Consulco has kind of a lot a low number But they're doing a whole bunch of stuff for automotive that's out of tree not I mean Like in user space and other places and the same with other companies But you could see kind of the top contributor for a lot of open source a lot of these Consulting companies and what they've been working on and it's kind of the same stuff you expect, you know, I to see Media tech processor support a lot of driver cleanups We client conic Pangu Tron is doing you it turns out that if you do a reaffactor you can get a pretty high commit count out of that So so that's just kind of interesting stuff. So now we'll talk about some technology areas We've noticed any an interesting talk from T.I. during LPC 20 23 so already a little bit Sometime ago about Looking into how we want to improve the boot time of Linux systems on embedded on on automotive Systems in an automatic in automotive you do not want to wait seconds long seconds until your system is ready so they were thinking about Using the bootloader to do device initialization Then it requires to pass the whole initialization to the Linux itself And Also working on Pre-initializing the market drivers. We know that synchronization of Ethernet that takes a little bit of time So if we can do it earlier, then we have it ready when a Linux needs that you can see the slides and the videos for this talk Then on the graphic sides, we can see on open source Vulkan driver fan video that Has been announced by Collabora and They have an article about this that you can have a look Then on the file system and IO This work on empty the SPI nor so Locking work to support reads While writing that could be interesting for embedded use cases area FS and and husband on the cache caching a speed and the block IO speed up work that Is the problem we haven't checked it had if it has been merged Then on real-time preemptive RT that is not the year when we announced that the whole preemptive T has been merged Not yet But what is left is? Is not that much and it's applying cleanly The big blocking point is the print K Apart from that there's a little bit of TTY serial DRM and Few things here and there are related to architecture But they're really the really the elephant in the room is is the print K on the security side So we have talked about the stacks It turns out that There was information About the kernel addresses from the elf nodes that was exposed in the system there was a work how to to remove that information because it could be used during attacks and BPF meets security in In many places there is a talk yet again about unprivileged Users being able to run BPF modules Well, we are going to see how how it goes Then also on the security side CV handling the big news at the beginning of this year The Linux kernel becoming a CNA. What is CNA? CNA stands for CV numbering authority If you are CNA you can create or assign the CV numbers So the identifiers for security issues The process itself Has seen a lot of discussions about the problems it has The big the big thing that the kernel community has seen that sometimes the project members disagree with the researcher or the project members disagree with them between them about if it's a really Security issue what is the impact and so on and so on and if you are a CNA you control what is assigned to your project so the kernel community has decided to make the pass and become a CNA to be able to design to assign vulnerabilities to the kernel before it was done by Well, basically everyone So they have decided to have a policy that is a little bit different that different that all the CNA policies So they are the assigning CVs only to fix the issues. They have a plan to invalidate Entries that were assigned by someone else. That is a pretty complicated thing to do talking from experience And the kernel community is not the only one that became a CNA in the recent months there's also gdpc there's curl python and The list is growing Kind of every week If your project is not the CNA the paperwork is not complicated. It's the cost is zero in money So you may consider doing that to to control what is assigned to your project Now we will pass to the System size Okay, so now I go on a little bit of a rant because The slob memory allocator was removed like okay fair enough. There's probably not that many people using it my main my main problem with that is that It was it was between the time it was deprecated in 6-2 and the time it was removed 6-4 It was only like two kernel releases and there's this like no way I'm sorry, but there's no way that the embedded people were going to notice that You know we should we're supposed to be working upstream But a lot of it, you know, it's not going to actually impact our our products for usually were a year behind or two years behind And so I just didn't think that was enough time. I would have liked to have seen a longer deprecation period And then the slab memory allocator also got removed in the same year So only slob remains. This is not the end of the world I think most people were using slob anyway and there is a config slob tiny option It's just not going to be as small though as slob was and so I I am not doing MMU list stuff But I worry that some of the people who are who are trying to get Linux to run on the deep deep embedded You know lost Facility here that was beneficial to them You know, I understand kernel maintainers don't want to maintain multiple memory allocators And so I understand the direction, but I thought it was unfortunate Really interesting in thing in terms of testing that showed up. This has not been mainline yet, but Kolobora Did this thing where they are submitting the CI pipelines upstream to put into the kernel source tree And Lena's initially kind of rejected this Didn't want it at top of tree But there was some discussion it looks like it could go in not at the at the top of tree in a in a directory Or he doesn't really have to pay attention to it. This is interesting because This is just one CI system that people use GitLab CI But the way they structured it was they actually had directory they put it in its own directory And you could envision other CI systems also putting materials parallel to this upstream So if you're using GitLab when this goes upstream or if it if it manages to get upstream You'll be able to just run CI right out of the box on the kernel source tree without having to add additional stuff And I think this is actually a really good thing I think other other CI systems may add their materials We may find ways to share CI materials And so I mean we have case self-test, but we don't have like kind of framework integration That's upstream and there's no reason not to be sharing that in terms of tool chains GCC was released 13.2, which is the current one was released in July. I didn't check in the last week 14.0 is supposed to come out This month sometime. I may have already come out, but if not, it should hasn't come out yet. Okay It's supposed to have more improvements for the static analyzer some actually really cool stuff Where it'll find problems before you even run the code so You can look at GCC actually as a test program It finds more bugs than probably any other test program out there, but luckily it finds them before they make it into your code and then the other tool chain that's used a lot is LLVM and there's kind of a They're improving their support for architectures risk 5 new ARM processors and Interestingly LLVM is not just used on the kernel. It's increasingly people are using it to Do whole distributions There are some resources I'll just I'm not going to stay on this one, but Bernard Did a great talk at ELC last year talking about a bunch of alternative tool Toolchain elements highly recommend that talk and then also You may not know this, but bootlin has this area where you can download Toolchains Pre-built tool chains that they've gotten from build route. So they're just all available out there This saved my life. The reason I'm mentioning it because I had a board that I It's a long story, but I needed an old compiler and These guys had a pre-built one that was lying around so Anyway, so and they're they're available in multiple flavors, you know for different C libraries and stuff And then on the tracing front I talked about perf data type tracing We did and then eBPF Increasingly is used for tracing and other things Including on itself. So there's now a tool Called BPF top which does pretty much what you would think it can monitor BPF programs and tell you how much they're using the system And they're overhead and stuff like that. So So that's a nice bit of introspection there and then I'll turn the time over to Marta for more stuff On the other project the 50 is coming any day or any hour right now Officially expected in April that will be the new LTS and what it has That is the support of the generic arm 64 for system ready setups So the first first time you can build a generic arm image and boot it on different platforms The default corner is 6.6 and there are hundreds of Package updates as you would expect the previous latest for Yachter is the 4.3 non-built Rest in November this one we've kind of six point five on the industry side Oh, we have chosen three events The first even the the C array that's the first time I'm talking at this conference about C array I've has been I have been stopping myself from talking about C array What is the C array for the US audience you may be less aware? C array is a mandatory cyber security framework for all software and hardware in they put on the market in the European Union it has Many requirements some examples of them being that Software updates will become mandatory by default five years for everything Every window will be obliged to do due diligence of all components that they use including all open source components There are requirements on documentation. It will be mostly based on self-assessments, but it's a little bit more complicated in that It it's extending the C marking it is Nearly done The final publication is expected in October 24 Then it will be three years for Up to full application. So we are arriving at 27 But three years will not be that long in this case, especially that Standards need to be created And the C array is referencing a lot of standards and they are not written yet So during the first part of those three years, they will be writing and approving of all those standards and Only then we'll know What the exact requirements actually are? Then HDMI when we are back we understand that's in some places We are lacking open standards in the case of HDMI the lack of open standards Causes the lack of a possibility to write an open source driver for HDMI 2.1 It's a little bit of impact for embedded development Yep Some people still haven't understood the open standards and the nice most helicopter probably you have heard it had an Accident the blades have been damaged and it cannot cannot the nice helicopter cannot fly anymore NASA announced end of the mission Unfortunately, but probably we'll see Many other projects and we are releasing many other projects with Linux in space happening back on the community to theme Okay, so I'm just going to talk really quickly about conferences and about the elinux wiki So conferences embedded Linux conference, okay, so our grand plan to do it once a year it only lasted one year and There's a lot of reasons for that. I'm not going to get into all the details You can ask me in the hallway and I'll explain some things but One can view this as a positive thing So we're in the spring in North America in the fall in Europe. That's the way we had been for quite a while We have this one going on right now. The next one will have a mini ELC in Vienna So it's only going to be two days and two tracks a little bit smaller in scope But the CFP is open for that now and the deadline is the end of this month. So So please consider Submitting some entries for that Okay, so why the change away from EOS is a standalone event Again, I don't want to get into a lot of details, but it had to do with some some aspects of sponsorship and some aspects of attendees we did a survey a while a couple years ago and found that about 50% of our end users like to be Co-located and about 50% of our users prefer a standalone and so it's like It's hard to please both camps Hopefully This this year. I was actually very pleased with the technical content. Sometimes open-source summit I don't know if I'm even allowed to say this They don't have like the technical content, but they have had this time. There's been several really good talks and Linux con and so Sometimes it's good to be co-located. You get some extra benefits from that This is in red. I believe yeah So I'm I'm planning an embedded Linux micro-conference of plumbers, which is also in Vienna That's one of the reasons we were not going to do ELC in Vienna as we're going to kind of replace it but plumbers and ELC are not the exact same thing right plumbers is kind of more of a deep dive on a Individual topic. The reason this is in red is because it's been submitted as a proposal But has not been approved yet. So don't don't count on that happening But I'm hoping to organize that I My personal opinion is that the plumbers content and embedded has been a little on the weak side and I'd like to beef that up Get more involvement with the the kernel engineers on on deep technical topics The other thing is the elinx wiki, okay, so we were concerned about the elinx wiki because we lost our funding for Our administrator bill trainer turns out bill was hired as a full-time employee of Linux foundation so and This is not the only thing he does, but he is paid to continue working on the elinx wiki As part of his job. So this site it could use a facelift There are some areas where the content is kind of stale But it does have we continue to maintain all of the slides and all of the videos For every instance of ELC and so there's a lot of great material out there Also, a lot of board vendors use this site to publish, you know technical specs and stuff. So I Think this site is underutilized. I'd like to see more people using it I am looking to get some volunteers to maybe help walk through the site and try to clean up and Update portions of the site. We actually do have some money the last gasp of the of the project that was funding this Took their remaining funds and they put it into a crowdfunding site So there is some money that we can use to add content to the site or do some other things And then I think I'm gonna turn the conclusions over to Marta. Yeah The conclusions is overall If it's not the year of the Linux desktop, it's definitely yet another year of embedded Linux We have embedded Linux deployed in billions of devices We have the core kernel supporting Very well the embedded use cases But of course we are getting new devices. So new drivers new things to upstream to modify to optimize There are investment needed in the infrastructures infrastructure and community not only talking about XZ but All all the related subjects and we are Looking for your ideas what we are missing for the support of embedded Linux and what Yeah, and you can you can Talk to us During the hallways and I think that we managed for the first time to have first time in the history of this talk to have Time for questions. It is So if you have questions, we have a mic People are not used to having questions to this session Okay, if there are no questions, I'm gonna I'm gonna talk a little bit about the Mars helicopter Because I think it's really cool. I just found out this week so the Mars helicopter is is Grounded it cannot fly because of the damage to its blades But well, I got to be okay You guys are getting a freebie because this is one of my questions later this week in the closing game So But the base station there's a base station on the rover That is also running Linux that continues to run and the helicopter itself also continues to boot up every day and it and They're going to over the over as long as it continues to do so they're going to use it to get temperature data and They're going to take pictures Because the camera is still working so it continues to be an active science instrument on Mars running Linux and they say that it has enough capacity in the storage to Handle 20 years worth of saving data So it will soon the rover is soon going to leave that area and it will not be in radio communication It can't send that data back to earth, but hypothetically When some human lands on Mars or some or the rover comes back around for a pass and gets near enough for radio contact They can receive all that data. So it's still doing science on Mars. So we're excited that we still have Linux running on Mars That's that's all I got to say about that. I find that so exciting that I just can't stop talking about it Any other questions about any other topics? Yeah, that's dangerous. Oh, yeah. Oh, I think it's on now try it We get the mic on Should Okay, just repeat I'll repeat the question or Okay, so the announcement was that I think we have a new mic back there, too Yeah, that's weird So the announcement was that kernel recipes will still be in Europe every year and they're Embedded recipes. Yeah, embedded well, so kernel recipes to embedded recipes will be in Europe every year, but they're adjusting their dates to not conflict with Embedded Linux conference Any other any other questions? There's also you for those of you in Japan make sure you go to the fan technical jamborees We have those about once a quarter and there's Interesting content about embedded in on those as well Okay, if there are no more questions, then thank you for your time. Thank you everyone Hello, okay testing Okay, so Good morning. I think it's time. We should be starting the talk My name is Laura more I work for a guy and today we're talking about Automated testing of embedded web applications I met the guy there since 2020 working in the webkit team Mainly I was working on key rate tasks and now there's I'm working mostly on the web drivers support for that be okay, but you take a and in the past I have been always Since the beginning of my career I've been working on offices tools like the in the old Nokia times of my ammo in the Python binds and we had back in Brazil Port of webkit called nix it was spread cut similar to WP I streamlined port a keyboard and also worked on the Enlightenment Foundation libraries also in the binders for JavaScript and C-Shop Igale is a Open-source consultancy. It's over 20 years old. It's from Spain, but we are fully emotes nowadays We are working on that cooperative. So we had a flexi instructor We are mostly known for our work in the web browsers and giants It's the main contributors to webkit and chromium outside Apple and Google respectively we also have been working on gecko and servo recently and Not just browsers, but we are growing our contribution in other kind of projects like different JavaScript and Giants compilers and lot of projects in Jeff or Jay streamer and a growing number of contributions to Graphics drivers in Mesa and the kernel due to our open source background we are very proud of our working in the Different working groups to standardize the technologies to work together with others and move the technology forward For today, we will be talking a little bit about waters. What do I mean when I talk about embedded web applications some challenges that we face developing them and testing them and how browser Automation can help you dealing with those challenges So why web applications? web applications they are like my colleague Adrian Mentioned in yesterday talk. They are eating the world. They are growing everywhere because they enable first development cycles the developer can See what's happening quite fast Even with the added complexity of today's JavaScript tooling it's still very fast to develop new applications It's also enables fast deployment like if you are using a remote server to serve the web application You can just update the new version on the on the seven instead of having to To deploy directly to the physical device with either a new major or another kind and It's also a larger existing of both different tools different technologies Vibrant community of the developers and It's growing but in real life web applications you write the answer and you run mostly anywhere for example, we are back. It's looks like in some ways we are back to the works better with browser days and Why does this happen? because despite all the work in the standards, we have some enjoying differences Blink which enables Chrome based the browsers might support Difference Have a different level support for a given feature to advocate versus Gecko. This is one of the most common kind of differences especially when we have some new shiny new standard that one of the engines might be a little bit more conservative to to implement but even if we get the same engine we might have browser difference some of the Features might be Implemented at the browser level So this we might have some cases where the Chrome behaves different for rave that they behave different for edgy and Because of the web kit we might have so far different from the number of it and Also, we have the trouble with fashioning because the one system might be using a Different fashion with a different kind of level of implementation for a given feature There's a great talk linked from the last year's Selenium conference by David Burns That talks how those kind of differences appears as browser engines are different from desktop browsers in mobile browsers Because it's also about other Is the web application running on a desktop like machine on a tablet on On a phone The size of the screen it's 4k Full HD. Are you using a scale of native resolution and What kind of interaction the web app will have it will be using if we just a display will it be using mouse or gestures and Given that what do I mean about embedded web applications? just Question first who here already works with some kind of embedded web application, please raise your hand Okay for people So embedded application differs from generic web applications Because it's not just a generic website that I have That's running on a generic mobile browser. Let's say just my sister just lost from me with some kind of stuff I Mean that I Embedded web application it will load some specific resources. I Can control the user will not get the access to the URL bar and load some malware website In some cases, I might even be loading the resources from Locally in the machine to isolate from the external and network and We control what kind of browser is running the Web application so we don't have to deal with some esoteric lab different browsers and ages and I think the most importantly the web application can explain the browsers with new APIs to access some Sapses from the system But even with this kind of differences in real life We might have different devices with different hardware capabilities for example different CPU Different GPUs the even different CPUs different screen sizes. For example, they are dealing with Project that the same application is expected to run I max 6 and I max 8 with very different GPUs with different levels OpenGL support. So this also brings some complexity to test Also, you might have different browser versions because Due to hardware capabilities or different device generations, we might have let's say another Webkit version running on a other version of the project, but it's still consuming the the new version of the web application But thankfully this is still less variables than doing generic website development and as I mentioned generic websites are usually sandboxed the browser prevents them from Accessing the system only through some very specific APIs defined by storage groups and even so it's still very limited But in better devices, sometimes they need to interact with the system. For example We have a pro we had a project where they Through a web application you control the mixer a blender for to So you need to be able to make the web application interact with the system and As such testing the web application in a live system is a must There are cases where we need to be able to test the live The web application in a running system It's big this differs from also from generic website development because usually generic testing tools for Websites they kind of launch a sandboxed browser just to start the load the page run the test Okay, it's over and in some cases the browser starting the the process of starting the browser might connect some systems in Some services in the system and we need to be able to to see it working on real life especially on Entrant tests for for those applications But then How do you test Web application first of all there's a confusion between automation and test Usually when people talk about automation they already may always lead to lead to Talk about testing But automation is just a tool that we have to Control a web browser For example, we can use automation tools to crawl the web it's having a Use it extensively in the past to Crawl the web For let's say getting some informations. I remember some friend friend that they had a project that was crawling some government sites for information We can use automation to field access for example to do some boring stuff but also we can use automation to running test steps which is our focus here and originally Automation was Designed in a way to mimic user actions for example navigate by giving URL find the input for Given form with the valid name Enter the text That input Then find the boot with the label confirm and click this button Also click this button search for the message user created so the whole most tools of the the tooling of automation was created originally mimicking user Actions and these led to some design this decisions that Still reflect today So in the beginning It was about early 2000s that the first real efforts appeared Namely with the Selenium and the web driver projects that after some time they measure together and we can what we call today the Selenium web driver that driven the development of those tools and And it was widely adopted, but it's still kind of depending on some browser specific Communication eventually it became WTC standard in 2018 So the browser zone had only had to implement the standard and the tooling that sits on the client side could talk with to any browser and And these web drivers standards nowadays it's supported by all major browsers and giants Chrome WebKit Firefox and it's what we call today by web driver class and It's this protocol that Mediates that the communication between the client the out the automation script or test script and the browser it's based on HTTP and It allows remote connections which which is interesting for the better those kids for example the developer can be writing his automation scripts on the desktop normally launching the scripts and it will connect it to a browser running on a embedded board like a has very pie or AI max and as it's based on HTTP under the hood the user sends comments to the browser which Sends the response the HTTP response back to the user With the result of the action This worked well for some time, but Just mimicking the user nowadays. It's it's not enough why because For our applications logic has been steadily moving to the through PEM We have huge applications nowadays running on the browser instead of thin just thin layers and Given the nature of JavaScript lot of synchronous tasks and the synchronous tasks does doesn't mix very well with the Common in the responses scheme of the original web driver protocol and Pauling is tricky in those cases because for example you send a common to do something in the web app and doing this will trigger some Task on the browser that will take an ink no time to finish The automation script is waiting for some kind of signal, but the browser can push back so the The web driver classic script you will be Pauling checking is this happening is this happening So this introduces some kind of sampling error that for example, you might be waiting for a specific value That might be in the past you might kind of lose some exception that might have happened in the browser So The web driver class key has this kind of limitation in the meantime also, we have been developing new tools like the Chrome developer tools the remote Spectrum and WebKit and Those tools are very very powerful they enable true two-way communication with the browser because we will open the For example in WebKit with the remote inspector you can open the Spectrum in a in a different browser targeting a Given browser that you want to expect and it will react to things that happen in in the browser and So given those capabilities we end up developing automation tools based on those protocols and one of the most famous ones on top of those protocols, for example, we have the Since the Puppeteer that's based on Chrome Developer tools, which is the protocol that Chrome uses to do some debugging inside the Chrome based browsers and those kind of tools Enabled new capabilities that help it address those limitations of web driver class, but back again They were some kind of browser specific solutions Based on those experience the industry sit together and we are Developing constantly they what we call the web driver by die protocol by die for bi-directional and It's already press well supported in Chrome and in the chromium engine chromium engine and in for Firefox and WebKit we at Igalia have been working to add support for it and It's progressing Prior to also be implementing the coming months under the hood Web driver by die As I mentioned it enables true two-way communications between the communication between the user and the browser because the user now can still send comments to the To the browser, but the browser can push back information to the user script and Web driver by dies implement these by using web sockets instead of HTTP. It's just just to set up the Connection It's built on top of the experience with a driver class. So if I drive a by die We should be able to do whatever we could do with web driver class For example, you can still perform user actions Like I said clicking getting information from the page Find elements so on but also you can move toward the browser state without polling Which is the game changer in for testing for automation What can we do it is we can mount or log messages by the heads in specified how the Client can subscribe to events and whenever there's a log message from the browser for example console.log message or a javascript Exception you can assign some cowbacks to be called from the client Also, we can observe for don'ts changes in the web application Which is something that was also missing as I mentioned you have to be polling now You can just react to to those don't changes and for testing it's also Very interesting that with by die we can intercept and mock browser requests So when the browser Going to start to fetch a network resource by die allows you have to have a by die allows you to Intercept that request create a mock response and the browser will use that mock at response so this also helps and Testing and as it's based on web drive a class key by die also allows you to connect to him or Browsers running on other boards And what about some browse automation tools because by die and web driver adjust specification just a protocol between the those two The area they have an implementation a reference implementation inside the web platform sample story called It was formally called WD client, but it's just a t-layer on top And Given that we now have That in specification for web driver and by die it makes it see easier For two developers to create new tools on top of those the most famous one probably is Selenium because it was let's say the one that started it all in the in the past and It deeply influenced the web driver class key standard Do it to these it was the first one widely adopted and And it's has implementations for JavaScript, Python, Java, Ruby, C-Shot, many languages And Selenium is not just the Selenium web driver library It's a huge project with a lot of the different tools like the Selenium and Anger for to handle different browsers And now so the Selenium grid where you can assign different Different workers that can run the test is in parallel to reduce the size The the time that they spent running those tests Another tool that's getting traction is what we call WebDriver.io it's also based on the W3C WebDriver protocol and It's written since the beginning in JavaScript. So it is deeply it's very well integrated with the The let's say the workflow of front-end developers So it helps it makes it easier for people doing the front-end development also use it for automation and It also right it it has built-in support to run with the frameworks behavioral driven developer frameworks like Cucumber, Mocha and Jasmine so It's also a plus that it's tightly integrated with the front-end development environment Another major player that's also gaining traction is Playwright That it has some oranges in the Puppeteer So it's tied to the kind of tied to the Chromium DevTools protocol Nowadays it's being developed by Microsoft it's under active development. We at Igalia work with Playwright in the especially in the supporting WebKit for it it has a Very interesting feature called outweights that Before performing an action the outweights features it performs some checks In the in the given element for example, and you go to click a button It checks if the boot is visible if the boot is enabled If it's visible, it's not obscured by some kind of other element that could Receive the event so it greatly helps organizing the and Reducing the flakiness of the the tests But unfortunately Playwright it depends on patched Firefox or WebKit versions for the Generic web app the developer this might not be much an issue but for if you we are developing a An embedded version of WebKit basic browser for example We might have some downstream patches specific for the platform and the WebKit patch for playwright is quite huge so it might add some complexity to testing Now we Do a recap of a few Points many points of the the talk Embedded web applications they usually run on more tightly controlled Scenarios than generic applications. So these greatly reduces the complexity of testing but at the same time that we need to test how it integrates with the With the system so We might need to to run different kind of tests as we might connect to The system running on a board. It's not that we can just do some Hemo testing on the desktop of the front end developer and everything is alright But thankfully especially the new protocol web that everybody has some Features like the bit by the bidirectional Communication that enables deeply inspecting the the the status of the web application within the the target browser and We can better test it That's it so Questions anyone Yeah, okay, the question is about maintenance of the automation script is how Often they change how well can we maintain the intro the lifetime of the application? One of the I think one of the important things in these To maintain this is the coordination between the automation team and the people writing the web application one one fragment pain point for example is The selectors used in the web application to identify the the a button important a field that we want to test especially nowadays with the Those frameworks that move elements in the dome around all the time for example some People nowadays are a bit wary of using XPath to select so The both teams need to agree in How how are we going to label to identify the the main elements to be to be tested there? I think there are some There are already starting to appear some AI power tools. I think to Identify to that visually we identify the the elements What is I think mainly it's something that need to be agreed upon between The people writing the tests and people writing the the front end code. That's why probably using the JavaScript based automation tools will help when talking to people from writing the JavaScript based web applications to make easier to make the Web developers help you if testing did find the the points and such Okay, the question was About integrating those kind of automation based tests into the pipeline to to make sure that you're taking account to the the The duration of the test is to make sure it integrates well with the the pipeline For this kind of tests the automation tools, I think they are let's say Very agnostic in the sense of they can be integrated in various kinds of pipeline For example, we are working a project that will be using cucumber to to run the tests So to describe it the tests I mean so they'll be written in that Language and the steps the test steps will be used We are still defining the the kind of the library that we be using either Selenium or web drive IO or other and In turn those tests that those and cucumber tests that can be integrated in other kind of testing frameworks that the company use they I think the main point is that the The way that web driver is designed allowing to connect to a running browser To a remote browser, etc. It's also great flexibility to choosing the to to using it with the tool that's already being used being inside the Organization, I think the the next talk in this building will be exactly about some testing tools to to integrate the To be running on embedded system, so probably the automation tools could be also be running on over Okay, so this is it Thank you very much and See you around Hello everyone My name is Paweł Wierczorek. I work with Colabora and I've been involved in testing automation efforts since around 2016 Maintaining various parts of automated testing laboratories ranging from devices under test Maintenance as well as their supporting hardware through server side components profiling and development Up to co-architecting custom solutions in this domain Today, I would like to share with you some of the experiences I had with the various Various software solutions and some of the encounters or let's face it Straight-up mistakes made along the way I would also like to share with you how you could make use of these lessons learned In your laboratories and your workflows As well as maybe having a better start in some of the topics in the domain of Automated testing labs. You might have not given a try before I Will start with a short introduction to why this talk Was created next I will move to describing all the candidates that I Chose for comparison After that, I will also elaborate a bit on the use cases I tried to cover and then I will give you a short summary of My experience with all those various software solutions Finally, I would like to share with you what might come next What would be my recommendation what might come next and also some final thoughts before we move to Q&A session For now, let's start with motivation for this talk. Why? Why we're all here? And let me take you a few years back to the ELCE 2017 in Prague where the topic of comparing strengths of various solutions got a bit more attention during birds of the feather session called farming together Led by Andrew Murray It was for me one of the first times when I started looking outside my bubble of automated testing laboratories Or maybe before we go there if you haven't seen this session, I Encourage you to review the recording From the link down below However, it wasn't the first or the only effort aimed at working towards experience sharing in this community There were others like two automated testing summits That were held in 2018 and 2019 with that 2019 not only covering Status updates for various test lab stacks as well as survey conducted by Timberd the 2019 event also had multiple talks on various tracks ranging from board and lab management Through covering test definitions up to most common QA systems and their issues It also sparked many follow-up actions like for example open automated testing standards Call which is held every month and provides a forum for discussing various topics Or the workflows mailing list just to name a few of those follow-up actions What I believe is that these efforts have shaped testing automation community To what testing automation community is today and Actually still play an important role in developing new solutions improving existing ones and onboarding new engineers to the topic of automated tests during the previous Embedded open source summit in Prague. I had the chance to discuss the topics that were raised in 2017 with Andrew And we were wondering whether the questions he asked back in 2017 and the discussions that followed Needs revisiting and if so What would be the best way to do it and Let me be crystal clear with you I cannot express how much I appreciate continuous work and support from members of this community and All that All the help you show on the daily on the daily basis Sharing all those Insights of what you develop what you use what you continue on improving is invaluable and I realized that many of you have gone great lengths to help others to try different solutions out and That is actually exactly what I intend to encourage you to do today If you're at the start of your journey I would like to provide you with some hints on what the available solutions for you are And how well they can fit in your requirements If you're already maintain a test lab or some of its components I would like encourage you to take a look at your solution then check out how it compares to other solutions and maybe think of the ways how both of them could benefit from each other and These actions were actually what I've been up to for the last couple of weeks But I made a few assumptions. I wanted to just start small maybe with a virtual device only and To make things a bit more challenging. I also Decided that I would like to Work with either no or as little as possible of external support What I meant by that Was that I wanted to find out how easy it is to to access and use the documentation already available when starting from scratch or Without much experience in new testing laboratory. I hope that by the end of today's talk You will have some ideas how to approach such a challenge yourself and you will feel encouraged to do so But please don't make a mistake that I did of Trying to tackle this challenge alone, but we'll get to this in just a second First let me introduce you to the champions will be taking a closer look at today And I would like to make a quick note that I chose Those only as an examples of various paradigms You can go with when creating or maintaining your own setup If you see an important one missing, I will be more than happy to learn about it And we can discuss those as well either during Q&A or we can take it offline Also, I realize that some of them support multi-paradigm and Certain functionalities can be achieved in various ways. I'm not questioning this at all. I Will be just highlighting what I really enjoyed about all those solutions the most and first Candidate is lava which in just Maybe over simplification but It's mainly set of Django apps that connect to postgres database So obviously that's a Python based solution it Also provides an existing library of YAML test definitions and is well equipped with dashboard results querying helpers and various other tools its performance in is continuously profiled and pushed and By that I mean that the performance limits of these solutions are continuously pushed forward And it supports quite large workloads. You can even say that it's Cloud deployment ready as well as it's well integrated with various external tools like Prometheus for monitoring its ongoing Its ongoing work or GitLab for submitting Submitting tests to be executed in lava The next candidate is Labrith, which is also a Python based solution Which introduced a place term, which is not only device under test that you would use in your testing But also accompanying support equipment it already provides supports for large set of such hardware and Also provides an integration for pie test for writing test scenarios Interactive use is actually a first-class citizen for this solution And what's also important and interesting is that there is set of Tutorials on how to get started with lab grids available as a playlist on a YouTube channel of the creators of this tool and Next one I Had a chance to play with is Bordswarm, which is relatively new solution and in And other Then the previous ones this one is based on the Rust tooling It also provides an interactive access to your boards But it is focused on providing Hardware abstractions to unified how you can access your devices for example from storage point of view or serial Console that you would like to access So the test definitions might not necessarily be useful in this context instead It is meant to be a building block for more complex solutions Or maybe just a unified base layer replacement for existing ones It also follows the UNIX philosophy and UNIX principle of doing one thing, but doing it well. I Also wanted to take a look at More at a solution that is aimed at more Enterprisey Let's call it that way environments and for that I chose beaker Which again is a Python tool It's not necessarily meant for embedded environments, but a similar set of Functionality and requirements on its side are also present. So power state toggle Some additional supporting hardware And so on this is also a fully fledged Solution with various dashboards and you guys available, but there is no obligation whatsoever To use all of those components if that's not your thing You can still benefit from its subsystems like for example cobbler just for having a bare metal Provisioning layer if that is your requirement. I Also gave a try to custom hardware assisted Solutions and I realize there are various solutions available for that ranging from USB maxes Micro SD card maxes to more complete solutions and one of those Which is Testing in a box by coating was the one that I chose to give a try I could try to Also specify what language it actually Uses for its supporting software, but That's not the feature I would like to highlight here What what is actually a really interesting is that the designs for this custom hardware and it's a software integration layer Although tailored to specific system requirements were kept generic enough and modular enough that they should be that you should be able to Reuse those tools in your setups as well But before we even think of trying to do so, let's make an important note that it's much harder, obviously to do iterations Using someone else's solution unless it ticks all the boxes for you There will also be some invested Some investment needed in both costs labor know-how Before you could even try to incorporate those in your setup, but it probably can bring nice returns in long run Also, this is one exception to the rule. I made that I would like to try all those solutions in virtual environment Because it's simply not applicable to Do such an approach? Another remark I would like to make is that I did not manage to get my hands on Testing in a box hardware. So I made use of what I have what I had available Which is a simple micro SD Demultiplexer and custom PDU To sum it up We've got the five contestants that That started in at various points in time as you can see most of them are Python based but not all of them and the test definitions or task definitions are Described in various ways I Would also like to Make another reminder that I do realize that That for example lava provides hacking sessions or multi-node tests And also lab grid works with custom hardware for assisting its test runs But I chose those Candidates as the just representation of the use cases we will be covering and now let's move to those use cases the first one which I believe many of you might be interested and Is interactive access to your devices under tests most often it's just to get an access to device that is Not available in your physical location, but instead it's somewhere in a server room on the other side of the globe possibly or just Downstairs for cooling purposes If more than one user wants to access Such such a device It's also important to remember that there will be a need for some reservation mechanism For example to time out sessions or just Kick users that no longer make use of device they reserved Also just Access to serial console is a good start, but for embedded purposes some additional control would really be useful for example to GPIOs to Massive storage or to additional measurement equipment like ACME boards Next use case I covered was patch processing of submissions And I just tried to throw a bunch of test submissions and see how well it would be handled by scheduler that That was either built-in or external to a solutions to those solutions of course some of the candidates I showed you earlier are Do not have an built-in scheduler, but instead only provide a reservation mechanism and for those I Cannot Recommend you know with how well this scheduler works because it's just an external tool Other than that it would be very useful to have some notification mechanism or preferably even a callback one and this is probably mostly used as Acceptance tests on wide range of device types when a single result does not really matter before the whole test suite is completed Up to the last device you would like to check Next I will also wanted to cover version control systems because while many embedded projects and Developers work using complex dependency structures some of the projects use just a single repository and actually need per patch verification in Such cases it might be a deal breaker if the test lab system integrates with your VCS or not Otherwise, it will also only bring unnecessary overhead to the projects you develop. I Also realized that all those features won't you won't get you anywhere if a new user gets stuck at first try and I Was also curious whether some of these solutions already come with some evaluation Environments to get started quickly and try out the basic features some of them in fact Come with extensive documentation some like I mentioned with tutorial videos but These and I guess you'd agree with me work best if they do not overwhelm user at first glance It Also might be a bit hard to find the right pace for the user to learn a new tool And from the developer point of view and I mean developer of such a solution It might be hard to provide Such a documentation, but I believe it brings many benefits in a long run for example by creating Scenarios for basic use cases If you automate them you already have a verification environment for your test lab solution So just to recap quickly the aspects that I analyzed was interactive access batch processing version control systems and integration with them as well as documentation and First steps that you would take with such a solution Now, let me share some of the experiences I had with those test lab software for interactive access Lab grid and board swarm performed really well for me. I enjoyed using both of these solutions And although lab grid provided much more feature rich experience Board swarm allowed me to keep things simple. I Realize I still have to evaluate for example new approach to hacking session in lava which now is teammate based But that's still to be evaluated Also lab grid ticks all the boxes I mentioned earlier When I described what I would like a solution that provides interactive access to be equipped with As for the batch processing Lava was the solution that ticked all the boxes for me So it provides it provided me with a scheduler with the right priorities with call back mechanism and even Q clean up in case the test submissions that I made Were waiting in a queue so long that they became no longer relevant and I know that it might seem like recurring theme that those solutions tick all the boxes for me, but I Think you would agree with me that these are very mature test lab software and That is why They they work so well in those use cases as for the solutions that do not come with Internal scheduler I had to pick one from the version control system that I tried to use with a given example and That's probably the best way to work with such Solution for yourself just pick the one that you would use as your version control system and Speaking of version control system When when we discuss the interactive Mainly interactive use for the solutions the Pipeline provided by your version control system are Quite a good candidate for a scheduler together with a reservation system in lab grid or board swarm for Lava the integration with the VCS Could be made just by submitting jobs and either poll for the results or use the Previously mentioned notification mechanism or even use custom runner integrated with VCS You might have noticed that Some of the text on the slide is slightly different color. I linked all the relevant additional software and If you'd like to know more about these solutions go ahead to a slides on the schedule and Try those links out as for the Tools that were built around version control system like for example testing in a box the key benefit for me was having the whole config of Such a solution kept in a single place So that I did not have to actually care Where do I get test definitions from? How is my laboratory configured and so on? On the documentation front I have to admit that the Beaker documentation was the most extensive one I came across Ranging from setup documentation through administration one up to user guides Not far behind was Lava document Lava documentation with both classic one and the new one hosted on read the docs and I think I already mentioned it once or twice that I really enjoyed the tutorials provided by lab grid development team And going to Examples that you could reuse as a basis For your own setup and they're testing in a box Initiative provides examples in most of their sub repositories While lab grid just holds an example sub directory in the main repository, which is also fully equipped And Lava already provides a library of various test definitions that you could simply reuse Many of those solutions also came with ready-to-reuse evaluation environments beaker with a set of Ansible Playbooks to just provision some virtual machine or Possibly other system that you would like to try it on As for the lab grid My colleague Chris Obart created also a set of Ansible playbooks to try Lab grid in your own setup and you could check out his talk from Embedded Linux conference Two years back Linked on the slides As for the lava you've got multiple choices Ranging from just a simple Docker compose which sets up a server for you to try things out Up to more device centric approach with Lava Docker that Generates actually a configuration for Lava based on the description of your testing environment and I what stood out the most for me from all those solutions was that some of the Testing automation software Makes sure that all the design decisions are well documented like for example lab grid and board swarm I was also really impressed by the amount and Specificity of the documentation provided by projects like beaker and Lava and what I liked about the testing in a box is that The hardware designs that were created for the specific use case Could possibly be reused and recreated in any other testing laboratory And now let's move on to what I would like to Recommend to you and I realize that some of them might sound a bit tribal maybe even Trueism like like a trueism, but I guess you'd agree with me that there is never a silver bullet in any of these approaches and if you ever find a Solution that is ticking all of the boxes for your requirements, then you could be called an extremely lucky Also each of the testing champions. I showed you might require some adjustments for your specific needs And that comes with some integration effort As well So it's really important to decide on what your key requirements are so that It's not just a guess what would what could work best for you, but it's a choice It's important not to over invest in a solution that could become a dead end for your use cases Also If it is a choice there will probably be a fewer integration bumps for the adjustment of the whole setup to your needs And that would also help you to set your expectations, right? So that no one ends up disappointed with any solution What probably could work best is a hybrid approach So for example Getting to work a board swarm and using it as a base layer replacement for some existing Testing automation solution that could make use of hardware abstractions or for example using lab grid with its dedicated hardware solution as well as Providing some additional integration to to an existing test lab like the git lab runner I mentioned earlier or for example exporter of various metrics to your Prometheus database But the most important recommendation I would like to Give to you is to not make the same mistake I did on trying to work everything out on my own the best decision I made was by it was to approach a User that already had some experience With one of those solution is one of those solutions when I really got stuck and almost gave up on Trying to to make it work Because it reduced It could have reduced greatly the time that I spent on trying it out for myself but also from a Viewpoint of a developer of such a solution it could provide a really important feedback on What works well what could be further improved and what use cases are Actually covered by a given solution As for the next steps I still need to upstream a few changes in my Evaluation environments, but only if they are actually applicable according to Core developers, maybe I made some mistakes Probably that's The reason for that is that I tried to do it without communicating first But those virtual only quick starts Probably could be beneficial for reusing if you Consider trying out a new software solution Also, maybe it would be useful to create a set of requirements for such a virtual only quick start environment but That's actually for you to decide and that's what I would love to hear from you Do you think that it would be beneficial? Would you like to use it? And if there is actually a need for that Related to it. I also Wanted to Provide you with some setup that is not only bound to virtual hardware, but Actually can be reused with the physical one. There were some previous discussions on choosing the right candidate for that, but that's also a slightly more complex solution to provide and Again, what I would love to hear from you is whether this action whether there actually is a need for Such an environment or it would never be used by anyone The last thing I had in mind is To try to scale up all of those solutions I know there are certain efforts in some of them to Keep it testing and Check how it behaves with higher and higher workloads and if you'd like to Get More details on this topic. I linked three different talks covering it for you to review the recording to sum it up I Wanted to show you that there are Many various solutions to choose from if you'd like to check something out and Maybe even migrate to a new one. I Realize this might be a high investment But I hope that some of the features that I highlighted Got your interest and maybe even encourage you to take up the challenge and again, I would like to make another remark that The biggest lessons the biggest lesson I learned is to try to engage with core developers and The community around given solution right away so that you don't get burned out. You don't Get stuck on possibly simple issues and You also provide a feedback to those developers and That's all that I've got for you today If you've got any questions, I will be happy to answer them so you mentioned a lot of upfront work to to Get everything working and that it is difficult how much time Does it take to or how many people does it take to administer these types of? Projects like a lava instance or so I guess that's hard for me to estimate and that's because I Did this experiment that I didn't want to Engage with anyone and try it only on my own. I believe that if you get some maybe mentoring or just Some hints from core developers that time to get your new system ready would be greatly reduced But for me that was just Fun experiment it was about two hours per week for a few weeks and That's what I ended up with Thank you Thanks for your talk Do you have any experience with some kind of peer-to-peer testing? You know often you have some kind of a device on the other side You want to test some interface and you might need to configure sites on things on both sides any comments on this So let me rephrase so that I make sure that I got you right You would like to perform a test that involves multiple devices under test that communicate one with another right, so first solution that comes to my mind is a Multi-node test on in lava, but I don't have much experience with that I mostly used the lava for just a single device under test As as far as I'm concerned although we would have to try it out we can do that in just a minute a Place in lab grid terms could also Provide you with multiple devices under test that would be reserved for your test run and Possibly communicate with each other And that would be the two solutions that I would try out first Okay. Thank you very much There's another solution that Intel had when they were doing their Android testing and it's called TCF stands for test case framework That one also has a lot of similarity to a lab grid and it also had the ability to Set up like a virtual network as part of your test framework and everything and that allowed you to do multiple devices and talk to each other Through a while it was in Python 2 and then it finally got upgraded to Python 3 So it's on github Intel TCF so that one's quite similar to lab grid though Thanks for your comment. I will definitely have a closer look at it But if there are no more questions Thanks for your attention And have a good one Hey, everybody getting quiet. I guess that means it's time to start so welcome I'll be talking about some adventures I've been having in the industrial IO IO subsystem and Of course what you all really came for hopefully something about some Lego robots So I'm David Lekner. I'm an embedded software engineer at Bay Libre Bay Libre is a consulting Firm work based in Nice, France, but we've got remote people scattered all over the world as you can tell I don't have a French accent We love everything embedded and open source especially Linux kernel Android we're starting to get into Zephyr and we also recently hired the code sorcery team. So now we've got tool chain services We can offer I've only been working at Bay Libre for about six months now So my background in Linux kernel hacking actually comes from playing with Lego robots the Lego Mindstorms EV3 that came out in 2013 runs Linux and I spent quite a few years hacking on it reported Debbie and Linux To it and I started sending kernel patches upstream and that's how I Learned everything I know about being a kernel developer maintainer and then more recently Another gentleman and I Lawrence Volk Started that project called Pybrics, which the more recent Lego robots don't run Linux there are more constrained microcontrollers, but we're running micro Python on them and Before that I was I actually started my career in industrial automation automating water and was what wastewater treatment plants. So I had kind of a bit of a non traditional career path to get where I am today, but it's all come together In the industrial IO system now, so I've got five little adventures to tell you about today If you're following along at home watching the reruns on YouTube or the PDF you can choose your own adventure But I've got a captive audience today. So we'll go through them all the first couple ones are going to be talking about Hardware aspects of what working on and kind of day-to-day how we Managed to get remote workers working on industrial IO analog digital converter type hardware and then the last three ones are a little bit of a report on interesting things that are going on upstream in the IO subsystem So for those of you don't know industrial IO subsystem is the corner of the convert kernel or Anything that's an analog digital converter or digital to analog converter hangs out that it's all about Measuring and signals and things like that So our first adventure the adventure of the remote IO developers As we mentioned we've got Developers all over the world at Bay Libra so our team that's working on this particular project of Upstreaming some support for some analog digital converters and also developing other User space tools. We're also doing a little Zephyr bare metal type stuff for the same analog digital converters We've got about six fully remote people a couple Hybrid that are in the office some days Wrote some days and one person the office full-time so We kind of like to share what we're doing and hopefully in the Q&A compare notes with anybody else who's Doing any similar kind of work like this to see You know, what's working? What's not working? so To start with Early on then on this project. We decided that we're going to keep all of our hardware in our lab in France So as a developer we're connecting remotely to a hardware stack like this and In the next slide I'll zoom in on What we're actually Looking at but for now. We'll just appreciate the aesthetics of it shout out to Fabian on our team for Making this all nice and neat getting all the cable management done so here we've got a Development board set up that people can remotely connect to to develop their drivers test them out And it looks pretty cool at night, too. So Here's what we actually have on each shelf there We've got of course our remote developer wherever you are connecting in through a VPN and I forgot my pointer so we connect to the Development board which is running Linux and So much for that idea So that's our z-board it's a development board that runs Linux and we've got an analog digital converter evaluation board connected to it That that's what we're developing a driver for and To be able to test it. We've got a what we call the m2k. That's like a Swiss army knife of signal generator logic analyzer oscilloscope all-in-one To generate a signal and do any kind of testing I have a motto that if we can measure it we can understand it So we need our instruments to measure what we're doing to be able to test and verify that our driver is working We also have a special power switch. This is something we've developed Internally at bay labor we call our co-pilot that can allow us to remotely Cycle power on our development board. So if it ever hangs crashes, whatever we can reboot it remotely and All these things are connected with USB to a little mini PC and We also connect to that through our VPN and On our mini PC We're running graphical tools to be able to actually visualize what we're doing the scopey is the graphical interface for that Swiss army knife m2k gizmo that we can use to generate a signal or Use it as an oscilloscope if we need to and then we're also using the IiO oscilloscope program to actually capture data From our analog digital converter to be able to see are we actually Capturing the same signal that we're generating to know if our drivers working or not so as Some of you might guess if you've tried to do work like this before Trying to do this remotely without being able to actually get your hands on the hardware has some issues with it So these are kind of the biggest pain points over the last six months that we've identified The First couple of ones we've more or less figured out The second two are still kind of causing us problems. So Sharing resources and when we first started we had at one point four different people sharing one laptop That was connected to four development boards and had about 20 USB things plugged into it That didn't work so well. So We sprung the money got eight mini PCs now. Everybody's got their own That problem solved just took a little money a little time But easy problem to solve the next one the booting files. We tried a few different things I've got a couple more slides on that. So I won't talk about the details just yet Then the last two is on our setups We often have to swap out that analog digital converter that We might have five or ten similar ones that we're developing a single driver for X number of chips and so we have to swap out the board to make sure it's working on all the different chips And so since we're working remotely Some we have to ping somebody in the office and say hey, can you come switch out the board for us? Which is you know kind of disruptive to them if they're trying to do other work, but that's also at the same time That's their job. So they do it at the Maybe the biggest pain point has been the debugging actual physical electronics issues that The issues that are like noise Cross-talk that's not a problem with your software. It's actually a physical electrical problem So the remote developer is Can't do anything about it at all. We have to rely on whoever's in the office to have the time and knowledge and skills to be able to debug that and so that one's particularly a problem because the remote developers the one who's been reading all the data sheets becoming the expert on that specific chip and the Person the office doesn't know anything about it and you're saying hey Can you fix this this for me and they're like sure yeah? I have to learn the whole thing how it works, but so jumping back to our Boot files this one we turn there's you know a few different things people recommend We tried using a SD card mux But those actually turned out to be more of a problem than a help we had lots of electrical issues They just didn't work The we would switch it and then not be able to connect to it anymore so we gave up on using those and You can also do like a TFTP boot boot from network We tried that, but we have three boot files that we need to update and Only two of those work over TFTP so it didn't really completely solve the problem for us So we decided to just keep things simple as possible our development board Mounts the boot partition automatically at slash boot already So we can just use secure shell as long as we can boot into linux. We can use secure shell the Copy files into the boot partition reboot our board It comes up Worst can I scenario you have a bad kernel it crashes or something well We've got our mini PC connected with a serial console so we can get into the u-boot prompt and Recover if we've got a bad kernel or bad device tree So that only leaves the one bad case of the boot bin which has the u-boot in it If we botched that one well, we're out of luck somebody at the office is gonna have to fix it for us But luckily that one just doesn't happen very often So that hasn't been a big problem so Kind of wrapping it up, you know, is it possible to actually work like this? You know some of the reasons why we decided to work this way of having the hardware remotely is that We need multiple engineers all sharing the same hardware because we're developing a linux driver a bare metal driver User space stuff all for the same board. That's not always the same engineer so there would be a lot of logistics trying to ship around the same board to every engineer and We wanted to avoid that things get lost in the mail the shipping costs add up and Not all the remote engineers have all of the equipment you need to test those hard problems So that's our main motivation for having our setups the way they are but You know, there is a loss of efficiency of not being able to get your hands on the hardware yourself so we have been making some exceptions for that that when One engineer or there's only one software task for one engineer then we've shipped them the board Or if we had a duplicate board or some of the chips are just so incredibly complex that it makes more sense to Ship it off and ship it back So I don't know if everyone else on my team agrees with this conclusion, but I think overall it's been working Well enough for us We've been managing to get things upstream so that leads me to my second adventure Which is the first Linux driver. I got upstream Using this method So this is the adventure of the resolver the digital converter. So remember these tires. They come into the story later so I'm guessing most of you don't know what a resolver is I didn't either when I started this so here's a 30 second crash course on What's a resolver? So it's this? two-piece thing here This is called a frameless resolver and it's got a rotor a little smaller part rotates It's connected to electrical coil that rotates and there's a stator That's the bigger ring it stays still and it has connected to two more coils that are at 90 degrees to each other and the rotor coil we energize with an electric field and That induces a voltage in the stator coils and those signals go into a Resolver the digital converter chip and since they're 90 degrees apart We have a right triangle the converter chip does some trigonometry And you can figure out what angle the rotor is Compared to the stator. So this is measuring The rotational position and it also has a clock in it so we can measure rotational speed of how fast this rotor is spinning So when I started this is our first test setup that they had set up at our lab That I would connect to develop the deriver It's got the same board running Linux Connected to the evaluation board with the resolver to digital converter chip and then we have our hardware store Set up of We added a shaft and some brackets to mount our resolver so we could actually spin that rotor But there's a few problems with this setup number one. It's human powered that Whenever we wanted to test our driver, we had to ping somebody on slack to say, hey, can you go turn that shaft so we can make sure my driver is working and Also since this has to measure speed Humans don't turns out they don't have a constant speed setting, you know, can you turn this one revolution per minute? Or they don't have a high speed setting. Can you turn this 10,000 revolutions per minute? And also since that came in two pieces That have no connection at all. It's called the floating resolver for a reason You can't touch and there is a point five millimeter air gap in between so Our high precision hardware from the hardware store Wasn't really up to that task, but do you know what is precision engineered to a fraction of a millimeter? That's right Legos So I went to the office and got to visit the office in France and I brought back one of the Duplicate boards we had with us and since I've got that background in Lego robotics I built a our version to set up or we can actually Test this properly So remember those tires I told you about from the 1980s Space sets well it turns out they're a perfect fit for the bevel in the resolver because I needed something to grip it to spin it With the motor and then I built these little Gizmos with some worm gears to get precision alignment that you can see there's four of them So that I can there's two degrees of freedom on each side of the resolver So we can get it exactly lined up the two donut shapes, so they don't touch each other at all so Here's what it looks like in action this is just doing a rotation test by hand to it's measuring and radians and then to measure speed the Lego motors actually have a Encoder in them as well So we can know that Exact speed of the Lego motor and compare that to our resolver So that's actually how I validated that our driver is in fact working and converting the speed correctly To radians per second. That's what the industrial Ios subsystem uses so hardware problem solved onto The next part of this is it turns out to get this driver upstream there was actually a Driver that's been in staging which is the corner alert of the kernel where Things that aren't quite ready to upstream sometimes go But need more work So there'd been a driver in there for 13 years waiting for someone to love it. So I Also got to learn the lesson in this adventure of how do you get a driver out of staging? So number one fix the bugs first always good practice when you're up streaming anything number two fix all the user space interface in Industrial Ios subsystem we have tons and tons of sysfs attributes for All the different aspects of these kind of parts and we try to use the standard ones as much as possible So that we don't end up with every single chip having its own unique Attributes so the driver in staging was basically reading and writing registers through these attributes, which is something maybe you do in debug FS, but not in Industrial Ios so we had to fix all that then Stuff in staging usually doesn't have device tree bindings So you have to write that and one thing to watch out for is don't let using a device tree property in your Driver sneak through without documenting it in the bindings YAML you have grumpy DT maintainers and And Then finally once you fix all those things we can actually move it out of staging I did a goat get format patch with no renames so that we could see the full driver on The mailing list give it one final review as it was moving out and then Once you finally get it out of staging then you can start adding all those shiny new features you wanted So mission accomplished just took a little ingenuity and about 40 patches and finally got that upstreamed So speaking of upstream that brings us to our last quick little adventures of Some other things going on upstream in the industrial Ios subsystem So we have the invest adventure of the spy offload What's the spy offload? Well the problem we're trying to solve is some of these analog digital converter chips can do a million two million samples per second and we need to get that into the Linux kernel over a spy bus and Using what's out there in the kernel right now we can get 10 maybe a hundred thousand if you've got a really fast system but yeah nowhere near two million and There's also all sorts of sources of jitter because when you're dealing with interrupts They're not always happening any at a regular interval So you end up something like this? What's wrong with this picture? Yeah, we missed almost the entire sine wave at least in this capture because the kernel must be any busy Sending the previous capture over the network or something. I don't know So Our solution to this is spy offloading so that's a way to do all of the spy transfers Completely without any CPU intervention. So avoiding all of the interrupts all together So the way this works is we basically Record a series of spy transactions whatever it takes to read one sample from our analog digital converter chip And then we're triggering it with a hardware trigger. So for example a pulse width modulation signal that can Make a pulse one million times per second at regular intervals. So we get evenly spaced sample our received data Gets piped to a DMA buffer So we're getting all of the samples in one big buffer At one time then we just get one single interrupt to tell us the buffers full and we can Read that buffer. So that's eliminating all the gender problems and it's allowing us to get way more samples per second so Ta-da, it works This is actually getting two million samples per second from a chip and Linux So the good news is it works the bad news is it's not quite all upstreamed yet So I sent a series of patches for this upstream and It got Quite a few criticisms, but it's also nobody said it was a bad idea It was just the way I was going about it Wasn't great. And so it's been suggested to break this down into smaller parts So the first part that's already been upstreamed already is this idea of pre-cooking or pre-preparing spy messages So this is an optimization that Actually was talked about about 13 years ago on the spy mailing list We resurrected this idea that Whenever we're doing a spy Transfer in the kernel There's a lot of code running to verify all the parameters of the spy transaction and so in a Like in a analog digital converter when we're reading samples, we're doing the same Transfer over and over and over again. So there's a lot of CPU time wasted Reverifying that same transfer over and over again. So now since the six point nine kernel there's a new API to pre-optimize a message or spy message to avoid some of that overhead and So the next steps for Breaking this down because remember that the idea is to eventually have a way to record and playback spy transfers completely from a hardware trigger So The spy maintainer wants this to be kind of something even more general purpose that can Solve more problems than just ours because that's how we do things in the Linux kernel. So I'm kind of looking for other people with these cases feedback hopefully to Figure out how we can make this work the best for everybody the In the IISO system we can already start using that pre-optimization to make II drivers more efficient The one of the ideas also thrown around was a lot of Spy drivers can do DMA mapping that that part could be moved to the pre-optimization to Optimize that even more A lot of people interested in this we're using can controllers So if you're interested in that, this is probably something you want to look into the There's a couple spy controllers that are specialized for spy memory That have a lookup table where you can pre-program spy commands and play them back by a hardware trigger So that exactly fits that paradigm that I mentioned before So maybe there's an opportunity to also improve some performance with spy flash memory using this and The last ideas from the spy maintainer of a lot of the Embedded systems nowadays have extra Microcontroller cores like maybe the PRU on a Beagle bone that we could use to also be a spy offload to improve Port performance in any way you want. So that was spy offload Next quick adventure is the adventure of the IIO back-end framework. This is also something new in the industrial IO subsystem What was the problem we're solving? Well some folks from STM 32 had this ADC IP block and a digital filter IP block that are separate blocks So have separate drivers, but they needed to somehow connect them into a single Logical device in the IO subsystem Analog devices had something similar where they have an analog chip and then they have this axi ADC FPGA block that reads data at super high rates that they need to somehow connect into a single Logical device in the kernel so we came up the idea of a front end and a back end So the front end is going to be that analog digital converter and the back end is going to be the digital filter or whatever that other block is and Now you can write Device tree bindings with a provider consumer thing to connect them together very simply and then your drivers You can register a back end and a front-end gets that back end that so the IO back-end framework is doing all the heavy lifting of connecting those together for you and Now you can have the easy way to write a single IO driver that's using multiple pieces of hardware And last little quick one is the adventure of the IO DMA buff the problem we're solving for here this one is getting more data from point a to point B for example this Use initial use case was some software to find radio stuff where they wanted to get 64 megabytes of samples From the IO subsystem out over a USB connection as fast as possible So we have this new DMA buff framework the way it works DMA buff is a API in the kernel to Move DMA buffers between subsystems or user space and the kernel So we can create a buffer in user space we can attach it to an IO device now using some my octals and capture some samples in that buffer Detach that buffer from the IO subsystem and Then reattach it to the USB function and then tell USB to send that out over The USB so that really improves performance a lot by avoiding copying the data from The IO buffer to the USB buffer. It just we just move the handle instead. So and This one test case they were over doubled their throughput While at the same time Reducing CPU usage. So that's a really nice optimization there, too so that concludes my adventures in the IO subsystem and we'll open it up for questions With the spy offloading my assumption is that you were starting not with bit banging But you started with a spy driver that had hardware support. Is that correct? Yes, yeah, and then offloading was one level beyond hardware support your That makes sense now, right? Yeah, anything that's going to do offloading is going to require special hardware that knows about Being able to have a hardware trigger or Piping the Received data somewhere else. Thanks So the the peripheral in question is an axi FPGA peripheral that analog devices makes you can find documentation it's open source can find documentation on wiki.analog.com and The hgl is also available Hi, thank you for the presentation and I think it was really great because it has always been very challenging to test Sensor drivers remotely at least So I just have a basic question. Did you require any kind of human input at some point? Or any challenges in that case While you were testing the driver for it. Yeah, usually we need somebody to set up the hardware and attach a signal generator or Logic analyzer whatever we need to the hardware, but once that's done We can connect to that remote PC and control all that remotely. So it's we're trying to minimize the extra human need but So once the basic setup is done after that you still need some human support for that Not unless there's a really strange problem. Oh, yeah Thank you. So on your oscilloscope and stuff. Are you able to Is the remote user able to see the The screens from the oscilloscope app or they yes We're using a remote desktop to access that many PC. Okay, so we can see everything on it and use it Just like we're there. Okay, that's great. Really cool So the the m2k that was being used is called other on 2000 and in fact, it's actually a complete Linux system so it's based on a sync MP no on a sync 7000 runs Linux and Also runs IO for all the different blocks in inside that that unit for Signal generator ADC stacks and so on and so you can also remotely connect to that unit So you you don't really need needed Kind of the scope the evaluations of running directly connected to the device you can also use that across a VPN and run that Software on on a different PC maybe in a different location The latency is a little better though when you're local Thank you for the talk I was wondering about for what I understand you were mostly using it for development your setup With the computer remote computer and stuff, but I was also using it for testing and see eye purposes Yeah, most of this is going just manual Testing since we have to generate a physical signal but I mean my typical setup is I'm writing the kernel Stuff at my computer at home Compiling the kernel there and then using SCP to copy the kernel to the remote board and then reboot the remote board and And then the remote mini PC is just the user interface for the logic analyzer any tools like that That fully answered your question. Thank you, right If you were to take it the next step and have it be more of a CI type of setup does the does that m2k? Scope and stuff provide the ability to script it and not have it visual But to maybe capture some logs and do some comparing. Yeah automated tooling. Yeah, like Michael said we can connect her remotely with Anything that talks IIO User space stuff. It's all using the IO Stack so any compatible tool Works with that and we've made a Python script So far just to create a simple sign wave, but it could be automated much more real quick Reservations for the boards so you can share them between developers. Did you homebrew a script for that or did you use something that you found in the community? I'm not sure I fully understand the question So like to prevent say two developers from trying to use a board at the same time. Oh, yeah the We have a we use snipe it for Keeping track of assets. So we made a Setup asset that people check out One board or two board or two setups and then we'll we can use that to see who's using them Okay, thank you But we try not to have two people working on the same thing at the same time so that it's just not a problem of having to take turns With the continued development of cheaper SPC's and such what stopped you from building more custom test harnesses to test some of these devices instead of building off of what looks to be I think you said it's a xylene zinc Development kit why not build your own platform The a lot of the analog digital converter chips we have are using an evaluation board with the FMC connector And that's already built into the zinc and there's lots of history of Testing developing and testing these devices with that hardware. So we're just sticking with it. All right. Thanks all for your questions And thanks for listening Test Everybody hear me good Let's go ahead and get started Want to welcome you to this session. We're going to be talking about quality beyond kernel CI in upstream Linux for TI SOC's Want to take a moment and highlight You know something that we feel is very important, you know, TI has a very long history in the open-source community and Collaboration as you can see from you know all the the different organizations there that you know, we have relations with with Including kernel CI Which we are very happy to announce TI is now a platinum member of kernel CI as of this week We're very pleased about that And I know you guys have probably seen this some of the other talks, but I think it's important to highlight upstream first very important In our view to have a upstream first mentality And as it relates to this presentation, if you're going to have it upstream first mentality Better have a way to test the upstream. So we're going to talk a little bit about that here Quick intro of speakers. My name is Barry Sharop. I've been with TI Long time going on 18 years now Started my life at TI and analog been in a AP for about two and a half going on three years I am the manager of the system integration and test infrastructure team We're responsible for managing the physical infrastructure hardware as well as software for doing test at TI for Linux I Am also pleased and privileged to be up here with Nishant Menon Who also has almost two decades of experience at TI and he is everything Linux? So Linux system architect Kernel development kernel maintainer He has a very very very deep experience in Linux We have also worked with Minas humbard Zimian Couldn't be here today, but his contribution is No less important He is the lead developer for the system test team that does a lot of the automation tools as well as the Kernel CI implementation the test and kernel CI testing that we do in this presentation Okay Disclaimers this is technology presentation. This is not a product readiness or road map commitment in the opinions here are Our own not TI So get mad at us don't get mad at TI All right quick overview Give a little bit of background a little motivation of you know why we're doing this as it relates to kernel CI And then talk about how we're doing upstream testing and this breaks down into a few buckets here You know, how are we doing the kernel CI testing? How are we doing? CI CD testing as it relates to upstream and then a full system test of our upstream builds and then I want to transition a little bit into You know talking about how do we get to a? Common testing framework, you know towards common testing. All right, so motivation. Why do we use kernel CI? Kernel CI provides a common platform Obviously to help kernel maintainers gauge our Lennox kernel And they do that by publishing the test results You know on the the website on the the the you know customer-facing portal We need to use that to get notifications if there is a regression We want to know about it obviously And it helps us gain confidence about our builds since the images are built outside of TI You know it sort of helps us make sure there's been some secret thing hidden in there that makes it work inside TI Whereas it's gonna break outside TI So quick aside on how we do tests so as I said we maintain a test farm my team does We have two sites that have test EVMs one in Dallas, which you can see on the left Please excuse the cabling. It's utilitarian and then one in Bangalore on the right We have about 300 boards roughly split between the two sites Those are used for both Lennox testing as well as our toss You know the key point here is that it is Truly a global test farm so we can initiate tests from one site run them on boards at either site in parallel Bring all the test collateral back to the original site. So it really helps with our Parallelism and scalability across, you know all these different test platforms Okay, so let's talk about the upstream testing Start with kernel CI So the upstream testing we're doing in kernel CI we do automated build a test collateral So we build our boot images locally But everything else is built at kernel CI and we use the API to you know pull it down to our site So, you know the the build has the kernel files the DTB files the file system And then you know we pull that down from the kernel CI cloud to TI and run those tests on our boards in our test farm You know in Dallas in Bangalore The results are automatically, you know collected and uploaded to the storage site on kernel CI for web-based review So pictures worth a thousand words it kind of looks something like this you have the API server the storage server Web dashboard query the recent kernel build run the tests on the TI farm Publishes us up to the storage server and then make that available, you know through the kernel CI web dashboard So what is the status of this right now? So we are running nightly boot tests We would like to get the LTB and kernel self-test those will be coming soon I think Venus is back in the office next week. So hopefully very soon We are currently running 13 upstream SOC's and that covers both our satara platforms and our Jacinto platforms And you can see the the you know the web screenshots there probably looks very similar to what y'all are used to and you can see that You know green is good, but it's not always green Okay upstream CI CD testing Our upstream CI CD testing does end-to-end automation of the latest upstream build test and release so the builds are Automatically initiated test plans are automatically executed again on the boards the results are tabulated into a report and The build collateral and those test reports are then automatically pushed up to a customer-facing portal So the way this looks you got a next to pride branch Do the build assuming that passes you promote to actually run the tests push all that collateral out onto the public portal if For some reason the build fails then that branch essentially is dead doesn't go anywhere until it's fixed And then the next cycle has the next set of results So these execute the CSED tests execute nightly The tests that we run tend to be a more expanded set of tests beyond what we're doing with kernel CI They tend to include device driver tests that are a little bit more complicated, but you know still fairly simple You know localized to a board One of the challenges with this since we do want to run it nightly is we tend to have to make trade-offs You know between the the test coverage versus the amount of time it takes to execute You know so we want to run this thing in you know, you know three four hours tops Across you know all 13 so see so you know that time tends to add up So the test reports that we have are really useful for our developers you can see that You got you know green for passing red for not passing And it gives you the ability to drill down into an individual test and Drill all the way down to the log file to triage and figure out you know what went wrong So for the the upstream snapshot we take As I said all the build collateral and those test reports push them up to our software DL website For anyone who wants to look at them to go see Much of the content is similar in the sense that you have all the SOC's that are supported in upstream You can drill down to an individual snapshot down to the test results for that snapshot You know all the way down to the log file to you know dig down and figure out what weren't wrong if a test does not pass Okay full upstream test So We recently started running full system test on our upstream so we automatically Generate those test plans based on requirements. So this is a requirements based testing strategy As you can probably guess that can tend to generate a lot of test cases So hundreds per SOC across all the SOC's we can easily run thousands of tests and these tend to be even Longer running more complicated tests than the CICD Test plan so we can have you know device driver tests that actually require the device to be plugged into the SOC, which is a challenge you can have you know test cases that you know could run for 20 minutes 30 minutes an hour I think we have a test in there that does a thousand boots. We can run for several hours So these test cases take you know upwards of two days to run in some cases. So it's a You know a very very long cycle We've implemented Automation to flash the SD cards And obviously there's automation to collect all the results tabulate them together generate reports for all the SOCs So as I said on a mega generate test reports, you can see an example here These are not published to the community today That is one of the things we would like to do if it makes sense One of the you know key highlights here versus the CICD Which is sort of a functional testing is these test reports do link back to requirements So you can get an idea of you know, how much of our requirements are satisfied with actual passing test cases and again from a From a triage standpoint you can drill down to the individual Test case results all the way down to the log file figure out what went wrong if something isn't passing Okay So to recap where we are We are running the kernel CI tests. So those go out to the kernel CI upload the results to the web dashboard We also have these sort of private builds these private tests for CICD and system tests Some of those are supported uploading into the portal and I suspect at least in the conversations with folks over the last few days that y'all have something kind of similar a Lot of you kernel CI tests Maybe some tests running privately Maybe some of those results are uploaded somewhere where they can be viewed But it's it's it's a lot of similar things implemented in different ways One of the nice things would be Can we get to a place where all of that kind of testing is common? So we're all doing it sort of the same way and Can kernel CI sort of be the umbrella to sort of bring that together a sort of a foundation? And I will pass it off in the shunt Dig into those details a little deeper Thank You very Good can hear me why I mean why would we care about doing this? Collaboratively we all talk about doing upstream testing. Why can't people make products from upstream? Simple they don't trust upstream it all comes down to this little keyword over here trust. I Don't trust the quality of software. That's the I can't trust that you won't regret in the future I can't trust the features that you have in the the solution that we are proposing here Involves the standard stuff to manage trust you standardize stuff so that hey if I see it on one vendor Is the same on other vendors too? You decentralize if I am able to do it you can replicate my results too If it's like it right look at the shark on it. Nobody has injected a new problem into the system Reporting can be trust the reporting that you're doing. There's no bias in the reporting What if you are changing the performance number a little bit skewing the result towards your product, but that that's not really true transparency is the key Today we don't have that transparency currency I is moving to the transparency What can we do next stage? so I understand most of the folks who Work on Linux our focus not purely on Linux, but the guys who are actually creating products They look at the product solution and most of these systems are heterogeneous in nature. You have Linux You have our tosses like free our tosses sapphire thread eggs. There's so many different noises coming to the They all have the same problem. How do I test an interface? How do I test a peripheral and camera sensor? For example, it doesn't change whether it's with ti and xp Adi, what are we interfacing you're using? If you're a seed like if you're using a serializer DC lights of it like fpd link is the same thing Why are we testing and differently if you dig down, right? I like to use this little thingy on the side. We all use different languages. This is the standard communication Delta We all would have sung Kumbaya if you were all talking English together But it's a diverse world I get it but if the communication were to get standardized between this it would be much easier for folks to reduce the cost down and Move between different vendors easily There is limited value in looking at testing as something of a differentiator. It is already commodity Let's come out of place it There are a few topics that we kind of put on here But it's just examples of various problems that we have some folks like lava. I'm a big lab grid fan. Sorry Barry He likes lava There are different frameworks that people have created ti decided fork LTP Linux test project and added device driver test Framework on top of it which ti Managers year on year. We have to rebase. It's completely out of tree solution We also on the whole side created something called a VATF There's actually two different frameworks that we have on the PC side Because many testing involving peripherals have to send something from the board and has to be cross verified from the PC side as well When you look at industrial protocols, for example, yeah, maybe you have a big list of things to do They're all standard problems. We all repeat the same solution, but in different ways Decentralizing and this is key. You saw the farm that ti has right. It's huge like we have specialized infrastructure backup power supplies all kind of crap going on But there is real estate limits how much we can grow today. We have 300 maybe we can go to 800,000 boards But that limit is coming soon. If I look at these products or 10 years We say in Linux. Hey, we love Linux We support any product that people use for years on the go really do we Do you do testing for the same product 10 years from now the way that you're testing today? most of us don't It's a matter of economics The only way to spread this cost out is to let the guy who's interested whose stake is high to give the importance that he wants Instead of doing this centrally Currency I does that today. It's already decentralized. There's a framework. There's a standard for us to do it We can leverage it to the next level So many things if you guys are in corporate world, you know that our business priorities change Ti has a lot of history of doing that often Our omap processors are no longer Actively supported. There's a lot of folks who have complaints about it. There are other products that we have slowly scaled back But there are real products out there that actually use these processors. What do they do? They have no way to replicate what we have done internally This gives them an option of doing that Anyway, long story short. There is so much duplicate duplication that is going on here The other thing which is very scary to my marketing people at least and again, I'll claim the disclosure I'm speaking as an individual right now Nobody likes to publish performance results out there Without marketing team review. Why because a D MIPS number change is millions of dollars lost But guess what? There's nothing hiding it. I can run Tristron as you can you'll get different numbers But if you could reproduce the same test results, that's great. We are scared about saying hey, we failed at this case It's normal There are regression like for today for example You boot master branch is completely broken None of the K3 devices boot today morning Yesterday it was today's broken. We know internally because we are monitoring Yes, we are monitoring both you boot master and you boot next and we know masters records and we know why by now There's nothing wrong in reporting that the other thing is The quality of reporting as well if you are sitting so the way that we looked at this problem is if we think From external out in what would you like to see? How would you trust the data to be able to create a product based on mainline colonel mainline you boot mainline TFA? everything mainline Call it your master directly You need to be able to reproduce the data. You need to be able to trust the trust the test reports that are coming out But there's no standardization. You're entirely dependent on TI's translation of the test report if we say that I2C's function you can't replicate that you have to trust us long story short Currency I today is solving a humongous problem already The infrastructure for that a distributed reporting system the amount of platforms that they're trying to solve is already a complex problem But that's just the tip of the iceberg The real thing that we as embedded developers have is the platform sensor the products themselves Here's a scope for us to grow further down buddy so Call for action common testing framework beyond Magnus kernel as much as I had leveraged the shared solutions, you know kernel CI can be that umbrella You know, it's it's part way there You know, it would be a good place to to congregate to do that. So, you know, our You know, our challenge is how do we work together to make that a reality, you know, and that's you know, that's kind of what we're asking But there's called action Okay, quick. Thank you Obviously Texas Instruments Linux Foundation Colonel CI Lonaro Panganitronics Calabra None is what it would be possible without their help So big big. Thank you and Contact information Myself and the shunt mean us emails and IRCs, etc. And We will open it up for questions We really wanted this to be like a conversation where What stops us from doing this? Okay, so I can tell you what's stopping me from using kernel CI right now and that's that I'm waiting for v2 to drop It's in its next generation phase and I'm a little bit worried that they're in a Mode well, I talked to you about this in the hallway I'm a little bit worried that they're in a mode where the next Version of it is gonna be like a kitchen sink version. They kind of have second system syndrome Anyone want to comment on that am I wrong I could be wrong I Think we gave that impression and I think we were right in the direction we've been fixing this strategy for the past three months there and We have a system that's working. It's supporting build results departing boot results there is much better testing for device probe for example enabled and We are scaling up that like slowly like involving a few maintainers Hipporting a lot of this they stuff like manually at the moment because we want to we want like a data that we can trust And we want like to not make the mistakes of the past system that was just if you if you start like looking To how the test results is a big mess and in the in the legacy system because we can't really understand like the bookness of the data so we're spending a lot of time on on on that process and More communication is coming up. We had like some blog post last week There's a bunch of other things on the way There is a community engagement working group that has been set up and Shua Khan is going to join us to leave that group So that's kind of the new states of things. I think that that kitchen sink is no more You say something who's a request race ability right and There is a tool called ballot that has been starting to get built and this will probably play a big role in As putting that in a story some toda along with trace ability that That's not the point which I want to make I was more on the Sigh where you had an old one slide where we're showing all the different kind of like like bridge or if you go With whatever you like It's not a little bit like you would like to look for how can we get more standardization? But the question is would it make more sense to go on like this is the agreed Standard we are talking not about the tool which we're using I'm asking this because I come from an environment where I have LTP test cases. I have robot fresh cases. I have CTS BTS STS test case from Android LTP I have all these different kind of things I don't want to spend any effort in migrating one test framework to the other But the nice thing about the test framework if they come up with saying I report a test case It goes with a zero one minus one whatever they have as a result and we directly translate and create like a mapping on it And with this we can see independent which Testing environment we are using we get a comparable list of results for the reports because at the end you want to see a green red cross check whatever and maybe find something like configuration and Often to try to get a standard So I see the Jenny V for example Jenny V started with generic IVI system There was so much discussion going on they couldn't agree on which audio manager to use which framework here or there Other parts there was no strong alignment with a GL What turned out is that the Jenny V changed into a coviz are where they do a lot of vehicle signal specification and this vehicle signal Specification give the room to say we are discussing. We're talking a similar language. We have an interface open standard and I Would see this is much more a way to go and they get the traction and it helps that OEM suddenly work together and say well, I need these kind of signals You need that and we make this as the interface and this could be an interface also Torx Colonel Si or whatever? Three different problems that you introduced but let's talk through them In terms of information exchange that has to be a standard And that's the best way to do it have the abstraction at API or data exchange level and we need to do that step number one step number two is the Even if we standardize something there will always be someone who is not lining up the standard There'll be some BTI be with anybody in that case. So there's a cost to move to the standard Why should I do that? Be the same source code beat a different source code or beat the same source code the translation has a cost We'll have to justify that to our management. Everybody will have to that the standard actually has value by moving to this API standard the third vector standardizing the test code itself There are benefits in some areas where they are commodity but as we mentioned it can be a very hotly debated topic and We know the VI versus emacs discussion, but we won't bring that up, right? Again, we have not set up the working group yet And this is part of the discussion in terms of we would like to do that and we would like members to come and say Where do we want to draw the line? We don't want to come and say okay. We are willing to provide the code that we have right We'll dump it out there and say this is what we do you guys want to use it Do you want to write rewrite it or do we reduce the scope to just purely API? From where I stand I Feel there's a lot of value in sharing code Directly and contributing together because from TI's perspective There are vendors like fight echo here who work across multiple associate vendors, right? There's companies like Toradex who do the same They all invent the same thing And there's a lot of value at least at embedded level that we can share and start differentiating at all Now there's a cost associated which means I will have to rewrite a lot of stuff on my end From my perspective it pays off in 10 years. It's worth it Now, are you convinced that's a different question? Okay, as you know, I've been You're preaching to the choir or maybe singing to the preacher here So About six years ago. I had Fuego adopt the kernel CI Format as it's out as it's native output format. Well, he's translated into json And then Colonel CI changed so that was a bummer because it was a lot of work and so If we adopt something It would be good if we kept it The only way I think you can deal with this is you're not going to go back to things like LTP these massive suites that already exist and Have them change their output format and and so I think you have to have you have to have a parser per test or at least a parser for per Test output category. So that's one of the reasons we pushed k-tap Into the kernel k-tap is not the greatest output format but it's good enough and particularly for the types of tests that the colonel was running at the time we adopted it and It's uniform, right? So and I really appreciate the efforts by the way to go into the existing kernel tests and make them Compliant with that output format because now you can at least run k-tap on you can run a tap parser on the kernel and put it Into whatever you want the Linguafrank would it be for for test output? You can do the same thing with LTP without you know, you make a single parser for LTP You get a huge amount of win off of all that But I would love to see a single common format Maybe kernel CI as it is today is now that current format I don't I haven't looked recently because it scares me to have to go back and rewrite my My generator again their generator was one of the hardest things to write To try to do do that in Fuego was to write a generator that could handle all the different cases And I have all these parsing modules But I would love to have them land somewhere that other people could reuse them, right? So I ozone results I can put out into old kernel CI. I can take Bonnie results I can take stress test results and we just need to Find a place to put that stuff, but right all my parsers are Fuego parsers All right, they're in Python. They just take output and convert it into a Python module But we need to have the discussion and we just need to pick one and actually start doing it But the harder part is not coding. It's it's finding someone who's gonna host that and You know, you have to have a place where this code can live This has been a big talk on the current cycle meeting the past like a few weeks and yesterday We spent like a few hours on this topic like not only test specifications per se, but the work texture of what currency I will be for now on and We feel that like how this how the different CIs they have like their own test specifications They're on like they start in their data that and we want to move that out of the different size systems And maybe move a lot of that inside the current tree like that's part of the effort that they're pushing with CII improving the test quality test quality of things like and There is a test catalog a proposal from from Nikolai Kondershov as well But how those things I think are taking us towards like a more common place to start like a Not on the tests the the specification for the tests the kernel configs like which bars should be running which things So I think that's that's where Kernel CIs is building a lot of a lot of grounds for people to come and share that I guess With all the power thing and adjusting to something which is a project at the question if this is really The way where you want to put your bad on if I think for example on container framework There's a large set of container frameworks available different kind of containers But there's also the OCI compliant container the open container initiative and what I hear in the industries I'm talking to that they say okay We may have like there is the LXC which is not compatible where they have certain use cases where they accept it And because I say that's for a performance reason or so we need to go with it But I know a lot of other places where they say we will only accept new technology which is in accordance to an open standard interface and then it may be better to have like an open testing infrastructure testing interface whatever which could be hooked under Existing an eye environment like Kernel CIs about having more open and of course you will also find then even in OCI compliance decks extensions so Where you just say okay the OCI is not fulfilling all my needs I need extra features I need extra things, but they are not for all cases Which allows you also differentiating parts maybe for testing is less on the differentiating elements But you can have all like extra fields You see this also with SPX where you have like mandatory things extension things You're more compatible as compatible and I get this is for me the perspective where you leave a lot of openness And you bring it to the standard Our framework and then you have maybe parts. There's adapters and so here this is how I can have the if we go to the Intermediate parting competing compatible with kernels the iron so on so to just have this as an interface and not changing all the Things again and again because one project decide that we change it to a different format because we like it better And don't see the full ecosystem of all our users I mean that's that the thing which you have in a traditional way also when you do Open source or a fork open source projects utilize them, but you don't give back your requirements You don't give back your modifications There is a strong risk of breaking with the upstream and this would be something similar like we have all this different kind of test Types all these different kind of things they all need to be then aware to the kernels CI environment and kernels See I need to know how they are used in order not by mistake breaking something and getting good then all the feedback of the Community and maybe this works more if we say here We have a standard you comply to issue change it and then we have like a governing board on the standard Which is not like a collecting pool of that's where we have the result or Texas test execution It's going to be a journey the first thing is to say that you know we can If there's one message we were to take from this presentation, right? We can go and attack the easy problems right of saying that a currency I is let's solve this problem of reporting and Reporting of boot systems for the maintainers. Yes, we can maintain some pressure that quite a bit. Thank you But the scope is much broader the real problem said that we have to solve this much broader It will take us a few years to get the we'll make mistakes in the route But this is where the collaboration comes into play. What are the right tools for us to use? What are the? What are the steps that we should really take and Do we constrain it to purely criminal or do we want to extend this further beyond current into other open ecosystems? Zephyr can be here can also benefit out of the same thing. I know the folks who like Twister and you know pie test You would use a spy test Zephyr uses twister. We use our own colonel self test k-tap Whatever that standard is right. Let's talk the same language as step number one The intent is to start start that migration start that discussion Let's look beyond what we are doing today And if you folks are interested in joining us in the journey We would love to talk Love to hear your ideas. We don't have a monopoly on ideas. We need your ideas to any other questions Comments thoughts So Gustavo and The girl CI team is here. You have our contacts as well. We are on IRC too. So you guys can just if you Are wondering how do you propose this to your management and you need help come talk to us? We can work informally. We'll try and figure this out Thank you. Now before we start I would like for everyone to be on the same page So that we have a good understanding and there's a brief background So let's take a look at the energy sources for embedded systems Energy sources for our systems can include a direct input via DC or you can have USB power systems even either net, right? Another class of devices would be battery-powered devices for example lithium-ion or a CR-2032 battery and A very interesting class of devices would be energy harvesting devices So these examples serve to illustrate that we can have multiple sources of energy in the system What is common to all of these classes? What is common is that we want we want to use the Maximum no you want to use the minimum amount of energy to do our task Why is that? Why do we care about using the minimum amount of energy? Because energy capacity or also known as battery capacity is a fundamental constraint in the divide in the design of any system For example microcontrollers or cell phones are constrained by the applications that they can run because of the amount of energy That is needed to accomplish the task With that in mind We have come to an understanding that we have a finite resource How do we manage it? Let's take a look at the problem statement first To introduce the concept I would like to bring to your attention a very similar concept of performance optimization If I ask you to optimize the performance of your program What would you do? You would say that okay? I'm going to measure the latency of the application using a mature tool for example Perf and I will obtain a metric for example seconds or CPU clock cycles and Then I will try to figure out a way to reduce this metric right now Let me ask you the question that is the focus for today What tool would you use to measure the applications energy? Any ideas? That's a great tool. We will we will discuss that. Thank you for that. I really like it. Thank you okay, so Before before we go deeper into this how have we come to this question or how did I come to this question? well many many years ago in a very faraway land there was a undergraduate electrical engineering student and this student was as Engineering students do studying at the last moment for an exam The exams tomorrow the students panicking today in the library Trying to figure out why the laptop keeps dying every hour right and In classic undergrad engineering mindset this student thinks hmm If I kill the applications that are using the most battery, I will be able to study longer and then I will I'll be able to pass so So this student does this this student tries to do this exactly He says okay, I'm going to find out which applications are taking the most battery and He cannot he cannot because this data does not exist and That is why I'm here talking to you today So towards the end of this presentation, I would like to I Hope that we are one step closer all of us are one step closer to answering this question We will not get to the answer, but we'll be one step closer and that's my hope Now let's try to keep thinking how undergrads think The undergrad said okay energy can be calculated using these two simple inputs power and latency The power can be obtained from the CPU for example the rappel interface in Intel tells you how much the power how much power The CPU is using exactly at that point of time You can also refer to the data sheet for specific systems Let's assume that the CPU reports you're using 15 watts of power The latency as you saw before can be measured using a tool such as time or per And for the sake of this calculation, let's say that your task takes five milliseconds of latency and Great, you know, we have solved it. We have done What we set out to do energy is equal to 75 millijoules Unfortunately, this is not the truth. This is very far away from what is actually going on Let's try to peel the layers and see what is going on here in the system The first oversight in our calculation model is that this is this calculation assumes a linear power draw I took the value of 15 watts and I assumed that the CPU is running at 15 watts for the entire five milliseconds of time That is not the case. Here's a slightly realistic CPU power curve You can see here that we have a lot of valleys and peaks in this curve And if you happen to measure the power at a valley, you would not even get the full picture. You would run with a Much smaller number than what is actually being used So the first limitation is that power is not constant over time and there can be a number of reasons for this, right? You can have power gating. You can have different types of instructions. You can have Different type of workloads, right? So we need to be very conscious of the power draw at every step at every every time in the workload second our calculation model assumes that we can get the number from the CPU and Unfortunately, this is also quite restrictive because we don't consider devices such as ED RAM Our sensors which are pulling every five milliseconds or the network interface which has an antenna which is constantly burning current so We have experimental data which demonstrates that these devices can often dominate your entire energy consumption Because the CPU peaks but these devices run all the time and our data is also corroborated by other research sources In a number of research papers and books So that is the second limitation that we need to break out of the CPU dominant mindset third The interface that we used to get the power value the datasheet or rappel is available only on specific classes of processors So it might often turn out that you don't even have access to this data in the first place Sometimes data sheets have conflicting values. I'm sure some of us might have experience with that And that brings me to the third limitation. We do not have a uniform interface or a reporting mechanism To make sense of this data right now, I would like to summarize all of these problems into this Nice crisp summary, which would be we are inaccurately calculating only a fraction of a highly specific systems actual energy consumption And I think that should be improved This is a complex statement, right and I would like to so I don't want you to focus on the complex We would like to simplify. I want you to take away this nice summary of this statement We cannot improve what we cannot measure. So my work strives to measure better. My work strives to improve our measurement infrastructure How are we going to do this? What is the goal of this project? My the goal of my work is to develop a framework to accurately and reliably Measure the energy consumption of a process on Linux Right now once we have this With this infrastructure, what is the use of this? What is the value that we provide to our users? Well, once we have this data, I want to report it to the end users So that our undergrads can study for their exam more Nicely so that they can make sense of it in an easy to understand and useful format We want to expose this data to programmers We have command line apis that improve their action ability now action ability is a loaded word here We're going to get back to that and see what exactly I mean by action ability And finally to system designers We want to give tools to system designers to enable them to iterate and explore the design space and pick a point in a design space intelligently Now this sounds very lofty. Let's try to break this down into exactly what we're going to do For that I would like you to focus on this term framework In the context of my work Framework means a set of models and tools What is a model? A power model is essentially how we reason about a device's power consumption So when we picked that number of 15 watts In our mind we had a model that the cpu is burning 15 watts for five milliseconds, right? All of us use models for everything in life I want to build better models for things that are poorly understood Some examples can be DRAM or network interfaces It is hard to reason How a device uses power if I don't understand what is actually going on Once we have this power model What we want to do is we want to build tools which can accurately calculate the power Based on these models One example of such a tool would be nvidia smi utility that is provided by nvidia for its gpu's The tools come after the models So in summary We need accurate models And reliable tools To calculate the energy consumption In a sane manner With that said and done Let's take a look at how people have tried to solve this problem so far In the embedded world a lot of people probe the hardware What you would do is you would take a set of a tool like a multimeter or Something similar and you would put the probes on the wire Either on the direct supply or on the specific component And you would see okay for for this particular duration The multimeter reports me x value Now this is a reasonable way but it's not scalable Because you cannot always have access to the device For example, if we consider a large-scale deployment of thousands of devices across a wide geography and one device malfunctions I cannot always run to that device Right so it would be ideal to have a software-based solution that can pull the device and send me the data over the network So I can fix it remotely Now how would you do it right now? What is the current state of tool? What is the current state of art? As we saw we have a tool called power top And here is a illustration of power top running on a system live This is a screenshot. This is not live And I would like for you to focus on two key data points The leftmost column here stands for power estimate And the rightmost column Tells you what is the process or the device driving that power estimate? Okay I would like for you to focus on one particular data point in this screen That would be 1.45 watts For gnome shell. Okay, we're going to see why that matters in just a little time So it is possible to use power top to view the power estimate of a process a device an interrupt Or a timer However, what did we gloss over here? What did we miss? First of all power estimate is a discrete time event Remember, we're trying to calculate energy And energy is what your battery has Energy consumption of your application has a higher correlation To your battery life as compared to power at a discrete point of time So we want energy estimate Second Power top has vendor specific implementation It would be nice to have a general generalizable tool And third Remember the two things that I asked you to focus on actionability And 1.45 watts from gnome shell Let's take a look at why that matters now What is the actionability of the data that we just saw? We saw that gnome shell takes 1.45 watts If I ask you to optimize gnome shell and reduce this Do you know exactly what you must do? Do you know what steps you need to take? And that's a gap So we want to provide Insights to the programmers that enable them to reduce the energy Right? Right? And that would be really helpful Now We saw our problem statement We saw our goal to calculate energy reliably And we also saw the challenges with the state of the art Let's try to take a look at the system design that I would like to build Or that I'm working on On the screen right now you can see a simple flowchart of it's a very high level System design for what we're building The leftmost blue boxes represent the inputs The brown box represents a regression model And the green box is the output Which is what we're driving for Essentially this is a regression model and regression comprises two inputs You have the regression parameters And you have the inputs to the regression How are we going to get these parameters and these inputs? Let's take a look at the parameters To get the parameters for the regression model we have an algorithm What we do is first of all we take the system at the baseline state Okay, we stabilize the system and turn off everything as much as we can to determine the idle power This is very important because uh this is the power that your system is going to burn anyways Okay Once you have reliable reading for the idle power What you would do is you would Turn on one device Let's take the example of the backlight the screen You turn on one device and you would observe a fluctuation on the value and that fluctuation is what Of course, I'm talking about a stabilized value. I'm not talking about the transient That fluctuation is what matters to us because that fluctuation is what the screen is costing me at all times Once we have the baseline for the screen What we would also do is we would sweep because I understand that the screen at full brightness takes more power than a screen at low brightness, right In my mind that mental model always needs to work And we sweep this value from low to high To understand how much the fluctuation varies And all of this gives you data points at idle rate at low brightness at high brightness And what you do is you repeatedly correct collect this data for different devices All the target devices in your system Ideally you want to turn them on off and sweep them And this data can allow you to solve for the regression parameters This gives you the regression models in the parameters. That would be the a in the ax equal to y Okay, so we saw how we get the one time measurements And that gives us the parameters Let's take a look at the third blue box on the center the process accounting infrastructure That gives you the input variables So our goal is to combine the inputs to the regression model And from this the algorithm goes like this The first step is that we identify the PID and the group processes We have a workload that workload has a identifier and the PID is used to identify it We pull the process accounting infrastructure or as it is known as popularly slash proc For this PID The process accounting infrastructure at all times logs the resources that are being used for this particular process That would include the jiffies or the time The network connections or the sockets the file handles or the disk usage The memory set size that would be the DRAM usage And we obtain this data One very interesting data point here is screen backups It turns out that if you frequently cycle through screen backups You're going to tank your battery That's not good. Don't do that okay, so Finally, let's say that you have this data over entire time, right? Let's say that you do the measurements for 10 seconds But your process was not running for 10 seconds We live in a multi processing world And in a multi processing world your process runs for some amount of time before it's being Before it is preempted. So we need to calculate the fraction that must be attributed to that process Let's say that your process ran for 1.5 seconds over this 10 second period So the fraction here would be 15 percent So I can attribute only 15 percent of the systems power used to this particular PID right And this data gives us the input to the regression model Once again, we summarize we have the device specific measurements for the parameters We have the inputs. We have the proc interface And that gives us the data that we need to predict an energy value Now are we done yet? I still have a little bit of time. So we're not done. You're not done. Okay What we just saw is a system design I want to discuss with you the challenges and things that I would appreciate your feedback on things that you can ask me questions on First of all This system predicts an energy value. So at the end of the day, it is an estimated value. It's not exact ground truth Okay, and I would like to Add this line. I I like it. It's very popular in the machine learning community All models are wrong, but some are useful and my hope is to build a useful model Okay Second challenge There's a bias and accuracy tradeoff Now if you think a bit Carefully, what you're doing is you're running a process on the system. That's your target workload and you're adding additional load Your your your measurement infrastructure is also doing some work on the system now So you have less resources available to do your original task And if you have a very complex model Which is super accurate Which takes in a huge amount of parameters That's going to create a huge amount of load So essentially the readings that you get from that super accurate model would be biased So we have a pareto frontier here between accuracy which would be a model with a large number of parameters and Bias And what we want to do is we want to sweep this this frontier to identify a sweet spot Between the accuracy and bias that the system can get us another challenge data collection The linux ecosystem is extremely diverse, especially when it comes to embedded platforms There are millions of devices thousands of vendors billions of ic's Very often even data sheets are not available for some of these devices And the power estimates can range across two to three orders of magnitude As an example, you can have an edge device which runs within a budget of few milliwatts and you can have a server class data center A server class data center device which would easily pull in a thousand watts So we see a huge difference between the range of the value An error of one watt for a server would not matter much But for a milli watt edge device one watt is going to fry it right, so We need to be very careful With the accuracy that this model gives in the context that it is running Being mindful of the context is critical here And continuing on the same line of thought There is a significant difference between the estimated value And the actual value that there is always a ground truth How do we identify Diversions from the ground truth I cannot always probe it Right, we establish that it's not always possible to probe it manually If I cannot obtain the ground truth Is there a way for me to identify a diversion from it in another way? I would love to know about this. This is something that is super important like we need to be sure That the value that we are seeing is useful relevant reliable and privacy This one is a bit more Let's say philosophical but I would love to get inputs Who should share this data? If you're not familiar there is a significant amount of research which demonstrates that I can identify what exactly you're doing on your device by looking at your energy and power patterns I can identify you're running firepox And a particular website on firepox by looking at the instruction trace and the power patterns So how do we collect this data? Because we do need the models. We do need the models to get the energy values So what is a reasonable privacy conscious way for people to share this data? And once we have this data who owns it? Who manages it? Who profits from it? Who organizes it? Very difficult questions great, so Few challenges that I'm struggling with Let's get back to the good stuff We see a lot of talk about carbon emissions We see a lot of talk about how do we power these LLM models and how do we constrain the carbon emissions coming from these LLM models Let me ask you this. How do we calculate the carbon emission of an embedded platform? And I'm talking about operational carbon emissions for the folks who are in this business So here's one way to calculate the carbon footprint what we would do is we would take the energy consumption And we would multiply it by this this interesting term called energy composition We just saw how do we get to the energy consumption? What about the composition term? The energy composition refers to how the energy that you're using to power your device was produced Okay, it depends on the number of factors which include Where the energy came from How it is stored What was the geography of your operation? What is the time of use? If you have a device operating in daytime from daytime power, it might be powered from solar And also great loads. So this is a complex electrical engineering problem But fortunately we have a lot of good people working on this there are libraries which can tell you the energy composition Assuming that you can provide them their inputs So I would like to constrain myself on the consumption problem My project focuses on consumption and I trust great people will solve composition So Let's get to the goods What is the product? This is the product for our undergrads I want our end users to have useful data That enables them to Do something in their real life We can see here that we are able to report data for our device Or a particular application I believe this would be very helpful not just for undergrads for everyone, okay For the programmers What What is the end goal is to have a command line api That indicates what is the energy value for a process And one really nice really cool use case would be that let's say that you're programming in your favorite ite And you write a triple nested for loop And immediately your ite says oh man, this is going to take a lot of energy, you know, maybe consider rewriting it I think that would be fantastic. It's it's an ideal end goal But i'm working towards it. So I want to have energy efficient code optimization suggestions live in the platform to improve programmer actionability And for our system designers We want to help them iterate in the design space the design space comprises performance energy efficiency carbon efficiency cost People typically have a good view of cost You also are able to iterate over performance and I want to open up this axis of energy efficiency Which can also help with the carbon efficiency So we want to enable device we want to enable system designers to pick devices Or to iterate over devices with different energy consumption values based on the power budget that they have And explore the design space better than how they can do it today so That is the product and that brings me to the conclusion of this talk First of all I really appreciate you showing up I understand that This was a bit too much and I would like to summarize this talk into two key distilled takeaways Like forget everything else if you don't forget, but if you forget everything else I would be very happy to remember these two things that I helped you get closer to today first We cannot improve what we cannot measure And this applies to many things in life not just energy And second we need to break out of this CPU dominant mindset Right like the CPU centric mindset in life that CPU is everything and that's all we need to focus on Because we have devices that are drawing enormous amounts of energy Do not quote me on this but in data center design Memory has already eclipsed CPU as the dominant energy consumer Do not quote me on this I hope we are one step closer to answering the question that we started with A tool that we can use To measure the energy of a particular process And that brings me to the end of my talk Thank you very much for listening. I really appreciate your attention And open the floor for questions Can you hear me? Yes, yeah Since I have the mic I'll ask the first question if you don't mind So at Texas Instruments also we have been facing this challenge where you did talk about systems In the context of a pc or a laptop where the peripherals are outside But today in the complicated ARM SoCs that we have Even inside the SoC there are many peripherals or many controllers whose power we need to measure So TI sort of came up with this power estimation tool So we ship out this excel sheet where we have an estimate of how much power each Peripheral would consume for example a GPU or a spy controller And at what capacity what clock rate etc And then you can feed in those values and then get an estimate of how much power your system would consume theoretically So I think it's a very interesting challenge that you have posed here and to go at the process level and Try to sort of measure that it's going to be a challenging problem The thing that you said About applying the model across embedded systems and server grade I was wondering do we really have to do that? Why can't we have different models depending on each system? So just to repeat the question for the audience. Um, first of all, thank you for the feedback. I appreciate the kind words To repeat the question for the audience Our audience member asks if we have to use the same model across different contexts That's a great question. I believe we do not need to use the same model across different contexts However, this brings the challenge that You need to get more data So the ability to pick a different model depends on the data availability for that context And data is not easy to come by in this world But thank you for the question. I support this. I'm with you on this that we need different models for different contexts. Yes Thank you for the talk. I think it's a really Nice idea I was going to ask it because in practice it seems like of course if you have a vendor like ti who is supplying these sorts of power estimates Your job at creating this model becomes a lot simpler in the case that for instance, you have some hardware accelerator and Perhaps you don't have either You know information from the vendor on what power it's expected to consume You know, maybe you can measure it yourself. The other thing that you may not have from the vendor is statistics on utilization of the device So for instance, I thought, you know Your measurement of network traffic is a good example of where it might be easy to do this modeling because for instance The number of transmitted bytes might be, you know closely correlated to how much transmit power is consumed by the wi-fi card But like have you considered some of these for instance neural net accelerators Where the accelerator may be doing some processing and it's a very opaque black box in terms of You know how this accelerator is actually utilized And you know linux, you know, whatever drivers are available in some cases don't report You know much detail about You know what the actual utilization of the transistors in this processor is So like have you come across those sorts of situations yet? Where it becomes hard to get that utilization data So that's a great question I would like to repeat the question for the audience the question is that assuming that we have a black box device which does not report statistics about use How do I attribute the energy in that case? That's a great question Um, if you remember we discussed this algorithm Of the baseline Even if it's a black box what black box means is that I lack the mental model of what is exactly going on But I can still observe the effects Of this device I can still measure my baseline Give this device a workload And then observe the fluctuation And that allows me to build a rudimentary Estimate of how the device will likely work Does that does that if I may give one example again in the context of a ti sock So we have linux running on the a53 cortex a cores And but at the same time we have mcu m4 mcu r5 cores coexisting at the same time So although linux has a driver called as remote proc that can help us load the firmwares There are other scenarios where linux may be completely unaware of firmwares running on mcu's that are running on the same sock So in that case, I don't think linux will have any knowledge Of how much power and how power is being consumed by the other peripherals so, um I understand I would like to Bring to your attention Something that is common to all of these systems. I'm assuming that these systems are being powered by a battery That is my assumption and often very often if you have a single battery powered source The battery always reports to you the amount of energy left in the reservoir You can still obtain a difference in the battery level By turning that device on and off And that gives you To get a rudimentary estimate It would not be perfect But it will be one step closer to the end goal Yeah, just to add to that. I mean, uh, there is one solution that uh, we sort of offer is in our evm's We actually have current sensors or inas to various rays like ddr Soc core etc So you can actually probe these current sensors Within the soc itself. So even linux can have access to how much current exactly the system is drawing So maybe that can help you in this research as well. That would be very very nice. I will take a look at that Maybe we can connect and you can point me to the resource. Definitely. Yeah. Thank you. Thank you for the questions. Great questions So regarding the problem of divergence Does this model expect the Power profile like the power usage behavior of a device to be consistent over the lifetime of the hardware and the device So to repeat the question for the audience Our audience member asks that Over the lifetime of a device We expect the power values to shift That is correct And I do not know how to answer that I want to but I don't know I'm not confident that I understand what you mean by divergence specifically But a lot of systems have temperature sensors if not power sensors and I am wondering if If you can use some measurement data to say like look where our the system is very hot But our measurement is consistent with like a lower temperature in the past. Is this like a source of drift Or or very cold when it's reading too high or something like the waste heat is a signal of energy use So to repeat We can correlate the temperature with the readings And try to get more sense That shouldn't be possible. I believe that is One approach, but I have not looked into it. Okay, but thank you for the input. It's really helpful Yeah, actually, uh, again, we have actually seen Difference in how much power a device consumes Just with the delta of four degrees Celsius. So if you operate at 26 degrees c Was a 30 degree c you will see a difference in the currents being consumed by the device So it is a very important and sensitive thing to factor in temperature Oh, it does. Yes Yeah Great point. Thank you. I really appreciate it How do you Attribute power usage of shared Preferrals like the screen or the backlight to individual processes So to repeat the question the question is A screen is a unified resource for different applications And how can we attribute? A single resource is powered to different processes At this point, I do not do that I Attribute the power of the screen to each process However If we build this model further It is possible to identify the fraction of the frame buffer that is being used for each process I believe it should be possible I think the valent subsystem I'm not sure but I believe the valent subsystem might expose this And that might allow it. But again, we have background and foregone problems as well. So I I do not Know how to attribute that But I would like to very much In the future Thank you for the question Actually, I'm wondering if the approach with these models where you then want to I don't know Account the power measure The amount of power that is used to particular events or devices is the right approach at all Or if we just say we observe and then we identify patterns which are good or bad And with these patterns over time, we try to find out why our I don't know like machine learning models can do that Which F which devices or which Sub-patterns have the biggest influence on either making requiring more power requiring less power So I would I will repeat the question to clarify because I did not fully understand the question What I understand is that you are indicating we could Look for patterns in power consumption instead of Exact measurements exactly. Yes Even to get a pattern even to get A pattern of energy We first need to measure energy, right? Yes, but we don't need to account that we we just take in all the measurements or all the sensors we have Like temperature power sensors also CPU utilization Amount of rights to this all this kind of data we have And based on that data we do pattern recognition So if I understand correctly You would like to have a mechanism which Does not calculate the energy, but instead tracks the symptoms to indicate a problem exactly is That could be quite useful however It does not allow One a few of the key objectives that this project tries to achieve and that would be measuring and optimizing the energy for a particular budget and also For the system designers To iterate over the design space It can tell them where not to go But it cannot help them pick a good point But it is a good idea. It is a more feasible idea. I agree Thank you for the question More questions, please Well, I'm going to run away after this. This is your last chance Okay Thank you very much. It was a pleasure to present my work I appreciate your audience And have a great day Oh, um, oh, sorry Cody and Shannon, I need you to act right Stop planes, gieball Start working a reverb. I can hear me I don't know. It's weird. It's fine Never giving this up Right Shannon Can I do something for tim bird's discretion? Yeah Yeah He enjoys hot tamales. It's been a minute since he's brought them Oh, there it goes Okay, welcome everyone to the closing game If you've never been to ELC before You may think well, this is a strange turn of events But this is Something we do just kind of for fun at the end of The conference, it's a tradition we've had for many years and so I am gonna these gloves are like super tight They said one size fits all but my hands are maybe too big. Anyway, but Thank you for attending ELC. Thank you for coming out And I hope we have some fun. We have some great prizes I need to do a little bit of housekeeping before we get to the game portion But let's go ahead and get started So like I said, I'm going to take these gloves off Otherwise my well There we go And then I also need to wear my regular glasses, which is too bad because it ruins the effect. Don't you think? So The first thing is a lot of thanks are due to a lot of different people that make this event possible Um, so first we have the sponsors a big shout out to them. We could not do an event in Like this with this nice venue and everything except for our sponsors. So we want to thank our diamond sponsors intel and ti and Really appreciate them also our platinum sponsor midakura and Our gold sponsors lanaro nxp and yakuto project And then our silvers analog devices canonical agalia and wolf SSL. So thank you to all of those so but The money is important, but so is a lot of the behind the scenes work. So we want to thank our program committee Jeff ozier mixon thomas petizoni some of who are sacrificing Those two in particular are sacrificing their opportunity to win prizes by being judges today But we also have kate stewart. Uh, yoshi kubayashi Marta rushinska drew fustini Walt minor and myself. So that sounds seems weird to give myself applause And then of course the speakers The speakers really are the reason we come the preparation that they made to research technical topics and present things that are working on Is one of the more important reasons we come together and then you as attendees The hallway conversations that we have all of that goes into making this A fun and useful event in terms of your role And I would be remiss if I did not mention the lynx foundation staff There are some of the best in the business And They really are some of the best in the business if you've ever tried to run an event It is there's so much stuff going on behind the scenes So now Speakers you are supposed to turn your slides in on monday, but if you have not Please get them on sketch.org. There's a couple of ways You can manage sessions on your session page You can if you can't figure it out just email them to cfp at lynxfoundation.org Please don't make us hunt you down for some reason. There's always like one or two stragglers And it's it's a pain the all of the eos sessions were recorded And those will be on available on youtube look for the lynx foundation playlist So, you know, sometimes it's like a three-ring circus If you need to go see Talk that you couldn't go to because you had conflicts It'll all be available And I think that's a really remarkable resource We do have a presentations page. It's already been set up. It's got nothing on it right now as of today But as the We'll organize that page and put the slides and links to the videos on that page when they become available so And then future events so coming up in vienna I know this is a little bit of a change of plans from what we've announced previously, but there will be a mini elc in vienna in september And it's going to be a little bit smaller only two tracks on two days So a little bit smaller than usual But the cfp is open now. So we encourage you if you have interested in In presenting at that the deadline is coming up pretty soon. So it's april 30th So only a couple of weeks And we've one of the reasons we went a little bit smaller is we didn't want to overlap with plumbers And so plumbers is the end of that week And so That's that Okay, so now it's time to play some games Um, I like games where everyone has a chance to win. I was just talking to Some of the members of the lf staff We used to have games where we would just invite a couple audience members to come up and play But in this game everyone in the audience has the potential to win There are two types of games. We have skill and luck skill so-called We will the the basic idea of the games is that we narrow the contestants down We start with everybody in the room We'll have you stand up and and then if you get an answer wrong It's a trivia game if you get an answer wrong we request that you sit down And then as soon as we've whittled it down to A number less than the number of prizes we have left We will we will Announce the winners So what will happen is we have some judges in the room That have no slips of paper and it corresponds to these pieces of prizes So they'll these these prizes on the table And then once once we're done handing out a round of prizes we'll restart we have multiple rounds of questions and Multiple games. Oh, okay. So everyone should have a set of cards They were placed on your seat a red card and a green card And it'll be obvious what to do. You're supposed to hold those up And our judges will be checking to make sure that if you got the answer wrong Use it down Let's see. Okay. So what have we got for prizes actually more prizes than usual Very generous donations from beagleboard.org and from texas instruments So we have a lot of development boards. We have a beagle 5 board a beagle y ai board And we've got 10 beagle play boards lots of fun to play with and Each of those comes in a bag full of ti swag So that's that's uh, if you while you're running your board at home You can use your ti pen to take notes on your ti pad of paper So, uh, we want to really thank texas instrument for for providing those And then we also have four mastering embedded linux programming books by chris simons and so In case you need a refresher if you or if you're just starting those are those are great Um, and then Also, we have gift cards. So we have gift cards ranging in value from 25 up to 100 dollars So nothing to sneeze at And then we have ldmi and end.net professional subscriptions. We have two of those And we also have souvenirs from seattle the lf event staff has gone out and and found some stuff Uh, like this. So we've got a little wooden cubic robot thing We've got the space needle and legos and a bag with a bunch of things coasters and and pens and and Things from seattle and of course hot tamales. So, uh, I don't know if you know, this is kind of an inside joke the official candy of the fuego test framework is hot tamales and uh So, uh, we throw that in every once in a while. We don't know if people like them doesn't really matter. Uh So but, uh, uh, if you win the lwm.net certificate and one other prize that I will place this next to You you also get some hot tamales Um So our first game is going to be embedded linux history Uh, technical nerd in space trivia. Okay, note the space theme Uh, maybe you've caught that already. Uh, and an important disclaimer the game is not fair Okay, so this is important to remember because there's always someone who says wait a minute It's like the the answer that the presentation has is the correct answer And so, um, okay, so everybody up everybody stand up. This is quite an athletic game actually for uh And so the current released version of the kernel is Is it green? So hold up the green card if you think it's six dot eight Hold up the red card if you think it's six dot nine rc4 Oh, okay. No, you cannot hold up two cards on this one You got it. You got to decide which of these And the correct answer is Green six dot eight if you are holding up red So rc stands for release candidate. It's not an actual release Uh, it says it right there in the name. Okay So this is actually I and I apologize if you're a newcomer This is the first question on every single game I play So people know my tricks by now Uh, well, let's go on. Okay. So richer researchers recently demonstrated The ability to double the resolution of optical microscopes What mechanism did they use to do that? This is this is like pretty earth shattering when you think about it They can double the resolution of optical microscope. Was it with AI super sampling in AI or was it using entangled photons? okay And the answer which is very cutting side Is Entangled photons the fact that they can use entangled photons to increase the resolution that is just mind blowing And so it has you have to be like a scientist doing quantum stuff to understand what the heck they're doing Okay, which of the following tasks has been automated recently using AI And this is important is scheduled for deployment in june This is going to actually be released to the public in june Is it ordering fast food at a drive-thru restaurant or loading dishes in a dishwasher? Okay, and the answer is Ordering fast food at a drive-thru wendy's has announced that they're going to have their customers talk to a chatbot Thing the order entry efficiency can only improve So, okay, we're getting pretty good. So you who are still standing you're you're in good shape Okay This is a tricky one Yeah The xz backdoor security flaw Was it green possible because xz was open source or red? Discovered because xz was open source. You can only you have to choose one You cannot do both Okay, you have to choose one Well, I am so nice both of those answers are correct So I made you choose but you're nobody got eliminated that round. Oh and pithy quip goes here Okay, this is harder Okay, this is harder How many instances of linux are running on mars right now? Right this minute Is it zero Green red one or green and red two How many instances of linux are running on mars right now? Okay This was hard to figure out And i'll explain it turns out it's one so red So how many do we have left seven? I think we can call you guys winners. All right Okay While there while yeah, stay standing until you get your ticket um So let me explain so there are two copies of linux running running on mars But one is in the helicopter turns out that there's a zigbee base station on the perseverance rover That one runs continually because it's powered by a nuclear thermal generator And the other one is solar and so it goes to sleep every night And it's currently where the rover is it's asleep right now so So i'm surprised more of you don't have mars time figured out Uh and here's the interesting thing you could theoretically If you happen to get some craft near the perseverance rover You could hack into that zigbee signal and send send all kinds of crazy data to nasa This is called high physical security Okay, everybody up Okay, the most photographed landmark in seattle is the space needle Or the pink elephant carwash sign Okay, there's a lot of chatter here. I hope the seattle people got this one right It's the pink elephant carwash sign Oh, that one was brutal. Hey, it is it is photographed more often than the space needle I find that unfathomable but people have done studies on instagram Uh And it does have some impressive engineering Uh, let's see. Okay. How many developers were involved in the creation of the 6.9 RC for kernel 6.9 kernel how many how many developers was it about a thousand or about 2000? Okay, and the answer is About 2000 and it's been consistently about 2000 for like, I don't know for the last three or four years Um, I have a tool that I use for my embedded status presentation that I look at this with And so that's it's pretty amazing that it's that consistent Um, okay, this was really interesting What company recently demonstrated an interactive dress that can change colors dynamically while being worn? Was it zara or adobe? Come on use your linux knowledge Okay, yes, this is called a wild guess it was adobe Go so if you get a chance. I didn't put the link on here Well, actually this uh, this source that I have does have the link is totally cool. Um, oh, are we down to four? Okay, you guys are winners Yay Yeah, this dress is not only with the flick of a button. Can you change the pattern on the dress? Uh, but they made it so that it had sensors. So as you're walking around that that last picture on the right there um The uh woman demonstrating it who was actually the software engineer and the seamstress on it She as it walked it was like this wave pattern So water would bounce and stuff totally cool Actually, this uh, this stuff was from um, a demonstration that adobe did in october But it actually has hit the runways. So it was put on the runway in paris in february So interactive fashion coming soon Uh to a store near you. Okay, everybody back up okay Recently linux was demonstrated running on a processor that cost 70 cents or 15 cents Okay, and the answer there is 15 cents which is pretty amazing. Okay, so I got to talk about this a little bit um The processor was a risk five, but it didn't have sufficient addressing or something going on And so it had to boot the kernel from eight meg spi nor and it had to use a risk five emulator to address the kernel So because it couldn't address it. So he had a risk five processor emulating a risk five. I don't know But 15 cents, I believe that's the lowest I've ever heard which is pretty cool um Okay, the lev 2 lunar nano rover this landed on the moon in late january And it was did that use the linux os is that true or false? So, okay, it's actually false I wish there's a long story behind this because the team that did this board came to me and asked if I could put linux on it And uh, and I said 1.5 meg. Are you crazy? Anyway, so it's running nut nut x so it's not linux, but it is open source So it's pretty cool. It actually is the rover that took this picture, which is a pretty famous picture of the of the japanese lander um Okay, for the first time this year linux has hinted that he may retire soon Is that true or false? And to my knowledge it is false So if you were green sit down, uh, he's uh, I he keeps I know how old he is and I know he's getting it kind of normal He's getting into retirement age territory, but I think he's said Uh, the last thing I've ever heard him say about retirement was that he has no plans To to retire anytime soon. So he'll he'll keep going. Okay university of glasgow Recently received funding to study an innovative mechanism to improve performance of solar installations Was it by using geo engineering to decrease cloud cover? Or was it to Uh by using orbital mirrors to increase the sunlight to the array Okay, and you'd think in the university of glasgow that maybe cloud cover was a problem But it turns out that they're all about orbiting mirrors So that's actually pretty interesting you've heard a lot of people saying Oh, we'll put solar satellites up there and they'll beam the the microwaves from space But turns out there's all kinds of technical problems, but this is this is eminently doable with existing technology They said based on their initial studies They could increase a solar arrays efficiency by like 30 to 40 percent with the extra hours of sunlight that they could provide Okay pike place market in seattle Is it known for being the oldest continuously operating farmers market in the us? Or is it the first farmers market in the us? That on the us mainland that sold pineapples And it turns out that this one is the Pike place market 1907 longest continuously happy over 100 years Okay, pineapples are good, especially on pizza Okay, let's see here. Okay Scientists have created the world's first working nanoscale electro motor. This is actually a pretty big deal I mean people working in nanotechnology Did they assemble it using an atomic force microscope? Or by folding dna molecules? Okay, the answer on this one is Folding dna molecules. All right, how we doing? So I think we need to go one more round Okay, when was eos last held in seattle? Was it three years ago eight years ago? Or eos has never been held in seattle Okay, this is interesting because it's never been held in seattle So elc was held three years ago. So I know why some of you answered that But elc and eos are not the same the game is not fair Okay, so let's go ahead and give these people prizes Okay, and then after this I need a can you guys confer and tell me what what we're at? Oh, okay, okay, so Okay, eight. How many do you have? How many slips do you have left? Okay, let's do one more round. I've got some I think they're interesting questions. Okay everybody up More than half of all spacecraft in the known universe Are running Linux? Is that true or false? More than half of all spacecraft And the answer Maybe shockingly is true Because the total number of spacecraft that we know about is less than 10 000. It's about 9500 or so And starlink and planet labs just their constellations alone Are about 5900 craft. So That's pretty that's pretty cool Um Okay, research This seems like a give me to me, but we'll see Research indicates that AI generated code gets reverted more often from projects where it has been introduced Is that true or false? Do you think AI code gets reverted a lot? It does It does I don't remember the exact statistics from the studies, but multiple studies have been done to look at reversion rates and The bottom line is that using co-pilot is strongly core correlated with mistake code Okay, so be careful using co-pilot double check it A tech startup recently announced plans to build a data center where was it on the moon or at the bottom of the ocean Oh, this is interesting. Okay On the moon Here's the crazy thing They already placed some prototype hardware on the moon In that in the IM mission that was just done in march Or it was late february. Anyway, they actually this was one of the private companies that had a module on that And so they actually plan later this year to put To put a storage center data storage center on the moon Very hard to break into So that's that's crazy the lunar economy is already starting Um Okay, this is also kind of mind-blowing an AI chip was announced recently that will have a large number of transistor elements Was it four trillion or a hundred billion? Oh gosh, I didn't get very many of you on that one four trillion before When I did this exact same question in at ELC and prog which was only like nine months ago And the answer was the other answer like It's crazy how fast the AI is driving a lot of a lot of innovation here um I think we need to keep going the official nickname of Seattle is Is it green the emerald city? appropriately enough Red the queen city or green and red the rain city What's the official nickname? Okay, you got to commit Turns out it's emerald city It what Seattle actually did used to be called the queen city and uh in terms of rain This was a bad week to do this question because the weather has been gorgeous But it's overcast a lot, but it actually doesn't rain that much But uh, okay, I think we how how we do we got we got to do one more Oops Oh, there we go Okay, the first transparent simon conductor was created in the last year Using a laser on a glass substrate. Is that true or false? We'll go with calendar year within the within the last calendar year And the answer is false And the reason for that is actually transparent simon conductors have been around a long time Actually, CES showed a couple of transparent TVs this year So that means it was developed way there But what did happen was they made one out of standard Tellerite glass using just a laser. So they were able to etch stuff into the glass Using using a laser. Okay, so I'm gonna wait one two three Okay, sorry, we got to do one more Okay, so busy box was initially created for mm. ULIS systems. You see linux. Is that true or false? Do you know your Do you know your history? Apparently most of you don't All right, so these guys are winners it was actually created for debbie and boot ploppies So this was back when floppy floppy's were smaller and actually existed So that was that was the use case for busy box Okay, I think Uh How are we doing do we have enough to do one more round of trivia? I don't know where I'm at on my trivia Let me 22 Okay, okay. Let's do one more one more trivia and then we'll go to luck. Okay everybody up Okay The cell phone in your pocket Has more computing power than the Apollo mission that traveled to the minimum back including onboard and ground control computers Okay, I didn't fool anyone with this one. Oh, maybe a couple people I apologize. Yeah, the ground computer is in the 60s. Come on Yeah, so Let's do this one. Okay, this one's interesting. What percentage of linux kernel vultures are first time contributors every release Is it 2% or 12%? I think this is kind of mind-blowing. It's red 12% Think about that think about the fact that every kernel release There's about 250 people who have never submitted a patch before to the kernel that we're aware of Okay, there's still an opportunity for you to get involved in kernel development, which is which is pretty amazing. Okay Um A research group announced a processor that could be produced for less than one cent per chip Okay, what was their breakthrough? Was it because the chip was made out of plastic or because they reduced the gate count? Hey, it was because they reduced the gate count So if your reds stay standing It turns out that people have been doing plastic processors for a while, but the yield has been way way low and so the actual The way to overcome the yield problem was just to have less transistors to keep the gate count down I think these guys got over 80 percent yield, which is pretty good for a plastic chip. So one cent per chip. That's amazing um, okay What is the average airspeed velocity of an unladen swallow? Is it 11 meters per second or is the correct answer african or european? Okay It turns out I'm going to give it either Either answer is correct. There's a there's a really funny web page that talks about The exact dynamics where the guy figures out how fast the swallow flies an unladen swallow okay Okay, according to the debbie in popularity contest. What text editor is more popular Is that vim emacs or nano? Oh, yeah, and there's the emacs user at work Okay This will blow your mind. It turns out it's nano So the the interesting thing about this is this is the first year I've done this question before this is the first year. It's nano. It's always been vim Uh, sorry emacs people so Lena's torvolds was actually bitten by a penguin. Is that true or false? Okay, not fooling anyone with this one either. This is a long long time ago But he was bitten by a penguin in cambera So I think we still have how what are we at? Okay, how many are standing? Okay, I'm going to do one more This is a repeat. So if you are come to closing games a lot How many processors are there right now running linux in low earth orbit? Is it about 6400 or is about 400,000? You got it aside It's over 400,000 Wow, okay. All right It it turns out That each starling satellite especially the v2s the v1s only had 66 linux processors the v2s have 85 linux processors And there's over 5000 in orbit So that's a lot of processors. I think I think within a couple of years It'll become self-aware and it'll be it'll be skynet So, okay, so that's it for that game Oh, I we're I'm not going to do this as a question because we're going to move on to the the rock paper scissors But I thought this one was funny So Does lenas torvolds prefer to do indentation using tabs spaces or a mix Actually, uh, yeah, it turns out Well, I mean he talked about this in the in the uh keynote And so uh for just for your reference, I put both a tab and a space on the screen And that's actually true. There is a tab and a space there. Um, we'll do one more This was something that linux mentioned linux first got into kernel development because He was trying to impress his future wife or he likes understanding the working of the computer hardware He mentioned this in his keynote as well. He likes, uh, he likes understanding the working of the computer. Okay So that's that's good enough for that. Let's uh Everybody just turn your head while I go through these real quick Okay, anyway, okay time for game one is over. We're running out of time. So we're going to run the last one really quick This is rock paper scissors. I think most of you know how rock paper scissors works This is against the presentation The presentation has canned answers. I mean, we're not doing lizard spot this time Or what did I do? Did penguin krill one year? Um So this is what the hand signals look like paper scissors rock Okay, so you're going to hold those over your head So you don't need your papers anymore And let me see. Oh, it's like so this is I think everybody knows what the uh the rules are in this game If you beat the presenter you get to stay in the game You if you tie the presenter. I'm sorry you lose So that means two-thirds of you mathematically are going out every round. It's a brutal game But so everybody up Um, I'm going to say one two three go And then you guys are going to throw and then I'll reveal the answer So don't change your answer after after you do it. So one two three go Okay, and the presenter is Rock So if you're paper you get to stay in the game Okay Yeah, this is brutal. It goes fast Okay, I'm going to say one two three go Okay, some of you know what I do here. I always do two rocks to start with if you've been I wondered if this time I should change it up, but Uh Remember I might not next time Okay One two three go Okay, what you probably don't realize is that from here on out I use I actually have a python program that generates the answers. So they're random after this And random sequences end up doing uh, okay, so if you're rock stay in the game Oh, wow, are we down to four? Okay winners So even I don't know what's coming up next. I mean I wrote the presentation, but I didn't memorize the Okay, okay, let's uh stand back up again Oh, and if you guys could confer and see how many prizes we have left So You got four three how many two? Okay, so we're at Oh five seven ten right ten left. Okay um Ready oh one two three go And this time the presenter is paper. So if your scissors stay standing Okay one two three go And this time the presenter is scissors All right people getting excited out there Okay, and One two three go This time the presenter is rock Okay, how are we doing? Oh, wow Okay, that was more brutal than mathematically expected Okay, so we got four left that puts us down to six. Okay. Oh, yeah four if I count correctly Okay, I think we have enough for one more round here We'll wait until we get Let him get his thing Yep, okay, we're getting pretty close to the end here. This will probably be the last round Oh, well, and then I just have a couple of closing remarks. So, uh, okay ready set go And If you are scissors stay standing Ready set go Oh Well, you guys mostly had it up. Okay, so if your scissors stay it stays standing Okay, I'm going to not advance it until I actually say, oh, wait, how are we doing? Yeah, we need to go one more. I think You have three Are you at zero Yeah, you're out. Okay, so we have five Okay, sorry. We got to go one more round. Oh, it's mathematically. That's a bummer Okay, one two three go Okay If you did rock you're going to be happy. Oops. Yeah Is that is that five? Oh, no, it's six Oh One two three four five six. No, no, it's seven. Okay Yeah, okay, we got to go one more time. Oh, you guys are so close Okay, one two three go Oh interesting It's paper Wait, all right, that's it I don't apply Okay I used to fast forward through these but since I randomize it. I don't really need to worry if you see them I could get out of the presentation Oh, okay So just time real quick for a closing thought Um So we're all here because we work on open source. We use open source. Hopefully we contribute to open source Just Just one thing as a final thought do something in an os s project. It doesn't have to be code contribution. It can be Documentation it can be infrastructure help Even I said this in my keynote Even if you all you do is get on the mailing list and tell people how you're using the software That is valuable as a maintainer myself. I really appreciate learning what the use cases are And so almost anything that you do in an open source project is helpful. Okay, not everything but almost anything And so don't wait to join a community don't think well I'll do this next year or I'll wait until the end of the project Just get on the mailing list and occasionally lurk and the very first thing is just to be watching See how the conversations go and eventually you'll find some place some way that you can contribute The first contribution that you make the first email that you send To a list is hard. The first code contribution you do is hard. It gets easier the more you do it The xc backdoor one of the reasons that it worked or one of the reasons it made it as far as it did was because we had an overworked maintainer who Felt like he wasn't keeping up and if people had been there to help him out You know the story could have been different. Luckily it was caught because of the open source effect The last thing okay, so your contribution can make a difference. So just the very last thing I want to say is A couple of weeks ago. I had a chance to go see The eclipse in the u.s with my son and my granddaughter and I just that you know eclipses don't come around very often. I think the next one in the united states is 2045 So I might not even be around I think sharing your enthusiasm for something is one of the most important things you can do Share your excitement about open source share The message of open source. We have this great resource that we're working on together That is making the world a better place And so with that I'll just leave you the thought that the future is bright We have go out and go out and contribute and be a member of the community I I hope that you've learned something from your attendance at elc this year and that you've had a good time And I hope to see you all next year. So thanks Oh I got to say this okay, if you are if you have a ticket for a prize The certificates the gift certificates we need to get your picture We need to get your name and and The elf event staff will take a picture if it's one of the uh boards you can just take that and go but if it's uh Well, make sure show us your ticket right so But anyway, the rest of you can take off But let us get a picture of you with your ticket and the certificate Your one I I validate that you are good to go on that one All right. Oh, did you get the Seattle swag awesome. Yeah, including a bag to carry it all in Next to your conference I oh, sorry