 Hello, everyone. I am Vaishanath Achyat from Texas Instruments and I would like to talk to you today about using Graebers for robotics and real-time applications using Sepharatos. I work with the Texas Instruments Linux development team and TI has a strong history of open source collaboration and contribution and we plan for long-term sustainable products with an open source ecosystem in mind from the device architecture phase itself. And we also focus on an upstream-first methodology in our internal and external development as well. As part of my day job, I work primarily on Linux kernel and UPUT for TI devices and I also maintain the TI devices in the Sepharatos. I was introduced to Graebers and Sepharatos through a Google Summer of Code project with Bigelboard.org in 2018, where I worked on enabling support for these micro bus add-on boards in Linux. So micro bus add-on boards are these plug-in boards that goes to micro bus sockets. So this is how I have planned the talk. We'll just go through an introduction to Graebers. Then developments on using Graebers for IoT, then BigelConnect and some limitations and how I have a proof-of-concept to overcome these limitations. And then I also have a short demo where I have something, a real-time control loop running through Graebers. So what is Graebers? So Graebers is an application layer protocol which was developed as part of the modular smartphone project from Google called the Project ARRA. So this was not released to market. So the concept of this modular smartphone was that you will have a base module smartphone which will have some basic features like your processor and all basic features. It will have expansion connectors on which you can plug in different modules to it and create your own customized smartphone according to what you want. So if you want a higher resolution camera, you can plug it in, you get the camera. If you want a projector on your smartphone, you plug it in, you get that. So unfortunately Project ARRA smartphones did not reach the market, but something similar the Motorola Moto Modes actually was released to the market. It uses similar concept. So we plug in modules and we have your own customized smartphone. So how does this work? Yes, you just plug in a module and it just appears on your Android phone magically. So the underlying main subsystem that is there is the Graebus subsystem in the Linux kernel. So Graebus, you can think of Graebus like a remote procedure called application layer where the host can control and discover modules that are connected. So if you connect the camera module, the host that's the base module can detect what's connected on it through the Graebus manifest-based identifier mechanism. It can also send and send commands to this module, get operations done, then get back results to the host. So what's different? So it's just an RPC protocol. So you just send messages, you have a handler on the remote side, get operations done, it comes back. So the main thing, the beautiful thing about Graebus is that Graebus has strong ties within the Linux kernel, different multiple subsystems, and also Graebus has a well-defined module identifier mechanism through which we can describe and discover the add-on modules connected. So if you look at Graebus and the Graebus underlying code in the Linux kernel, you can see that multiple different peripherals or the buses have equivalent Graebus virtual representations in the kernel. So if you look at I2C, UART, SPY, USB, power management, PWM, camera, audio display, all those things have a virtual equivalent of Graebus in the Linux kernel. So with Graebus, what you can do is that like, let's say you have your, let's consider your modular smartphone, that runs Linux or Android, and you have your module, let's say it's a module that has a GPIO expander, or let's say it has an I2C bus. You plug it in and through the Graebus discovery mechanism and it's actually appearing on the host as an actual bus. So you are mirroring and a peripheral on the add-on module to the host Linux system. So the main concept, concept behind Graebus is that to keep all the most of the intelligence within the host system, and these interchangeable modules can create your own unique kind of systems. And these remote devices, the add-on devices always run some generic firmware. It does not have, so let's say, it does not have any custom firmware, it's more of a generic firmware. And here I would like to talk about some advancements in Graebus for IoT. So Graebus for IoT was first proposed by Alexandre and he wrote the base framework for using Graebus and then Chris who is a sufferer maintainer who he took up the he ported the, originally the Graebus device site firmware was in the Natex RTOS and Chris ported the Graebus subsystem as a sufferer module which has, so then now we have the platform code and the application code loosely coupled and any platform that supports sufferer can also use Graebus for sufferer. So, and the primary transport being implemented is TCP IP because you can send TCP IP or anything and there are also other transport supported. So, and there is also user space applications written by Alexandre, it's G-bridge, so you can think of it as a bridge between your Linux kernel and your remote device. So in the actual project era or the motor modes these modules were directly connected to the smartphone or module and it was talking through a transport called UniPro which is a standard in the process communication protocol from IP. So now the, but the Graebus being an application layer there is no strict requirement that we should use it, the XR transport we can use any transport, so we are just passing the messages through the transport. So, what actually happens with this Graebus for sufferer module G-bridge and this ecosystem is that, so we have a remote node that's around sufferer and we have the sufferer, Graebus for sufferer module works on it. Now what happens is, so let's say there is a GPIO chip on that remote module microcontroller, so that GPIO chip is getting mirrored to your host system through the Graebus and the all the underlying maybe the complexity of the RPC layer is abstracted away from the user. The user just sees a standard GPIO chip on the Linux system like it was actually physically present on the particular host. So now let's say I use, like I try to toggle this GPIO. I use some libgp, GPOD standard commands and toggle this GPIO. So this is a standard GPIO chip interface in the kernel, but it's exposed by the Graebus GPIO driver, then it goes to the Graebus core, it creates an operation, it goes to the G-bridge and it goes to the remote node. So it, how it goes to the remote node can be any transport, it can be wireless, it can be wired, it can be UART or anything, so on all my demos from now I am planning to use the 802.15 for sub G, low power wireless as a transport. And now it reaches the packet, reaches the suffer node, it goes to the RX handler, then Graebus handler, then it goes to the suffer standard GPIO call. So that's how a GPIO works. It's not only GPIO that's supported, earlier we saw what all peripherals or subsystems had equivalent Graebus versions. All those are supported. For the interest of suffer, Graebus for suffer or IoT, we are just focusing on spy, I2C, GPIO kind of process. So I'd only like the right now we can use Graebus for IoT and like I would like to introduce you to BeagleConnect and BeagleConnect Freedom. BeagleConnect is a revolutionary technology from BeagleBoard.org, which uses this Graebus for IoT and device drivers in Linux kernel to virtually eliminate any low-level software development. So earlier we saw that GPIOs on a remote microcontroller appears on the host as if it was already on the host. Now we can have I2C devices or spy devices appear on the host in similar manner. Now think of it in the next level. So you have a device on the I2C bus. So now it seems the I2C bus appears as if it was on the host. You have that device also appearing on the host. So what you can do is you can reuse the existing device drivers in the Linux kernel to work with the sensors and actuators and devices that's connected on your remote microcontroller through Graebus. And so now we are using the Graebus for IoT with the G-bridge and different utilities. We are not tired to be using any project or a specific hardware or anything. Your host can be any Linux host. It can be your laptop. In my cases I am using a BeaglePlay development board, which is one of the newest single board computers from BeagleBoard.org. I am using that as a gateway and it talks to the BeagleConnect freedom, which is a wireless MCU from BeagleBoard.org and it talks our low-power sub-gigahertz 802.154G. So now what I have is a small demo. So I have this BeaglePlay running the stock Debian image from BeagleBoard.org and I have the BeagleConnect freedom connected on a separate room far away from this gateway. So now what I am doing is this there is a script called Start BeagleConnectGateway. So it just sets up the network and pings this remote node. Then also starts the services like G-bridge. So now what you can see is it's the host system. It's running Linux. This system is running Safari. This is far away and I am logging into the host BeaglePlay. So in the kernel logs, you can see a light sensor and a humidity sensor popped up. So it was not ever actually connected to this system, but it was actually here. So this light sensor appeared on the host. I can use standard Linux utilities like IAO info or any other higher-level middleware utilities to get the sensor readings, make measurements and all. And I did not even have to know that what sensor was connected or even the register map of this particular sensor because all the drivers are we are using from the Linux kernel. And also in Linux, almost everything is a file. So reading sensors, reading, controlling actuators, all just is simplified as simple file operations and you don't need to do any lower-level development. So this I showed for the light sensor and the humidity sensor that's on this board. You can even add on all these microbus add-on boards and make it work with the same system. We have tested with around 150 add-on boards. You can have just plug-and-play support and make it appear on the early Linux host through Gravers. Maybe I'll stop a bit if you have any questions regarding how like this is the state of Gravers for IoT and we will connect now. Okay, so right like earlier what happened during like the sensor reads and all was that so we have a light sensor that works through like I2C. So it needs some I2C transactions to set up like trigger the measurement then get back the measurement. So through Gravers, we were sending all these transactions and like continuously making all these measurements. So one failure, one this can really simplify a lot of IoT applications where you don't want to do lower-level development, but you want to focus on your middleware or your actual application and you just want to prototype different sensors. You want to try out different systems and you are more interested in your final application more than getting any particular sensor work on any different particular board. So I experimented with the Bigel Connect Freedom and Gravers for IoT and a few things I find was that so in the project are a smartphone the host module and you are actually physically plugging it into the add-on modules to the actual smartphone. So there is so the and the purpose like and the functionality is only useful when the host is active. So in IoT applications, your central gateway or the node is sending the commands to the remote node, but even if the gateway goes down, you might want to have some operations happening in your remote nodes. Let's say for example, you are deploying a weather monitor or an air quality monitor. So you are continuously pulling this data from the sensor, but let's say your host goes down or the connectivity between your host and the remote is not working. Then your complete system kind of goes down, but let's say what if you could have off-load to the remote node, it compute the average sensor readings continuously for a day and your host will go and read back the sensor reading after a day or once per day and in the previous case, you are continuously pulling, so you are continuously sending Gravers operations to read all these I2C transactions and making the readings. So it's also not very power efficient in some use cases. So if you could off-load to do it something like let's say take a reading, then wait for a time or like maybe suspend, wait for a time or wake up, then make the reading and do average. So you could optimize it for some use cases where power is also critical. And we can so in the project-array use cases, the transport was very, the continuous, reliable and very high-speed transport. In IoT use cases, we could have low data rate transports like the sub-gibbards, long-range communications. So before how we could just like make some changes to overcome these limitations, I would just like to talk about how a Gravers operation looks like. So I'm not going into the details of Gravers setup and also one of these operations, you can think of each Gravers transaction as an operation and all these operations will have a short header than a variable length payload. So this payload will be depending on particular operation. So let's say you in the earlier case, we talked about toggling a GPIO. So how the Gravers operation packet for the toggling a GPIO looks like is we have an operation header. It has fields like size, ID, then there is a field, important field type. So with the field type is how the particular handler is being called. So if the type is the if the type is GPIO set value, then the GPIO set value handler at the remote node is called. There is also concepts of C ports, but for simplicity, we are just talking about how the handler is involved. And the payload is actually specific to each transaction. So for this particular GPIO set value, you'll have a payload saying which and value. It's the offset on the GPIO chip and what's the value to set the GPIO to. So one of the difficulties I face during this prototyping or working with Gravers is that Gravers has a complex stack starting from your Linux kernel, then netlink to the user space, then the user space application that bridging to the remote node. And you need to have a Gravers for software remote node functional to do any prototyping. So I wanted to isolate some parts of the system and test few parts. So I kind of combined the Gravers kernel functionality and GBIT functionality into a small Python library. It does not implement all the functions, but only few of them functionality that's of interest. And with this, what you can do is suffer has a native POSIX target. You can do CI and testing or you can do prototyping on your host itself. Your remote node will be a suffer POSIX target executable that runs on your host and your Gravers host side and the kernel side will be a simple Python library that triggers this transaction. So here is a proof-of-concept I have for overcoming some limitations we discussed regarding using Gravers with kind of slow or unreliable transports. So we can just send transaction Gravers operations one by one and execute at the host together. So we can do some let's say you have a sensor which has something like an enabled GPIO, then you enable the sensor using a GPIO toggle, you read the data over I2C, then you disable it. So maybe you can combine all these send one by one and ask the remote node to maybe execute it together. But this kind of approach is not useful in most cases because you need to have some decision-making capability at your remote node. So what I have added is that you had allocated some scratch pad kind of registers or you can say it can be considered as temporary variable, a temporary memory region where the host can write constants, read back the values, and also the host can perform arithmetic operations on this scratch pad. So all these are new Gravers operations I have added. So you can read to this register region, you can use this as a temporary holding register variable, read, write, and also do perform arithmetic. You can do a few more things. You can do make another Gravers operations payload to this scratch pad or use this scratch pad region as the payload for another operation. For example, you read an I2C sensor data using let's say I2C read word, you get the sensor data, you can actually put it to the scratch pad region for further processing. Then we need some decision-making and nitrating operations. So I added two operations for if and while all this takes the input conditions or operands from the scratch pad region. So you can do like if scratch pad register five is true, then perform some operations or while scratch pad register is true, you can do some operations. Then also a simple playback option where you can combine all these previous building blocks, you can kind of put a sequence together, send it, you can play it together, or you can even ask to play it indefinitely as well. So this is the operation. So one deviation I made from the Gravers specs or the existing Gravers operations is that so far all the Gravers operations have a short header and single payload. So in my extended operations, all these payloads can be different operations. So you can put multiple operations inside the if, while, and all. So this is the scratch pad of operation. Like you can specify an offset. That's the kind of you can say register number or variable number. And then the you can also I already told like you can write another operations result also in the scratch pad region. Then you can perform mathematical operations as well. So most of the usual ones are implemented. Then there is decision making and loop operation. So I have just put like there is an example of any if operation. So I just wrote to the scratch pad register five one, then if up and five means that if that scratch pad register is true, then we'll go to the if, otherwise we won't. So I am doing a manifest read operation, then decrementing it. So if I run this again, then this won't be executed on the remote node. So this is actually going to be executed on the remote node. Then I'm just doing an I square C read by it and write that particular read by it value to the scratch pad resistors. Similarly, you're performing a while. So this will this while will run five times. So I just wanted to see whether can this set of operations allow me to make a robot. So I wanted to run a control loop on using this set of operations. So what I did was I used a vehicle connect freedom hooked it up to a small line tracer robot. So it has some few IR reflective sensors which takes inputs. And so if it deviates from the line, we get the IR sensor input. So we can compute the error according to the error. We control the motors. So in the normal gravers over like for IoT, you can send this transaction over the remote or the sub-gigards wireless. But I did try the latency. This was kind of 50 millisecond for transaction and you cannot actually do a control loop with that kind of latency. So what I have done is that like I have put the sequence in the same with using the similar the previously described operations and try to offload the control loop to the remote node. So here I have the demo. So this is also being developed run on a big old play as a gateway. And so I'm doing the network setup. And it's running the same gravers firmware as earlier. And also you can see the, so I'm just showing you that there are onboard sensors that shows up in the kernel also as before to just show that it's the exact same firmware. So I just started a Jupyter notebook on the big old play and I'm just doing all the, since it's a prototyping, I'm just doing it on using the Python. So, so I'm just like running all those. And so in the end, what you'll have is that you can describe different transport also using this library. So I'm just using the TCP IP. So you can see I am so in the end user gets interfaces like the big old connect freedom dot get manifest or big old connect freedom dot set GPIO offset. It's an easy user interface for the end user. Like, and this is not using the kernel grab us now. I'm just showing the grab us manifest, which is the module identifier information. So this Jupyter notebook has all the different operations implemented and I have added it in the references as well. So here I'm trying to read the onboard sensors on this board you through grab us transactions, but not generated from kernel but from this experimental Python module. And for that robot there was also a PWM micro bus add on click board that was used to control the motor speeds. So this was some failed prototype. I tried to maybe try to run the control loop over wireless. It did not work. So this is the final code that it look kind of looks ugly like assembly and you have different operations where I kind of calculate the error control the motor according to this. And so right now I have dispatched the sequence to a big old connect freedom on a line tracer robot. And you can see that's running a control loop, but there is no actual firmware Lord or firmware change on that device. I'm just sending a sequence of operations that I'm offloading a sequence of operations to that robot and the control loop is run locally. So once that set of operations are send out, the even if the gateway goes down or the host goes down, that will still continuously be running. And in addition to that, when it's running, you can control parameters of that control loop as well from the host. So that also I'm doing it now. So I have some helpers to just the update the speed just changes the PWM duty cycle. So the control loop is running offloaded to a separate thread and it's running in parallel. And then you can also send more sequences or more operations and do more things in this at the same time. So I'm changing the motor speed from the it's being done remotely, but the control is loop is not affected at all. You can change do other things. You can interrupt the control loop as well. I stopped the robot. I can make it go back then do the control loop again. So all this you are not doing with any firmware Lord. You are just offloading different operations over grab us to the device. So one nice application I could find was so when we used to make these kind of robots while we were in college. So what we used, we had a challenge of tuning this PID parameters. We used to take out the robot and update the parameters, then put it back, update and do it again. So now you have you are developing all this on your host system and you can have like interface with more and more middleware. Like, so let's say you have on the scratchpad registers, you have something like a K P K I K D and your control loop takes that as an input and it runs in parallel. And you can send a message to update these values from the host continuously without taking out the word. So you can interface with more and more middleware like rose or other middlewares since you are doing all the development on a Linux whole system. So yeah, I admit the interface available to program or send the sequence of operations. It's not very straightforward. It's kind of some like you are sending and if then it's slightly difficult and not very straightforward. But the way how grab us for IoT with the Beagle connect freedom comes up. It's very straightforward and you are just using the sensors and devices on the remote module as if it was on the Linux, your host and you also have device drivers getting probed with them. So you may not be able to directly use it at that maybe and as an add on to that if you want to do offload control loops or other operations, you can use this. And right now how it's implemented is you send a group of messages and it's offloaded to a thread. So if you run more than one kind of loop, you will need to take care of the synchronization yourself. So I have attached the references for all the related works. Also, I have added the details and instructions this Jupiter notebook. You can you don't need actual hardware. I tried it on Beagle connect freedom and Beagle pay, but it runs on the host itself. You can use this effort native POSIX target build it and do development experimentation on the host itself. So this work takes input from a lot of different related works. So I would like to thank them all and thank you all for listening. Yes. Yeah, I'm planning to find time for that. We have plans. No, not yet. So you want a way to make do some decision making on the remote node. So for that, you are using the scratch pad as temporary variables. So let's say you perform an I square C read, put it to the scratch pad register. You write another constant to another scratch pad register. You perform a comparison. You can do different actions like that. It's kind of temporary variables. I think both of you have. Yeah. So if you so you want to create a great virtual device with a keyboard on your remote node, if you wanted to make it appear as an HID device on your Linux host. So only thing. So and everything that's written is generic. So I don't remember if we had implemented HID on the suffer side. There is already support in Linux side. So we might need to enable that. Let's assume that's enabled. You will need to change the, the grab us has the add on module descriptor format called grab us manifest. That's where we describe what all are there on the module. So you just need to have update the manifest. So right now all you will need to refer to the grab us pex document and update the manifest. And there is also a manifesto tool that creates this manifest binary. That also is a good, there are good references on that tool as well. No, I have not talked to anyone yet. So I think like a byte code interpreter like ebpf, that would be great. But one advantage that we have here is that you can have like very large kind of operations as a single transaction. So if you think of ice and I square C read, if you want to do it using raw or you want to call it from scratch, then it will be slightly difficult and it will be platform dependent as well. So with the grab us operation and the, the, the grab us module running there, you can just send the standard grab us message, which says to do an ice square C transfer. Yeah, I think that's just one benefit I saw. But if you really want to do any like complex development, this way might not work really well. Yeah, Linux driver because the Linux device drivers does not have the information about that. I need to think about it, but right now all these device drivers, the OPT 30 and all this we use was the direct device drivers. There is nothing for grab us that we did on that. So what one way I can think is you can do some more complex operations using the Linux device driver than do repeated transaction using this. Let's say you prop the device driver, perform calibration, all those complex operation, you upload it and get help from the device driver in the kernel. Maybe repeated measurements we want to do, we can do it from the user space. I did not do a measurement like that. Sorry. No, I did not do a benchmark benchmarking we should do maybe I was so without this extension, I could not run that control loop even if I was sending the transactions one by one. But with this, I can run that control loop. So I did just did that kind of comparison. Okay. Thank you everyone. So without. So I think the update rate I could achieve was around one kilowatts two kilowatts range. So without that I was kind of at 25 hertz. Thank you.