 Thank you very much for coming. This is sensors and PWM control from Linux. If this is not what you were expecting, now is the time to leave. It's kind of like when you get on the airline and they go, yes, we're headed to Denver. And if you're not headed to Denver, you're on the wrong plane. But then you wonder, how the hell did you get on the plane in the first place? Because they check every ticket when you're coming in. It's like, I don't know, well, all right. So we're gonna talk about PWM. Now, who am I? I'm Mike Anderson, and I am director of technology at the PTR Group. The PTR Group, I founded the company back in 2000. When I sold the company last year, we had 24 engineers. The company we sold it to then immediately got bought by Huntington Ingalls Industries. So if you're not familiar with Huntington Ingalls, they are the Newport News Naval Shipyard. They make nuclear aircraft carriers and submarines. So if you need a submarine or an aircraft carrier, hit me up. I can put you in contact with the right people. But yeah, they had no idea that we were in the purchase that they made. And so it was kind of like, where'd you come from? What do you do? You do that. Well, in that case. All right, so we're gonna talk about, in this particular session, we're gonna talk about sensors in the real world. And then how we interface sensors into the industrial IO subsystem in Linux. We'll differentiate between analog and digital sensors. We'll get into some of the other specific types of sensors and what exactly they measure, various position measurement modalities, whether it's acoustic, ultrasonic, infrared, laser based, what have you. We'll get into attitude and alignment. This is where we talk about gyros, accelerometers, things of that sort. Inertial measurement units, IMUs. And then we'll talk about how all of that plays with something called the Pulse Width Modulation subsystem. So we're actually gonna be talking about two different Linux subsystems in this session. We'll be talking about both the industrial IO subsystem and the Linux PWM subsystem. They are related, but not closely related in the code. They're related in what they do and why you would want them. But they are completely separate subsystems inside the kernel. So let's talk a little bit about Linux and the real world. Now, when I first started working with Linux, which was back in 2000, that was kind of the joke in the embedded and real time world was that there was no way Linux could ever be real time. That was just impossible and anybody who thought they were gonna use Linux for an embedded system was out of their mind. And a lot of that was true back then, largely because the people who were writing device drivers back then would write a device driver for like a sound card, let's say. And that sound card would work fine on a 1.8 gigahertz Pentium. But they never thought anything about how that sound driver is going to work on a 400 megahertz ARM processor. So fortunately, most all of that code is now gone. It's all been replaced, rewritten. And effectively, with the introduction of the preempt RT code into the Linux kernel, now we have the capability of doing what most of the real time operating systems can do. Now, we're not gonna be talking about microsecond resolution on many of these things, Linux still can't do that in a lot of cases. However, it turned out that if you took a look at where most of their customers were actually doing real time processing, their requirements were nowhere near sub microsecond. If they were in the few hundred microseconds, that was good enough. And when we think about real time, real time means being fast enough. Now, for a purist, real time means being deterministic. And that's a lot of what preempt RT does, it brings determinism into the Linux kernel. However, the determinism does not, just because it's deterministic doesn't necessarily mean it's fast. In fact, we've seen many benchmarks, and of course there are lies, damn lies, and benchmarks, we understand that. But there are many benchmarks that actually show the difference between the voluntary preemption of the kernel, which is the default in the kernel these days, versus the soft, the preemptible desktop, which is the soft real time option, versus the hard real time preempt RT options. And you find out that the soft real time options in many cases are actually faster than the hard real time options. But the advantage of the hard real time option in preempt RT has is that it's deterministic. So deterministic does not necessarily mean fast, it just simply means that it takes the same amount of time every time. And that's a lot of what we're trying to achieve, especially when we start talking about PWM control of either servos or stepper motors, things of that sort. So with the preempt RT code in place, we can actually use Linux as a real time operating system in the real world, and there are many examples of that. As a matter of fact, there was one just before the break where he was talking about an industrial controller, his Epic controller, and that Epic controller was running a quad core IMX6 with the preempt RT patches and doing real time industrial control. So industrial control is very unforgiving. If you make a mistake in an industrial control application, it's not unusual for people to die. So people, as a general rule, the industry, the industrial control industry is very picky about what they do and what operating systems can be used. So the fact that we're starting to see Linux in a lot of those applications is a real testament to the abilities that both Ingo, Molnar, and Thomas, Gleitzner, and Steve Rosdett, several others, have done in making the kernel capable of doing hard real time. Now, when we're talking about sensors, usually we're talking about sensing something that exists in the real world. And the very first documented case of this is actually back in the Song Dynasty where they found a piece of lodestone and they actually made, that's that thing down in the corner, they actually made a compass out of it. So that was the first known real sensor measurement unit. Now it's, of course, obviously sensors have come a long way since then. We're now seeing a proliferation of what they call micro-electromechanical sensors, otherwise known as MEMS, and there's a new class called a nano-electromechanical sensor or a NEM. MEMS operate on the micron level. So this is a MEM accelerometer that we see here. And a lot of the issues that we're trying to figure out when we're dealing with sensors is, first of all, what is it exactly that we're trying to sense? Again, oftentimes it's something in the real world that we wanna try and figure out what the attitude, roll pitch and yaw, or what direction is something headed. So we're trying to figure out something in the real world, and that could also include things like gas detectors, chemical parts per million, we could do radon, we could do all kinds of sensors. If we have the appropriate sensor for it, then the Linux subsystem that's gonna target this is basically fairly well suited for a lot of these applications. So when we look at sensors in the industrial world, it turns out that for industrial, especially if you're looking at industrial IoT or any of the older industrial processes, they run until it breaks. They do not go out and replace things. They are basically operating in technologies that could be 30, 35, 40 years old. And good case in point, I know of one major metropolitan region whose entire water system is run on Windows 95. And it's like, you gotta be kidding me. Really? Windows 95. And they go, well, yeah, because the people who designed it, designed it back in the 90s, Windows 95 was new at the time. Those people have all retired and left. And now we have a system that we have no idea how it works. And therefore, we don't dare touch anything because if we did, we might potentially break it. And if we broke it, we wouldn't know how to fix it. So, and that's a metropolitan region that serves as over two million people in its greater metro area. I mean, it's like, well, this is scary stuff. Oh, and don't even talk about cybersecurity. No, no, no, no, don't even bring that up. When the cyber guys come in, they say, keep your hands in your pockets because if you touch something and zap it, there's no way for us to replace it. So that's bad stuff. But that's what's happening in the industrial world. What will happen generally is we will find ourselves with interfaces like RS232, 422, 485, multi-drop. We'll see TTY 20 milliamp current loop. All these are ancient, ancient, ancient interfaces, but they're the interfaces that are still currently in use in most of the industrial applications that we see across the world. So when we're dealing with trying to go into a brownfield application, now brownfield says it's been around for a long time. We know how it works. There's nothing really new here. There's nothing to be learned by changing the way things are done. Therefore, the impetus to upgrade a brownfield application is simply because you can't get the parts anymore and things are breaking and you can't replace them. Now, on the other hand, greenfield applications are fantastic because with a greenfield application, nobody's ever done it before. And because no one's ever done it before, you're free to use whatever technologies are at your disposal. And this is where we see a lot of wireless technologies like Lorawan and Sigfox and these other techniques, they're not IP-based, but they're great from a cellular perspective. So they operate somewhat similar to the cellular network and you can actually have a private network using these technologies and do a fairly large area. For instance, Nigeria has a very large, fixed Sigfox interface. The folks in, let's see, there's another one that's in South Africa that uses Lorawan. Here in the United States, we use Lorawan quite a bit and we're seeing more and more examples of arduinos and beagle bones and things of that sort that can speak into the Lorawan network. But those are relatively new technologies. And as a matter of fact, tomorrow I'll be giving a presentation on all of the latest alphabet soup on Wi-Fi interfaces. And Wi-Fi is trying to move into the industrial IoT as well. And we have several different problems with that happening. So if you're interested in that kind of topic, I'll be talking about that tomorrow afternoon. Now, with a lot of these devices that are out there currently, and the most of the sensor systems that are in the existing Brownfield IoT or industrial IoT, we're seeing UArts, we're seeing just serial ports. Now, even some of the more modern sensors today still use serial ports. So that becomes a bit of a problem for us because serial ports aren't like they used to be. That is, in the old days, we knew that an RS-232 port had a certain voltage on it. And we could always just depend on that. Well, now we see TTL level serial ports that are at 1.8 volt, 3.3 volt, 5 volt, 12 volt. Some of the TTL stuff is still 12 volt. So you can't just simply plug into a serial port and think it's going to work. You could plug into a serial port and the magic blue smoke could escape. And that's never a good thing. So when we're dealing with serial ports, it's not as simple as just saying, oh, transmit, receive, and ground, it's now the voltage. And we may have to have a level shifter in there to make sure that we've got the correct voltage to keep from toasting the hardware. Now, with a typical MIMS sensor, they can be anything from GPIO bit banged to I squared C or spy buses. Those are all fairly common approaches. And each one of those approaches has their own pluses and minuses. We'll talk a little bit more about that as we get further into the material. The newer systems, they will use even more exotic interfaces. They've got MIC BSPs, which are buffered serial ports. We have sports and MIPI and CSI-2 and there's a whole collection of new interfaces that are all typically specific to a particular technology. Like, for instance, CSI-2 is specific to cameras. So if you wanna run cameras, you gotta get a CSI-2 camera because that's what most of the camera interfaces are these days, certainly in cell phones. We also see, of course, a proliferation of custom FPGAs. One of the things that I've run into more often than not, as it turns out, when you have somebody who is an FPGA person and they've decided they're going to implement an ethernet interface, the ethernet chip itself, they'll buy that, but they'll implement the PHY inside of the FPGA. And because it's a custom PHY, we don't have software for it, so the money that he saved by putting a custom PHY in there is gonna be spent in all the software support that we have to have in order to make it work. So, when we're dealing with sensors, we have to work with the hardware people to a certain extent to make sure that they understand that the decision of the interface that they use is going to have an impact on the software. And it could be such a significant impact on the software that it offsets the cost difference that they're trying to save by switching to this funky interface. But that is entirely up to the, between the software people and the hardware folks to kind of work that out. Now, in the traditional Linux world, we had a couple of different input, we had a couple of different subsystems. One of them called the input subsystem and the other called the HWMON subsystem. The input subsystem is largely targeted towards human interface devices. So it's keyboards, mice, joysticks, things of that sort. And the characteristic of the input subsystem is that they're relatively low data rates because a human can only type so fast and incredibly non-deterministic because humans aren't very deterministic either. So the input subsystem didn't have to be very fast nor did it have to be deterministic. In the HWMON subsystem, this is one where we actually see a lot of interfaces like I squared C and spy but the sample rates on these things are relatively low and that's because the HWMON subsystem is targeted at things like fans and temperature sensors on motherboards and things of that sort where we're only polling them maybe once a second and if we miss the polling deadline by a few hundred milliseconds, ah, nobody cares. So it's not a big issue. So again, even though the HWMON subsystem is using the same sort of interfaces that we would expect to see on some of these high end sensor systems, these spy and I squared C subsystems, they're not doing it very fast. It's not made to be a very high polling rate nor is it meant to actually be, you know, very high speed interrupt driven either. So those two subsystems, although you would think they might be usable in an industrial control environment, actually turns out they're not very well suited. They're fine for monitoring the speed of the fan but not so good for other things like figuring out where you are in time and space due to attitude, role, pitch, and yaw. So we enter in, in 2009, and this was in staging in 2632, the industrial IO subsystem. Now the industrial IO subsystem when it was first put out there, a lot of people were looking at it kind of scratching their head going, well, I don't know that I really need this. Why am I doing this? Why do I need this? And because what it did was it was just largely analog to digital converters or digital to analog converters and it was again running at a relatively low data rate. So they said, well, this is no better than HWMON. Why am I putting another subsystem into the kernel? But as time went by, they started adding other sensors that did not make sense for HWMON like gas sensors, being able to determine that you have radiological sensors, chemical sensors, a number of sensors that just didn't make sense at all in the HWMON subsystem because that's built for hardware monitoring of the motherboard, this is actually something that's happening external to the device. So if we look at the 5.x kernel series, and I'm running 5010 on my machine right here, I've got a little over 200 different drivers for things in the industrial IO subsystem. Now, a lot of these drivers are also kind of niche drivers that didn't fit very well in other places. For instance, things like clock generators and potentiometers didn't fit very well into the MISC subsystem, nor did they fit in the HWMON or the input subsystems. So the industrial IO subsystem becomes a convenient place to put those kind of oddball drivers that are only used for very specific sorts of applications. We also support, one of the other things that it can do is it supports triggered sampling and the use of DMA transfer regions. So this is something that we don't see in HWMON. HWMON is largely pulled. We're gonna go out once or twice a second, we're gonna ask the fan how fast are you spinning and if it's not spinning fast enough, we're gonna spend it up a little faster. So that's what HWMON does. In this case, we can now have a completely interrupt-driven subsystem where the device itself tells us when the data is ready. This also gives us the ability to tie in microcontrollers into some of these systems. So if you look at a lot of the inertial management or measurement systems, the IMUs, you'll find there's actually a little microcontroller out there that's actually doing sensor fusion and integration of the individual components that it's reading and then sending that information across into the industrial IO subsystem using DMA. In the most part, even if we're not using DMA, most of these sensor interfaces use a kernel FIFO back-end channel. So reasonably fast, not as fast as the DMA, obviously, but with the use of K-FIFOs, we also have non-blocking operation. One of the concerns that we have with any sort of industrial application is if I'm not reading the data fast enough and I get more data from the sensor, what happens to that data? Does the sensor stop working because the channel is blocked? Well, in most cases, we don't want that. We don't want it to stop working. So in a K-FIFO back-end, what that channel is going to do is if the channel is full and you try to write more data into it, it just gets dropped. So it's entirely up to you and user space to be able to read the data fast enough and process it that it doesn't have a problem with backing up into the K-FIFO and overrunning the K-FIFO back-end channel. Now, the types of sensors that we'll find in the industrial IO subsystem, we have, of course, accelerometers and gyroscopes and magnetometers and IMUs and all these other things, but they've added a whole bunch of new ones recently. One of them is a %02 sensor, so like on watches. So now you have the ability to determine how oxygenated you are. So those are available now. And color detection sensors, gesture sensors, and, of course, the hazardous gas and chemical detectors, which have been a lot of what's been added in the recent couple of years. If you're really interested in the details of the implementation of the IIO subsystem inside of Linux, then Matt Raniste has a great presentation from ELC 2017. It's available on the web. He'll go into, he goes into a lot of agonizing detail as to exactly how the subsystem was put together. So if you're interested in that level of material, I highly recommend you go out and take a look at Matt's presentation. Now, when we're dealing with sensors, we have to understand the nature of how a sensor works. So sensors are typically thought of as either being an analog or a digital sensor. An analog sensor, because we're using digital circuits to work with the analog sensor, we're going to have to sample the analog sensor and quantize the data. So what we will see here is something like this analog signal and we're measuring the analog signal several points in time along the way and this measured digital version of the analog signal approximates what the analog signal is actually doing. The key thing here is the sample rate. Now, when we talk about A to Ds, we're usually focusing on how many bits the A to D can do. So if we're talking about a typical Arduino processor, the Arduino processors will typically be on the order of a 10-bit A to D. That means it's a value between zero and 1024. In the case of the Beagle bone, the pocket Beagle, like this guy here, the pocket Beagle is a 12-bit A to D. So we have values between zero and 4095. Now, what does that mean? Well, that means that it's measuring the complete voltage shift from zero volts to maximum five volts in some cases or 3.3 or 1.8, whatever it is. And that represents 4095. So somewhere on that scale is the measurement that you're taking right now. You save that piece of data and you can convert it into an actual voltage. So with digital sensors, not all of them are complex. Most of them actually are fairly simple. We'll have things like limit switches and we'll talk about that in a bit. And of course, the inertial measurement units are probably some of the more complex of the sensors we will have to deal with. Unfortunately, that's all on the input side. Now, how do we deal with output? Of course, some of the sensors, the actuators may actually be driven by serial ports or I squared C or spy bus. But if we're trying to approximate an analog voltage, then we have a couple of different ways that we can do that. Either we have a digital to analog converter, which takes a digital number like 1024 and then converts it into a voltage that goes out onto the interface. Or we use a technique called pulse width modulation. And we'll get into the details of about pulse width modulation towards the end of this particular presentation. Now, the data rates coming from the sensors range anywhere from about 100 to 400 kilohertz in the case of I squared C. Those are two modes of I squared C, 100 kilohertz mode and 400 kilohertz mode. All the way up to 100 megahertz for some of the interfaces like the serial peripheral interface, the spy bus. So, fortunately though, a lot of microcontrollers and small development boards like Arduino's actually support these interfaces and they're very easy to work with. So, in many cases, if we're just trying to do a quick prototype of something in the industrial world, we'll pull out an old Arduino and we'll plug the Arduino in and we'll just write a little quick Arduino code just to see if we can do it. And then if we can actually prove that it's functioning, then we'll go to the trouble of actually trying to hook this into the industrial IO system through Linux. And that's simply because the Linux interfaces, when you get ready to write a device driver for Linux, it's never a simple thing. Even if you understand exactly what the hell you're doing, it's not gonna be easy and it's gonna take you several days to debug it. Whereas I can whip out an Arduino, an application on an Arduino in about five minutes and we've got it working. So, from a testing perspective, we'll oftentimes use these dedicated microcontrollers just because they have pre-canned libraries that deal with I squared C and SPI and they know about accelerometers. There's usually a library already available for the accelerometer. So, we don't have to write anything new just to test the circuit out. But, once we get ready to get to the point we're trying to field it, now we gotta integrate it into the IO subsystem. And, fortunately, the IO subsystem with a little over 200 drivers at this point actually has examples of many of these types of devices. So, that's a good thing. We're not writing a completely different driver from scratch. We can go into the IO subsystem and find a device that's similar to our device and model our driver on that. Obviously, that's all covered under GPL. So now, another aspect that we have to deal with with sensors are are we talking about absolute measurements or relative measurements? An absolute measurement would be, say, turn due east. There's a way that we can actually measure that and guarantee that we are, in fact, pointing east. Or spin the revolutions of the wheel up to 3,500 RPM. That's an absolute measurement that we can then make sure that we met that particular requirement. But relative measurements are also out there where we'll say, okay, no matter what direction you're pointing, reverse. Go 180 degrees the other direction. Now, that means that I don't know exactly where I'm pointing right now. I just know that when I'm finished, I'll be pointing the other direction. I'll be pointing behind me. And I may not need to know the absolute value. I just need to know that I'm not facing the same direction. On the other hand, I also have cases where I will say, well, I wanna spin the wheel, the shooter wheel, for instance, and I do a lot of work with the first robotics competitions. So we'll have applications where we need to just spin the wheel a little bit faster. So they'll say, well, software guys, we need you to spin the wheel 25% faster for this range. And they're like, well, 25% faster than what? So if you didn't have an absolute measurement, then 25% faster is a little tough to measure, actually. It's a little difficult to figure out exactly what it is other than, yes, I'm measuring a voltage and I'm gonna add 25% to that voltage. Whatever that does to the motor, I'm not quite sure, but it's 25% more than it was before, which is kind of the thing that we have with relative measurements. So however, we may need to be in absolute measurement mode in order to figure out what the relative change is. So if they wanna do spinning a shooter wheel 25% faster, then it may not just be a voltage that we're measuring, it may be that, well, we're actually going at 2000 RPM and I need to be going 25% faster than that. So I need to have that absolute measurement first before I can then add the relative delta on top of it. So whether you're using absolute or relative measurements is completely dependent on the application and whether or not you actually have sensors that are accurate enough to be able to give you absolute measurements. Now, also when we're dealing with sensors, we have to deal in terms of these things called degrees of freedom. Now, a degree of freedom otherwise known as a DOF, we'll say that this is a six DOF sensor or this is a nine DOF sensor. What that means is I have X, Y and Z, that's three degrees of freedom and that may be on the accelerometer and if I have three degrees of freedom on the magnetometer, the digital compass and three degrees of freedom on a gyroscope, that's nine degrees of freedom. Now, they're not all related to each other, they're from different sensors. However, usually what I'm trying to do is I'm trying to do a sensor fusion of all of those individual measurements to derive some physical knowledge of where I am in space and time. So when I'm dealing with devices like an inertial measurement unit and IMU, I'll oftentimes find that IMUs will have nine degrees of freedom associated with them but if I'm trying to do this as a drone, I wanna have a tenth degree, which is the altitude. Now, altitude is measured as barometric pressure. So I have to be able to add a barometric pressure sensor to it and then calibrate the barometric pressure sensor so that I know how high, high actually is. Because if I don't know where C level is and I haven't calibrated the sensor, I have no idea how high I am. Of course, there are other ways to measure things like height and this is where we start getting into position sensors. One of the simplest position sensors is this one that we see up here in the upper right hand corner. These are limit switches and you would think that, well, exactly what am I limiting? And oftentimes when I'm trying to do robotic arms, for instance, I'm going to have a limit sensor associated with that because the arm has a motor attached to it, the arm is moving. When I get to a certain point, I'm supposed to have a sensor that tells me to stop but if that sensor, the measurement sensor that's actually measuring the height of the arm isn't working correctly or is intermittent in some way, I'm going to want to fail safe. I'm going to want a way to guarantee that it doesn't break because I've put too much torque on the arm. I want to put a limit sensor there, a limit switch that says, okay, either you're told to stop because of the real sensor or you've hit the limit and you need to stop now. So oftentimes these limit switches will be there as fail safe operations. Now my company, we do space robots for NASA. We're actually on the space servicing mission, we work with the space servicing directorate at NASA Goddard and we're working right now on a project called Restore L which is a mechanism for being able to fly up and grapple a satellite in orbit, basically pull back the mylar, cut the safety wires, unscrew the filler cap, refuel it, put everything back, close up the mylar and then let go of the satellite without having perturbed its orbit. And this particular mission is in low earth orbit, so we're traveling a little over six kilometers a second. And trying to sneak up on a satellite that you don't exactly know what its orientation is and grapple it at six kilometers a second is a challenge to say the least. And if we didn't have extreme sensor systems to be able to do this, there's no way anyone would allow us to even come close to trying to accomplish this mission. Because as it turns out, there are international treaties that say if you create debris in space, that's a violation of international treaties. So of course NASA's folks that are in charge of all this are like, you better not run into this thing and cause a debris cloud because if you do, we're in deep trouble. So that's, but those are the kinds of things that when we start talking about position, we have several ways of measuring position. Now in the case of a limit switch, that's a digital sensor. Basically it's either on or it's off. So I can use a GPIO and I can just simply pull it. I don't have to do anything significant out of that. I don't have to have it interrupt driven or anything crazy like that. I can just look at it periodically and say, hey, is the sensor engaged? And if it is, stop doing what you're doing. On the other hand, some of the position sensors are actually analog sensors. And we see here a potentiometer. A potentiometer has three inputs or three wires coming to it rather. We have the input and the drain and then we have the wiper here. So what's going to happen is I've got power coming into the circuit and then the wiper here is pulling power off the circuit and where the wiper is determines the voltage that I'm pulling off the circuit. So this is an analog circuit. And potentiometers are one of those things that in the industrial IO subsystem for Linux, it inherited potentiometers. They actually were in the MISC subsystem for quite a while and they said, well, okay, it doesn't really fit in MISC. It's kind of a sensor, so we should move it over to industrial IO. The issue of course, we can also get into things like this little beastie. This is a rotary switch with a resistor ladder on it. So the problem with a typical potentiometer is you have effectively an infinite number of voltages between completely off and completely on. You just wiggle it a little bit and the voltage changes. On the other hand, oftentimes we want to have detents. We want to make sure that we only get certain voltages in certain positions. So that's where things like this rotary switch with potentiometer, I mean with a resistor ladder in it can come into play. So when we're dealing with an A to D, we're doing analog to digital conversion, we need to know the voltage and we need to know the resolution. If we know the voltage and the resolution, make sure that we've matched those things up. And for instance, I've had cases where I had a 3.3 volt circuit, but the sensor was a five volt sensor. So what I needed to do is to put a level shifter in there and shift from five volts to 3.3 volts and then my full length of measurement, the full 4096 value, which was a 12 bit A to D, I'm now measuring the 12 bit A to D scaled down from five volts to 3.3 volts. Fortunately, you don't have to build those kinds of circuits because companies like SparkFun and Adafruit and several of the other hacker, maker community sites sell those things and you can buy them for five volt to three volt, five volt to 1.8, 3.3 to 1.8. Most of those voltage combinations already exist so you don't actually have to go and build anything like that. You can just simply buy the little circuit board and have it implemented directly. So it's a little easier to work with. Now for position and speed sensors, there are a couple of things that we can use. One of them is something called a Hall Effect Sensor. And with a Hall Effect Sensor, we're looking at a magnetic field being generated in the sensor based on the presence or absence of a magnetic core. That is some chunk of metal here that's spinning. And in this case, when it comes to the gap in that chunk of metal, it then says, oh, there's no voltage here. And then as the gap spins by, now we get a voltage again. So I can just simply count pulses. And this is the way that you're, well, this is the way you used to do this. You don't do it this way anymore, but it's why your automobile used to implement their cruise control. Basically it was looking at the gear associated with the motor control. And as the gear was spinning by, basically it would then count the gear teeth. And you can actually count the gear teeth as they go by. So these can be very accurate sensors. We also see these things being used as a tachometer. So we're trying to figure out how fast a particular wheel is spinning. Well, we'll just attach a piece of metal to it and we'll count the number of pulses that go by and then we'll be able to determine how fast the wheel is moving because we know the diameter of the wheel and things of that sort. So something like this would typically be attached to a GPIO. And of course the pulses would just be counted with an interrupt or they could be polled depending on the speed that we're dealing with. Now a good application, this is a good application for something like the Beagle Bones Programmable Real Time Unit where you can basically just kind of dedicate a processor to sit there and watch this. One of the things that we're concerned about when we're dealing with very high rate polling sensors or very high rate data, high data rate sensors is we don't wanna put too much strain on the Linux subsystem. And it's simply because Linux has got lots of other things it's doing. This is not the only thing it's doing. It's not just simply counting pulses on a tachometer. It's doing that and a whole bunch of other things. So we don't want the high speed sensor to override the other capabilities that we have to have. The reason we're using Linux in the first place is because it's got this fabulous network stack and all these other capabilities, multi-core and multi-processing. We don't wanna screw any of that up by having an incredibly high polling rate that it's going to consume most of the CPU. So now, hello. My thing is not sensing. What is it doing? Let's try this again. I'm not sure why I did that, but let's try and see if we can get back to where we were. There we go. Now, some other sensors that we have. This little gizmo that you see up here is called a rotary encoder. This generates a number of pulses per revolution. And by knowing how many pulses have gone by, we know how far it's moved. For instance, in robotics, I happen to use a lot of 360 pulse per revolution devices because then it makes it real easy. One pulse is one degree. And I can explain that to high school students, even if they've only had geometry, I can explain how to translate that into a distance that's actually been moved on the floor. These have a number of pulses per revolution. And the problem is that most of these sensors will output this as a raw pulse count. So I have to have something that's sitting there being able to get those pulses off the device as fast as the device is actually running. Well, this particular one is from a company called Grayhill and it can go up to 10,000 RPM, which is fairly fast. So you need to be able to sample at those kinds of data rates. And that, of course, gets very complicated. So we need to watch out. We can have very accurate sensors, very high data rate sensors, but we have to make sure that the Linux system that we're actually using to read that information can handle the data rates that we're talking about. And then kind of the last one of these that we'll talk about is this, one that's referred to as a quad shaft encoder. It's also known as a quadrature encoder. And what a quadrature encoder does is it has two pulses that are generated. And it's got an A and a B channel. And if the A channel leads the B channel by 90 degrees in phase, it's going forward. And if the B channel leads the A channel by 90 degrees in phase, it's going backwards. So if we're trying to determine an absolute reference, then we need to figure out, are we going forward or are we going backward? And you may in fact have one wheel going forward and the other wheel going backwards, because you're spinning. So when you're doing things like that, you now have to be able to keep track of two very high performance data pulses coming in and understand their relative phase from each other. Is one is the A leading or is the B leading to determine whether we're going forward or backwards? Fortunately, a lot of these can be tied into what they call ESCs, electronic speed controls, which are off-board devices that are then talked to by Linux. We will send a command to the ESC and go turn at 10,000 RPM. The ESC will then handle that because the ESC has a microcontroller that's built inside of it. So it offloads a lot of the details of keeping track of all this bookkeeping. Now, when we get to proximity sensors, here things get really interesting. In most cases, proximity sensors that are not going to cost you a fortune are going to be time-of-flight sensors. So we're gonna send a signal out. We're going to measure how long it takes for that signal to come back and then based on that measurement, we'll be able to determine how far away you are. So it's like a radar or sonar. And these can be in different modalities. This particular one right here happens to be infrared. We have ultrasonic and this is referred to as the infamous ping sensor. It has one that actually emits and the other receives. And then we have these new ones. This is a LiDAR. So this is an infrared laser. It's a class one laser. And with the infrared laser, we've got one emitter and our collector over here. And the nice thing about the LiDAR is that it's extremely accurate. It's very precise. Unfortunately, sometimes almost too precise. We did have one that was attached to one of our robots and the game piece that we were working with had, it was like a castle and it had a little hole cut for a door, a portcullis. And so as we're scanning to find out how far away the castle tower is, so we can figure out how to shoot this rock at it, we encountered the wall, we encountered the door and the laser was so accurate that it told us that the wall was an extra three feet away because we weren't measuring the leading edge of the tower. We were measuring the trailing edge of the tower because the laser was so incredibly accurate. So some of these you do have to kind of take with a grain of salt. Most of these sensors are interfaced using either I squared C or spy. But in the commercially available not so incredibly expensive world, you get a range of about 30 feet or so. In some cases you get them up to as much as 30 meters. Question. It depends on the sensor. This particular one from Sanyo actually uses time of flight, but there are those that do use reflected intensity. So it just depends. Now reflected intensity has its own problems because now if I'm against a shiny background versus a dull background, I may or may not be able to actually see the reflected intensity. So when we're dealing with these things, we have to know what's the expected range of the sensor? What does the pattern look like coming out of it? In the case of the ultrasonics, they can either be a very tight beam or they can be a very wide beam. And in the case of the laser, they're typically spot on just a particular spot in space in space. So fortunately what's happening is rotational lidars. That is the ones that actually spin that you see on the, has anybody seen the self-driving Uber cars? Couple of you have? Yeah, we have them in Pittsburgh. And you see this little bubble on the top of it that's spinning around. That's a rotational lidar. And it's basically measuring at like 5,000 Hertz, 5,000 samples every time the thing goes around and it's using that to create what's referred to as a point cloud. So I'm taking the distance that I'm measuring and I'm creating a model of everything that's around me based on that distance. Well, those rotational lidars used to be very expensive. They were always a thousand dollars or more. Now you can actually get them less than 99 dollars. So there's a couple of them that they've actually repurposed from Nido robots, the little sweeping vacuum robots. They actually had a lidar in them. So they've taken those out and they've repurposed them. So you can now pick up a spinning lidar for less than 99 bucks. And that one happens to use a serial port I think is its interface. So that of course adds more complications because when we're riding the driver for the industrial IO subsystem, now we need to take into account the physical mechanics of the system. It's polling rate, what its accuracy is, what kind of distance can it actually report, all that sort of business. Now when we're measuring acceleration, acceleration is measured in terms of Gs and normal gravity at sea level is considered to be one G. So you'll notice here on this particular chip we have an X and a Y and a Z indicated on here, which indicates the orientation of the sensor itself. So when we're trying to use a sensor like this to determine the roll pitch and yaw associated with the device in space, then we need to understand how that sensor was mounted. If it's mounted flat with X, Y and Z, so I know Z is this direction, X is that way and Y is that way. But if I mounted it like this, because I didn't have room to mount it like that, now everything is shifted. So I need to make sure that I understand the orientation of the sensor so that I can understand what the measurements are. Maybe X is not X anymore, maybe X is Z. So I have to understand exactly what that is and this is one of the things that I always get into case with my electrical folks on the kids on the robot team because they'll just say, oh, it's a sensor, just stick it in there. It's like, no, no, no, no. Some of these sensors you have to know exactly how they're stuck in there. Excuse me, we have to write the software accordingly. These can also be used for detecting impacts. And this is important because of another sensor called the gyro. The gyroscope is very sensitive to things like impacts. So if you've got two robots, both of them are doing like 12 and a half feet per second across the floor and they hit each other, you've basically got a 40 mile an hour collision. And when you have a collision like that, it's gonna screw your sensors up. So in the case of the gyro, the gyro tends to drift anyway and then you get a strong hit like that. You need to know that you've got to readjust the gyro because it's probably not where you think it is. So gyroscopes, they don't actually measure an absolute value, they measure the rate of turn. So it's not like they're going to say, oh yes, you just turned to 90 degrees. It's no, you're traveling at six feet per second in a particular direction. So gyroscopes can in fact be jockeyed around to produce an absolute direction, but it requires a lot of sensor fusion on the case of the gyro, which adds a lot more math to the problem. So you need to make sure that if you're going to do that in your drivers, the drivers have to be able to do that. And of course there's no floating point in the kernel. So it means you have to use fixed point operations in order to do that kind of math. Or you offload it to a microcontroller someplace and let the microcontroller do it. Another way of measuring distance, we have a magnetometer, it's a digital compass. The problem with magnetometers are they measure magnetic north. And unfortunately, if you're near very large motors, very large motors have very large magnetic fields. So magnetic north may not be necessarily where you think it is. In addition, if we're trying to use this to measure true north, we need to understand how far off the magnetic north pole is from the true north pole. And we now know that the magnetic north pole is now drifting. So it's not where it was 20 years ago. And it continues to drift towards Siberia. So that poses even more interesting problems for navigation. And of course, if we're trying to use Linux in a navigation system, we need to make sure we take that into account. And then we get kind of the Mac Daddy of all these sensors, which is the inertial measurement unit. This particular one has got an STM32 microcontroller on it to do the integration and fusion of the sensors. But it turns out that the IOC, excuse me, the industrial interface there in Linux, actually has support for this MPU9050 9DOF sensor. So we now can use either the raw sensor inputs or we can use the cooked sensor output from the microcontroller, depending on what it is we're trying to do. And if I just add a barometric pressure sensor to this, now I've got the makings of a drone that I could then use with the Linux drone project. So now other sensors, we've got strain gauges and pressure sensors. There are a number of these things that are out there. Gas sensors, whether we're measuring carbon dioxide, CO2, liquid petroleum, methane, you name it. The weird thing about some of these gas sensors is that they actually have a heated element in them in order to be able to measure the gas they're trying to measure. So if your gas is explosive, like isobutane, then be careful what sensor you're using. You don't want the sensor to actually set the gas off. That would be a bad thing. Fortunately, the industrial IO subsystem has examples of all of this. And so you're not actually cutting any new trails here to add these kinds of things. So that's all on the input side. Now we're gonna talk about the output side. And the output side, yes, there is a possibility that we actually have a digital to analog controller, but more likely than not, we're actually using something called a pulse width modulation signal. So now a PWM signal allows me to approximate a voltage. So here is an example of PWM. In the PWM world, we have a frequency that we're using and a duty cycle. So let's assume that just from a simple perspective, let's say that our maximum voltage is 12 volts, or minimum voltage is zero volts. So if I'm on and off, if I switch on and off, and I'm on and off 50% of the time, so I'm off 50% of the time, on 50% of the time, I average out to six volts. And that's basically the concept that PWM actually works on. So it's looking at how often you're on, how often you're off, and then measuring that against the maximum voltage to determine what voltage you're actually outputting. In a Linux, the PWM subsystem is actually used in many, many different places. First of all, it's often used with LEDs. So if you have, you know, if you go into suspend mode, and there's a little LED that breathes back and forth, well, what it's actually doing is it's turning the LED on and off very, very fast. And so the more it turns it on, the brighter it gets, the less it turns it on, the dimmer it gets. So that breathing LED effect is actually a PWM circuit that's making the LED do that. Other places that we see PWM are in backlights for your laptops. You have the ability to change the amount of brightness in the backlight. Well, there's actually a PWM in there that's turning on and off the LEDs in the backlight to approximate a voltage. So when you tell it to go a little bit dimmer, it's basically cutting back the duty cycle on the PWM. And of course, the other one that we see this used in is fans. So oftentimes we'll have fans inside of our units. These fans are also controlled typically through PWM because we don't want the fan to be 100% on all the time. That gets really noisy. We wanna be able to have the fan adjust itself based on the temperature of the core so that it will be faster when it needs to be faster and less noisy when it doesn't need to be as fast. So all of those are controlled with PWMs. The problem that we run into with PWM in the Linux world is jitter. So jitter is defined as the amount of time that it either, we get variations in the amount of time it takes to either turn the signal on or variations in the amount of time it takes to turn the signal off. And this example right here happens to be an oscilloscope plot of a trailing edge jitter. So this is trying to turn the signal off and it's not terribly accurate. Now, depending on how we're implementing the PWM, this can be either an incredibly bad problem or a minor annoyance. In this particular example, this one was actually a Python GPIO library that was approximating a PWM signal. So this was on a Raspberry Pi and the Raspberry Pi actually has a Python library called PWM and you set the duty cycle and you set the frequency and it starts generating a PWM signal but it's doing that through a GPIO. So that GPIO is being controlled through software in Python which is not all that fast in environment to begin with and now we have the potential for being for context switches and things of that sort to take it offline. And that's what's causing this jitter. Now, if we're dealing with LEDs and laptop screen backlights, this is not as big a problem because the human eye has persistence. That is, what's the refresh rate of the human eye? It's 24 hertz. So that's why movies are shot in 24 frames per second because the human eye, if it's 24 frames per second or faster, the human eye integrates the motion together so you can't see the individual pictures. It looks like continuous motion. So in cases like that, and an LED also has persistence. When I turn an LED off, it's not immediately off. The LED dims and then finally goes off. So when I take the account into account, the human eye's persistence of light and the LED's persistence, if my jitter is out there, it's not gonna be so big of a problem because the LED is not quite off, but the human eye integrates that anyway and makes it all look smooth. Where you do see some problems is when you get like 60 hertz coupling in from fluorescent lights and it starts to cause it to do a pulsing sort of thing. That can be a bit of a nuisance, but again, for the most part, if we're just talking about LEDs and fans and backlights, the subsystem using a, even a software-based GPIO PWM is good enough. However, that puts a significant load on the system to be able to be toggling that GPIO at a particular rate. So we need to find some other way of doing it. And oftentimes this is implemented using an actual PWM circuit. So even with preempt RT, giving us the real-time capabilities, we are still not able to meet these jitter requirements for some of the more sensitive objects like sensors and stepper motors. So we wanna be able to fix that. So in order to fix that problem, we have several different approaches. One is we can put a PWM generator circuit out there. And into the circuit, we just simply load the duty cycle and we load the frequency and it starts generating PWM. And it's a completely separate circuit. It's not controlled by any user space software other than to set up the registers. Once the registers are set, it's done and it starts generating PWM. The other option is to take advantage of a microcontroller like the PRU and the Cetera parts. Or we're also now starting to see many cases of mixed architecture ports where in the mixed architecture, we're seeing a Cortex-A and a Cortex-M in the same physical piece of silicon. So if your system on chip has kind of mixed architectures like that, you can carve off the PWM work, dedicate it to the microcontroller. The microcontroller isn't running an operating system usually. So the microcontroller just simply generates the PWM signal and that's all it does. And that offloads that effort from the processor so the processor can go off and do other things. So Linux actually has a PWM subsystem. And for a requirements just such as backlights and fans, that's originally what it was designed for. Now the PWM subsystem is manifest typically in slash sys. Occasionally you'll see elements of it in slash dev. But if you're interested in looking at the kernel documentation for how the PWM subsystem is implemented, then I highly encourage you to go to the documentation inside of the kernel source. That documentation is actually pretty good. And that's one of the things I found in general for the kernel source documentation. For the most part it's pretty decent. But it gives you a pretty good idea of what's happening in the PWM subsystem. But let's quickly go through this so that we understand exactly how this thing is supposed to work. We have of course, assuming that we've got sysfs turned on inside of the kernel and that means most modern kernels do. Some of the older kernels may not, but most modern ones do. You will have then the slash sys file system. And if the PWM class exists in that file system, we will then have sys class PWM. In that PWM directory, we'll see a whole bunch of things that are exported as PWM chip in. And there may be more than one PWM circuit in there. And we may have more than one PWM channel per PWM circuit. So it depends on how the interface board was created and how many PWMs it has. Also understand that the number and location of the PWMs are typically gonna be determined by the device trees. So if you're using ARM architecture, for instance, you're gonna have to look in the device trees to figure out where the PWMs are and how many of them there are. They may not necessarily have a device driver in the kernel at this point. You may have to write your own. Again, model it after existing PWM device drivers in the kernel so that you don't have to reinvent that wheel. But inside of that directory, we will also, if we cede into the PWM chip slash in, you know, a PWM chip in, whatever that one is, we'll see a value in there called in PWM. That's the number of PWM channels that that particular chip supports. Then when I export a PWM channel, then it will basically start clocking the PWM out. And there are a lot of additional properties that are made visible. The properties that we'll typically encounter are going to be the period, which is a read-write. Now, of course, all of this is only accessible by root. So you're gonna have to be super user to dork around with any of this stuff. But you can do it straight from the command line. So you don't have to write a piece of code to do it. You can do it from the shell. So we can write the period, which is the total period in nanoseconds. We have the duty cycle, which is the active time of the PWM signal. So if we put in a 50, then it's going to be on 50% of the time. If we also have the option for polarity. Now, you don't normally use the polarity option because what that does is that switches the values from zero to positive to zero to negative. And some hardware circuits actually use negative voltages as positive numbers. So in those cases, you could potentially have a requirement for using the polarity. And then we finally have the enable option out here, which will turn on or turn off the PWM signal. So obviously the PWM channel has to be connected to something. Otherwise you're not going to be able to see it. And this is typically going to be done either via hard wire or via the pin mux. So if you've got a board that supports a pin mux, you can then take the PWM line and allocate it to a physically accessible pin on the part so you can then attach it to your external PWM device. So now what happens if I'm trying to do a drone? If we take a look at the PWM subsystem in Linux, it's typically got circuits that are dedicated to functions inside of the board. So for instance, I've got LEDs that are going to be pulsing. I've got fans, I've got whatever. So those PWM circuits are almost always already used. You might find one or two of them that are not being used, that you might be able to wire out. But if you're trying to do things like control a 3D printer or control a drone, chances are you're gonna have at least four external motors. So these motors have to be very accurately controlled. If you've ever seen a stepper motor that is not getting an accurate PWM signal to it, it literally vibrates itself to death. It will destroy the motor. And some of these motors are big. They're controlling big arms or big servos, whatever. And as a consequence, if you destroy it, you're out thousands of dollars. So being able to guarantee that you've got a rock solid PWM signal going out to it is a very important piece. You need to know what the frequency is and you need to know what the duty cycle is. If we take a look at a typical hobby RC servo, that operates at 1500 Hertz. And then 1500 Hertz plus or minus 20% means it's either all the way to the left or all the way to the right. So again, we have to be extremely accurate in the values that we send out in order to be able to get it to move the way we want it to move. So we do have cases like here, NXP has their IMX8. You'll notice that it has both an arm, Cortex-A part, which is up to four cores and a Cortex-M part here, which is responsible and actually attached to the PWM lines. So I might be able to wire something out there, yeah. Oh, I'm sorry, yes, you're right, 1500 microseconds. You are absolutely correct. I stand corrected on that one. The other aspect that we have is we can have boards like this. And this particular board is a USB controlled 16 channel stepper motor controller board. So we can get multiple devices attached to this. We send one command using USB, LibUSB, it then talks to the device, sets all the stepper motors into the right position and holds that PWM signal for me. This would be the kind of thing we would use for a 3D printer, for instance. So sometimes these individual boards can be, of course, using UARTs. In the case of the USB interface, it's actually a UART interface. We can also see I squared C, spy, or others. One of the things that I do frequently is I will tie an external microcontroller, an Arduino or something like that to a controller board. Just because the Arduino is easy to code to write and get everything working with the PWM, make sure everything's functioning correctly and then I'll take the time to actually port it into the kernel and implement it inside the kernel. Assuming that I have a PWM signal that's available to me, a PWM line. If that's not the case, I may actually leave the Arduino in the circuit. I may actually just simply go, okay, yeah, just send the command to the Arduino, let the Arduino do the real-time piece and we'll do the rest of it inside of the Linux side of it. So there's an incredible variety when it comes to sensors. And depending on the application, fortunately, the industrial IO subsystem already has a lot of implementations of these circuits. So if we're just dealing with backlights and on-board PWM sources, they'll likely work fine for what they're designed for. But if we're trying to control external devices like stepper motors and servos and things of that sort, we'll oftentimes have to go to off-board circuitry in order to make that function. And that may then be integrated into the industrial IO system as a device driver that simply talks across I squared C or spy to this external device. Or it may be done through either Ethernet or USB, whichever one happens to work best. So let me just show you a real quick example of this. So I have here a Beagle board, I've got a Pocket Beagle and in the Pocket Beagle it has a PWM system out there. In this case, we see SysClass PWM, we see PWM chip zero, chip two and chip four. So if we CD'd into chip four, let's say, and do an LS in there, we see this NPWM. So let's take a look at MPWM and it shows two. So I've got two PWM lines inside of that particular chip. So now what I would like to do is I would like to make sure that this particular device is on and active. So what I'm gonna do is I'm going to export it. So at this point now I have exported this and this is actually clocking at this point. It doesn't have any values in it yet. So it's not turned on, but it's enabled. And then once it's enabled, I can now go into, if I take a look at this, you'll notice that it just popped up that thing that says PWM four colon one. So that's chip four channel one. So let's go into chip four channel one and take a look in there. And you'll notice there is a polarity, there's a period, we have the duty cycle out there. So if we were to then take a look at this, we could do a cat on run. And then, whoops, that's the wrong place here. Oh yeah, so what we can do here is we can set up the duty cycle so we can do an echo. Let's see if it's stopped. My little thing here's goofing up a little bit here. So I want to set the duty, I want to set the period and the polarity and then enable it. So let's assume that we're going to go and set the duty to 250 nanoseconds. 250,000 nanoseconds. Can't really see if that, oh permission denied. Yes, of course permission denied. Oh, that should have been rude. That I just seeded someplace else. All right, all right. So now let's try that again. We will echo 250,000 into the duty. That still says permission. Yeah, no, that should be, I'm rude. All right, I didn't sacrifice enough to the demo guides this morning. Oh, duty cycle. Oh, yeah, yeah, you're right. There we go. Invalid argument. It's just not going to work for me. But in any case, I would typically then, yes, yeah, invalid argument. Okay, whatever, I'm messing it up somehow. Yeah, huh, well, in any case, if I had this working correctly, if I could actually see what I'm doing here, then I could effectively say that on this particular device, the PWM maximum voltage is 3.3 volts. So if I set the duty cycle to half of that, so if I set the duty cycle to 5,000, excuse me, 250,000, this particular one is supposed to be at full voltage at 500,000 nanoseconds. So if I did a cat on the period, oops, yeah, cat, period. So it's currently off. So ideally, I would set a value to that. I would echo, it's supposed to be 500,000. Yeah, it's a mist of no. And that's supposed to be echoed to the period. Okay, that worked. There we go, cool. Now let's try and see if I can echo this to the duty cycle. Echo 250. Yeah, well, no, no. In this particular case, it's measured in terms of nanoseconds. So if the maximum period is 500,000 nanoseconds, then if I wanted to be half of that, I would send it 250,000 nanoseconds as the duty cycle. But, well, that depends on how this implementation is done. So let's see what happens if I just echo 50 as in 50%. Maybe they're doing that. Yeah, hey, yippee. All right, so this one measures it in terms of actual percent duty. So now I've got a frequency of 500,000 nanoseconds and a duty cycle of 50%, which is giving me one half of the maximum voltage swing. The maximum voltage swing on this particular card is 3.3 volts. So I'm actually seeing about 1.65 volts if I could actually set there and measure it. I do have an oscilloscope, but there was just no way I was gonna be able to get it set up and projecting correctly. So you'll have to take my word on that one. All right, so that's kind of the idea behind how this system works. So you should at least have an idea at the conclusion here. We have the industrial IO system. It is made specifically for those kinds of sensors that are very high polling rates or very high interrupt rates, much higher than what we would see in the HWMON subsystem. There are currently about 200 of those drivers that are in the kernel. Some of them are in the staging tree. Some of them are actually accepted at this point. But fortunately, you have examples of almost every type of sensor that we've talked about here. And you can then use that as a mechanism to build your own drivers from there. Obviously, when we start increasing the number of hybrid SOCs where I have both Cortex-A and Cortex-M or Cortex-A and things like the Cetera with the programmable real-time units, we now have the ability to do much more fine-grain control over things like PWM inside of the Linux subsystem because we're basically offloading the actual PWM circuit, the actual PWM generation, to a separate circuit someplace. And that's actually doing all the real work. We just, from the Linux perspective, we're just using Linux for command and control and the rest of it is being done through an external circuit. And we can always use external microcontrollers like Arduino's and things of that sort to build these things. Just as a way of getting started, I've got several robots that actually, if you have a Raspberry Pi, there's a thing that I think Seed Studio makes, which is a plug-over Arduino that plugs right onto the Raspberry Pi 40-pin header and then gives you all the access to the Raspberry Pi signals at the Arduino level and of course all the Arduino signals coming off the Arduino as well. So it's really handy. The weird thing is it doesn't use the serial port coming off of the Pi. I actually have to run a USB interface from the Pi to the Arduino to make it work. That's really weird, but in any case, we're starting to see a lot more of these hybrid circuits, especially with the IMX-8 where they actually have, effectively, and it actually uses the Arduino IDE. You can actually get access to the microcontroller using Arduino IDE and write your code using just as you would a normal Arduino. And they will often even have the Arduino pinouts on the board itself. So you can then use all the Arduino capes or pads or whatever they call it. They're mezzanine bus, whatever they call those things. Shields, shields. So you can actually use the Arduino shields and do all the electrical interfacing that way. So with any luck, this has been of some use to you and I'll even get you out five minutes early. So any questions? Mercifully no, oh yes. They actually have two different ones. So there is one that's through the PRU. So if you wanna go through the PRU, that requires a whole bunch more of additional programming. That particular one that you were seeing was just on the normal SOC. Yeah, that's correct. Yeah, and that's why I don't normally use the PWMs that are on board these things because they're usually made for LEDs and back lights. So they're not terribly accurate. And it depends on how that particular PWM implementation was done. You have to basically look in the implementation to see exactly how it was implemented. But yeah, I wouldn't, I mean, it depends. In this particular one, you were giving it a number between zero and 100, but not all of them worked that way because I've worked with ones, which was what I was trying to do before where you actually specify how many nanoseconds the thing is active and how many nanoseconds is its actual period. Oftentimes, but just as often I've come across boards that say, oh, here you go, it's got a PWM on it. That's fantastic. Where's the driver? We have a datasheet. If you write it, we'll be very happy. And then of course, then it's up to you to try and upstream it, but they may just say, well, send it back to us. Oftentimes, I've actually negotiated free equipment that way. I say, well, yeah, you've got something that I really wanna use, but I'm not gonna pay you for it if it doesn't work. So give it to me for free and unlimited tech support and I'll write a driver for you and then you can use it for anybody else who wants it. And that model actually works reasonably well. A lot of manufacturers will do it that way because they don't have access to Linux kernel developers that can actually write this kind of code. So they're willing to cut a deal. So if you feel so inclined, you can get a lot of cool hardware. I got tons of cool hardware in my shop just because I needed to write a driver for one and I convinced them to give it to me. Any other questions? Cool, thank you very much.