 Technical difficulties with slides, so I'm going to go out without the slides. So you can enjoy this. This is Grand Rapids. This is where I hail from. And welcome. So my name is Josh Francisco. The title of this talk is Hitting the Pavement with Python. And just a little bit about myself first before I get into it. I work for a company called Blue Midora. We write monitoring solutions for enterprise-y things, mostly VMware stuff, but we're branching out. We have like a database monitoring service as well. So I got my start in Python by just kind of necessity. Started working with like bash grips and stuff like that when I started working at Blue Midora a lot. So naturally, as you evolve from scripting, you move on to like real languages, a lot more robust, things like that. So I started out as an intern, then I was a QA engineer for a while, and now I'm a DevOps. So Python is becoming more and more a part of my everyday life, basically. And that's not just at work, that's at home too. So the project I'm presenting to you guys is a personal project of mine that's all done in my free time. So the only reason I'm here is a friend of mine just was like, hey, give this talk at the local Python users group. And so I did that, and he's like, you should submit it to PyOhio, and I'm here. So I did not actually expect for the talk to even get accepted, so thank you. The project started originally in Ruby. That was an experience. So after experiencing Ruby, my friend Dan, he's in the audience, was like, hey, you should try Python, and Python just worked for me. Something about writing Python was a lot easier for me to wrap my head around, because I don't have a software background. I was a machinist before I got into Blue Midora. So I was in manufacturing and things like that. So it was a lot easier for my machinist brain to take in Python than it was to take on whatever Ruby is. So today actually marks the public release of my project, so you guys can check it out. There is unfortunately a little bit of hardware dependency because of the nature of what I'm doing, but so this is an aftermarket engine management controller. So if you modify engines up to a certain point, the factory computer can no longer compensate for what that is doing. So eventually it makes sense to move to something more robust. Some controllers you can reprogram, and that's fine. These controllers are unique in that they're very generic, so I can apply this same controller to different engines of makes and models, cylinder, configurations, everything. So what that means is I have the same set of tools and approaches to every single project I do with this. So sort of the open sourcey, Linuxy version of fuel injection. The factory application is in its own right very good to interact with it and do tuning and everything like that. Great application. There's a paid for version that unlocks more features and can do more of the things that I'm doing, but it's still all just communicating with it through the serial bus. So if you can figure out how to talk to it over the serial bus, you can do all the other things that you couldn't normally do without buying the software, which it's worth buying it and all right, but where I'm going with this is the factory tools logging is not very good. It's not very fast and it outputs to a tab delimited text file and you can't really do a lot with that in real time because you'd be trying to read from a file while it's being written to constantly and that just smells bad no matter how you look at it. The application is also this big heavy Java application, which is great because I can run that application on Linux. It's really cool that there's a tool like that that is completely cross-platform, Windows, OSX, Linux, and you can tune your engine on any of those. So engine management controllers are pretty cool little devices. They're actually more akin to an oscilloscope. They measure all these sensors and equipment on the machine operating in real time and we're used to in Python and Webland dealing with millisecond latencies. My code is still operating in that realm of speed, but the controller itself is operating in micro and nanoseconds as far as how it's looking at the engine. It's very, very high speed. So the closer we can get down to that micro and nanosecond range as far as monitoring, we can get a lot more resolution out of our data. Eventually, you just get too much, right, but I digress, so wake up. So why you want to gather data like this is it serves as an objective tool for making changes, right? You make a change, you can see that change, how it's reflected in either gas mileage or performance or whatever thing that you want. So I focus on taking all that data, gathering it and stuffing it into a database so I can look at it and do stuff with it later on because when you're driving a car, you can't really be looking at all this data coming in real time because you'll run into things. So it's a good idea to save it. You can also look and compare a pass at a drag strip. You can compare multiple passes throughout the day and you can see, oh, as the air temperature drops, the car starts to get faster from these sort of things. Now you can see that in your time slip at the drag strip, but what you'll be able to see, and I'll show you here in a second, is you get a lot more resolution. You can see where you are gaining the most performance in a specific spot. And maybe you gained overall, but maybe the engine was actually accelerating slower at a certain point, but your time slip won't reflect that 500 RPM window, right? So Pi EFI is the name of my program. It's a high speed data collection display and analytics platform. So this right here is actual real time data coming from this controller on the desk right now. It's coming in 28 times per second right now. Depending on some parameters and things like that, you can up the collection rate even faster. Hi. No, this, okay. So we'll talk a little bit about my rig here. Yeah, no, that's good. So the controller needs to see an engine doing something in order to generate like RPM data, right? Because that's rotations per minute. So I don't have an engine running on the table, obviously, because we'd all be dying of carbon monoxide poisoning. One sec here. So what I have is an Arduino that is a signal generator basically. So it's generating a square wave signal that represents what an engine would be outputting, right? So that, once you configure it with the tuning programs, because there's different trigger wheels that determine position on the crankshaft and things like that. So you could have a single blip for every rotation, right? That would make sense. Some of these trigger wheels have like 32 teeth on them. And with a square wave signal, that's not 32 measurements, that's 64 measurements because of the up and down every single time. So you can get tons and tons of resolutions. So what that means is the computer can see if the engine's accelerating or decelerating in a lot more precise of a manner. That's kind of outside the scope of this talk a little bit, but still interesting, right? So one of the reasons I ended up using Redis for all this is pure speed. When I first started writing this, I was using Postgrass because everyone uses Postgrass. But what I found out is in my main collection loop right here, it's just over and over when I was writing to Postgrass, that interaction was taking long enough, it was affecting my collection rate by like five or ten times per second. So I was having to do async to fire the data off somewhere else, let Postgrass actually put it in the database. So once I actually kind of fell into Redis on accident, Blue Midora was writing a monitoring product for Redis at the time. So I got to use it and I was like, this is awesome. And it actually happens to be way, way, way easier to use than SQL. It's really, really easy command line, right? So you can even write to Redis from bash if you want to, like in bash scripts. You don't have to use the Redis CLI or anything like that if you don't want to. So how many of you are familiar with NoSQL stuff at all? A few? Okay, so NoSQL data stores, it's usually like key value kind of stuff. Redis happens to translate really well to how you would normally code in Python. So you can, you're used to working with key value pairs all the time, that's bread and butter stuff. So it just translates really nicely into Redis. You're not having to write select statements and things like that. But the downside to that is you don't get some of the relationship, the relational goodies. But you just have to create that relational stuff on your own. So as collections are coming in, I basically store them in a, what's called a Redis hash. So the first collection comes in that is saved into a Redis key, which is different from your normal key value pairs. So and then, so Redis key would be like collection, right? I don't have it displayed right here, but it's similar to this, right? So this would be your Redis key and then spot one on the hash for your first collection and then the result. And then the second one and so on over and over and over and that constantly increments. And as you can see, you don't have an idea of time, of when that came in, right? So what you do is at the same time that you're putting that data in, you write to a separate list in Redis with that index, one, two, three, four, and the time that it happened. So now, if I need to get like 60 seconds, the last 60 seconds worth of data, I query something else that gives me the list of indexes and I can pull those indexes out of the main hash. So make sure that stays awake. Where are my slides? So yeah, Redis was a big godsend that really enabled this to be as fast as it is. Right now, what you're seeing again from Redis is a pub sub mechanism in Redis. So if you can kind of think of it as chat rooms, but they're one directional. So this collector process here, well, actually I'll back up a little bit. How many of you know what Tmux is? This is Tmux. So everything is sort of microservices architecture in here. So I just use Tmux as like my little environment to fire up everything I need to run all at the same time. So Tmux works really great for what I'm doing here. Where was I going? Where was I? I forgot. Redis pub sub. Yes, pub sub, pub sub, yes, OK. So as a collection comes in, I save it and I index it and I do all those things. And then I also publish it to the Redis pub sub channel, if you will. So then this is the display side, and that just receives that pub sub message. So I'm not actually going into the database and checking for the latest record. It's just getting it sent to it automatically. So I'm sure you guys are thinking of like, oh, that's really cool. There's lots of little inter-process communication things you can do with that. And there is. So it's a really simple mechanism. It's not too way, though. So you can't confirm a result like out of the box. I can't say that, oh, yeah, I got that back. You'd have to set up another pub sub in the other direction and write your own sort of handling for that, right? So the display side of this, as you notice, I kind of went for like a retro sort of look, right? Everything's done all completely in shell and Python. There's no other crazy display libraries or anything like that. The colorization stuff, I wrote all that. That's in the repo. I'll pull that up, too. And that is basically, there's a list of colors. And depending on the range, I just picked the one at the top or the bottom or in the middle. And you set the range, as you can see down here. This is sort of defining what a dashboard is. So this right here is health. So we got battery voltage, coolant temp, and we got another coolant right here. So this is actually three columns. It just sort of looks like two. So this is voltage right now. So 14.7 volts. I'm not handling decimals very well in this yet, but you get the idea. So the bar mechanism was kind of fun to write. So check this out. It reacts very fast. You guys can come check this out in a minute and play with it. But, and I also have a little potentiometer up here. And it's very responsive. It's instant. There's very little latency as far as what is actually happening to display time on the screen. I haven't figured out a way to accurately measure it. So if anyone has ideas on that, that'd be great. It'd just be nice to know what sort of latency I was seeing on the screen, how old that data was. Because right now, I can't tell. But it's still very, very quick. So do you guys have any questions? I kind of just wanted to cruise through my spiel really quick and see if you guys had questions or other ideas and stuff for me. Yeah, go ahead. Okay, yeah, that would probably work. So he suggested to have a button and drop something on it and then have a camera to measure that latency. That's not a bad approach. Anyone else got questions? So this is all print statements. This is all print statements. And then I do the ANSI escape sequences to jump in and out of colors and things like that. But this is all just print statements. There's no crazy colorization library here. So I'll tell you a little bit of a trick on how I got it to do this. Cuz it looks really cool, doesn't it? So what I do is I print two lines. So I print this line and then new line. And then I print this line. But then I tell the cursor just to return back to the beginning. So you can see the cursor here. So it gives the illusion that this is static at the bottom. But it's actually just printing over it every single time. Yeah, so right now we're doing it 30 times per second. My system I have at home does it 45 times per second. Thank you, yeah. Yeah, it's just, it's all print statements. It's all very core basic Python. I'm only using the Redis library, trying to think what else. I hate curses. It's some sort of cruel joke that they call it curses. It really is, for some things it's probably better, to be honest. It does handle drawing of basic boxes and things like that a lot better. But one day it just clicked to me, the stuff I wanted to do with it, I could do with Tmux, you know what I mean? Which gives me the boxes and the isolation which I would have had to do in curses. And it'd be all weird and abstract. For this, one of the libraries I'm using is Tmux P, which is a Python Tmux wrapper. So you can figure this dashboard with a YAML, which I can show you guys. Well, this is the dashboard config YAML. I'll cover this real quick. So in Redis there's a concept of a sorted set. So this would be the score of the dashboard. And then this would be the actual dashboard dashboard string. So I just pull this into Python, split. I expect it to be the same number of elements every single time. It's really simple stuff. But I'll show you the Tmux stuff. So yeah, this is Tmux P, really easy wrapper for Tmux. It's, you can see what it's doing right there. So for my command structure, I'm using this library called PythonFire. It's a pretty crazy library. It lets you expose a class and its methods to the CLI directly. Sorry, we're gonna pivot here for a second. So can you, you guys can't really see that, can you? Yeah, all right, that font's so easy to read, right? So this is what Fire does. So this is my pipeline class. The initial bootstrapping into my CLI. So all I do is expose, I pass this class into Fire and it just takes care of this for me. So these are the methods within Pipeline. So then if I go to the next one, these are the methods for that method, which that method dash just returns a class. Nope, nope, it does it. Automatic, super magic. So then like add, those are the parameters directly. There's no guesswork. I don't have to deal with parsing arguments ever. It just does it for you. It's really awesome. If anyone does any CLI stuff, I highly recommend it. Yep, PythonFire, yep. Google wrote it actually, super, super handy, super, super handy. It's one of my bread and butter ones. Yeah, so that would kind of be up to you how you want to get the data to the dashboard part, right? So from an Arduino, you have a couple different routes with an Arduino. I'm using my Arduino as a signal generator for the engine control module. I'm not talking to the Arduino or getting data from it. It's just there, yes. So you can pull up a serial interface with an Arduino, though. So you can gather data from an Arduino that way. So it'd be the same kind of thing that I'm doing. Just you'd have to code yourself around the Arduino's parameters, right? So that's one option. Otherwise, if it's something more like a Raspberry Pi, the way my dashboard stuff is working, the way this stuff is working, these are just key value pairs that I'm passing into this display process. So this dashboard config right here, key, what I want it to print as, that's why it says temp here, not coolant, because coolant is too long. And then shiny is the display parameter. And these are ranges and things like that. And then anyone recognize these guys? What are they? They're from the AML, but more specifically, anyone? It's a Python thing. Yeah, it's Python string formatting. So I'm just plopping that in to do the string formatting, and it justifies it left or right or center. Yeah. I'm lazy. I don't need to write justification. They've already taken care of it. So yeah, what else can we talk about? So serial data is fun to deal with. What's really interesting about this controller is I can send it different commands and get different results back. I'm not doing any of that yet. So this controller, I send it the letter A. That's all I send it. And then it sends me back 212 bytes of just data. It's not JSON. It's not ready to go. It's not pretty. But I know what that result looks like. Every single time. So this is the configuration for that byte string. So this is the position. Now, so these are set bits, like on-off switches. I'm just collecting them as a whole as an unsigned 8-bit integer. So if you didn't have this, you would have literally no idea what you were looking at. I mean, you could start guessing. Because there's only six possibilities for the orientation as far as signed or unsigned, 8, 16, or 32 bits. Question. So this is actually defined by the controller's firmware. So this is the actual config from the controller. And I just parsed it in and did useful stuff with it. So yes. So it's pretty standardized. Now, I can reprogram the controller so that the a command will send back a different arrangement of this. So I coded into this configuration on purpose, because I wanted it to be future-proof for if they update the firmware and change this, I wanted to be able to use it. So this, has anyone dealt with byte streams at all in Python? Not a lot. It's OK, because it sucks. But so you use pack and unpack to do that. You'll have to look into it. But basically, I have to build a character string that represents the orientation of these signed and unsigned 8, 16, and 32-bit integers to unpack the 212 bytes. And then on top of it, because they are forward thinking and serial communications isn't perfect and it's susceptible to noise and things like that, especially in an automotive environment, cars produce tons of electromagnetic interference, tons and tons of it. I don't know if you guys have ever been in a car and the radio has a weird buzz that seems to be attached to the engine? That's what you're hearing, is electromagnetic interference. So there's a CRC32 psycho-redundancy check that you can verify that you did indeed get a good packet back, basically. So I checked that. I think the only times I haven't gotten one back is when I didn't know what I was doing yet. But we were testing this on this laptop here. And I had never run it on this laptop until last night. And I was getting some weird controller problems and things like that, whereas this machine's fine for some reason. So I think it has something to do with either the state of the serial device when it's come up or the Linux kernels or something like that. How are we doing on time? Anyone else? What else we got? Yeah, so his statement was, how would you handle logging to an outside Redis? For that, I would probably want to have something to gather up a bunch of it and just quickly dump it because I don't think the internet would handle this rate of exchange. There's quite a bit of latency, comparatively speaking, sending it over the internet, making sure the request got sent, and then sending another one back to back. And I guess you could do that asynchronously, right? But if that was in a blocking loop, you probably couldn't hit 40 or 50 times per second on a normal internet connection, right? That's kind of my thinking. So to save it to the cloud or something, I'd probably just have something to dump it in a larger packet to another Redis database or maybe Postgres or something, right? Something more suited for archiving. Yeah. So I let this run for eight hours overnight, and it was like two or three gigabytes of memory. I mean, it's trivial. It's 212 bytes. And I'm not even being smart with the data I'm saving. I'm saving the raw data that I get back. So this is what the raw data looks like when it comes back. It's not a byte string Python when I printed this, converted it to whatever it wanted to. But so I'm saving that and then this along with it, which is that just deconstructed into human readable form. I'm not even being smart about how I'm saving it. Yeah, I mean, there's even some additional things you can do with Redis. And different data types that you could probably get it down to storing even less. Because right now, there's certain parameters that are just always like zero. So you're saving 0, 0, 0, 0, 0, 0. Well, if you could dedupe that, you could save even more space. So if I extracted this out and did I don't know what you'd call it a delta situation, I could save even more space. What's that use? Oh, yeah. So he said I could use Zlib to compress it. I can do that in Redis. That is a possibility. Redis will just do that for you if you want it to. There's a CPU hit. I haven't been worried about space yet, so I haven't looked into these possibilities. Yeah. Yeah, so let's check that out. Well, it's not running now. That's not going to work. So granted, this is a quad core MacBook, but CPU usage is still very, very reasonable. Under around 10%, let's just say. I'm not sure how much of that is top and whatnot. But yeah, yeah. And that was another reason what he was saying is if you cut the display, you could save even more CPU. And that was another reason I went with sort of the microservice sort of architecture is CPU performance and things like that. So when it is in a car, I could have a pie using almost no electricity just running and collecting, and I don't have to have full blown hardware in a car. Because the car environment is a really tricky place to put a computer, and it's not just space considerations. You have to think about temperature swings. We're Midwesterners, so it gets below freezing every single year, and it gets pretty hot in the summer. You have a really wide range of temperature fluctuations to deal with, and average consumer hardware isn't built to handle that range. Then you also have vibration. Your Sunday driver car is nice and comfy and doesn't have a lot of vibration, but a high-reving Honda that goes to 8,000 RPM that's singing, the whole car is vibrating. And that does fatigue electronics. I mean, that is a real thing. The boards can vibrate, and the traces can come apart over long times. Yeah, no. So his observation was, does the display limit my collection rate? Because they're decoupled, and I'm using Redis Pub-Sub. If the display, and for some reason, can't catch up, what will happen is the Pub-Sub queue will just grow, while the collection is still going as fast as possible. Yeah. What's going on? Yeah, so the light version of the product collected at 15 times per second. And in my lab setup at home, I'm getting close to 50. And here I'm doing about 30. So I'm still, in a less than ideal situation, I'm still doubling what I had with the factory tool. And it's way more flexible, uses less resources and everything. Yes. Yeah, I'll put the slides up on PyOhio and stuff like that, so you guys can go through and absorb them. I had some interesting metrics around the speed of Redis. So off the top of my head, setting a key in Redis takes three microseconds. And getting it takes one microsecond. So I mean, there was little tidbits like that in there, but I've covered most of the stuff I wanted to get to. Yeah. Yes. Yeah, so eventually, I could code some logic into this and say, I don't know exactly what the situation would be, but I could tell the controller to do something. Perhaps make tuning adjustments, fine adjustments, and things like that. That is a very real possibility. It's just a matter of communicating with it and telling it to do what you need it to. Yeah, anything else? Yeah, check out the repo. Can't show it on the screen, because my slides are busted. But if you just search PyEFI on GitHub, PyEFI, it'll come up. There's probably little sections of my code that would be useful to just about everyone in this room. I mean, it's just kind of like a whole bunch of tools I stacked together to do something useful to me. Like the display side of this could easily be retrofitted to, like this could be ping times or some sort of server health or something. And you could just use the colorization library that I've got here. Or it could be something you hooked into an Arduino and you're getting a temperature from a room and you're just updating it every 10 or 15 seconds. And you could just stream that to here. You can see like this. I didn't really touch on this, but this dashboard configuration updates live from here. So the configuration for this is stored in Redis. Every time I get a new request in on the Pub sub channel, I check the configuration. I pull that down and update this. So if I remove one of these, it just drops out. Or if I add one, it just appears. And because it's in a sorted set, I can put something sandwiched in between here if I want to. So again, something that's probably useful to just about everyone in here. Anybody else? Time's up. So if you guys want to check this out up here for a minute, you can touch it and look at it. Feel free. I'm going to hang out for a few minutes, but we've got a long ride home. So thank you for your time.