 And we're ready to start. So next we have DIY OSC Debug multi-tool app. Alexander's gonna be presenting, so please welcome to the tour camp stage. Folks, oh, there we go. Thank you for coming. My name is Alexander Alsayad, and we'll be talking about debugging simple systems when you're a little input or output starved. And this is also my excuse to talk about two things I'm really passionate about these days, which is OSC, which is open sound control and open source protocol. That's really awesome. We got into details. And ESP32, which is my favorite microcontroller of the year. So first off, who the hell am I? I'm a bit of an oddball in this mix. I am by no means a hacker. I've been a game designer for about 16 years, started in the games industry in 98. It's been a while. I've worked for people like Microids back in the day, Ubisoft on stuff like Splinter Cell, Mass Effect, Bioware, Burkitt Riot Games, or Phoenix Labs, Little Outfit, and Out of Vancouver. And I'm currently working as a senior technical designer at Magic Leap, where I basically get to experiment with the future of virtual and physical interactions. So I'm looking to see how we can inject magic into reality and have the real world actually be the output of virtual decisions and virtual decisions actually having output on real live objects. I'm one of the rare people in the company who actually keep playing around with that. So I've been basically playing at Maker for the past year and a half, just making weird shit happen with my hands and my eyes and all the good stuff. I won't be able to show too much of that, but that gives you some context. In experimenting with hardware, I developed a need, which we'll go into soon, to basically create myself a multi-tool because I had all these heterogeneous devices and I only needed to debug them. I was input-starved and I was output-starved. And so I made myself a little phone app that allowed me to basically push a configuration to my phone, regardless of what I'm connecting to, without having to edit the app and just control stuff remotely. I needed something that was platform-agnostic because I'm working on our prior to your hardware. I'm using PCs, I'm using Arduino's, I'm using our ESP32's. So I needed something to be able to easily talk to all these devices. I am also not the best programmer, so this needed to be simple to implement. And I kept coming up with different ideas, so I needed something that was very versatile and allowed me to just extend fairly easily. So this is kind of what I came up with and all the things I learned along the way. Why? Basically, because yeah, if you're using an Arduino or an ESP32 on a Wi-Fi network and it's controlling a set of LEDs, that's all good. But when something doesn't work and you don't have a display attached to your microcontroller, it can be a pain in the ass to debug. And having to constantly, if I'm using a thing that's got external power, suddenly I can't actually have the LEDs connected while I've got serial going on with my laptop. And basically, I just need to make myself some better tools. So, in my experimentations, I came across one of my favorite protocols right now, which is open sound control. How many people here have ever seen or used open sound control? Cool, we've got a few converts. I'm pretty sure that anybody who's used it is in love with the fact that it is one of the most lightweight, versatile pieces of communication protocols ever invented. You can use it on so many different hardware platforms. We'll get into that. But it is basically designed to be platform-agnostic and extremely lightweight. It's got, at its core, a very simple way of addressing things. So when you're communicating with a device that implements this protocol, okay, you basically send packages to specific addresses. And those addresses basically just look like a folder structure, which may be very easy to have good semantic labeling. You can also make fairly complex trees. This is useful for a number of different reasons. Actually, let's skip here. Because, for example, you can easily do dress matching. So if you want to send a specific message to a number of different subsystems that share parts of an address, you can just use wildcards. And so send all the signals to all things, or specific things. You also can easily, when you send a message, append the data types of your choice. So it supports 64-bit integers, millisecond precision time tags, massive floating point numbers, strings, characters, RGB colors, MIDI signals. I mean, this thing was originally designed for real-time audio applications. So the amount of precision you get is extreme. This is useful in a lot of different contexts. Yeah, I mean, this stuff is made for real-time applications. Which means that if you're actually trying to synchronize multiple effects from multiple different devices, you can actually make this happen. And you can also compensate because every message has this very precise time tag. So you can actually synchronize a bunch of effects by going, I want this to happen at this time. And I know when this message was received, so I know exactly how much time to wait. You can also bundle things for simultaneous effects. So you can send a series of different messages that are all meant to be treated at the exact same time. Again, for real-time applications where you've got complex systems, there's really nothing quite like it. This is basically what an OSC stream looks like. At the lower level, you've got this OSC message. The OSC message contains an address which will be very easy for your program to basically just parse. Like you can literally just have a switch statement that says, if I get a message that the first part of the address is this, then I know to route it to this specific function and deliver the payload to that function. The data is fairly easy to read. It's basically just an array of, and you have an array of the types that this data is gonna be in. So again, you don't have to worry about typing too much and you just rely on this stable structure. You can actually read what the information is gonna be in your data stream directly. Now these messages can be bundled, basically into an OSC bundled. So we've got like the length of message, the OSC message, and multiple messages encapsulated with a specific high-precision timestamp for each of these bundles. So again, if you even have some delay in transmissions that vary from system to system, you can actually compensate for them. And then this gets bundled into a stream, which basically just is a stream of bundles. And as it get received, basically you just can have this constant stream of data fed into your different functions on different hardware. The cool thing is that you can make that happen on UDP. You can make that happen on TCP. You can roll your own. You can do this on system on chips. It supported pretty much the libraries for this on every little system on chip I've ever come across, platform agnostic completely. You want libraries, name your language, somebody's implemented it. You want to use high-end tools like, I'm a game designer, I use Unreal, I know a lot of people use Unity. They are plenty of free-to-use plugins that you just basically dump it in and you can start using it extremely simply. Implementations for this are so lightweight that literally in a couple of hours, you've got bi-directional communication without ever having to worry about sockets or anything. You're just, as long as you get your ports right, then you've got communication. We'll get to that. Now, I'm gonna take a small break to talk about my other big passion these days. This is probably gonna sound like an ad because I love it so much, but no, they're not actually sponsored. The ESP32 microcontroller. How many people here have actually used an ESP32? Oh, good, I can get some converts. If any of you have used Arduino's, TNZ's, or any of those varieties, I strongly suggest you take a look at the new player on the field. They've been around for about two years, like main production. They are incredibly powerful and will go through why, but they are basically this small and have the same amount of GPIOs that you would expect from most microcontrollers, except they do everything better. Let's take a step back in history. So, Espresso for the company who basically does this chip set, started, I think, in the early aughts with the ESP01, which some of you who use it, may have actually used this as your wireless, as your Wi-Fi controller. It was like three or four bucks. It gave you Wi-Fi. It was simple to use. It had pretty good range. I mean, this thing, even though this is this tiny antenna, this thing can broadcast up to like 100 meters. It's nuts. And it was basically the cheapest Wi-Fi solution for any Arduino or Teensies or whatnot. Supported, you know, BGN. It could run as an AP, which was great, but it was still limited by whatever controller you plugged it into. Little later on, the 8266 came out with an ODEMCU, basically, architecture. It's basically the same chip set. You can actually see like the entire thing is just right there. But it's on a board so you can actually have like USB and you've got CPUs and you've got all your GPI opens. And you can kind of use it as an Arduino, but from what I saw, most people mostly use it as a wireless LED controller because out of the box, it just did that awesome. But, and I mean, it had way more memory than an Arduino so you can actually control, say like a thousand LEDs with one of these. So there were a lot of like Burning Man projects that used these for wireless controllers because they were just super cheap and pretty reliable. And like with 80K of user data, that's pretty sizable compared to like the 4K that an Arduino has. But now big brother is here and the USB 32 is just freaking awesome. This thing actually has dual core processor. One of the cores is dedicated to the hardware. So whatever you're actually running on your sensors won't actually reduce the rate at which your program runs. Like dedicated actual hardware is makes a really big difference because if you're running on timings and suddenly because you've changed how one of your sensor works, you're just running slower and suddenly your delays don't work the same, it can be messy. This makes it much, much easier. Also 160 Hertz at minimum, 240 megahertz. I mean, this thing just runs. You can throw a lot of math at it and it won't care. 520 kilobytes of SRAM for a tiny microcontroller is a lot. You can control tens of thousands of LEDs with this one chip. Also on top of surviving you super easy 2.4 gigahertz Wi-Fi, this thing also has Bluetooth on a chip because you know, why not? So just to put things in perspective, like the biggest competitors, like the good Arduinos, 2.5 kilobytes of RAM, 16 megahertz clock, like that's almost 10X on every metric. Even the bigger boys like the Duo and the Teensie, like they're still really far back and sometimes they cost more. Now, if you're into input outputs, this is where this thing gets incredible. It's got DMAs. By default, some of these are just already touched capacitive. You just can just use touch directly on the pads themselves. They're a little tiny, but just connect wires and you've got touch controls immediately. It's got ADCs and DACs. Take that Arduino, take that Raspberry Pi. It's got I2C, so you wanna have good little screens. No problems. It's got UART, obviously. It's got Can-2. I haven't played around with it, but it seems like an incredibly powerful way of actually doing hardware networking of multiple devices at super low latencies. It's got SPI. It's got I2S. I don't know if any of you have ever tried to run audio through microcontrollers, but it can get gnarly. If you're actually looking for quality audio, there's anything under I2S will just shred it. Like there's just no way of using analog GPIO or power modulation to have sound that sounds good. It's just not made for it. This thing will. It's actually designed from the ground up to be and it is completely supported on a number of these pins. Obviously, it also supports power modulation, but that's pretty much every microcontroller these days. As far as programming for it, you also have your choice. Off the bat, it supports the actual Arduino IDE. So if you're already using that, you can just continue using that. It'll just work. There are some library support issues. I mean, this thing's only been around really in lots of hands for about a year, so there's still a lot of work on libraries, but I haven't come across anything yet that I wasn't able to find a specific library for. That does mean I'm using NeoPixelBus for LEDs instead of FastLED, which I was used to, but that's about it. There's always a good solution. Expressif also provides their own IoT development framework, but you can use Python, you can use Lua, you can, I mean, it's native language, it's actually C. So basically, if you program, there's a comfortable place for you guys on that platform. And I mean, there's just so many. This is just a small portion, but lots of people have just made plugins for the ESP32 on whatever ID you want to use. And wait, there's more. This thing's actually 11 bucks on Amazon. It's actually cheaper than most Arduino's. This makes no sense to me. Like, why isn't everybody experimenting with this thing? Partially because not a lot of people are just used to using older microcontrollers, but this thing's just amazing. I just strongly recommend you play around with it. It's got a few weird things to get used to, like sometimes connecting to it can be a little weird, so it's got two reset buttons, but outside of that, it's totally worth the little bit of friction that there still is. And if this community grows, it supports just gonna get better. All right, back to our regular schedule programming. Let's go back to what we're here to talk about, which is my debug app, which basically uses these systems together. Gonna show you a little use case of, whoa, no video, why? Let's do this stuff for this one, or not. So this is one of the things I've been toying around with. Oh, I see, offline apparently. I may just have to come back to this later. Oh, there we go, we're good. So I call this the world's most expensive LiteBright, because it basically uses a Magic Leap device to control a board of 880 LEDs with your finger without actually having to touch anything. So this is gesture-based, uses this expensive piece of hardware that's not released yet, and you can basically just use your finger at a distance and just draw to your heart's content. This is fairly real-time, and I'm basically passing in RGB values directly interpreted from a gesture directly to a small ESB microcontroller over Wi-Fi in basically real-time. But when things go wrong, it can get hairy to figure out why, which is why I made this. So at first I was like, oh, there's this thing, it's called TochOSC, it can do all these things that I wanna do. And then I realized the problem is that you need this proprietary piece of software to actually make layouts. It's really cool if you're doing music, because it's got all the right tools to do little faders and little sliders and volume controls, and you can put text and colors and all that jazz. It's pretty good. But if you're really looking for custom functionality, say, I wanna push a string, well, that's not gonna help. This is really made to be an extension of your synthesizer controls. It's really made for people who do audio and video live, which is great. This thing, again, it's not a hard-coded interface. You just basically push a file with a specific file format to use their editor, make a thing. You can also, in theory, make those files on the fly if you understand the structure well enough, but it's a little clunky, and it's very proprietary-ish. And the app's not free. There's an app for iOS, there's an app for Android, and a lot of people use them because for three or four bucks, it saves development time. But if you've got specific needs, I can tell you, in a few tens of hours, you can get to something much, much better and much way more flexible. So my design goals for this app were pretty simple. I wanted something that I could actually code, and I'm not much of a coder. And I needed to be extensible because I knew I was gonna make more types of hardware, and I didn't wanna have to rewrite or hard code too much shit in my app. I wanted to be able to run it on my PC, on my laptop, on my phone, on other people's phones, on multiple, like on iOS, on Android, just because I give a lot of demos and sometimes being able to cheat during a demo is really powerful. And I wanted to also allow the app, like the controlling thing, like the application, the hardware that I'm using, to be able to push the configuration so that no matter what I'm using, I get custom controls that are meant for that IoT thing or that application. And I wanted to basically support clients that were either applications running on my phone, applications running on my PC, or hardware running custom firmware. Made it easy for myself, basically. So basically this is what it looks like. It's fairly bare bones. By default, there's this page, which is the connection page. You basically type an IP and a port and you can connect to it. If there's an OSC server receiving, you'll just get a connected message. Otherwise, I've also implemented a basic registration system. So with OSC, it's fairly simple to just send messages to whatever subnet you're in at 255. And then anybody else who's listening on that port will receive that OSC message. So I use that to basically populate a list and say, hey, this guy's ready for connect. Hey, this guy's ready for connect. So if I have four or five objects that are currently waiting to be debugged, I can just see them appear on the list, click a connect button and boom, I'm now connected to that thing and I can debug it. Next, I can actually push configuration in different tabs. In case I get really fancy and have lots of different controls, I basically have these four tabs. I'm looking to eventually make that configurable so I can have as many tabs as I want, but four is enough for my needs. And I can basically create widgets on the fly when I receive configuration information to say, okay, I've got a text field. And when I press the submit button, it will send it to the device with the right address and send that string around. Can do that for floats. I can have my state button so that it actually tells me if it's true or false. I can just have an event button, which is just spam it and just something happens. And you've got sliders so you can like just have fun with the actual touchy-feely variables. And then lastly, I made myself a little logging pane. Basically any message that I send to the slash log address just gets printed there. So at any time I can say, oh yeah, cool. My wifi is actually connected. And well actually it would be hard to get there if the wifi is not connected, but at least I get a confirmation. And I can actually, as the process goes in and as the hardware initializes, I can actually show what the status of any system is. And if I'm giving a demo that's got say multi-stages, I can actually have this say like, all right, I'm in this stage of this software demo. If I have to reset, then I know what to do. If something breaks that I know, did I get to the next state? Is that where that broke? Or did I just fail out of the previous case? So going into detail, for registration it's fairly simple. Broadcast the message to 255 to a specific client port. This is the only bit of information that basically both sides have to agree on. So you have to have the client port of the app known, which if you're doing both ends, it's fairly simple. Basically, send a message to the address slash register. The data in it is just an int, is the port. The server in this case, like the device I'm trying to control is actually, sorry, the client is whatever device I'm trying to control. The server is my phone in this case, running the app. Phone receives the register, it has the port, basically adds it to a list of clients. I now know an IP and I know a port. It has an index. I know how to communicate with that thing reliably, even though I didn't know what any of that information was before. The phone can then reply with a handshake message. Doesn't have to have any data in it. It's just to make sure that both ends understand that a communication channel has been opened. This allows me, for example, to say, hey, you're connected on the receiving end. Once registration is done, and both sides agree that they can share information and both have IP addresses and ports for the other side, whatever I'm trying to control just sends a message to your address slash setup. In my specific implementation, I just basically use an int from zero to four for the type of widget I'm gonna spawn. So whether it's a text, float, state, event, or slider. The label that appears, so if I go back here, see this here is the text that appears here. That also happens to be the address I'm sending stuff to. Just keep it simple. And then which tab I want this control to be in. So I can actually group things into logical groups if I want to have like, these are actually like tab one is going to be me being able to override next chapter, previous chapter in a thing. And then the fourth page, for example, is just general debug where it's reset buttons or send a generic message, for example. And then you can also set default values in your setup information so that the slider or the text value actually has information populated to give you an idea of what type of data is actually expected on the other end. So this is an example of implementation in Unreal. It's Unreal may not be a very common tool here, but this is just to give you an idea of visual scripting. Basically the general idea, you get some information, you spawn widgets dynamically. There are a number of different tools that will allow you to do that. If you use iOS, obviously, Coco is great at just spawning stuff, but there's a number of different ways you can create your UI. Just choose your tool of choice and this will be very easy to implement because you just basically get an array of information and just use that information to spawn your controls. Step three, now that you actually have an app that's populated with all your controls, it's very trivial for that app to go, okay, I have the address. I have a payload that the user has just entered and he presses the submit button. I'm going to create a NoSC message, send that over wifi to the right port because I know what I'm connected to and the thing can now just receive that data and do whatever you want to do with it. I'll show you an example in order to be no idea what that looks like, but basically you're sending a message to slash label and the payload is whatever data you want to use for that event. And then slash log, I just use a general purpose tool to just send strings, just get a status report and everything. And usually I just, every time there's an event that could trigger a log, I just call an update log and the update log just fetches all the variables I want, print it out and just spits it out the log. So for example, this is one implementation in Arduino. It's incredibly simple. You have a new UDP packet that comes in and the OSC library basically gives you a dispatch function. Dispatch function says, okay, I've just received a message at the address test float. Trigger the function that's called right there. I call them the same just for shits and giggles, but this basically can call any function and it will pass in as a parameter the data that was sent to that package, to that address, which means that now you just have a function receiving data just like you would call any other function and you can just treat it as its unitary thing. So that means that you can actually call a function that already exists. You don't have to be specifically for that use, which means that you can easily debug existing functions as opposed to just make custom ones for debug. But yeah, that's literally all you need to do to start using external data. It's just that simple. You just include library and then dispatch. Blown through this a little quicker than I thought, but step four, obviously profit. If you jot this address down, this is basically tinyurl.com. slash OSC debug app. I actually will provide you everything you've seen here. You will get an Arduino art implementation for ESP32 of a simple debug app where you can just basically log to serial everything that you're receiving. You will actually get a packaged APK version of my Android build for this. It's just functional. So you can just follow the instructions in the documentation and it will just work. And I also provide Unreal project files for Unreal Editor 4.18. So if you wanna play around with it, you'll see how I'm spawning my widgets dynamically for the app. And probably also get a, you know, it's also very trivial to implement this and have say an Unreal game be the receiver or the thing that you control, which makes it incredibly fun when you're actually debugging wireless things, say on a wireless headset that you're walking around with that you can just start using your PC and have full logging over wireless and trigger things at will just, you know, from the comfort of your phone or, so I use it a lot for demos because then if something goes wrong, I know I can overwrite anything and the user usually doesn't know that I'm actually doing anything. Just, oh, you're stuck. Oh, yeah, it's fine. And that's all. That's all I got. Do we have any questions? And uses of this? By all means. So what good uses? I've used it for controlling the rear on some clutch and both magnets. I've used it as the entire protocol for a programming system, for a test bench and programming system, for actually writing the firmware to those devices. So I have had JTag controlled over it. I have Wi-Fi devices controlled over it all using twisted Python. I've talked to Kivi actually, Kivi the graphics environment in Python has OSC implemented under the hood for its multi-touching environment. You can totally use it again and in fact, I monkey patched it to allow for all the rest of the data types that I was using. And it's just fantastic. It just runs in your main loop. I've used it in microcontrollers, STM and U32s, one of the C, ESP266s, and in fact, I have all three bussed together so a message could be arbitrated through from an Android device through an ESP266, through to the ST, and all all traffic was actually mapped through it. It is the most useful protocol and the overhead is tiny. Absolutely tiny. It's like the string at the beginning, you can minimize your strings if you want, you don't just have addresses if you really want. There's a special spec that adds on uses it to do instead. And beyond that, it's like one extra word or 32 bits and then your data. I encapsulated it over serial, I fired it over USB, I've used it over UDP, PCP, I've had pass-throughs through them. I've actually even done it over Webkiller because why not? Is there anything to rest of development easy? Writing it up in JavaScript took me a day and I'm not a JavaScript developer and that's to make a whole library, it's so simple. Yeah, I started with WGS with it and it was again, trivial to implement, like an hour and I was already communicating back and forth. Goodbye, REST calls. Oh, and a side note of the other thing that I'm promoting, ESP32s, over the air updateable, because it actually has writable prom that you can actually write to while it's running. So again, you can just basically do anything you want with it. I have an outside to the note. Oh please. Power, oh, oh, if you're on battery, lower the clock. Fair, I recommend actually, yeah, I recommend external power for these guys, but I mean they're powerful, so it's gonna happen. Well, oh. Is there any extra security concerns with these or are there things you wouldn't put over OSP? OSC could actually be encrypted that actually has some support. I believe that specification two is actually gonna have some more embedded support for it. But by default, my current understanding is that the addresses are basically gonna be clearer, but any data that you have inside can easily be encrypted. That's just a question of both ends agreeing on with the encryption, but that's gonna be it. I mean also, this is a UDP packet or a TCP packet, so you can also just do your encryption after you've crafted your OSC bundle, encrypt it, decrypt it as a packet, and then you're fine, because it's literally just a data stream. What about UDP and TCP? Oh yeah, OSC does. No, it does your app. Currently, it only uses UDP. And in that, in the register step, you said you do a broadcast, so it kind of implies that you're on the same local network as the device you wanna. Yeah, yeah, yeah, you basically, the other prerequisites is to make sure that all your devices are actually on the same subnet. Who you thought about, or how you used KS4, you needed to talk to a device on another network? So one of the steps that I plan on doing is using the ESP32's capacity to be its own access point. So then you can just have your device connect directly to its Wi-Fi network and connect to it directly. That's not super useful if you're trying to have multiple devices talk to each other at the time, but if it's gonna be a one-on-one conversation, then you can just use that, and then it's pretty much AP agnostic, because you're connected directly to it. I do believe you can, yes. I haven't experimented with it yet, but the ESP32 can complete, you can use it to make a mesh network or ad hoc connection. Does your app have feedback integrated into it? Like if you have a slider and the other device moves the slider, is it good? Currently not, that'd be fairly easy to implement because both ends can actually just be an OSC client. Yeah, I mean, I don't see how that would be a problem. You might get some collisions, so depending on how you're polling or pushing might change. You may want to move to like a polling system. It depends, I mean, basically just creating extra Wi-Fi traffic at that point. So depending on your network environment, your mileage may vary. So about the ESP32 access point mode, does it have the same core client limitation with the AP266 AP mode as, or? I haven't yet run experimentations with the AP mode. I don't think it currently has that core client limit. I think that was more of a, like the AP266 is way meeker hardware. But it's something I'll definitely keep in mind. I know some people have had issues in the past using SPI and the Wi-Fi at the same time, but I've been actually able to run like an OLED screen while I'm on Wi-Fi. So I guess that's still some unknown territory as to what types of issues with AP. Maybe with too many clients, you might run into some problems. Also, you might have slightly different results with different implementations. Like All-MX makes a development board. It's way bigger and it's got like, it's got relays integrated and it's got RG45 connection and it's got some of the hardware is different. And I think, so in those cases, like there may be slightly different implementations and limitations or less so.