 My name is Alex Chenoy. I'm a software engineer here at 4MOMS and basically today I'm going to tell you the story of how I went from being a web developer here at 4MOMS to being a hardware developer. And so this story starts as most stories do with a young, incredibly naive hero in college who's listening to his CS201 professor tell him all about how bit shifting and who's compliment math works and he's sitting there thinking I'm never going to need this. It's abstracted away. I'm going to work in a VM all of my life. I don't actually need to know any of this stuff. And he's about to find out that he's wrong. So basically I had this opinion that the web was important. This was before the iPhone. Servers were the most important thing. So any problem you had you could just throw more server power at it. So fast forward to about probably a little bit less than a year ago. I had to take the average of some samples. And so basically if you want to take the average of something you sum up all of your samples and then you divide by ten if you have ten samples, five by the number of samples. The problem is that division is a computationally expensive operation and so if you want to divide by ten that takes a really long time. But you could do something like say take sixteen values and just bit shift off four and this is much faster. And the reason that this works is because you're dividing by two. And to prove that this works if you wanted to take one million divided by ten you just move the decimal point. So it's the same thing. If you want to take eight and divide by two you convert them so it's a thousand divided by ten. That gives you a hundred hundred and base two is four. So and this technique is basically the exact same thing that I was learning in my CS201 class. I was learning about doing a lot of this low level math and this is all stuff that I thought I'd never need. And then now I use it all the time. But really I'm here today because of the original Xbox. So if you don't know Justin was asking for speakers for code and supply talks. If you give a code and supply talk he'll give you an original Xbox. So you should do it just for that. No but seriously I'm here because I was talking about the original Xbox and the security mistakes that Microsoft made. And I was just sort of talking in the office generally to a bunch of coworkers and sort of explaining how a lot of it worked. And they had a lot of problems with overrunning memory. Not hashing things properly or not using a hash for security while they thought they were. So it's covering a lot of this stuff and it was all super low level stuff. This was all stuff that was happening in the kernel and dealing with how they were dealing with addressing memory and where they were storing things. So coworker overheard this and an opportunity to join our embedded team and take the skills that I learned from test driven development on the website and bring them to the embedded side came up. So I moved over and that's kind of the story of how I moved over. But I've learned a lot since then. I've learned a lot about the embedded space and there's a lot of things that are different but there's also a few things that are similar. And I'm going to cover those today. So perhaps the biggest change that I've had to deal with is a change to my workflow. And that's because I've had to slow down a lot. So this is a picture of a post it note that I have on my laptop. Because on the web it's pretty common. You just want to make a change. You do it real fast. You run the test. If the test passed, the change is good. And if they don't, then the test is bad. But it's not uncommon to just be typing at full speed and just see what happens because it doesn't matter. It's safe. You can't really do that on the embedded side. And in fact, you know, Facebook's motto for a while was move fast and break things. So they wanted to just get stuff out the door and even if it broke, it didn't really matter because it's a safe environment. So what I'm going to do now is I'm just going to take a, we're going to look at a change in Ruby. If the microphone will reach. Okay. So we're going to look at a change in Ruby. And so I'll just show you, I've got this thing that's failing. And I'm cheating because I already have all this stuff backgrounded. But this test is failing. And I know that it's in the cell test. So I'm going to go and go here, see if I can figure out which one it is. Just be alive. Oh, okay. So I must have uncommented this line. Again, I'm cheating because I wrote this already. But let's just see what that did. Okay. So now all the tests passed. So I'm good, right? And that's kind of what making changes on the web looks like, right? Like you have some tests that tells you something and you make the change real fast or relatively fast. And then you're done. This is kind of what looking like changes in the embedded world looks like. Like I've got, you can see I'm using a meter. I'm trying to figure out, you know, what's going on. You can see a bunch of wires coming off of that board. I'm actually testing those wires. And so the process is much slower because you have this hardware component. And you can kind of see that when you want to build a new feature on the web, right, what do you do? You probably start with a cucumber test. You probably write some UI, write up some HTML. You probably make a presenter and a controller, test that stuff. You probably write a model then, finally. And then by the time you've gone down this stack, hopefully all of those tests pass and you know you're done. So it's very top down, right? On the embedded side, what I found is that when I want to work on a new feature, the thing that I have to do first is start with this tiny, tiny thing that almost seems obvious, but isn't and then work upwards from that. So another good example, I'm going to show the same picture again. But another good example is sometimes you just have to verify, do a continuity check on the board between wires. Make sure things are wired up properly, right? That's a test you have to run. And so it almost seems a little bit backwards from what I was used to. So if you've worked on the web, you know that most web software looks something like this. This is a simplification, but you're going to get a web request in, it's going to go through some web server, Apache or Nginx. It'll yield to an application run time if you're using Rails, it's going to be Ruby. It'll run your application code, you hit some database, you get the response back, you generate a response, it goes up to Nginx or Apache and ships it out, right? And because I know there may be some .NET people here, in case you're wondering, it's the same thing on .NET. You're going to hit IIS, which is Microsoft's web server .NET framework, your code, SQL server, and then back up the chain. So it's the same thing. What you're doing in an embedded system, you're really writing an operating system for the controller that you're using. So you're sitting in an event loop and you're just sitting there waiting for things to happen. Or maybe you're monitoring, you're asking if things have happened. But you're waiting for all of this stuff to come in and you're going to be checking this stuff. Now some of this stuff is going to be an interrupt, so it's going to be a line coming into the microcontroller or into the processor that basically says, hey, I have something for you, right? Or it could be that you're asking, you're pulling some bus, like, hey, what is going on on this peripheral? In the case of interrupts, that can be really, really fast. And those interrupts have to, you want them to hit, when they hit, you want the code in those interrupts to yield, to do whatever it is they have to do, and then yield as fast as possible. Because the last thing that you want to have happen is have an interrupt or a timer, have the same interrupt or timer trample over itself. So if the code takes too long, the interrupt could fire again. And now the code from the previous interrupt didn't run, didn't complete execution. So now you're stuck in this loop where the code takes too long and you never get out of it. So the way around this is to typically want the interrupt handler or timer to just set a flag somewhere that you're then going to check somewhere else. Another way would be to use an observer to do the same thing, which is probably a better way. This is a really rudimentary example. But you're going to set this flag, and then you're just going to monitor for the flag in your main loop. So if the flag has been tripped, do some stuff, and then set the flag back to full so that the interrupt can change it again. And this is a very common pattern in embedded systems, which sort of leads me to inputs. On the web, when you click a button, it just fires off a request, right? It's a very atomic operation. You click the button, you're going to send some data, you're going to get some data back. That whole exchange happens atomically. Well, sort of atomically. One of the problems that you have in embedded systems is that your inputs aren't necessarily atomic operations. Because it's a circuit, and because there's current in the system, the current can bounce around a lot. So if you're looking for a specific value, you're going to have a button that's just bouncing around. And when you see a spike, that doesn't necessarily mean that the button was pressed. So what you have to do is see a spike and basically debounce it so that you know that it's been pressed. If you're lucky, you can have this done in hardware. So the button can do it itself if that can get expensive. If you have to do it in software, you basically have to write an interrupt handler and write a debounce controller into your microcontroller in your code. And this is all done just to verify the inputs that you're seeing. So you constantly have to be doing this kind of stuff. The next big thing that I learned is dealing with blocking code. So on the web, because things happen atomically, they happen pretty fast. Servers are very, very quick. You can do something like, hey, I want to do something where I'm going to write to a database. And I can wait for that write operation to happen. I can trust that it's going to happen in time enough for my request to go back. Unfortunately, that's not always the case in an embedded system. If you send a write to a peripheral, you may have to do other things while you're waiting on that peripheral to acknowledge the write and actually take care of whatever it is that you told it to do. So this would look something like this. You would start the write, and then you keep going through the loop, waiting for the write to be finished, waiting for some flag to tell you or some callback to tell you that the write has been finished. And you're waiting on this peripheral. So these are just sort of differences to how the code works that took me a little bit to wrap my mind around. And they're sort of fundamental to the way you make embedded systems. But that's not the only side of the story. So there's a lot of things with frameworks and libraries. If you're familiar with Rails or OS 10 development or Python, you're probably familiar with Bundler, Cocoa Pods, if easy to install. These are all package management tools. And they manage all of the versions for whatever packages you want. And these are great tools. They're awesome. They make handling your dependencies and handling all of your packages easy. You don't have to think about it anymore. You just put a line in a gem file, bundle install, and you're good. Unfortunately, on the embedded system, you don't really have anything like this. You don't have, you're largely reliant on vendors to provide you libraries. You're not always guaranteed that the vendor is going to give you source code. A lot of times, they'll give you object code. And so now you're reliant on their object code and the documentation. And if the documentation is wrong, and then you're kind of phishing, you have to run the function, see what it returns, see what you're getting back, and try to figure it out. It's almost like reverse engineering it if they don't provide you with good documentation or source code. And I think the biggest difference here is that the tools on the embedded side of this stuff are not as sophisticated as what we have in the app world. And I think that that's due to multiple things. I think it's one, vendors are largely trying to protect IP. And so they think they can manage it on their own that they can protect their IP. And I think it's also due to the fact that a lot of these systems are in someone's IDE and you're inside a specific system that not a lot of people are working on. Whereas on the app side, there's tons of people writing rails and the whole thing is open source. I'll see if you can get in this one. So basically what I'm saying here is that you have no training wheels at all. And even down to the way that you sort of structure programs, so Justin was talking about the Starter series on Saturday where they're going to deal with object-oriented programming and the NBC pattern. So we're all familiar with NBC, with Model New Controller, right? So this is separating your code into what it's actually doing in the discrete part. This is actually baked into most web frameworks and most app frameworks. Now you can get something similar on the hardware side. There's a pattern called Model Conductor Hardware. It's pretty much exactly the same thing. The model is what would be the equivalent of your business logic. The conductor is handling between dealing with the hardware and the model. And then the hardware is what the view would be. But this isn't baked into any sort of embedded framework. You're largely on your own to do this yourself, to be disciplined about it, to structure the code this way. Which, when you're starting out, might be difficult because you just don't know how to structure it. But if you can get to this point where you structure using this abstraction and using this sort of architecture, it makes things like doing test-driven development really nice. So one of the things that is the same, or sort of the same, between the way that I worked in the web and the way that I work in embedded systems, is that you do have some pretty nice test-driven development frameworks. So if you're an app developer, you're probably familiar with some of the things on this slide, R Spectrum, that you come up with. You need to try and figure out a way to test the system without having someone press that button. Because you want it to be automated. You want to be able to test your logic without having someone have to go over, press the button every time you run your test suite. So this can be a bit tricky, but you can just sort of look at the hardware. So if you have a microcontroller and you know that certain pins are mapped to certain inputs, you know that basically it's just memory. Like you're talking to a register, you're talking to some memory address location that is reading from the button, or that will tell an LCD what you're looking for, or that will tell some periphery what you want. So since it's just memory, can't we just create some section of memory on our own that would stand in for that thing? And the answer is yes, yes you can. So if you are using this model conductor hardware pattern, you can then just sort of create your own software version of hardware. And recognizing that that's important, you can design this in a way where you're not entirely abstracted away from the hardware for all cases, because you do still have to do some sort of full test, but you can test a lot of the interaction this way. And so just as an example of what this might look like, like let's just say you have an array of memory, you've created like an 8-bit integer, and you've got some button that you want to create, and you know that the value needs to be zero. So you can just create a variable that now stands in for that button, and use that as your hardware, and then set it before a test, run a test, and test against what you expect to find. So this was the same, but a little bit different, right? Because I'm still testing, I'm still using TDD, but I'm also now having to divorce myself from this other piece, this hardware piece. But testing is good, and I can find failures. I'm used to finding failures. On the web, my failures usually look something like this, and I can just find out what I'm looking for. And that'll help me find software failures, right? So I can run the Unity suite, and I can find software failures, and I'll know logic problems. But sometimes the problem isn't software, right? I just divorced myself from this hardware, but when I'm running on the hardware, the hardware is also a component that I have to care about, and I have to think about. And that's not necessarily something you're used to doing on the web, because the server is there, and all that stuff, it works, right? But the hardware can be a problem. So maybe your software works perfectly, but a trace on the board is wrong, or is going from one peripheral to the wrong pin, or maybe you shorted out some component. This has actually been a pretty big thing for me to learn, and I'm still sort of learning it, is that sometimes the software is not to blame, and I'll go chasing a software problem, and then it turns out that I blew out some hardware component and just didn't know it, because I wasn't thinking about it. So we were talking about memory a little while ago. Memory is incredibly important on embedded systems. On the server side, we've got basically infinite memory. If you run into a problem, you can buy more memory and just throw more memory at it, or throw another server at it. Again, I know that that's not always the case, but in general, memory is infinite. On embedded systems, it's not infinite. So as a quick question, does anybody know how many bytes this function declaration takes once it's compiled after optimizing for code size, too? Takes four bytes. So just the function declaration, no code inside it, just declaring this function takes four bytes. And now, I found this out because I was working inside of a very small amount of memory. I was working inside a 4k of memory, and that 4k had to hold some vendor libraries. It had to hold my own code, and it had to hold basically the frameworks that I was using, all inside a 4k. So I got to a point where I had extracted some code out that didn't necessarily need to be extracted out into its own function. I extracted it out so that I could read it, because that was a very common thing that I was used to doing on the web, like, oh, I can make this function smaller by just making some human readable function. And I did this, and I compiled the code, and it didn't fit on the chip that I wanted to put it on. So I went through, like, what can I do, what can I do? And I finally found out, like, this is four bytes, and I had done this a dozen times, so now I had to take all that code and put it back. And there are ways around this, too, other things. But these are things that you kind of have to think about, and they're way more front of mind. So the story just illustrates that sometimes the solution that you have to use might not be, like, the prettiest solution, or the most elegant solution, because you just have to fit inside this constraint. And these limitations are things that you'll encounter. The last thing that I kind of want to talk about are real time operating systems. So I've been saying that you're basically building an operating system, and you're basically sitting in this loop, and you're waiting for things to come in. And it's still largely true. But when you get to a certain complexity of system, now you might have many peripherals that you need to talk to, and you might need to be talking to them and scheduling jobs and dealing with jobs. So the idea here is that you have a microcontroller, you have some scheduler that's in charge of handling all of these jobs, and then you have a bunch of jobs. And so what the scheduler is going to do is a scheduler is going to start with the jobs and just sort of determine execution time. So this is really what you're doing here, as you're saying, I know these discrete jobs that I want to use, and I know roughly how long they should take. And I'm going to schedule them based on that knowledge. So you want to just sort of jump from job one to job four to job two, job three, job five, back to one. And this is something that I've looked at a little bit. I'm still sort of exploring them and still sort of learning about how these scheduler algorithms work. But it's pretty interesting. On the web, we pretty much our releases look like this, right? Push to master, Jenkins will get it, build it. You're good, right? Deploy it. Cap deploy, maybe. It's like one line. For embedded systems, you're going to compile your code, and then you're going to transfer it to a supplier, some supplier. You're going to issue a change to the supplier, say, hey, use this new code. And then you're going to put that chip on a product, and that'll go to customers. So this whole turnaround time, it might be a week from the time that I actually compile the code till it actually gets put into a product. It could be months before it's actually deployed. So the cycle is much longer. And that kind of goes back to the whole slow down piece. My workflow time is much longer because I'm dealing with just a slower workflow and more challenges. So if you're interested and you want to know where to get started, Arduino is a great place to get started. It's very, very beginner-friendly. You can get up and running. There's plenty of tutorials. There's a lot of shields and a lot of peripherals that you can buy that are fairly cheap. SparkFun and SparkCore, another one. Adafruit has some stuff, too. And yeah, so if you're interested in embedded systems, definitely check these out. It's a lot of fun. It's been challenging for me, but it's been a lot of fun to figure out how these things work and how a lot of this stuff works. And then that is all I've got.