 All right, first of all, thanks, Jesse, for organizing this open dev room for games, which is really cool to have it here at the open source conference. All right, so my name is Pedro, and I'm a research and human computer interaction. I work at this place called the Hasselplatz Institute, which is in Potsdam, which is very close to Berlin and Germany. And so you're probably wondering now, hey, there's a faker in the room because you're not a game developer. And that's true. And the games that I'm going to show are super, super lame. And you just saw, like, I've made some unity, and the punging unity is just like really horrible. But hopefully you'll kind of agree by the end of the talk that maybe, like, the idea of using more kind of muscle-based force feedback is interesting for the future of games becoming really physical. And we just saw, like, a really interesting example today with a tangible orb. And also I'm here because I made this toolkit, which is called the Open EMS Stim, with the help of other people that I'll talk about as well. And this is fully open source and open hardware. And I just want to share that with you because that might inspire you. So first I need a brave volunteer that wants to play a very special pung. Come here. All right. So come here. You are the blue player, so you should stand here. And yeah, we can do this arm. It's going to be fun. All right. So what's going to happen is that electrical stimulation is going to try to pull your hand upwards as you're playing pung. And you guys, it's hard for you to see, but you're going to use the O and L to move your paddle up and down. And I'm going to use these ones, so you can start playing. All right. And I don't know if you're ready or not. So when he loses a point, it's even harder to play because your hand goes away from the screen. I'm going to demonstrate that. So that kind of stuff happens. This is like a really lame game because I'm a really bad game programmer, but I'm going to lose one again just so you guys can see it. So you could do that for like tens of a seconds or something. All right. I'm going to stop it. Awesome. Thank you. Thank you very much. You could do that, like, with different things. And if you're a game designer, if you're a game designer, you'll have interesting ideas of what to do with this. If you're just a researcher doing basic technology, you have no idea. All right. So there will be more demos along the way. So what is this all about? Disclaimer is that all these projects that I'm going to show are research projects in HCI. They're not no products and nothing is built on this or sold to Apple or something like that that uses muscle stimulation. But I think they offer an investigation of how these things could look in the future and could potentially make interfaces more physical. So all the prototypes that you see here, they use these type of devices, which are medical compliant muscle stimulators that you might have from your doctor to do rehabilitation on your muscles. So if you have degenerative disease or even if you're just building up muscle tissue because you had broken leg and you had your leg in the cast or something, your doctor might give you something like this. Technology that is very old comes from the 60s forward onwards from rehabilitation medicine. This is just a little example, like me controlling one of each of my other fingers just via a smartphone. All right. Types of games this is enabled. The first thing we built with this was just a game that had force feedback with mobile, which seems like a contradiction if you think of arcade machines. So here I'm playing a game and there's actually a force feedback, but it's my own muscles that does the force feedback. And so the device is actually pretty small. It sits in the back of a smartphone. You already saw that it can actuate like the fingers can also actuate the wrist and whatnot. And if you think I'm playing a little game here, I'm trying to fly an airplane and when the windmill comes, my airplane gets derailed, but that's involuntary. It just moves by itself. And now I have to fight that and put the airplane back into trajectory, right? So it's a very simple kind of approach to having force feedback. Now, if you think about it, what is kind of interesting is that the device is really tiny. It just has no motors, no actuators. It's this little thing. Now it looks a little bit different because there is a way to create force feedback sensations. It's called an exoskeleton. There's other ways as well. But this is the most portable one that you have, which is using motors, pulleys, big batteries, a lot of electricity, amps of power to be able to externally move your muscles. So what we're doing is exactly doing the same thing, a little bit less power, because we just use a 9-volt battery and we just send these tiny impulses to your muscles and they do their job by themselves because you have all the hardware already. All right. So this essentially allows you to do the force feedback in a semi-wearable or mobile form factor. Now the question is, where can this be also be useful? And the question becomes really obvious these days because VR is moving into the real walking. So not that you're just with the headset sitting in front of the computer, you're starting to walk in physical spaces in VR. That's called real walking. And I think VR is a really interesting step right now because what you see is super believable. If you look up there, I put this HUD there. But if I hadn't put that thing there, this would be like a picture of, I don't know, probably some forest in Germany. But so graphics is really realistic, but the sensations you have are kind of boring in VR. You walk around, you touch all these air spaces, and it's all like kind of haptic-less, right? So there's nothing there. Maybe there should be a wall in there because the normal games have walls that we just saw. You can also paint them in GIMP. Maybe there should be a box there that kind of punches you through. I mean, there should be like physical effects if you're seeing physical realities, right? If the game is about ghosts, that's all fine if you touch nothing. But if the game has a wall, it should touch a wall. And if the game has a boxer, you should feel the punches. So I'm going to tackle both. So first, let's tackle the boxing example. So this is Impacto. This is a project we did a year ago or so in which Sieging is my colleague, Sieging. He's actually feeling the punches from this virtual avatar boxer. We've demoed this set of maize. And the way he's actually feeling that is because we're trying to render this same sensation, right? The sensation of impact. And there's two things about that. If something touches me here. There's a tactile component, my skin, and there's a force that moves me backwards. So we render them separately. We have a little solenoid that taps your skin, right? Tuck, tuck, taps your skin. And we also have the muscle stimulation that pushes your arm backwards. If you combine both, you give an illusion to your brain that there was a tactile sensation here and a sufficient force that pushed you backwards. And your brain is going to interpret that as, well, this force was big enough to move me backwards. So he's feeling those forces in that game, feels like the boxer hitting him. And he can also hit back and stuff like that. I'll show a little bit more about that later. So impactal kind of goes this in a sort of wearable form factor. And when we played the set of maize with people, it was really interesting. Because they were hardcore gamers. So they were jumping around and going out of the tracking volume and things that we've never done, which was really cool. I always just played like this. But people were really engaging on a kind of physical level. And the key for this is really the muscle stimulation. As you saw in the previous one, the solenoid that we actually use is something super tiny. It's even smaller than all the ones here. Because if you wanted something to push through and feel like a boxer, you would end up with a robotic arm that would hit you with, I don't know, 100 newtons, 1,000 newtons or something, it would probably break you as well. Our solenoid, when we measured it, it's something like 200 grams. This is like tap, tap, it's nothing. But because of the muscle stimulation, it tricks you. So we played around with different sensations. You've seen the arms already. What's interesting is that it can actually hit back as well. So if you put a little redirecting thing for the solenoid or three-printing lever, you can kind of feel like you hit the boxer as well. We've created a little bit of a juggling football thing. That's kind of cool by putting the solenoid in the foot and the muscle stimulation on the leg. So when the ball hits you, your foot kind of goes downward. So it would feel all those impacts for the ball. Combinations you can hit back with your legs. I'm going to jump that one. It's just like tieboxing. And this one is my favorite. If you put the solenoid onto a random prop, it could be this water bottle. The vibration is going to be, when it hits, it's going to be propagated through here. But we're going to put the muscle stimulation still in your arm so we can move your arm. And that creates things like baseball. So you feel like something hit the bat, but it's your arm that goes backwards. So it feels like that ball kind of force. And this one is kind of fun. I'm going to play a little bit around with the visuals as well. So that's just a dummy thing that you're just holding like a filer. But in reality, in the game, you could be holding a bat, ping-pong paddle, fencing sword, different kind of props that represent physical realities. The hardest one that we just, this is recent. We're going to publish this soon. And this is trying to investigate what if you could create walls, making games even more physical by creating obstacles, like heavy objects and walls. So this one asks the following question. What if there was a wall there? And sieging could feel it somehow. So this is a longer experience. And what we're doing is using the muscle stimulation to try to pull you backwards to not enter large objects, like electro walls, like this button here. As you push it, you feel the resistance. And you're trying to push a soft button through. There's a cannon now that shoots propellers that you or things like that. And so you also feel them just an impact to your hands go backwards. And the most interesting one, I'm just going to skip a little bit, too. This is actually a very large VR experiment. You can be in this world for a while. So this I find kind of cool. Actually, when you grab the cube, you try to push through because you don't know where your cube is. And the system stops you at the moment where you actually face the cube. Also, it pulls you downwards to emulate the gravity, right? So to emulate the weight under the gravity that the cube would do. So you feel some resistance, which is kind of cool because when you throw it, it is easier actually to throw a cube because you just feel like, oh, now I don't have these forces, so I don't know what's happening. All right. So this is kind of the part that we do as research. And as I started to try to experiment and give this to my friends, to other researchers, I faced the obvious problem, which is all these things that I showed you were things that I engineered myself. And it's not like it can give it to you and say, oh, just activate this command in this pin, and this is going to work really well. Just as an example, the bracelet is actually not that complicated. But at the end of the day, it's pretty custom made. So there's a processor there, similar to an Arduino. There's Bluetooth. There's the muscle stimulator. There's a way to control the muscle stimulator without hacking it that's somehow way to find really important about my contribution here. Is that you see this thing is not open wide, and I'm not like taking out a capacitor and putting another one that I don't really know what it is. So I try to just modulate the amplitude that comes from this one. So if this one is medical compliant, at least I'm kind of medical compliant as well. It's very safe for you to use it. There's batteries. These things are not medical compliant. They explode accelerometers. And in the case of impact, there was a solenoid. And obviously, there's always the electrons. So now I was starting, like how can I share this amongst others? And actually, Jesse was a key person in this. Because I was invited by Gammerier and Jesse to give a workshop at the iGamer 2014. I didn't even have the kit back then. They all built something like this with relays and potentiometers and whatnot. And it kind of gave me the idea, OK, I have to structure this somehow in an open source way so people can actually reuse it. And then the other side of the story is that I saw a lot of people online that kind of like my work. And they were trying to replicate or tinker with it. And they were kind of putting some things in the wrong way. And I was like, OK, maybe they can hurt themselves. Let me start doing workshops or at least publish everything in a way that it's fully available. I can understand at least my decisions. And there might be wrong decisions too, but at least my decisions. So that allowed to create this thing that we call Open EMS Tim. The boards were designed by Max and Tim, made by me and Max and Tim. And then I remixed this into the version that it is now. Max was at the time at the University of Hanover. He's now at Moonstar. And his research is also super interesting. So if you have time and the slides are online, you can check his stuff. And then what I really did was I created this board for a competition at the UISD conference, we call it WIST conference, where it's like a game gem kind of thing where students come together in teams and they build prototypes. And then they demo at the conference. And so this was the hardware chosen for the last competition. That's why I kind of remixed and made an open source and so forth. So what the hell does this board do and why is it connected to this thing? So I told you guys that this is a muscle stimulator. It just creates a signal that would activate muscle fibers when connected to electrodes. And this thing is essentially just a way to computer control it. So it sits in the middle. Right now it's connected via this way, but it could be connected via just by a battery inside. It doesn't have to be over serial. So just a little 9-volt battery. And all this thing does is to modulate the signals. It has an Arduino inside. It has a Bluetooth, but it could have other things. And it has an understanding of how to modulate these signals without potentially being dangerous to you. All right. So that's what it does. It controls the amplitude of the haptic input. This is just a screenshot of the GitHub. You have all the kind of walkthrough on how to use it, how to build, and so forth. So who has used it so far? I mean, before there were people at the game jam at iGamer and so forth. But this particular version has been used by all the teams of this competition. And I just want to now stress out a few of the projects of really incredible stuff they built. Let me say, all right. So this was one of my favorites. It didn't win, though. This was rock, paper, scissors. There was a lead motion controller. So with one hand, you would select what you want to do. Like, I want the scissor. And the other hand is computer control. You just play against your other hand. You're like, listen, I won, and I lost. And it was working pretty darn well. It's hard to make the scissor gesture, so they just kind of approximated a gesture, and they considered to be a scissor, something like that. But the implementation was really great. The lead controller, the way they did it, was kind of even detecting a little bit before you actually finished your gesture. It was like 3, 2, 1. And the 2.5 already had decided what to use again. This was really cool. And this was, by the way, his rifle. He was our sponsor. He's a Google VR. We kind of sponsored the whole competition. And he's trying the project that actually won the competition, which is called a vibrato. So they were teaching people how to do this vibrato effect, where you're like, I don't know how to do it. Would your voice, by using muscle stimulation here. And Hayes was also trying it on his neck muscles. And they made a karaoke machine that was essentially, not just show you letters, but at some point, automate the vibrato and like, do, do, do, do. To you, it was really good. I think these guys also won an award. This was an air guitar. I called it the first real air guitar. So your muscles would essentially move automatically to any beat that you would play on the smartphone. So they also created like a tracker for beat on sets and so forth. Yeah, a lot of cool stuff. Some HoloLens things, a lot of VR. VR immersive experiences were also made. This is an interesting one. It kind of makes you smile while in VR. That's my advisor as well. And one of my favorites, talking about alternative game controllers and physical tangible orbs and things like that, was a ping pong paddle that was just for a physical ping pong game, two person. But I could hold the smartphone with my other hand and send like evil kind of things like shake your hand now. And you're like trying to play, but your hand goes crazy. And you had a limited number of power-ups that you could use and things like that. So it was a very interesting augmentation of ping pong. All right, so how to use it? It accepts commands via Bluetooth, 4.0. So that's the low energy that's what most smartphones have right now, and also accepts serial. And there's an onboard Arduino that kind of does that part. So just to show you how the Bluetooth part works. All right, so I'll turn this one off. So we made a Bluetooth because it also simplifies people just trying it out for the first time, because it's easier to do it over a computer. So if you want, you can tap on one of those buttons. I'll probably feel something. Channel one. Yeah. Yeah, so the longer you tap, the longer I feel. That was very short, yeah. So that's this one, and that's this one. So you can pass it along. I'll kill it at some point. So that's interesting, because in the workshops that we give, people can build games just out of even that controller, which is like Wizard of Oz, kind of just experimenting. You don't even have to build a full game. A game can be like a set of rules, and then if something happens, it does that. One team also built a board game, which I found really cool, that when you're just playing a normal board game, at some point something happens, and your hand kind of goes left or right. So that was cool. And then, yeah, you can also do it over USB, and we can try to do something kind of fun here. All right. We still have some minutes, so let me see if I can pull this off. Have you tried your game? Ha-ha. Let me see if this is running. So let me kill this one. So it's so hard to double screen. So where are we? I'll get this one. All right, I think I need my phone again. Who has my phone? Because I think that's my hotspot. Oh, you guys lowered the amplitude. That's why I wasn't feeling it. All right, hotspot. Am I connected to the hotspot? Oh, I can't see that. Let's hope that I am. OK, yeah. I am connected, but I didn't feel anything. Let's try that again. It's a little tricky on the big screen. Right? Fred. Oh, PowerPoint crashed. That's kind of fine. All right, so the board needs a little bit of time to set up the Bluetooth usually. So here you see one of the interfaces, which is, I made it small. It's not even an API, just a little interface for Node.js. So people can also do these things over the browser. And oh, yeah, I felt that one. Yeah, so when you load this page, you just zaps me. So if you would go to my IP address, which I would have on the slides, but we just lost the slides, part of the slides problem, I'll keep it running. So maybe it will work. Maybe you should jump this one. We already went a long way. That's nice. So here we are. So yeah, if you would go to this address, but you have to log into my phone's Wi-Fi, so that will take too long, but I would feel your HTTP gets. All right, so that sounds complicated APIs. It's not really like I made APIs, but what I made was a bit of like module for Python, like a very simple Node.js kind of library. I also did a processing thing, Unity 3D, and time is up, so I'll just jump. This is just screenshots from the GitHub, so all these things are online. If you want to make one, it's not super hard. There's also a tutorial online on how to bake your onboard and make one. We have only a very limited amount, but if you really need one, you can send me an email. I'll try to make one for you. And yeah, we make them like this in another one. So that's the end. I hope you kind of enjoy this thing. If you want to learn more about this, you can always organize workshops either with me or just by yourself. And I'll just help you online on how to set these things up, organize game jams like the committee has done. And if you want to get involved, there's a lot of hardware that needs improvement in software, definitely and mostly. You should just make a lot of weird games. Yeah, that's it. I want to show one more thing, so I'll sacrifice question time. If you like weird games, I think this is the one that you will like the most. This is a game without a screen. It's this children's game that you normally play with two people. You try to slap each other. But here, Patrick is playing against the computer. So that hand is the computer, and that's his hand. And this is really, really hard to win because you have no reaction time to kind of run away from the computer. So there's an interesting also game design challenge on how to design a game that has no screen and just works on your body and works on your muscles. Yeah, that's it. Which muscle? That's a great question. So there's a lot of things that require knowledge of what you're doing. So you do need to have a sense of the anatomy. And the things in the arm are really easy, and then things in the rest of the body get really complex because muscles are very layered and things like that. So there's electroplacement charts for all these machines and all the muscles you want to actuate. And that's usually the best way to do it. And the question for those at home was, how do you know where the muscles are that you want to actuate? Where do you place the electrodes? The MST line is just outputting sine waves? So question is, what is the MS device outputting? So there's a lot of people that also build their own and just output simple sine waves or output square waves. The medical way to do it is to output a thing called a biphasic waveform that looks a little bit like a sine wave, but has a very long tail. Definitely AC, it has to be, it goes up and down, but it has a very long tail, which now I would take 10 minutes to explain, but it allows your muscles to do a certain process that they need with that inverse, so the negative part of the wave. So yeah, you can totally use these, do yourself kind of things, but if you want to be on the safe side and have a really good experience, these medical devices that output that particular waveform. And your box is then modulating? Yeah. Yeah. Yeah. And how expensive is such an MS material? This thing costs 20 euros. How expensive is the MS device? 20 euros and to make a kit like this, the most expensive part is the Arduino, so it depends on how cheap your arduinos are. Mired out of the ripoffs with the non-FTDI driver, $4.00. But then the driver sucks. How much can you miniaturize the way up there? Yeah. Question is, how much can you miniaturize? I think the answer is a lot, like really, really a lot. This is self-engineered, kind of not an EE student kind of technology, like this could be a computer chip that big. Amplifiers are always going to be a certain size. So this has high-voltage amplifiers that do something like from your very small 5-volt signal to 40-volt, 50-volt. And that will always have a certain size. It's a transformer. But we're talking about something like this. It could be very small. It could be in a smartwatch. It could be in a bracelet form factor. OK, I think we're going to have to stop there. Thank you so much.