 I'm Phoenix Fairy. I'm going to keep it pretty chill tonight because we are a small group of people. So if you have questions, you can ask me pretty much. I'll take questions at the end. But I thought I would talk about alternative controller games or how to deal with ridiculous problems you didn't even think were possible. So I'm going to tell you the story about how I started making this kind of stuff and how I ended up here. I originally kind of got into games as a musician. So I have arrived at what I like to call an anti-formalist approach to games. And my games are a lot more like a musical experience or a DJ set. They have these flows to them that go through time and sometimes they're rules and sometimes they're not and sometimes they're things you can discover and they're very free flowing. So my music project in 2010 was a band called Black Swan. And we thought it would be, yeah, okay, come on, get it out. I know, right? But it was a chance for me to kind of experiment with making music using machine learning. So I was working with my friend Margaret Shadell and we decided to place a bunch of sound in a game engine called Unity, which is a 3D game engine. And we put a bunch of sounds all over the world. And then we had been working to embed sensors in our live performance to make it so we could play music without needing to necessarily have a keyboard or anything in front of us. So we were using cameras to watch us. This was back before the Kinect originally came out. We were just trying to observe ourselves and then train gestures into our system. Meg was working on this thing called the Cabo, which I guess you can now buy, which has an accelerometer and a bunch of sensors in it with Keith McMillan. And what we were doing was hoping to use machine learning to say, okay, we want to build a control space where if Meg presses the bow down on her cello really hard and my hands are up, this kind of behavior would happen. So the goal for doing this is we really wanted to explore what it would be like to have the interface wrap around us versus be forced to conform to an interface, like Ableton Live. So when you're playing live, it's really kind of awful to be hunked over a laptop. That's not the kind of experience you want to be having with your audience or with your group of people who have come to see you play. And this has kind of become a manifesto of mine that I've taken into a deeper, I don't know, I think it might be something that kind of weaves through my experience of being a designer. But basically, I don't think the user should ever have to conform themselves to the affordances of the interface. I think it should go the other way around. So I was using machine learning to take things like gestures and train them into a neural network to kind of generate a prediction. So for example, if my hand was up, I would maybe want the, you know, animations to spin right. If my hand was down, maybe I would want them to spin left. And if my hand was somewhere in between, I would want to get like that nice, rich gradient. And machine learning is really good at doing that kind of thing. You can kind of have these nice in between places in your models. So we were using a neural network. And this is kind of a nice neural network display that I think kind of makes it pretty clear. So if my arm was raised and, you know, the cello is being blowed lightly, we would take that in as our input. And then we would run it through our neural network. And we would associate certain inputs with certain outputs, like maybe a color change or volume in a clip being changed. And it really allowed us to create this really highly, almost random, highly responsive performance environment, which was really kind of nice. It allowed me to kind of have something to respond to versus just have that was preexisting. I sometimes didn't know neural networks will give you very weird valleys between, between classifiers. Sometimes it can be really interesting, particularly if you're instead of going for like a one or two, you're going for some gradient in between one and two. So we were, we were playing with that. So I started realizing that my kind of creative applications could really have this kind of wonderful sense of play. So does anybody remember this? Anyone here? This is an application called Auto Illustrator. It was, came out in the 90s on OS9. It was a random generator, kind of procedural generated Adobe Illustrator. And I loved it. I made some of the best art in this. It would do things like, you know, notalyze polygons, instant Bauhaus style, mutate into, colorize, and it, it was really random. Like parody was one of the options in the filters. And I really enjoyed how playful the interface was while taking something that I knew and really destroying it. So this kind of led to the very first game that I made that used my body in, or used a player's body in a kind of environment. So I made this game called Nightmare Kitty. So I was in a really good position when the Microsoft Connect came out because I had already been playing with camera tracking and machine learning in game environments. So when the Microsoft Connect came out, I kind of got curious what it would be like to try and create a game using body and position tracking and machine learning. So I could record different positions and associate game behavior with those different positions and then let kids play them. This one was something I was really interested in because this is kind of what it looked like when, when kids played it. And I really wanted to give them that feeling of feeling powerless and feeling scared and feeling small and then giving them an opportunity to overcome that by like standing up and going into like a very high power pose and feeling very strong. So I did that over and over throughout the game. And I really tried to like put my argument for what I wanted these kids to feel into the kind of physical form of the game itself. So I was like, I want you to feel empowered, but I'm going to do that through the actual physical shapes that your body makes over the, the, the place man of the game. And I started really thinking that emotions can really be triggered in response to an ecology. So this is kind of the big leap that ended up with me making hardware games. And I thought to myself, how can I build these like giant ecological spaces? So my first attempt to do this was night games. And this was a project where I essentially took my kind of performance environment and handed it over to kids. So I used a hacked PlayStation move and we managed to convince my friend Colin to dress up like a magic monster and hand out the PlayStation moves. And we had different gestures embedded in the kind of accelerometer gesture data of the PlayStation moves. So if you like would rotate it really slowly, you would get a really like slow spinning snare like but if you'd spin it really quick, it would be like, you know, so the different gestures would kind of change the sounds and like change the filters on the sounds in fun ways. There was also maracas and like ridiculousness. And I would have played you a video here tonight, but let's just be happy. We have lights. So this is a really, I got really lucky. I had a drum core come and play at one night and it was perfect. It sounded more amazing that had maybe ever sounded before. And I got a video that but it was really cool because I had all these instruments and these kids were just kind of playing this sonic environment in this very fleet free flow open ended way. And I was very fortunate in that I was working with like a very commercial controller. So this is probably the most spoiled I would be in every game in any game I would build up until now because all the problems were just on the software side. I didn't have any hardware problems. This was a beautiful thing. I had a preexisting hacked piece of hardware, a hacked connect, a hacked PS move. These things are manageable. They've already been tested for children in violence. So as we will get into in a second. So I began to look at like how games could be ecologies. And I was really interested in how they could be ecologies versus economies. So often in games there's game economies, right? There's value that you can get for a token or a system. And I really loved watching these kids. They invented this entire, they did what children did. They were well rather what children do. They invented their entire rule structure universe around the kind of behavior of the PlayStation moves. So call and response was incredibly popular. Follow the leader was another game that I noticed kids were just like constantly. And I was really like, wow, how can I think about ways that instead of crafting rule systems around this, I can just embed triggers and sounds and, and little hints of animals or fun things in this world at these places so kids can find them. So I really wanted to start looking at games as a possibility space and the rules are kind of created by the play and response to the world versus the other way around. And I kind of see this as an ecological approach to play where systems can learn to adapt and players can evolve rules as the play experience unfolds. So I think that what this ultimately does is it creates a really empowered agency driven environment where people have control to kind of create their own kind of experience. So I was like, okay, that was really fun. So now what if I actually build something? So I'm designed this and I thought, okay, I'm going to build a sonic world. I'm going to make a synthesizer and clouds. And I have this idea that the clouds would sing to you. And the synthesizer would be this fun thing, this dome you could play inside of and touch. And this tree would be musical. And it would light up at night. So this was my very first hope of being able to make this kind of musical world that was actually beyond just a controller, but could be like a physical space in the world. So I had this idea that I would break the sounds up by like height. So I started off with the rocks. There was a little rock that had a bunch of low sounds on it, like when you would shake it, there were clouds ended up swapping the clouds with the hut. But it was rocks hut. Then I think it was the tree and then the clouds and the stars was the kind of the way that I broke the pitches in terms of the tone. So the higher up you got in the interaction space, the higher that the tones you would generate were. And I designed a bunch of this stuff. So this is the very first, what is that even? Oh, there's a beam. Okay. So I designed the clouds in Illustrator and then figured out how we 3D modeled them and then we kind of figured out how to get them flat and print them on a laser printer. So eventually they were these down here in the bottom and we sort of assembled them. You know, I think these are like, they're not even all even, but kind of like either hexagon or octagonal face by face. And I decided to build this geodesic dome, which was really, this was an interesting situation. I found out there were guilds in Germany. And if you're actually wanted to get like a ton of pipes and a lot of wood, it was actually a really big challenge. So I rolled up to the game installation and had a ridiculous time getting all these pipes. But I had a team of volunteers and this is what the synthesizer was going to look like on the inside. So each one of these is a kind of a touch trigger. So is anyone here familiar with conductive paint? So I painted my touch triggers and I wired everything up and in my house it all worked great. But there was a problem. They moved me outside at night. Conductive paint outside at night. Moisture. End of story. So about half the time this big, beautiful geodesic dome I made just did not work. And I thought to myself, okay, I'm going to have these rocks and I want kids to shake them. So I'm going to design all the electronics inside the rocks and seal them with earth magnets. This plan actually worked really, really well. This went great. They're very hard to get apart. They kind of like bite you. So designing the magnets to sit down inside a 3D enclosure to seal it. Really good idea. And I just lithium ion battery it. And so I 3D printed this enclosure, took a little bit of modification. And by the time we were done, I was really happy with the way it turned out. And I made them with glow in the dark filament, which was pretty ridiculous. And then they'd light up from the inside, which was fun. And with the tree, you know, we covered it, we painted it, we did this whole thing, we built the enclosures. I was convinced this thing would work. I did a lot of planning. So this is what the pads looked like on the tree once the tree was assembled. And this was the final piece, which given this was the very first big thing I ever built, I feel like it was a really close match to the model. But I learned a lot of amazing things about this. First of all, like things don't always go to scale. And one of the things that I had never actually accounted for before was the resistance of the wire over space. So these wires actually around this stone that was about about it was not this large, but it was like, you know, it's good, like, you know, four or five people could easily stand in it that much wire actually builds up a lot of resistance over time. So actually trying to conduct capacitive touching something without maybe not a good plan. But you have to be really adaptive in these game installation environments. So just took my my bear conductive board and literally copper tape it onto the side of the pads. And this actually worked. And I actually got my installation up running when this incredibly ridiculous manner where I just taped it on this side problem solved me gone. And this is what it looked like in the inside when people played it. So the lessons I learned are resistance of wire really freaking matters. Conductive paint and scale is an issue when it's moist. So always make sure that somebody is not going to move you into an entirely different environment. Because when you test and you stress test these physical games, something like just it being damp out can really kill all your stuff. And you really cannot be loud enough when you need you want to clarify your needs to someone like a games curator and arts curator about what you need. Just be really insistent, give them a tech writer, ask them to read it back to you, make them sign it and maybe go over it again like once a week for a month. And that's actually like a really realistic thing to do because I don't think a lot of people understand if you're building these big hardware projects, they just see the final thing and they don't understand like, hey, something like just damp will kill your sensors. So it's really important to kind of communicate that repeatedly to non technical people. Okay, so this is Babybot. I discovered this magic. This I'm going to tell everyone about this. This is heat shrink with solder built into it. This is magical. So you can just take your heat gun, line up all your wires, heat shrink them and solder them with your heat gun. This is really lovely stuff. It is worth the money if you're in a pinch and like a bind to build a project with a ton of knobs and a ton of stuff. So this is 100 pieces. It is absolutely worth the 14 pounds. Just like saves you so much time. So this is Babybot. She had a ridiculous number of knobs and LEDs for a small project. And I didn't really quite know how much soldering I was in for when I built her because I just, I did not click to me that every joint was going to have multiple sides and like it was going to take me forever. So eight LEDs, eight knobs just on her sequencer, then another LED on her head 200 sides, then every jacks because I wanted to sit in installation. So it actually like plugs into the wall and there's like a, you know, all those circuitry to make sure that all that is sorted is fine. So I installed her in this game installation and walked away. I thought there's no way children can destroy this. So wrong. So direly wrong. The inside was the most secure I'd ever made anything that I had built to date. There was hot glue. There was heat shrink. You know, the board was really secure. Like everything felt really legitimate. And what actually broke was the clock. So this is the clock for the step sequence are here. Some little kid turned it so hard. They dislodged the wires from the knob on the inside, even though it was like bolted on the inside. So, you know, perhaps less knobs is the answer. So lesson learned, glue is not enough. You might also consider some sort of like super glue, hot glue, it's not strong enough screws and bolts are not enough. And children are insane monsters, which have they're capable of intense creative destruction, limit the part to make the interaction really simple. Also children don't understand knobs. This is a very strange concept to us because we grew up without iPhones, right? They really were like convinced the knob was only to be jammed. They didn't understand that it had a range. Like, and I watched kids under five and six really not understand that a knob had more than on and off. So that was interesting. So I decided to evolve it. So this is bop party. I decided to see if I could make different sounds by saying if I could get the kids to kind of touch each other and like grab hands. So I decided to hide little animals all over the world. And if you grab hands with someone, the little animals kind of come out. You can hear them. There's like a tiger that roars. And then there's like a little like laughing critter at some point. There's a ton of little animals. My friend Frida Aduban made the sound for this one. And it has an accelerometer in it. So when you pick it up, it kind of responds. And there's eyes and buttons. And the goal is to get you to reconnect the humans. So it's really fascinated by modular synthesis. So this is actually a modular synthesizer. The step sequencer will totally remove from the synthesizer, which is the head. So I was thinking I wanted to make a ton of them, but a bunch of children with wires after seeing how bop party went or how baby bop went. I knew I had to have a more like bulletproof way to do this. So I decided to make hand holding the kind of interaction model. So I can tell when two people are holding hands, when three people are holding hands, there's not much of a difference between after three, I can have like I've had up to eight people hold hands and it's not broken my interaction yet, which is pretty great. And then I can get really nice tight interactions for when they release hands. So if more than three people hold hands for more than like, you know, half a second of storm starts, and then if they hold hands for like a whole minute, it builds and breaks. It's just fun little world. And I was like, Oh, I give them cute little faces. Right. And I'll CNC aluminum the size and sand them. They'll be really safe. It'll all work out. And then, you know, I designed them. They look like this when they were done. They had LEDs and buttons. And I do not even not even joking you, we were not two hours and in the installation to a child pulled the faces off the bots. And I asked, why are you doing that? Isn't it? Don't they all just change like potato head? And I was like, no, I had to go to maplins and buy super glue and super glue on the faces of the bots. So that was something I learned. Also, I learned that people will use your your game like an exercise toy if they think that it helps them get fit. I had a woman, I had a woman because the accelerometers respond to motion and twisting. And if you shake them really hard, they would like respond differently than say, if you just turn them, I had like one older woman just like with full force, like doing like, you know, I was just like, okay, whatever, this is not designed for this, but we'll just go with it. So this one made it through the entire installation of about like it was at Somerset House in London about like 6,000 kids over a couple of days. I was pretty impressed with that. It's the last day when one of the accelerometers had been had been shaken so hard, it had come off the actual circuit board. I was just like, oh my God. So the lessons I learned are all cases must be a single unified piece. Children will shake loose hot glued circuit boards on acrylic, screw everything down internally, super glue always. So this is thromb. This is my new haptic grid. So I decided to make a giant haptic array. So there is a haptic motor in each one of these. I've gone down to one haptic motor per hexagon because you can't actually delineate the four. It's something I learned when I was testing them. But originally it was 28 haptic motors on these hexagonal surfaces. And I was like, okay, all right, I'm making a circuit board. Y'all can't be trusted. Like I'm just going to burn it to circuit board. And this is what it looked like when people played it. So you would try and find a B. And this is my first playtest. And there's an idea is that there's a B that lands on the surface and you follow her to the pollen. And then you collect the pollen. If you go the right direction that she's telling you to go and you get the right pollen, you get the point. So these are the sort of boxes ended up looking like this. And this is what it looks like when no one is playing it. So I feel fairly, fairly certain this is going to be okay. So I'm going to get rid of that one final wire. You can kind of see it up here. And I'm going to run the entire thing just wirelessly. So no exposed wires. Everything is on like a really thick laptop level lithium ion battery to run like, you know, a couple of LEDs. It should last like, I think I haven't seen it die yet. And I left it on for about eight hours. So I think it'll be good for a whole gallery day. Like I think it'll manage to make it through an installation day. So what you have to really kind of aim for if you're trying to put things in game exhibitions or museums or art shows is that that cycle of business hours, because you can either do triage at the end of the business hours, or they can charge it, right? So with thromb, it's going really well. Nobody's tried to lift the hexagons off the floor yet. I might need to nail them to the floor or screw them into floor if I was going to put it somewhere like the science museum. But it seems to be pretty safe so far. And one of the things that I really thought about with thromb was how to hide the wires. So we draw, we dug channels in the wires all the way around. And even in the circle that goes around them, all the wires are dug into the channels. So like into the circles. And the other thing I really wanted with thromb is that it would be really easy to carry. So this whole thing folds down into, you know, almost no space at all. I'm aiming for a suitcase, like a carry-on suitcase. So I CNC designed the entire thing to be like flat packable, which actually makes a really big difference if you actually want to have any luck of touring your installation or your piece. So this is, it's adorable when people play it. This kid was named Casper. He lives in Belgium. So anyway, lessons I learned, all wires must go. Like if there's a wire, just get it completely out of the way. Like the install must include a way to make the wires secure and dock the electronics into lockable systems. It's actually really hard to get the middle of that, of thromb up, which all the electronics are hiding in that white centerpiece. It requires like you need something to actually pry it open because it's tight, it's really tightly cut around the rim. And wireless solutions will sometimes interfere with each other. So this is my new lesson is we have, we're sending just binary between the different systems of the wireless. And I really didn't think that I'm going to have to write the radios to not be receivable by each other because one of them is actually causing interference on the other one. So there can be interference on your radio channels. So other problems that have emerged. One is timing. Timing on microcontrollers can be really, really, really wacky. And you can end up with weird timer issues. The other one is portability. This stuff is really like if you want to be able to show your game, you need to think how can I get it in a suitcase? Like how can I make it cheap? Because nightgames, the first version, cost like 5,000 something to ship because it was so large with all those pipes and all that wood. Timing on the chips are not 5,500 rather. Timing on the chips, that can, like I said, it can be really, and also for time I mean like this stuff is going to take way longer than you actually think. Like when you go to CNC something, even if you've tested it on a laser cutter, it could maybe not work out for you. Like you could have forgotten to fill it. You could have forgotten like how to like dog bone one of your joints. And then all of a sudden you'll have to go back. So always give yourself plenty of time. The other thing is be just like ridiculous with your writer. Like, okay, I need this. I need these extension cables. I need this much time to set up and just iterate it over and over and over and be like, okay, go over that with me again. What do I need? When do I need it? Okay, if you can't get it to me, you have to let me know by these dates. You know, just be like really clear about it. And the other one is funding is a challenge for this stuff. It's actually really expensive to make this work. And I think that kids museums are a really good venue, but I'm in the process of looking for funding. So if anyone has solutions that has worked for them for funding, I'm really open to hear it because it's, you know, this stuff is actually phenomenally kind of crazy. So those are my problems and interests. This is me on the internet. I'm at Phoenix Prairie, but I thought it'd be really cool to talk about problems that other people had for like building physical installation stuff and just open it up to questions. Are people cool with that? Awesome. I kept this one really chill. This is like not at all as crazy as some of my talks have been. Questions, thoughts, problems, Q&A. Just tell me things that have gone wrong. Here, take the mic. You know you want it. Do it. I have a question. And then you said you were working with neural networks. What were you using? Is it Wekinator stuff? Okay. Just a small anecdote. I have a colleague of mine who works in, they built huge installations for children's playgrounds, electronic installations. They had to make their own production line just to get it child-proof. It is crazy. Because they break everything. Yeah, everything, everything. The person I'm talking to now is a Taiwanese toy manufacturer and he's been giving me pretty good advice. Like you need to think like on the level of something that could almost go to manufacture for children. But even then, like one, the wearer that one child can give a toy is very different than like a thousand children. Yeah, you, I've heard people, I asked one of my friends and he's been building to scale models. So he's been building like scale at scale hardware. So he has a set of hardware that never leaves his studio and he has another set for deployment. Just kind of interesting. Other folks? Yeah. Related to that, I think if you build something like this and you want it as a playground, and then it needs certification. But if you qualify it as an art installation, you sometimes don't need this certification. And it's, yeah, that saves a lot of money because it's over 10 grand for certification. Holy Christ. Yeah. At least here in the Netherlands. Right. So that's a remark. So sometimes it's safer than to go for art installation. But the responsible parties and the owners have to agree upon that you do this. Yeah, I think that that's probably very true. I mean, we're, I'm, you know, lucky enough that I've been only operating in the games context. So we have a huge amount of latitude that I don't think other things good. Because people go, oh, it's a game. And then they don't take it seriously and actually works to your advantage. Yeah, okay. And another remark I recognize the ink problem. We had the same problem. Mojter was the problem also. And then another tip for you. You said you used Illustrator to to unfold the 2D designs or the 3D designs. It's the very first thing I did. I know I now know there are better ways. Okay. Yeah. You use Blender. Yeah, we ended up going that route by the end. Because Blender Blender has a paper, paper model. We've learned. Yeah. I learned this after I did the first one, my friend suggested. Who? Say it again. Bit bar. Say it again for other people. Oh, okay. Oh my God. That's amazing. But still, still, it's the core. It starts from a 3D model. That's really cool. Other folks of my little bots. No, no, I have like bits of thumb in the car because it's I'm using it for my PhD. So I'm doing PhD research on haptic games. So I'm working with a bunch of designers to so Heather Kelly is the designer on thromb. So I'm building this with her to kind of develop a haptic platform for alternative control games. So I'm trying to meet other people who want to make alternative control haptic games and then figure out is what they need and then build stuff for them. So other folks, I build my own controllers like everything in bot party. I even built the sensors in bot party except for the accelerometer. But yeah, I mean, no, I mean the microcontroller like Arduino. Oh, Jesus Christ. It depends on what I'm building, right? So like I don't really have a favorite. I choose based on the implicate like the the ramifications having it. I have a whole box, you know, it's sort of the microcontroller box. I'm sure a lot of you have that box. Yeah, it's just like Jesus. I am a fan of teensies, but I have had some issues with getting the capacitive stuff working because you know the capacitor charges up like there's a weird delay. So I don't like that. I mean, other than those, I have a ton of those. But I've been having the USB ports fall off on those recently, which is making me a kind of bummed. And I think a lot of it's because my teensies get more abuse than normal teensies. Other folks questions, even thoughts or things you learned in your practice, you think other people I have no authority here really. Other people might want to know about. Go ahead. Go on. So I heard about the paper recently. I saw there's a multi touch capacitive surfaces with the conductive paint. You can do multi touch with multiple wires to with electric with CK, electric with CK. So it's a trick. It's like the very conductive paints very similar. Oh, does everyone here know about less EMF? No. All right. If you don't know about less EMF, you owe it to yourself. All the conductive materials, a lot of the conductive paints, a lot of the stuff that you pay a lot more for actually comes from a strange distributor in the United States called less EMF. They sell ghost hunting equipment, but amongst the ghost hunting equipment is all this conductive material and conductive fabric. And you can buy big gallons of conductive paint and like conductive lycra and like Velostat. I bought a roll. I carried it. If anybody's in London wants Velostat, just call. I bought a bolt and I carried it back from New York because the bolt was $50. It was just so cheap. Does people know what Velostat is? It's a material that changes resistance with force. So it's really you can make your own force sensors with it really easily. But it's really expensive if you buy it from like any like, you know, distributor, like Adafruit or something, but you can get the bolt for like $50, which is like this much fabric. It weighs like, you know, 75 pounds, you know, and they have like conductive lycra and like conductive shielding and it is kind of a magical place. Less EMF, L-E-S-S-E-M-F. Even with shipping it to Europe, you're still going to pay less. Yeah, particularly because they'll sell really nice. They have like conductive sample packs and stuff. You can call them and be like, hey, can I get some samples to test your stuff out? And they'll send you samples really cheaply, which is pretty great. And then you can see if any of their stuff actually works for what you're after and then, you know, it's worth it. Oh, just conductive lycra. And then the other stuff is Velostat, V-E-L-O-S-T-A-T. That's the resistive stuff, huh? Yeah, they sell it by the bolt. But yeah, they mostly target folks who like do ghost hunting, right? So they're terrified of EMF. They got to shield them. It's really good. But, you know, you need a lot of silver stuff to keep out the ghosts. It's most excellent. And they've got like every kind of, like they've even got like, like all kinds of fabrics and materials that do conductive stuff, conductive knit, like knit, like yarn and about kinds of stuff. You can knit your baby. Yeah, there's a lot of crates of people out there, especially in America, in rural America, some extra special fun people. Anything else? Y'all should go drink and dance now. It's late. Yeah, thanks. See you.