 Alright, hello everyone. So our next speaker is working at Body Hacks, the Body Hacking Convention. And his focus is to introduce hackers to the concept of sensory augmentation, basic science behind our senses, and neuroplasticity. So here to present his talk, Hack Your Senses, Sensory Augmentation and Neuroplasticity. Ladies and gentlemen, Trevor Goodman. Thanks. Alright, so like he said, this is sensory augmentation one-on-one, I think on a schedule, or I renamed it the last second in DEF CON fashion. Hack Your Senses, Sensory Augmentation and Brain Plasticity. I'm Trevor Goodman, I'm the Events Director for Body Hacks, the Body Hacking Convention in Austin, Texas. If you want to ask questions afterwards or anything else, the QA mic is great, but if you forget that, this is me on Twitter at bellowosa. Let me put my notes back up. Don't mumble to yourself when you have a microphone on. So, Body Hacking is about using and applying techniques and technology to make your brain and body better, and this includes technology and techniques that are old as well as ones that are new, so you'll see both referenced here. But today I'll be focusing on hacking your senses specifically, because your senses are the only mediator between your mind and the outside world, so that includes your body. So I believe sensory augmentation has the most potential for helping us really understand our environment. But what is sensory augmentation? This is Neil Harbison here on the right. For those of you who don't know him, Neil was born monochromatic, so he's completely colorblind, he sees in black and white. But while he was at university, he worked with an engineer to develop this antenna that is osteo-integrated into his skull that lets him hear through bone conduction in a wider range of colors than you can see. So Neil's one of the most well known people to have augmented senses, and is certainly one of the more dramatic cases, but there are likely people here at DEF CON who have magnets in their fingertips or probably some people who have cochlear implants. So sensory augmentation is extending the body's ability to sense aspects of the environment that are not normally perceivable by your natural state. So you're born, anything you do on top of that is sensory augmentation. For the most part, sensory augmentations currently rely on using sensory substitution, which is using one sense to replace another. We'll touch on a lot of sensory substitution as we go along, because the vast majority of sensory augmentation right now relies on that. Direct brain interfaces are being tested right now at DARPA, and DARPA related re-net projects, including one at Case Western University where they've linked a prosthetic to a sense of touch directly into a guy's brain. So if he's rummaging in a bag or picking up objects behind a curtain, he can have some sense of what he's doing, even though it's through a prosthetic. There's another related project through re-net where DARPA has hooked a paraplegic woman up to a flight simulator directly through her brain, and she can fly the jet using her brain, but I'm not sure if they have the jet sensors feeding data into her brain, or if she was just using the monitor. But we're really close to getting into some of that space. Is this me? Is this it? Okay. But how does that work? How can we essentially just plug electrical current into our brain and get our senses on the other end, because that's crazy, right? So neuroplasticity, that's how that works. This is Paul Bakke-Rita. His father had a stroke in the 50s, and when you had a stroke in the 50s, the doctor said, you're screwed. You'll never walk again. Sorry. But his brother, George, was a psychiatrist, and he felt like his dad could walk again, so he made him crawl, and he made him go through all this really intense, horrible physical therapy for the time. And all those doctors said, you're crazy. You'll never walk again. This is stupid. Why are you doing this? But eventually, he did walk again, and now we know that's what we do with stroke patients. We make them go through really horrific physical therapy until they can relearn to walk again. And after his father's death, they did an autopsy and found that the stroke damaged areas of the brain were still there, so it's not like they're just gone. So it seemed like this was evidence for neuroplasticity, and that the brain had rerouted its connections around the damaged areas. And, oh, that's the wrong button. Paul Bakke-Rita went on to study neuroplasticity by creating sensory substitution devices. One of those devices originally was built to teach a sense of balance to people who had lost it in injury. Eventually became the brain port. I think this is the original balance sensory substitution device, but it may also be an early version of the brain port. The brain port takes data from a camera, so what it does is it's intended to help blind people have a sense of their environment. So it takes data from a camera, it transmits that, or changes into a grayscale pixel grid, and then it takes that grayscale pixel grid and transmits it to an electrode matrix on your tongue. And so you have this little rubber thing with a bunch of little electrodes on it, and you stick it on there and it vibrates, and it buzzes on your tongue in intensity based upon how dark or light the colors are. And with modern brain port and proper training, a new blind user can safely navigate a brand new environment in less than 10 hours of training. And so now we can see on brain scans that activity is registering in a visual cortex even for people who don't have sight through their eyes. So Paul Bakke-Rita always said you see with your brain not with your eyes. As Tim Cannon said first thing this morning, using these exact same words, which I thought was lovely, your brain is a pattern recognition machine. That's mostly what it does. It takes in sensory data in the form of electrical signals through your eyes or your touch, or any of your other senses, and then layers it on top of your expectations of your environment. So your brain supersedes your senses, so you see with your brain not your eyes, but your senses feed back into that and develop predictions and expectations of your environment over time. So you have this feedback loop even though your brain tops down your senses. And we know your brain tops down your senses because of stuff like optical and auditory illusions. Your brain is trying to figure out and guess what's happening here as your eyes are confused. So we know we can augment our senses. But why would you do that? So this is my snail friend. If you were a garden snail, you'd be able to taste and you could smell. And you'd be able to feel your body balanced over a leaf. We also know that snails can traverse over the sharp edge of a knife blade without cutting themselves. So they have some sense of body position. And your little cute cup shaped eyes would be able to tell the difference between light and dark. And you could see things a few inches away. But the snail's sensory bubble is pretty small. The part of the world that can know and can understand is tiny. As most of you are human, we've got a wider array of senses than that. They're commonly broken out in five, but it's really easy to identify more than that. Hearing is an obvious one. Smell and taste go together. Vision is, we just call it one sense, but it's also light and color and edge detection and motion. So there's lots of other things that fall under your visual sense. Touch comes into pressure and pain and cold and heat. All of those get lumped in there. All of those are your somatic senses. And then you've got a whole host of interreceptive sensors for things like blood pressure, lung inflation and hunger. And balance and joint position go into those things too. So if you don't understand what all of these are, for instance, we'll talk about joint perception. It's one of my favorite things that you don't think that you have it as a sense. But if you close your eyes and move your arms around and you stop, and you think about where your arms are in space, then you know where those are. So you have some sense of your body position that's separate than anything else. Loss of proprioception, which is the position thing, that's what we call it. It's a stupid name, we should just say position. It's a rare disorder, but it can cause lack of coordination and eventually complete lack of awareness of the body. So it makes you feel disembodied as if like you weren't really in your body. And this is also what the police test when they think that you're drunk and they're making you touch your nose or walk a narrow line. All these things together are what we call perception. German philosophers would call this your Umwelt, but we'll stick with sensory bubble, I think it makes a little more sense for most people. You're probably pretty happy with these amazing human senses that you've got instead of the snail senses. But there are a lot of things other animals can perceive that you can't. So the visual range, let's just talk back about site because we're mostly site oriented. The visual range that we see is a tiny sliver of the electromagnetic spectrum. That includes everything from gamma rays and x rays on one side to AM FM and microwaves and cellular on the other end. So imagine for a moment, all the different types of radio signals moving for this room. And you just can't see it. You don't process them. They're here. You can pull out a tool to do that, but you can't understand it. But if you look just outside of our visual ability, you'll find ultra violet and infrared in a lot of animals already. So bats will do echolocation. Snakes can see infrared, but it's a molecular sense. It's actually not visual based. So it's a chemical thing. And then bees can also see an ultraviolet. Also, Neil Harbison can see an ultraviolet infrared. So it's not outside of what we can do. Additionally, even if you just want to look at color, there are a lot of other animals that have different color cones, more color cones than we have and who can differentiate between different wavelengths better than we can. So like, for instance, oh, we'll talk about that in a second. Catfish are fun. They've got about 250 times as many taste receptors as you do, and they're all over their skin. So they taste their environment as they swim through it. That is an entirely different sensory experience of their world than we have. Imagine tasting as you walk through death con. For dogs, this is a good dog. He's also wearing AR glasses. And they, I mean, he's already augmenting his senses. I mean, it depends on what software he's running. But they possess up to 300 million olfactory receptors, which is compared to 6 million in us. So whereas we are visual, we understand our environment visually. Dogs understand their environment through their smell for the most part. So you can blind a dog and it really doesn't lose as much of its understanding of its environment as we do. But it's seen chemical trails pass through its environment all the time. But we don't stop with sensors inspired by other animals. We can build senses that let you sense the ozone level in your area. Moon Rebus has an implant in her elbow that lets her feel earthquakes as they happen all over the world. You could build something that lets you know the amount of email in your inbox or stock prices or since other cyborgs walking down the hall. Personally, I've got a horrible sense of direction. So I've always been interested in stuff like the beast and a lot of birds have a magnetic north sense. Oh, before we go anywhere. Narwhal is really cool. We skipped over this, but so this is actually a tooth. It's not a horn. And that tooth senses salinity and temperature and water pressure. And so like they're sensing a lot of stuff about the salinity is really cool because they breathe air. So when they're able to sense it's something about when the when the water freezes on the top of the ocean, the salinity changes and so they can tell when the ice is frozen above them or not. So if they need to get up for water, they know where they can air they know where to do that. So here's a fun example. This is a compass. You'll probably know what those are. You hold it in your hand and it tells you where Magnetic North is. This on here on the right is North Sense. It was designed by a company called Cyborg Nest. They released it about six months ago. Their mission is to create new senses. On the board of that project is Neil Harbison. Also, they've got some body modification experts in there as well. Steve Hayworth, that's who it was. Steve Hayworth is also on this project. So at face value, oh sorry, Cyborg Nest's mission is to create new senses for humans. And it's pierced on your chest. So it's right here and it's I think it's about one centimeter by two centimeters. And it vibrates when you face north. So face value both of those are really similar. You're like why would I do this? Why would I put this thing on my chest when I can just use a compass? But the compass is a tool that relies on your constant focus and attention. And North Sense is designed to be worn and felt all day every day. It's designed to be slept in it. And so over time, these become part of your experience and it blends into the background like the hum of the AC or things happening in your peripheral vision. So it starts to sound a lot more like how your senses work than how compasses work. Because a lot of the things that you're processing with your senses are processed in your unconsciousness. And until something pops out and stands out, that's unusual. So now everybody's paying attention. But if somebody made a noise in the back, you might not even notice. So here's the thing about the brain. The thing it does best again is to see patterns in data. And so when you put that sensory data on your chest every day, and it creates a predictable pattern in your environment. So every time you face north, it buzzes and tells you that you start to feel and dream and think in north. You have attachments of memories relating to where they were facing what direction they were facing on the planet. And North Sense has actually got enough users finally that they're starting to work with, I don't remember the university, but there's a university study with North Sense users that are doing an N of about 150. So they're going to get some decent data. And they're going to be able to see how people are processing this and what it's doing to their, the way that they interact with their environment. So again, I argue there's a huge difference between these two things. This is the big argument on the internet whenever somebody talks about North Sense. Well, I'll just get a compass. Other big projects in this space that I think are important really help do this. This is NeoSensory's vest. This is one of my favorites. It has vibrator motors attached to the side here. And the vibrator motors translate sound waves into tactile feedback that are intended to give people who are deaf a way to process sound. So if you're wearing the vest and I say technology, it will go. And if somebody else's technology will take those same sound waves and it will go, but it'll change based upon pitch and everything else. So it's not taking the words and then converting it into tactile feedback. It's converting the sound waves themselves. And this is at the wormhole. So we're playing with it here. And you can see it light up as it vibrates the music. But the applications don't necessarily stop with playing at dance parties and helping deaf people here. The vest can be set up to translate stock data into feelings. So you can feel the dow fall and plummet all day long. Or to feel the pitch and roll and yaw of a drone. So if you're wearing a VR headset and you're wearing a vest and you're flying a drone, you're going to have a much more I'm in the drone experience than you're going to have I'm sitting here watching it on the ground with a thing. In the middle, this is an AR helmet. I don't actually consider this sensory augmentation, but I think it's an important example. This gives a heads up display to the motorcycle riders. And it's definitely more wearable tech, but as some of this stuff gets better, you can certainly see how if you took this augment in reality and you applied it to your vision constantly, how it would become a part of your subconscious reaction to your environment. They're this on the right side is Daniel Kish. So you don't even need really cool technology. He takes groups of blind people out in the wilderness and trains them to use their hearing to echo like a using clicks or cane taps. And there are other projects that are using gene therapy to add red, green cones to people with color deficiency. It works in chimpanzees and others working on making people tetrachromats, so having a yellow cone. But this is DEF CON, so if you can't do it yourself, why do you care? So here's some examples. This real quick is I think it's a Northpaw or Southpaw. This is a variant of another north sensing device. They built this based upon a project at a university where they had built a belt with vibrator motors all around it and it would vibrate where north was all the time. They're seeing, they saw similar results in their stuff as north senses seeing. This is the same thing it's on a smaller scale so it's like a little prison brink bracelet on your ankle that vibrates. We've already talked about magnets a lot today so we'll not do too much of that but there are a lot of other projects. A bottle nose for instance that is supposed to go over that and help you see this haptic since haptic stuff is really interesting. This is again another way you can do ultrasound. This is a Arduino project on Instructables. And this here is the voice so there are a couple different apps even. This is a different visual program. This is like the brain port but instead of translating it into an electrode grid what it does is it scans vertically every half a second or second and it gives you an audio signal of what's in front of you. That's an Android app. It's also for iPhone. It's also for Windows. You can turn it on your and it's the voice for Android. You can download it and start playing with it right away. I've got it on my phone if you want to see it. These are all projects from somebody that is really awesome. She does a lot of research in this space so she has a lot of really fun projects. Her name is Isen. This is kind of like the brain port. This is her version of trying to recreate the brain port. This has a much lower resolution than the brain port does. She's also done an ultrasound that was intended for being underwater. This is another echolocation project. She found that if you blind people they actually start to use their visual cortex really quickly and adapt to it a lot more quickly. This is a watch that gives you different smells depending on the time of day so you can get a time sense that's substituted with your smell and then this is a bone conduction headset that she calls the playa grill that has an MP3 player that plays through your teeth and then through your jaw. I have another cool thing you hear about and I forgot it. Oh yeah so there are a lot of different groups that are putting some of these different projects together. There's hack the senses in Budapest. There was also recently cyborg futures at Parsons where they're putting together workshops to help people design and build new senses. I'm hoping to see some more stuff like this at Grindfest soon but I hear they're mostly interested in shocking each other. And with open source hardware like there's a company called Bitolino and open BCI both of those do different biosignals and the basic stuff you can get a Adafruit spark fun you can do a lot of fun sensory stuff right now. So if you want to hear more about this and related stuff here's my here's my stupid pitch that everybody hates. Come to Body Hacks first week in February Austin Texas here's how to reach us here's me there's my thing questions. I got like two minutes right morning. Alright cool. Who wants to be hit with a squishy brain. Cool questions. Nope. Cool. Yeah you can ask me here if you want to talk about it later. Thanks. So you had about 20 or so human senses listed which is different from the five senses. Yes. Which one does the five cent the fifth sense sound fall under is it pressure. Sound. Yes. Sound is its own thing it comes in compression of way like air pressure waves into your ear. So it's it's totally its own thing. So it's a type of pressure. I guess it's a type of pressure reception. I'm a lay expert in this stuff so that may be over my head. Alright neat. Yeah I mean it bounces off the your ear drum. Thank you. No worries. So for the RFID chips I know that there is documentation out there that you give to a piercer or body modification expert and they are pretty easily able to do it. Do you know if the north sense is the same or is that a lot more difficult. So this is funny actually was reading some stuff on it this morning. There is a lot of disagreement about whether the north sense it's mostly in like whether it's a good place on the chest or not because a lot of piercers will say hey it's pierced on transdermals. So they were using around ones but they've moved to flat transdermals and north sense says that they think that those are better and they haven't had any issues in the people who they've been testing with. But people like Sampa Von Cyborg are like hey well let's just wait and see. So part of this study of like 150 people is I think hoping to figure out whether like it's an actual good test location and I'm really excited about this stuff but I'm super worried about knocking all that and hitting it and having them come off. So personally I don't think that's a good location for me for that but a lot of people are trying it and we'll have to see. Are they going to place it on like a back? Has anyone tried anything else? I don't know if I don't know for sure where other locations have been posted. I know there's a guy Russ Fox who I think has placed them in a couple other places but I don't know specifically. Thank you. I know those flat transdermals are a lot better for piercers. Yeah well that's what they're saying too but we'll see if it works. All right. So how do you explain all this to your doctor? How do how do I I mean I'm lucky enough to not really get sick. But like what questions would your doctor have? For example you be the doctor I'll be you. All right. So why did you decide to pin this magnetic north sensor? Oh well OK in that kind of case I would I like to ask that with another question. It's like OK well if you if you could sense more of your environment around you why would you not want to? All right. Second question. You need an MRI. Do you have any magnets on you? Your fingertips. Yeah well I mean so the fingertip magnets I think it was funny we actually talked about this at the Q&A for the last talk. There's a lot of argument about whether it actually causes issue with with MRIs or not. I don't have any personal experience but with I mean the north sense you could unscrew the balls and pull the thing off if you wanted to for an MRI and hopefully you're not getting MRI every week. And I mean I most people probably don't get them every year. So I think for those kind of things those are those are niche issues which we have issues with any technology. So I don't think it's any different than any other thing that we adapt. All right. So a couple years ago I saw a demonstration from Tachi Lab on YouTube they had a some some texture based on a glove with a little thing on the inside and you would touch a touch screen and you would feel the synthetic texture on. Nice. So they're doing a haptic thing with it. Right. So then then you've got like sense egg right which is the sense to you or whatever it is where you have to drag your finger across it. So everything I could find. Then they're all Japanese. Like how do you get that that fine grained texture thing without the movement. I don't understand the science between a lot of that stuff. And you're talking about projects that I actually haven't heard of. So we've we've had a lot of trouble reaching people in Asia for stuff like this until recently and so a lot of that is just now starting to change those lines of communication are opening up. So I'm not familiar with those projects so please please tell me about them and give me links so I can go look at them. Cool. My background is in running events and psychology.