 So yeah, these are our names. I'm Alessian and he's Christian. We are both from Kunmari University. We are PhD researchers and We work with sound my background is in architecture and his background is in sound design, so I am interested in space and how space is used and how sound actually is part of this kind of use of the space And so we'd like to present you some of the project we are working on and And some we are worked on previously and some future directions as well So there's an important component that is the Design methodology for this kind of objects which are very physical and their way to be played It really depends by how they are designed and how the sound is designed as well So we're gonna talk about The technology we use for for these objects. So the first Creation is a is a broom. So There's an Arduino which is controlling all the input from the sensors on this broom including an accelerometer and an embedded speaker and the idea was to create a Sensitive object which could interact with the space and was an everyday object, which every person could use But I was mostly interested in how The interaction with the floor and the space could come out of the expressivity of the instrument So I tested this Mozi library for real-time audio on Arduino and and Later I changed a bit the design of the interaction including a distance sensor to control the pitch and Some kind of gesture sensing as well and you can see some examples from And after this experiment I went into Like trying different time from for this instrument But I was really interested in what people could think if you actually play an instrument like this in a normal house And what can happen? So I had many interesting Opinions from my musicians friends, which thought I was a bit crazy So Yeah Already said that the second experiment was Odette with Lidati Odoru and in this case we wanted to experiment with Kind of skeleton tracking from Kinect and we wanted to create an alter ego mirroring a movement of a user And in this case the user had to become part of this kind of dialogue this performative interaction in front of this image So the participant in this case was becoming the puppet master in a sense and the The shadow created by the object and the lights and also the sound was part of the Performance so when people could come and see this scene they were thinking what's going on and they wanted to be part of this kind of dialogue and The interesting thing is that the object was kind of fascinating in a sense, but also made by a very Common materials as bubble wrap It was clearly a closed anger with an Arduino and some servo motors which also produced sound which was very creepy and The sound I decided to program in super collider. I used an already existing flute Patch, but it was creating kind of different pitch of sound. So when you were waving your your arms In the air then you were creating like spooky sounds kind of 80 You also have a video to show So it really seemed like a ghost And it was really In terms of understanding how I was working it was really clear to see all the connection Which was also part of the of the design of the of the piece in this case the like In this case The Kinect was in a black box Below the the wings so we used to call them kinetic wings at the beginning So we were experimenting with kind of responsive structures And then we decided to like play a bit on this kind of personification of the character playing in front of you You were also animating and we use open and I for the Kinect libraries and processing So processing was sending serial messages to Arduino And from this I believe like Okay, so I'm gonna talk about this one briefly now. So this instrument is called the air harp So the idea behind the air harp was that it was originally started out as I'm this true box project So the idea was that you had a box on the table And it was all up to the person interacting with the box to figure out how to play this instrument and I really wanted to Design something that allowed the person to make music pick it up and make music within less than 10 seconds basically So this is the way that you interact with the instrument I'll actually try and give you a live demo first So basically it's completely embedded. So battery speakers and all this are in the box Nine physically model strings going across the length of the box and then there's some physics simulation which Changes the position of the string of the object which in turn plugs the strings based on the orientation of the device So the idea is that the interaction should be very very natural and very easy to understand This is also the second generation of the air harp where we laser cut this this nice front panel Reason we did this actually was to let the sound come out of the box because initially we just had one single f-hole on the front And so that was nowhere near loud enough to amplify the instrument So here's just quickly how it works. We've got a virtual mass on two springs And as you tilt the box we can derive the resulting gravity forces that would need to push this mass across the strings And this is all there is to it basically And here's just a picture of the inside. We've got a single accelerometer a bella board, which I'll come back to later Battery and two speakers and a line out and the potentiometer that does nothing. Okay I think I just went through all of this okay, so Physical physical modeling of sound is really useful for this kind of thing because it does a lot of the work for you You just tell it and I'm gonna put in This much force and it's gonna behave exactly how I expected to within the constraints of this physical simulation But I found especially coming from a sound design background that this is often not enough And I wanted to be able to perform Environmental sounds as if I was performing an instrument So here's a picture of somebody spring WD-40 on a door hinge And I say no don't do that Because I can use it for making synthesizers. So the idea with this project was to Use a trackpad to a very simple input mechanism to produce squeaky door sounds And so there are two approaches you can take you can either physically model the entire system Here are some papers that use equations right from tectonics seismic research to produce the sound of friction I Took a different approach to this which is to kind of approach the sound of a door like synthesizer kind of like a modular synthesizer where Instead of thinking about physical processes you're thinking about oscillators and filters and you name your parameters according to what you think It sounds like and it's a bit of a iterative design process. So this is the block diagram of our squeaky door model I've got an impulse train generators and amplitude modulation bunch of band pass filters and resonators And I should say this model was Developed based on Andy Farnell's prototype So I'll just play a video of this So over here you can see the trackpad Position the position of my finger as I played it and here you can see the parameters moving and how they are mapped to the parameters I won't play the whole video, but you can look it up later. Watch it yourself Okay, that's the wrong title basically so the model was designed in pure data and We use touch size from a surface to control the model And The main thing here is that if you want to perform environmental sound don't necessarily go for physics simulation And think about it more like a synthesizer. I'll just hand over back to Alessia after those experiments you saw before I decided to Start being a serious person and do some like kind of soundscape research in Related to architecture and so I decided in London to look at the area of Greenwich because of the architecture there and The river and a lot of green and so I was interested in how A location which is quite known and used as a restorative Place by many people including tourists could be understood in a different way Just because of its sound and so I organized a soundwalk which is part of my PhD research Who's updates you can see at this link and And I decided to embed the recordings in an interactive object and I decided to use fabric for this The the idea behind the fabric is that you can immerse yourself in this situation Recorded by someone else just touching the correspondent buildings and this is possible because I used conductive thread and An embroidery machine we have but it can be possible also just doing by hand And so for every one of these places I associated one of the recordings I made so This can be done With Bella because Bella allows you to to detect the touch and also To do some nice fading which is very natural and it's what I wanted for this object because after the strange Sounds coming out of the first two pieces. I wanted something a bit more relaxing to be my part of my research And so in this case You have the surface of the fabric which is kind of the conductive part you can touch and then There's a layer of insulating fabric with some holes and then the same capacitive part Which is sensing the touch goes to traces which go to this red ball and PR one-to-one and So I had to stitch all these wires And put them in place there and everything is going through I square C to Bella and Please help me like programming the code to To read these things in real time But it was amazing that everything could work just by plug in the cables because the fabric is actually really resistant compared to some other electronics So in this case, there's a strong meaning because I'm trying to look for interfaces that I can bring to other People which are not so familiar with what I'm researching which is sound and acoustics related to architecture and try to to bring something intuitive to them and This is also what like the methodology of use for another project So I went To China recently and we were supposed to create music boxes in a local village So I was really interested in like The sound environmental sound there because it was very relaxing with mountains and river But also I was interested in the local music. So I decided to record these sounds from local people and My partner in in this project is Shen. She wanted to have these Music boxes very tiny objects projecting shadows and this shadow was supposed to be the local Decoration patterns that they also use in weaving and so after some research the project had to be delivered in Seven days basically and so I tried to find the easiest way to To to be able to have local people interacting with these objects and and see the reactions And so I use in this case Arduino and other fruit soundboard and an accelerometer and So everything had to be embedded in this box, but the interaction should be like really straightforward and It worked well because even kids could get it easily. I have no sound So they understood immediately There was this other feature you could connect two boxes and play another song just by sensing a magnet and It was really interesting to go with these boxes There are spaces in there like social relax spaces just used for gathering and talk about local affairs of a village and just sleep there or chat and relax and They were really interested in hearing of course their music and they said this object is too simple for us We want our decoration you have to carve something on the sides because we we don't really recognize this cube as something which is ours so I needed to To be helped by a girl another design student explaining what they were saying and Was interesting to see how different people also were Giving different advice on the boxes. So they were saying embroidery is gonna be really expensive You don't have so much money for that amount of time Yeah That's it So you just rotate the side and change the song and then if you put it back to its original position then It stops working. So it's it's really simple thing, but I thought that for in my case, it's important like to do more in the research part like gathering the right sounds and then Leaving the design to the minimum, okay, then I will go back to Presentation Which is still okay? I'll talk about the lightsaber in a second But basically while Alessio is doing all this amazing soundscape research I was in my research group developing this board called Bella, which is a audio platform that works with the Beaglebone an audio framework really And this is what we've been using for lots of these projects. We've been talking about basically So the nice thing about this board is that you've got extremely low latency Audio output. So that means if you press a button, it will take you less than Less than two milliseconds or less than a millisecond sometimes for the sounds to come out of it Which is very fast And also the sensor Bandwidth the resolution is very high and the sensor analog sensor sampling rate is very high as well Which is at audio rate. So this allows us to make some very very The objects with a very very high level of expressivity in the interaction so to demonstrate this we built a lightsaber So you might notice that this lightsaber doesn't look like lightsaber. It's just a cardboard tube And this because we want to focus Only on the sound we weren't interested in the in the lights and so on So the way this works is we've got Bella inside one end of the tube and the battery and These are all the components that we're using so we've got an accelerometer over here a piezo desk over here a speaker our Bella board and the battery And so we used a piezo desk to detect hits on the sword So when you're swinging the sword around and you hit against the surface it will go You know And as you move around with the accelerometer We use that to we integrated the accelerometer value to get a velocity reading and we use that to produce the iconic lightsaber sounds And I'll just play a quick video which I may skip through a little bit which explains how this works So lightsaber is one of the most iconic sound effects. Everybody knows lightsaber from their childhood And I'm just gonna skip forward. Now these delay lines are a variable delay. So okay, so basically what we did was we Just go back here Okay, so one Ben Burr designed the lightsaber is what he did was he played the sound of a TV hum Coming through a speaker and then he used a very long microphone like a shotgun microphone and recorded the output as he imitated the swings of the Jetta is with their lightsabers in the film and We replicated this process inside pure data. Are you all familiar with pure data? For those who aren't it's basically Digital processing language a visual language where you connect lots of components to each other with cables And we can use this on billa basically so we reproduced this idea of Doppler effects delay lines and And movement using the PD patch and Then assembled it all inside lightsaber cardboard tube Unfortunately, we don't have that one with us to demo today I'm just gonna move straight on here, which is I'll just quickly wrap up this actually so basically what we're interested in is How does it feel to interact with the lightsaber when you have this low latency? What would feel like if you actually had a lightsaber? What would sound like and so on? and this is something that was really Made possible through the low latency and through the high bandwidth resolution sensor data Okay, so this is the final instrument will show you today, which is called we call it the vanguilla sizer Also known as the shaker and The reason we called it the vanguilla sizer was because we wanted to make an instrument that contains Literally just an accelerometer a billa board a speaker and a battery so something could hold up Something that you just requires the most minimal kind of interaction basically so So it's a couple long story short basically we started by making a synthesizer that emulated Vanguilla says soundtrack to Blade Runner and the synthesizer settings he had And then what we did then was we had a very iterative rapid process of switching between the patch designing the sound and designing the Sensor mappings to the sound and because Sound and sensor you don't really distinguish between them when you're programming for a platform like this It made it very easy to prototype stuff very fast So Basically what I'm trying to say with this one is that if you have very very rapid very iterative design Then you may end up with some strange results, but it will be very very carefully designed because you've had a very very short iteration stuff Okay, so just to wrap it up here are some little nuggets of wisdom. We thought we'd collected by Working on these instruments. So a keep it simple There's an enormous amount you can do with Just one single sensor Almost all of these instruments were showed today work with one single sensor or two sensors And we find that constraints are often extremely useful in making an engaging experience also Have this desired experience in your mind when you design the instruments Obviously people will misuse it or use it in unexpected ways, but if you've got one Design principle in mind, then you'll come up with something very strong Don't buy us to focus just on the sensors or just on the synthesizer, but think of the whole instrument as a whole And it will help otherwise. It's very easy to forget about You know, how am I gonna map the sensor range to this parameter and oh actually doesn't actually sound the way I expected it to and so forth And finally as I said before make sure you're comfortable with your design development environment and make sure you have a fast The ability to have fast iterations so the more quickly you can see the results of what you're prototyping the better your results will be and I think that is all of it from us. Thank you Can I just ask is the next speaker in the room right? Okay, if you want to come up You could ask Anybody's got one question that they want to ask Okay, this person here Hi, when you think about a Musical instrument one of the things about an instrument is predictability I mean, I know if I do this thing I will get this response each and every time and that would enable me to perform and have the same thing happen Several times most of the things you've shown me seem to be much more like I don't know what's going to happen I mean, could this be an instrument that someone could perform and consistently produce a repeated outcome Okay, so Yeah, the instruments we showed there was an element of unpredictability and they're actually designed some of them on the Principle of I am I don't know what this thing is, but I want to be able to find out very quickly what it is Because they're unpredictable in the first in this discovery phase doesn't mean that They're incontrollable. It still means they can learn how to use these instruments. So for example the vangelicizer with a shaker and a Few days after we designed that we realized oh, we can do this twisty thing. It sounds like a pizzicato string orchestra basically and This and that is completely repeatable and the same applies to lots of the other instruments as well So I don't know if the answer is a question but basically If you design it carefully, it will be controlled and repeatable. It may be strange, but it's repeatable and you can become good at it So we're gonna have these two instruments around if you want to play with them. Otherwise, thank you very much