 Willkommen, Julian Vogels, und ein warmer Applaus. Danke. Sorry, I made a terrible mistake. 8011, der Talk wird nicht auf 8012, sondern auf der 8011 zu hören sein. Please, have fun. Thanks for the introduction. Yeah, I'm really happy to be very excited actually, maybe a little bit too excited, so please bear with me. Today, I talk about the evolution and design of digital musical instruments. The talk will be not so technical, but the purpose is actually more to inspire makers to pursue this field and join me in my passions for these instruments, because they are amazing. So, just briefly about me. I'm Julian, I'm from Southern Germany. I studied in, what you just said, in Quebec Music Technology, and right now I'm running a little music tech startup company in Hong Kong as the CTO, and I'm a drummer. So, first of all, where do DMIs digital musical instruments come from? They actually have a really long history, despite what you might think. So, acoustic musical instruments like the piano, the violin, they're all like, their century is old and they were refined and refined and they're really very good instruments. But at the verge of the 20th century, people discovered new technologies and they wanted to utilize this in the context of music. So, already in 1897, there was the Telharmonium built, which was kind of like a huge organ kind of thing, and the idea was, there was no radio, so they wanted to distribute the sound to the living rooms of the people, so it was like a huge cable thing, it weighed like 200 tons, it's amazing. And that was the first one that someone spent his whole fortune on. The first, what we call open-air controller, because you don't touch anything, is built by Leon Teremin, or Lefterman, his real name. Excuse me, I think there's no sound coming out of my laptop. So, this is how a Teremin sounds. You might know it from the X-Files, for example, they were used, was it used. In 1928, M. Marteneau built the Ond Marteneau, which had a very unique control, you see there's a ring, which was on a slide, so you could have super nice tremolo effects and a very slight control of dynamics. So, this sounds like this, and Radiohead actually also rebuilt one recently and used it in a lot of songs, because it's really an amazing instrument. So all those early music instruments, they enjoyed a lot of approval by society just because it was a very futurist time. The same also is valid for the Tritonium, which is a predecessor of the synthesizer and had a resisting wire where you play. It sounds like this. Then later the Hammond organ was invented. It doesn't really classify as an electronic instrument, because it's actually an electromechanical, but it is such an important invention that I just wanted to show it, because it was very beneficial to the whole field. And later the electronic sac butt was invented, which was the first instrument, which was like a synthesizer, was voltage controlled and had extreme precise control, or accurate control of timbral properties of the sound, a sound like this. Then a new age commensis, which is computer music. In 57, Max Matthews at Bell Labs used the telephone network and the facilities that they had to actually build the first computer program for sound synthesis. At the same time also the Mark II from RCA was published or invented. So it sounds like this, it's one of the first program synthesizers. Then a very popular MOOC, invented the MOOC and the mini MOOC, which until today super famous for their fat sound and were very like a huge commercial success. Then another new age, everything became digitalized. So the Fairlight was one of the first digital synthesizers and the MIDI was invented or specified as a specification, so you could actually now connect synthesizers to another to use, actually this is very important because now the interface was separated from the sound source. Now you could just send control signals, which is very important what we see in a minute. Also 1980 was like the first popular, let's say unconventional controllers that get away from the whole keyboard thing. At the same time a lot of research institutions were founded like STIME, IEM, IRKAM and so on. The CMI Fairlight sounds like this. Still in the 80s, of course as you probably all know, this sound is the popularization of synthesizers which were so commonly used until today in popular music. And one of the cornerstones of that was the DX7 from Yamaha, which was the first one that used FM Digital Synthesis. And also in the mid 80s Miller Poquette invented Max. Max is until today used as somehow like the most important or one of the most important computer music softwares for composing, but also sound synthesis and mappings, sort of like a visual programming IDE. And out of this also pure data was coming out of this same invention. And then controllers get more and more unconventional. So for example Max Matthews, the same guy who invented computer music well, he developed the radio baton, then there were controllers with like more advanced sensors and for example there's also the lightning which used infrared and sounds very different, like now it's more like a digitalized analog, a conventional instrument, but very different playing techniques. And in 2002, the KNIME Conference was the first KNIME Conference which is until today the biggest conference for new musical interfaces and digital music instruments. And between 2002 and 2008 there were 266 music instruments presented and I don't have data for 2008 until today, but there you can see how many new inventions there are and how popular this field became. So yeah, I think it will be even like we have an increasing number of new inventions and great new instruments, because we have more accessible sensors and materials and also I think the community of musicians which tend to be a little conservative and not very keen on trying new instruments, they kind of open up a little bit. You see more and more like new instruments being played in professional musicians. Yeah, but at the same time, so I reviewed a lot of patterns in this field and unfortunately there are so many inventions but they're not really known, they're not really commercial or even just normal like public success. And that's also why I'm giving my talk, I want to inspire more people to build them in a way that they don't like stay in a garage. So now to the design part. A DMI is composed of three parts. So you got the interface where the musicians interacts with the instrument, but this distinction is very important that you have the interface that is separated from the sound source. So the sound source is normally a computer and the mapping maps this input parameters to the output parameters, so synthesis parameters. So that's pretty amazing because now all of the DMI's are flexible, you can customize them. You have completely, it's completely open, you can do whatever sound you want with them and you can tweak the controls beyond what is just a physical object that is like, well you have to rebuild a new one, you can just like it's so fast to get to a nice musical expression. So in my personal research and work, I focus more on the gesture control side, so less on the synthesis side. Because I found that controllers are really very interesting, we use them all the time. For example, you use a keyboard to type. It's specifically designed for this purpose. You use a gamepad to game, but sometimes you even use a keyboard for gaming or a gamepad because you have to write something on your console. So they see that they have to be really for one purpose designed for this thing because that's for every instrument is the same. And there are different types of them. So I found this very interesting to classify them like this. So you have augmented musical instruments, you have instrument-like DMI's, instrument-inspired DMI's and alternate controllers. For example here, augmented musical instruments are real instruments where you stick a bunch of senses to them. For example, touch buttons or FSR's or whatever you can imagine, he is also like in Neofilm for us at the Trombone. It's like a distance measurement with ultrasound. And all those things are then mapped to synthesis that runs like in parallel to the actual instrument output. Then a more commercial type of instruments are instrument-like controllers because they kind of try to emulate the musician's gesture of a non-instrument perfectly. So imagine, your trumpet here, you get this instrument and you can immediately use it because it has the exact same gestures but the sound can be different. Then there are instrument-inspired controllers where it's a little further from the instrument-like controllers but you can still use some of your gestures that we're used to as a musician but it makes more out of them. For example, here's the pretty new instrument, the Seaboard Grand, the one with the wobbly texture there. It's like a piano but you can press down and it has a subtle control of vibrato and you can slide pitch. Yeah. There's also the Eigenharp which is on the top right and the instrument one which was a very new instrument and an alternate controller has nothing to do actually with any music instrument that was there before. For example, there's the audio cubes which are actually cubes that know about each other's position and rotation so you can actually imagine that the conventional music instrument is more like for sequential control of things more like it's called an ambient instrument. Then, for example, Harry Cook here, he's a famous instrument maker so he used everyday objects like a mug or a shoe because he said people can relate to that so immediately that they just pick it up and they know how to hold it and if I place the sensors at the right spot they know how to use it way faster than if it's just an arbitrary object. There's also the Alpha Sphere which is also a very interesting shape and light. Also, for example, a graphic tablet I like this as an example because it has pretty accurate sensors and it's also an everyday object that you know how to use a pen and you're very skilled already at it so why not use it as an instrument? And this is also part of the alternate controllers collaborative controllers. At the right you see the two-car where two people play at the same time and the react table on the left from C.R.G. Jörger where it's a tangible interface so you interact with the digital digital information on the physical space it's all projected from the bottom and it reads whatever objects you place on them and you can rotate them and change the distances between them and make music like this. Then there are body controllers which are very popular at the moment too. You want to use your body gestures whatever gesture you want you want to just acquire all data about your hand movement or body movement so for example there's Michel Weiswitz it's called the hands it's a very famous one and Imogen Heave she developed the gloves in the middle on the right there's Onyx Ashanti with beat jazz and on the bottom is Marco Donorama who uses biosignals from his arms directly to make music and also open-air controllers open-air controllers you don't have any well any feedback anything it's just like your hands you don't have any sensors there's always computer vision for this and for this in the future we can all imagine where the journey goes to augmented reality and totally immersive systems so to the interface part how do you design an interface how do you start doing this so for me what I like to do is I think about gesture about how does a musician move, what movements could you could you do what can you imagine would correspond well to dynamics so to volume and to pitch would it be discreet, would it be a sliding pitch and what corresponds to timbre of the music so timbre I I'm not sure if everyone knows the term it's about if the sound is up or more mellow then to to get an idea of what you could do the first thing you do is you observe musicians how they play so then you know because that's where they're coming from and there are a lot of interesting gestures in all musical instruments for me for example I did a study where I studied the gestures of harmonica players and they have a lot of interesting gestures like the hand cupping gesture or just how they move the instrument and they have different techniques with the fingers and also like angles and so I did a whole emotion capture study which is amazing to get lots of data and numerical data they are optical ones, they are exoskeleton ones and also magnetic field emotion capture the problem is that they are super expensive you only find them in like research facilities so what would be better for the DIY maker is you use videos you just look at musicians perform and use open source software my favorite is Elan and you just annotate the gestures that they make like you say okay it's like huge upward movement all these things you annotate them and then you also listen to the sound or you can even analyze the sound as a a different sound properties for example the sonic visualizer I didn't put it on there but it's a great software for analyzing sound and then you compare what property of the sound changed when a musician did this gesture for example for harmonica it was like many people leaned backwards when they were more expressive and more high pitched and those things you can use them to those gestures annotation software is pretty simple you have different videos for example on what I labeled the three it's just some data some spectral centroid I think and then you just say okay what happens and then you look at the graph and you know what sound property is affected and then you can use that in your considerations what gesture you want to use for the acquisition of gesture I could elaborate on that but I wanted to keep it short use all kinds of different sensors they have to be suitable to what you want to acquire for example FSRs for force sliders and LVDTs for displacement infrared and ultrasound you can use them as displacement or general if there's a hand here or there or whatever so be creative and if you want to look more into sensors and stuff you can also use sensor fusion to get even more data out of two different like if you have two sensors and you use sensor fusion you can get more information out of it than you could use you can get with either one of them or both together now the second part is the mapping part which is kind of like distinguishing a bad instrument from a good instrument because you can have very easy mappings like one to one like you know you do this and it turns out a lot of good music instruments are not that simple because of course you understand it really quickly but it gets boring and then your instrument has some kind of like a toy like character which you want to avoid you want to make it a little bit more complex because you want people to be able to master your instrument and it shouldn't be easy so a many to many mapping you use multiple input variables and you map them to multiple synthesis parameters so a musician doesn't like you can try out some stuff but you wouldn't really get what's really going on because you have to change all of them at the same time to get to a result that inspires a more like holistic kind of use of the instrument so the musician more or less or just learns gestures instead of learns parameters which is like a totally different paradigm of using instruments interesting mapping ideas also you can use machine learning to classify a gesture for example you could record a gesture like this with any sensor you touch or just using the instrument you record a gesture and then you use a classification algorithm to say you have three or four gestures and you say which one was the closest and you can use it based on that you can also use indirect data let's say well you have a slider and you can of course read where it is but also you should think about can I use the velocity of someone sliding the slider and map it to another synthesis parameter then also you can user define mappings although you should be careful it's more flexible but then you lose kind of the face of your instrument because it was designed to sound like this and if you make it totally random then it's random so you have a lot of cool open source mapping tools like libmapper where you can even map over a whole network using OSC open sound control and also other interesting toolboxes from IRKAM this is where you can do the whole gesture recording and classification then the sound synthesis as I said I'm not talking so much about this but there's several different techniques additive, subjective FM, you have cool physical modeling stuff where you actually emulate reload instrument just digitally and you have a lot of great open source software to build your own and feedback so if you're playing a violin then you bow it and the whole body vibrates many musicians don't even recognize it but it's very important for them when they're learning an instrument that they get the vibrations on their body if you would take that away it would be way harder for them to learn the instrument than the state of the instrument the problem is now with digital music instruments this doesn't occur naturally so you would have to add actuators to the instrument to give some kind of primary feedback to the performer you can do that with vibration motors servers, whatever can convey information to haptics visual or even if there's a special sound that occurs so I want to inspire the home making community to do to build digital music instruments I think it's a very exciting field it's complex enough so you can spend your whole life in this field and you have a lot of tools at your hand everything is getting cheaper the sensors are now really like those instruments they saw from the 80s they were super expensive now this is really doable then systems like Raspberry Pi, BeagleBone Black et cetera where you can do no synthesis on this space on a credit card space which is amazing for most people that built those other instruments 30 years ago so the challenges though are there so for example at the KNIME conference there are so many instruments but a lot of them sound kind of the same because there is the interface, they map it somehow and then it's just noise and nobody can really relate to this other than people that really like noise music and often in my eyes it's also an excuse like oh yeah I do noise music so I don't have to care about the sound so don't make these mistakes and also don't be the only one who masters the instrument that's something that I did in the past and I tried to be good at it but actually what I should have done is give it out, make two or three of them and give them to musicians and have their feedback immediately and that's also why you have to make the setup easy for example an audience doesn't like it if everything is cabled and you actually want your Mac than on your instrument so make it, that's also like the coolest thing is to have an embedded system a box, like an instrument what people are used to and that you that a musician can talk to your audience that your audience sees like big gestures and can relate to what the musician is doing if they don't even see what you're doing on the keyboard then they won't find the performance interesting it's not only about the music it's also about the visuals a nice quote that I wanted to bring is this so this is Besson Reiter talking about DMI's what it looks like and it has a low entry fee everyone can pick it up and immediately make sound without knowing much about it but it has no ceiling on virtuosity it means that anyone who wants to master it has to spend 10.000 hours to be perfect at it just like with the violin or the piano here some of my own creations on the left top is what I call Philomis it's a rubber chord on a sound body and then a joystick on the top so that you can plug the string and then modulate the sound in two dimensions or even circular one the lower box is a device where you press record and then use it as a tape a tape player and you can play your recording back and forth and then you can destroy it with effects and you destroy it every loop so expect exactly what you shouldn't do so in the middle this little thing is my first commercial object it's not really an instrument it's a music accessory it's a feedback device so expect exactly for haptic feedback you can use it as a metronome etc reproduzend is now in china too so just a short demonstration of one of those freaky the virtual string so it was very experimental and here is another one that is this one an electronica monica okay I think I just concluded like this thank you wow hey that's awesome what do you do basically we are we don't have any time anymore we are three minutes late but we are it's a last talk before the lunch break so we have some time for Q&A if there are some questions please there are the two microphones ask your questions may you play again may you play again your music I think it's better than answering questions I thought my talk I knew my talk would last a little bit more than a half hour so I didn't want to do this as well I can set it up in 10 minutes or you can come to the white village and then really try it out yourself later if you want so please did you ever try the brain waves as a source of sound yes actually that's right I left it out but there was a famous musician and developer that used this brain wave to make music with huge helmet although I think it was very difficult because it's very noisy to actually think about music and then have it be music I think we're still a little bit in this stage but there are people working on this I think even at the sea base as I heard I was thinking about something more earthly like moving your arm and the brain lighting up in some zones did you try that well if you move your arm you can also just measure the EMG signal that comes directly from your arm and Ghee she asked Method Actors to have the arm pierced with a syringe in the arm which is very painful but she got the best signal out of the arm and then she asked them to act emotions and she decided those emotions into music that was made by robots so you can do a lot of stuff with EMG signals and thanks someone else just a very meta question what instruments do you have with you in the village what can you try out there the only one that is re-set up is this one the other one if someone is interested this is also the whole instruction on how to build this on my website so we can actually make it work it's not working right now but we can make it work thanks okay, someone else good thank you Julian for your very interesting talk