 Previously when you wanted to compose music or do music, you would have to go through a digital audio workstation You have to use a DAW and DAWs have drop-down menus. What is nice about VR is that it opens up this kind of new realm of user interactions What I've been doing with this most recent project, which is the interactive beat drops simulator It's primarily built around two gestures, which is one the up gesture which Starts intensity rising and then if you want it to keep rising you have to gesture up again And then you do it again and again and again and that's the intensity builds up In addition to this like the music builds up But also ideally the visuals around you start to get a little bit more intense the world starts vibrating colors start shifting a little bit Choreography happens in the world around you and then there's a downward gesture which triggers the beat drop And that's supposed to be super satisfying where I chose When that would happen. I did it when I felt it was right, you know I did With the gesture and the visuals Take take the inputs that you put and like react to that Boom what's up everyone welcome to simulation. I'm your host Alan Sokian We are at the Massachusetts Institute of Technology in Cambridge in Massachusetts We are now going to be talking about interactive audio visual creation. We have Gregory Osborne joining us on the show. Hello. Hello Thank you so much for coming on Very excited. This is a topic that we really have yet to dig deep into very pumped to have Greg joining us He's a recent graduate of Berkeley College of Music in Boston focused on developing the next generation of interactive audio visual art Using video game music implementation Taking advantage of virtual reality and new types of user interfaces and you can find Greg's links below Check him out Greg. Let's start things off with this perspective that we find ourselves as stewards of earth What is your current take on the state of our world? I? find it the Recently especially with with the kind of advancements of technology We've had especially in the last half century if not the last like century itself We've found that technology has made it way more accessible for For people who have no formal training in art or science to be honest In and very many fields to start participating in those fields There's a lower barrier of entry because technology has made it way easier for for people to One learn what it is that's important for the performance of that job whether it's art or science or whatever But also makes it easier for them to share what they have learned so that they can Like really expand the pool of people who are interested in that subject So for example if I have found this really cool cheap easy Form of music to create it becomes really really easy for me to spread that to my friends so that they can also Make that music and you build up your kind of you know culture around that where everyone is sharing It's really easy to share it's really fast you can do with a click of a button check out this track I made check out this art that I put up on tumblr or whatever Has become really shockingly fast To to share your advancements in a field So what I found is that? This does this hasn't necessarily diminished like the the value of having really deep knowledge of a thing Because you still can make like incredibly nuanced and complicated pieces of art because you have studied this for 20 years But what it's done is it's opened up this kind of avenue where that's not the only reason to to do art you can also do it just for your friends because you got home from work and You have Not been given the amount of like expression self-expression that you desired in your day I I'm sure that's relatable to a few people and they can go home and they can express themselves With you know their hobbies and not only it doesn't have to be alone in your basement that you do this You can also share it with all of your other friends who also probably possibly work similar You know jobs that don't fully like give them the potential to express themselves And that's not saying there's any shame in that but what we've seen is technology has made it way easier to continue to express yourself Despite you know these kind of primary activities that you find yourself doing yes The people that push the edge of knowledge and build the new tools and technologies that then Democratize people's creative ability to contribute is such a great synthesis on the state of the world. I love that Let's jump into the journey. So you were raised in DC Yes, and then you went to the Berkeley College of Music here in Boston I want to know tell me about the journey and what sparked your interest in music. Well Musically, I've been playing classes. I was classically trained in piano since the age of five It was kind of thing my parents were like, do you want to do this? And I kind of said yes to most of their suggestions at the time I was five So I did I found that I really enjoyed it Just playing piano Interestingly, I don't really play that much piano anymore. I kind of got sidetracked into composition around the ninth grade because because of my Like piano playing skills They put me into the honors program at the Levine School of Music where I got access to Most importantly this one workshop Which was everybody writes a song as a group and I remember like writing the song as a group with the lyrics and everything And I enjoyed that song so much that when they failed to send that song to us after the event was over I asked them multiple times and I never got it. I went back and I spent an entire weekend rewriting it for myself It was it was the song called my GPS lies It was basically comparing your GPS to a girlfriend like a bag like a bag There was like a poor dysfunctional relationship and so I enjoyed that idea so much I was like this I want to hear it again, but they don't send us the file of us singing it So I ended up remaking it and so that kind of started me on the process of composition As opposed to just playing music and my father Kind of was a DJ in his in his spare time. It's like as a hobby and so he used Ableton for that Ableton is this program that Can be used for DJing and it requires a bit of setup and you do it and that was kind of how he performed it But it's also primarily a music making tool So when he heard that I had an interesting composition He was like you should try this program and so I got that program and that's the program I've mostly been using for the past five or so years to create music till today till today Till today Ableton Ableton. It's a it's a digital audio workstation You can call the doc doc for short and you might hear me say that But it's it it's mostly used for electronic composition So that's kind of the music that I make right now is mostly electronic though That has changed since I came to Berkeley and is this still with when you were using Ableton last five years Is this really still the clicking and dragging into timeline sequences? Well, one of the one of the nice things about Ableton actually is it kind of revolutionized a new type of composition Which was there? Not arrangement view, which is what you were mentioned, but this is called an arrangement views There's the arrangement view, which is where you click and it's linear linear Yes, but Ableton also has the session view Which is where you amass all your clips in like this kind of spreadsheet essentially and you can program behavior into its playback So that you can trigger clips non-linearly So you can change your arrangement on the fly based on the music that you know It becomes a live performance and so that's what a lot of people will use especially now for live performances for electronic music because you're not necessarily up there with a guitar shredding but what you can do is you can show off your prowess with all these kind of inputs and like hardware and Triggering the right dials and buttons at the right time to perform the piece of music that you wrote electronically So that's kind of one of the things Ableton's excels at and that's also partly That's where you can add a sequence of a couple things and then have that become a program that repeats Yes, and you can just loop it you can have a bunch of clips trigger at the same time You can design Instruments with like macros that use specific dials that you adjust to change parameters live And so it created basically a new Type of performance for electronic music the digital audio workstation. It's a digital. It's a digital audio workstation It doesn't most digital audio workstations don't have that feature that digital audio workstation mostly just for creation of music Yes, it's just the the program that you use for that and then you can actually get your your ability to actually do To perform it live. What is the what is the? Scratch is this what is this? Well, this is like a table Is this is a scratching? Table this is that's not what Ableton is mostly. Yes, but what is what is that? This is scratching a turntable So there is an additional Potentially a physical unit that one can use to have a scratching of turntables as well Yeah, do like a physical kind of yeah, and you can record performances of that with your that would normally be separate gear though I'm sure you can find a plug-in that helps you do that just in the program and did you end up picking electronic music? Like around like when you were starting doing this and then you kept doing it through Berkeley Yeah, it's sort of I found electronic music a little bit earlier than I've had the access and means to produce it myself But it's kind of just like my musical tastes evolve kind of slowly based on YouTube videos that I watched and and the the music that they used in like Gaming montages was often electronic So the particular artists that I listened to ended up like fueling my my my love of electronic music But now I've you know gotten incredibly nuanced and deep as most musicians at Berkeley will with their musical tastes You know that people make really deep dives into specific things and not different. Yes And now this is quick you guys you teaching me about CWP Which means contemporary writing and production and this is for all different types of things film games Etc. So teaches about what this major even is Yeah, so this is at Berkeley College of Music in Boston It's one of many majors that you can take here contemporary writing and production tends to be the kind of Generalist composer where it's not necessarily you as a like specific artists producing and Producing music under your own name But it's for all the rest of the music that surrounds us in daily life be it in advertisements In YouTube videos for example You can arrange Sometimes when somebody writes up a cool piece of music, but they need it like arranged in for different bands like one for a band one For an orchestra one for five flutes in a fiddle or something They generally need they hire somebody to just arrange that for them So that's a skill that you learn in this major It's kind of just like general Writing of music as well as recording it you kind of it's it tries to be a mix of a lot of different things And do you need to know sheet music? Yes to do this So you need to know how to position musical notes on On a musical bar. Yeah, and there's programs that help you do that And how long they go for yeah, and what what is it an accent at articulations like the different types of notes that you can That you types of notes that you can interesting. Yeah, and and there's pro I actually have a friend of mine who specializes in Just Getting music onto the page in a way that's clear and legible Wow So people will have like, you know orchestral scores or whatever like music that they have written in their head and they have like You know this kind of mock-up and and he will go just into the program and make sure the scores clean And all the parts are very legible and very clear about what they do because that actually it's a process It takes a bit of time. I have he's actually doing one right now for me But and that's just what he does is his specialization. That's kind of a beautiful thing about Berkeley is that they're all Geniuses, but like at very specific things. Yeah, all the things are different Which is kind of like score cleaner or like music sheet clean up just just because it's tedious It's like it's a hard thing and there is a certain amount of art to it to make it as clear and concise to the player Who's reading the music so that they can accurately reflect the wishes of the composer. That's not that's not easy It's not necessarily always like just oh it'll happen It does take a bit of work and effort and a score can be used for a film or a video game or a commercial or is like so many Yeah, music is everywhere like people write music for lots of things. There's like entire Libraries just where people can go is like I need music for this kind of mood And you'll just go to the library and and pay for that music really fast and just use it Like that's another for example option for for CLEP people is to just write music for libraries that gets used in Lord knows what place because there's so many places that use music, but you know Somebody has to make it first and what would a CWP kind of Process maybe you look like for give us like what you would learn and maybe a general or two like that are important And then you how you focus So there's a lot of different paths that you can take with with this major They teach you how to arrange for specific groups of instruments like how do you even write for horns? How do you write for strings and for flutes and clarinets and the woodwind family and how do you effectively use percussion? And you have to like learn about the roles of this Of all these instruments you would learn like just basic techniques that have been used in in recent like contemporary music So like different harmonies how you stack up harmonies together How horns are used effectively in one genre versus another genre how they are typically used like genre tropes Of course you can get CWP offers a lot of deeper dives into specific genres that you might be interested in Just to specialize in a specific type of music writing I specialize more in electronic music But especially recently like hybridizing that with orchestral written music with live instruments and and kind of blending that together And that's that's my specific kind of focus that I did within the CWP major Of course my primary focus at Berkeley was was video game music and and composition for that kind of Medium and so that's that's really what I've been mostly focusing on of course Berkeley If you take the contemporary writing and production major you will be good at writing music like at minimum So they they teach you that pretty well electric and orchestral music for video game Yeah, it doesn't have to be for video games necessarily But that is the kind of that's where I tend to use my compositions That's the kind of field I'm trying to focus that your focus was you also made it Crazy interesting that you have to learn how to write for horns and for women. They have a stack for harmonies Interesting. Yeah blending different orchestral textures as it's whole entire art Like wow one of my teachers is just like just listen to the way he orchestrated this listen to the choices He made and like where the in octaves They are playing or who is on top of who and how they change the character There's a a lot of nuance to everything about music. It's not it's it's not easy Yeah, necessarily and to get great And there's just so much nuance and I love how you know you then Explained to me earlier within video game music composition, which is where you're super passionate about there's this big There's a huge difference and this is hard to capture when we're just you know Engaging with content. We're not necessarily thinking about how difficult it was to create it in the new order But like just like at a restaurant, you don't think about all the operations in the background And they don't want you to that's also kind of similar to video game music You don't want to make it obvious What the systems are behind it, right? Oh? the most mostly most of the time the purpose is to just Have it just happen to perfectly score what it is you're doing like you want it to be so like So I want to be able to double-click in and investigate when you want Yeah, and that's kind of like that's the stuff that video game composers will do when they play a game It's all deliberately try to break the music playback just to see how it works But under normal gaming circumstances, you're trying to make it just so that it happens It happens magically and without anybody noticing in as gorgeous of a way as exactly you you made this really clear to me I was like holy crap. This is crazy. So like when you make the score for a film the all of the music is Linear and then when you are making the the music library for a video game There is what's the sound when the door is open? What's the sound when you enter into a shower, right? There's like This is really complicated. When do these things play? How how are these catalog cataloged and right? Did you create them in the first place? Well, how do the all the algorithms and math work as well? Teaches about this is sure. Well, there's um, there's a lot of different parameters often that you have to take account for one of the kind of most basic building blocks of the music part of video game music Implementation is the loop so you have music that will continue indefinitely as long as you need it to Because one of the key defining features of game audio is that you have you can't make them do a thing Most of the time although you can't you can't sometimes script things and there's some games like art games for example that like Have specific timed moments That but normally want them to feel natural like it feels like a narrative progression that that that made sense But often the time you'll you'll you'll be defining how loops transition into each other and with what music And how fast how reactive so for example In some cases where you have a bunch of pieces of music that Are all the same tempo you can kind of Define the transition so that they always happen on the beat so it always makes sense And you can also add on top of that another piece of music that plays to kind of mask that transition So one very common trope I hit to ruin games for you now is a cymbal swell which is And on the ch is when the music changes And and stuff like that. That's generally called like a transition segment or or a stinger A stinger is more of a thing that happens at any point Just a piece of music that plays when you either like open a box like in zelda add Add And so that's like a stinger that just plays at any time there's a piece of music, but there's also That's an algorithm So it's actually attached to the treasure box. Yeah, so when you open that that when So you have to define Either if it can be like Opening a box it could be moving from location to other locations enter into another scene Yeah to change the music. Yeah, and you don't know, you don't always know exactly where you'll exit from the last piece of music unless you specifically define that. So there has to be a transition to make it kind of smooth? Yeah, it doesn't have to be smooth necessarily. There's perfectly valid places where it just fades out and then fades back in again. That is a perfectly natural, but the artistic decision of where that's appropriate versus where you really want to keep the pace going versus where you want them to notice often the distance, a piece of music that is directly spatialized to that location. That's a place I want to go to. That often will be tied to the specific piece of music that you're playing. It might not even be heard ever, but if you get close to that location, you'll hear this new layer of music on top of the music that's already playing and that's defined by question. Okay, so when we're moving more towards the surround sound and then to the ambisonics, these 360 degree places, we're saying that there's potentially going to be, let's say, a building that has some sort of music that's coming from and we're hearing it like in our background and then the closer we get to it. So it's always playing. That building in the video game music is always playing, but it's spatially in that location. So that way I can, wow, that's some complicated stuff to know how loud to play the music based on how close the character is. That's part of the mixing process. So a lot of game audio implementation is playing the game a lot to make sure that that layer of music isn't drowning out all the other pieces of music that might also be playing. Maybe as you get closer to that building you encounter enemies and suddenly battle music plays or something or gunshots happen and you have to make decisions as to what you want to be the focus. Like do you want the building sound to be the focus or do you want the bullets whizzing past your head to be the focus or all those kind of decisions are part of what makes up a general kind of game mix for the non-literary music and often you'll have competing sources of music that's like there's battle music but there's also this exploratory music that has this radio playing or whatever and there's the kind of difference between music that kind of just plays underscore and music that's specifically sourced in the game that you know where it's coming from. The same thing for kind of like sound effects like there's some sound effects that you need to know the position for like where enemies are versus getting hurt that's kind of an internal thing like you're getting hurt so you don't have to worry about where that's coming from you know where it's coming from that just happens in 2D and so you can just play that directly into your ears and you don't have to route that into whatever spatializer you're using for for all the positioned sound effects. It seems crazy thinking about every step that you take makes a sound in a game. Footsteps are like a meme because every person who does video game sound effects has to make footsteps it's like a part of almost every game. So either you have your library of footsteps that you just use on everyone but what if people will notice so then you have to change it or find different footsteps. When you're walking in grass versus on dirt. Yeah there's a lot of different materials sometimes games have like metal and then glass that you walk on or have different maybe you wade through a river and when you wade through river you have to make sure that the footsteps start getting wet and then fade out completely as you start swimming and then as you get to the end of the thing so we can't always just be playing you have to define those states. This is crazy so give us an idea of how many audio designers work on and engineers work on just a single complex video game like a sandbox potentially. Okay because there's different layers of video games right there's like indie games which are generally smaller which sometimes work with as little as like one audio person because you know they don't have the budget for a composer and a sound designer but like AAA games will have like flocks like you know many sound designers working on the different footsteps you know especially if you're on a tight schedule you know you you have to organize a structure whereas where the creative vision is being explained to the one guy at the top and he's like all right I don't like that sound effect because it's too magical and we're too much in a sci-fi universe so try and change that they have to make those creative decisions and they kind of have their teams working on either footsteps for characters sometimes one one you'll focus on one character and all the sounds associated with them things things like that so yes there's there's a lot of people working on sound that's an important part of the game it it makes a massive impact on your enjoyment of it especially if it's poorly implemented then you're like ah I hear that sound again and again and again and it makes me tired and not want to play the game and then there's a composer and a sound designer often it doesn't that doesn't there's nowhere is it written that that has to be the case but it tends to divide up on bigger projects better if you have one person working on music and one person working on like making all the individual sounds and they'll probably work together very closely to make sure that the general vibe is similar and and and mesh as well often they have to choose a key to make sound effects in to to make sure that harmonically it doesn't kind of clash too much especially for the more pitched sound effects that you might have in a game so they work collaborate together but there's a lot of work that has to be done so they do separate it out often into like music writers and then there's their team of editors that goes through and and through all the either recordings of an orchestra that they did to make the music or the session file that they if it's electronic music for example they'll get the actual file and then they'll separate it into different layers that can play at different times because layering is a very common implementation technique for video games where it's like you want to build intensity we'll just add the music in and then we add the third level of music in and then you add the fourth one oh my god you're succeeding so much but but somebody has to go in and like divide those and often you know a composer especially for larger games a composer will just write the music record the music um have a like have a mixer mix it and then you have like a team of editors who go through it and make decisions as to what goes on what layer at what time um and how to implement it um yeah and the way that you're teaching us about um the the video game music composition is so similar to the video game that we're in right now because there's cars on Memorial Drive back there that we can hear there's people that walk in and out of the doors in the hallway right now that slam in and out when you walk you hear the footsteps there's just this is such an interesting sort of world of all of yeah I'm glad you give us this perspective because we don't typically just you know think about all of the different noises in the kitchen all the different noises in the office yeah but it's you know you can take some time to think about that like nothing is stopping you from from just like sitting in a place and being like what do I hear right now and that's often an exercise that they ask us to do I just love that exercise because because often in video games the things that really sells the world and really sells immersion is like a bunch of tiny little details about the room that when you notice them you know it's like oh wow wow especially in sound design for example like when you build a like a sound like a sword like hitting a person right you have to try and put in as much detail into that sound as possible so it's not only the swoosh of the sword and the collision of the sword on the person but sometimes the enemy that you're hitting is going to be wearing armor and so you have an armor sound or it makes a different sound when they're not wearing armor because you broke it or you also want to have a different sound um for it if it's a bleeding you know sword it causes bleeding or poison or whatever you can add that kind of sound effect on top of that based on the type of weapon you're using um if it hits the center of the blade versus the tip of the blade yeah and that will be more important as we go into VR because often in in video games you don't necessarily have that direct control over an object in the sound that it creates but as we go into more physics-based kind of virtual reality games um how sound is created and propagates becomes even more important so like where the collision happens on the sword becomes important you know so that you can when you cling the cling happens from there if you like if the sword like collision is right here you want it to happen here and not on the tip of the sword or at the handle you need to make sure that you position the sounds where they would be in real life and um a lot of that is it gets more complicated as you get into virtual reality because the haptics from the vibration I mean this is as you embody the space you notice the failures of sound implementation more you know it's different when you're looking at a 2d screen and you're like oh there's an enemy over there you know in that general direction and you move your mouths to look in that direction and it pans so that it's in front of you um but but when you're in VR it matters not just what direction it's coming from it matters the elevation it's coming from it matters what objects are around it that how the reverb works how how the sound kind of echoes and reflects off the room so for example this room um you can hear a bit of reverb especially when we talk a little louder and that's because we've got glass here we've got glass on the other side and it kind of just balances back and forth for a bit and it makes it sound a little bit more live than it would be in in a room that was like all carpet and so especially as you go into VR which is why people put up the sound proof yeah for example prevent the reverb from yeah but but as you go into a game the kind of environment that you're in if that footsteps sounds like it's in a completely dry room you know but you're in a cavern or you're in like a church you know you want that footsteps to have the same amount of reflections as it would in that real space and it matters even more in VR than it does there's a couple of really you have to measure you have to you literally have to maybe take like a boom mic to a church and then maybe record it that way versus like in a field do you do that that's called convolution reverb where you like fire a gun or more commonly now you have like a kind of sine wave to measure how all the frequencies reflect off the space that's convolutionary room that's for like really accurate representations of one position in a space what's interesting though is that when you make that measurement and let's say you're in the center of the church the reverb is going to be different when you walk over to the wall and then when you walk over to the altar you guys are crazy how do you do this so one way is to like do the calculations using ray casting which is a thing where you like fire a ray in all directions often one of the games that does this really well is The Last of Us they have their there when somebody talks and walks behind a wall then the reverb changes based on where you hear them another game that does this incredibly well is Rainbow Six Siege is this it's a shooter team-based shooter and then but it's it because the environment is destructible around you it becomes really important that the sound propagates to that environment super realistically otherwise if you're standing next to a wall like if I'm if there's a wall right here and there's an enemy on the other side of it and there's a door right here when the enemy walks by I should not be able to shoot him through the wall based on sound alone I need to hear that sound through the door over here so that's where I hear so I can tell that there's somebody in the hallway but I can't necessarily tell that they're right next to me across from this wall and meanwhile someone on this side of the building is breaking down yeah and so all and especially if it's like on the third floor a massive explosion happens you don't hear the explosion you hear a bunch of reverb from that explosion and you can it positions itself from the doorway from which you would hear that okay we had we weren't we spent so much time talking about this great incredibly complicated it's so beautiful to talk about it and you'll watch and I think you'll actually end up teaching us more about it as we get a little deeper into some of the hardware and software side of things which are let's dive into this so you're very interested in making albums of interactive music so yeah and and I want we'll have you explain but you know we're talking about how to make this video game music composition this requires hardware it's required software it's things like unity unreal also all the VR hardware AR hardware is this now out there but this this albums of interactive music thing this kind of loops us all the way back to the democratization right so teach us about this and how you're building this well stick for example unity which is free to download and you can make a game in unity for free and if it like achieves I think a certain amount of success they have they take a cut of that basically but if you want to learn how to make games you can just download unity and make one using the tools that they've got there and there's also a large amount of free assets on the asset store on unity that you can use to to mess around with and see what it would look like if you made a game like this or like that and and that kind of that kind of experience to be able to just like go out and use it of course it's good for unity because if you want to make a big game you've developed all these skills and you're used to this particular engine in the way that it works so of course when you make a big game you're going to be like yeah I'm going to use the thing that I know how to use of course you can also take it as a challenge to try and do it in another platform like Unreal or Game Maker or there's a couple of tools out there that you can you can use to to make a game but the entries do you have enough space on your hard drive to download this program and then go mess around with that but cloud is now in effect or not yet the cloud computations I'm not sure I that's not really my I don't know much about that necessarily hopefully at some point it gets so democratized you don't even need a really powerful hardware locally you can just yeah it'll happen at some point I imagine um but in addition to that there's I there's sound implementation in unity that you can do and one of the things about unity is that if you know enough coding um you can make you can set up some crazy things and then people will sell packages that they made to do stuff that you that would be very difficult to to make just coding yourself but you can just go out and get a package that does that thing in unity for you but one thing that you can do is from an audio perspective right there's a couple of programs that are middleware audio middleware engines that essentially take the audio functionality out of unity and just put in a different program and then those tools are designed to be much more efficient like for from a user perspective it's like I want this thing to loop and then every time it loops I want to make sure it plays the reverb tail at the end while still starting at the beginning that's difficult in unity because your loops have to be perfect and you have to have the reverb tail from the end of the song at the beginning of the song so that you don't hear a click but if you're in wise it's really easy to set up you just put the beginning marker you put the end marker of what you wanted to loop in between and then all the sound that comes after that it just kind of continues playing as it goes to the beginning so it it makes it easier um this is wise there's also f-mod I think audio middleware interesting these integrators that work that's so cool that there's now markets for that yeah and also it's I want to see um something that makes it easy to be able to uh like you said to take it take what you have as your current audio um in unity out and then bring it in with the middleware that you use I just want it to be more frictionless too many times we hear about people that yes that like lose like yours worth of work because of software updates or whatever all different I mean it does take a bit of time I uh had some embarrassing an embarrassing long amount of time before I got wise in unity to work together because of one step that I missed in the process of just getting sound uh and that's kind of a thing I've gotten used to especially because I work in kind of newer VR spaces as I often just build in five hours of time at the beginning of every project to have nothing work and frantic googling to try and figure out why the project won't even load like that's just kind of like I've come to expect that but but it it it a lot of it is like YouTube tutorials online that you can just watch and and the companies will put up there and you just watch them and then you're like oh that's what I missed I'll just go and do that and then the sound works and then you kind of build up honestly YouTube tutorials are the most underrated source of learning in this world I agree so much yeah and mentors one-on-one yeah I I just for example I just sat with a friend of mine who for a class has to implement wise in unity together and in that process we ran into one of these bugs that it wouldn't generate a sound bank for Mac or we didn't know how to get it to do that so it would only work on Windows but not on the Mac that we were using at the time and so struggling through that process and just like being in your head against the wall just trying to figure out why it's not working that is very common but it it it doesn't happen again every time as in like once you figured that out you get you get further it's like a video game you get further and further and further as you unlock the skills you you have the keys to unlock that door and then you know the puzzle that you have to solve for that door and you know the quick shortcut um through the Dark Souls level that skips past the first boss um and you get better at at developing games and implementing audio and and whatever it is you kind of want to do and what's nice is that there's an often entire communities dedicated to helping you out which it just makes me feel so at home it's like ah yes like I had this exact question thank you for explaining it to me in 15 minutes it would have taken me literal hours to figure that out if you hadn't done that correct and you know yes yes and I want I want you to give us the you know this ties us into the fascinating intro um music systems that non-musicians can exercise creative control over this is the albums of interactive music right you gave some beautiful examples of being able to build up the beats yeah and then drop them yeah in the spatial environment that's uh I call it an interactive beat drop simulator the idea being that previously when you wanted to compose music or do music you would have to go through a digital audio workstation you have to use a DAW and DAWs have dropped down menus and DAWs have there have been like significant um advancements in trying to make it as easy as possible for non-musicians to do what they want with audio in these digital audio workstations a garage band has like a bunch of behind the scenes stuff that it does to your sound like adding reverb to it and creating buses for you when you create a new track and it really tries to make the process as simple as possible but ultimately um making music that you would like is difficult um it takes a bit of practice um and investments of time that not everybody necessarily has although I would encourage you to invest that time to try and give yourself some amount of self-expression but what is nice about VR is that it opens up this kind of new realm of user interactions where where you put your hands and how you move them can be used in a kind of more natural way I suppose um then like using a mouse and a keyboard to kind of click around it like music amazing music happens that way but you do kind of have to know to some extent what you're going for a lot of people who start out have like music in their head that they know they want to make but they don't know how to do it and I I still run into that problem right now um I beatbox music that I know I would not be able to make in the amount of time it takes for me to beat boxes like this is the sound I want to go to you know I want to do that I would like that would probably take me 30 minutes an hour and and this is me having like five years of music experience to just recreate that sound that I just made like it's there I can't just use that but it's and you can capture you doing it and then import it uh yeah well you can you can capture that what I've been doing with this most recent project which is the interactive beat drop simulator is while I'm trying to figure out the music system um I've been just beatboxing the music so when you when I'm playing it right now it's just me beatbox so the the way it works um is when you it's primarily built around two gestures which is one the up gesture which starts intensity rising and then if you want it to keep rising you have to gesture up again and you do it again and again and again and that the intensity builds up um in addition to this like the music builds up but also ideally the visuals around you start to get a little bit more intense the world starts vibrating colors start shifting a little bit um choreography happens um in the world around you um and then there's a downward gesture which triggers the beat drop and that's supposed to be super satisfying where I chose when that would happen I did it when I felt it was right you know I did with the gesture with the visuals and the visuals take take the inputs that you put and like react to that of course this is this is not composition right this is me creating an experience for you that you can play back you know and I compose the music so that it would sound a dramatic and I I made the system in wise so that it would you know it would time it to the beat so that no matter when you drop down it'll always happen in time and things like that but as I as this kind of field advances of using gestures to make musical flourishes when you go and then when you when you select specific items or if you point especially as we lose the controllers and move towards just knowing what your hands look like you'll be able to more and more easily create an experience that you just dance music into being and it can be to some extent you can have it be random um a lot of work will be put on defining what some gestures are teaching people what the teaching new users what gestures they can even do like I want you ideally to experience all of the cool stuff that I made but to teach you how to unlock those gestures there's a you know a certain amount of time that you would have to invest into learning how it works and whatever and that's mostly just art but I I hope at some point the kind of boundaries become such that you feel as if you are an integral part of this composition you know that you you chose these things when the beat drop is playing back you make some decisions to put effects on top of the digital signal processing or you you can smack objects that appear in front of you that temporarily play one like piece of music instead of the music you would be listening to otherwise to to give them tools so that they have as much control over what they listen to as possible but it still sounds good no matter what they do yeah that's a huge part of it yes and and then you're also moving to be to build an album so it's the beat droppers just one of many then I can do this immersive beat dropper share it with other people make it social there's so much and there's there's other ideas which is like just a gesture based remixer where no matter what music is playing when I do a gesture it'll either mute it or I'll go or I can like chop it up a little bit I could like make a gesture to repeat it over and over again and so for example I have I had a little bit of experience teaching kids in it was they got access to a studio as part of the boys and girls club there's a music clubhouse thing so they got access to a studio and what they wanted to do was not compose music as much as they wanted to rap over it you know they would go on youtube and they would download free they would just youtube to mp3 and then they would go into the studio and just rap over it and they would share with their own friends you know that's who they did it for they did it to show off and it's like to show the bars that they had but what if there was a really easy way for them to not only just like not there already is an easy way for them to rip music off of youtube but what if there was a way for them to exert creative control over that music what a spatial in a spatial environment where they could perform it and possibly record themselves doing that like put it on social media so like this is the bars that I do and in addition to those gestures that change the music they can also change the visual environment around yes they can record that and make creative decisions based on the visual environment that that's so cool this is all in your vision roadmap vision roadmap yeah this is like a long way down I am but one person and hopefully the the projects that I do and release I intend to make a youtube channel dedicated to how did I do that this is how you do it you know quick 15 minute video how do I set up this B drop simulator yes you know quick 15 minute video how do I make it so that the visuals react to the bass drum in one way but reacts to the higher frequency snare drum in a different way so that when I give them a drum set with a bass in a snare the boom the visuals are reflected based on that that's so you know and whatever music is playing that are easy how to then make and build on top of where you've been pushing the agenda these spatial environments I love this this is one of the biggest futures of like the way Millennials and Gen Z think is that it's not just about pushing the edge but it's about capturing how you push the edge to help have other people push it I want to see other people's better art look cute and of course I think everybody a lot of artists will be like ah like the imposter syndrome things like what am I even doing I don't I don't deserve this kind of thing and for me is sometimes it's like that with like I mean yeah I designed this cool system but like somebody will make better music for it or somebody will make it smoother like the gesture it's really clunky right now it doesn't perfectly sync up with the beat or whatever or the tools that I'm using just aren't fluid enough but the hope is that I will get to experience other people's improvements on that and I want to I I hope that people make it better because I yeah if you started walking us through some of the design tools of the future in this audio visual engineering space teach us about where you think that's heading the audio visual space and and even in the virtual and the spatial space as well of course well I think a huge amount of of where we're going is is being able to collaborate um in the same space using the same tools when you're physically separated in distances and we've seen this in like conference calls for example like there's you know you call in and everybody's in VR and there's a guy with a presentation at the front of the room right doing a presentation that's one way like the the star wars kind of hologram effect right for VR um where everyone's sitting around the table and some of them are virtual but in addition to that I think if you build up a community of people who who use the same artistic tools in different ways which is a very common experience people use the same DAW like digital audio workstation and make wildly different music right but it's difficult for them to work on the same one because most of the time there's one keyboard and one mouse and one way to input those shortcuts that you know and like so often when you collaborate it's one person inputting all the things and another person being like what if you did this drag that onto here and see what it sounds like try and automate this and so they're like giving suggestions but there's not as much of a way to directly do that thing but for example one of the things that VR is going to do is going to if you have a comprehensive enough set of like gestures and controls um such that everybody can be collaborating on the same piece of art at the same time using the same tools um then in the same space for example and talking to each other in the same space and having all those tools perhaps like in different areas so you can tell who's working on what um that can be used for music for live performances of music even you can use it for art you know like everybody engineering everybody who who like I have a specific artistic style I use this brush in such a way and that's kind of my unique identity is because I've gotten really good at effectively using this one tool in this particular way I think it'll become way easier to get agnostic collaborations and digital spaces and everyone has the same tools right in the same place and they're all working on at the same time yeah yeah that seems to be such a massive part of our future yeah yeah I it and it'll be fun you know to to be able to create with other people in the room with you yeah yeah but there's this is also something about the uh the what we get from the millions of years of eye to eye human human so there'll be a the balance that we strike with that for at least a while we'll see how that ends as we get better I mean it's we're not the hardware is still and I hope this always stays the case does not live up to our imagination of what it could be um but it will get better I assume I hope until you're indistinguishable you know yeah from the physical environment and then the it'll be easier to work with other people um doing the same things and and exerting your kind of your creative impulses on a piece of work as other people are doing the same I hope I hope that happens sooner yes yes and this is an interesting lead into our last two questions of course as as great becomes digitally indistinguishable from his physical self we must we must ask are we in a simulation are we in a simulation um I mean the easy cop out is does it matter uh mostly because I mean for me I enjoy this I I am happy with where I am um and to to to say that there's something that's more real than my lived experience it's like no sure I mean let's say there's like a Greg that's in a vat somewhere or something and like with electrodes tied to his brain I'm still I feel making meaningful creative changes to this world you know and I am experiencing their repercussions you know I'm I'm doing cool stuff I'm working on cool projects I'm creating stuff that I don't see in the world right I'm making those changes and I'm seeing how they impact the world hopefully yeah right and in that case that's meaning to me I don't really need a deeper one and then the last question Greg is what is the most beautiful thing in the world I I enjoy staying in one spot and just letting things happen around me in ways that I can try to come to understand like to to to sit and just as things happen be like how how how does that work you know it's this happens a lot more with sound than it does with other things for me but it's like that was a cool effect whoa what just happened over there that kind of thing like how did how did that reflection occur why is the sound behaving that way you know how would I try and imitate that like to follow up on the simulation thing I am in the business of making simulations so if I was in one how did they do that yeah yeah yeah right I want to do that I want to make that for other people like deja vu because it's cool how did they do that how do I make that for other people it's so it's cool I think it's good there's some really if you stop and listen there's some really cool effects that you notice that you wouldn't have noticed if you didn't stop love right things like how do you make love happen in a digital that you build love into a digital world right that's that's separate I found that outside of virtual reality Greg this is such a good conversation thank you so much for coming on the show and teaching us there's so much to still understand about interactive audio visual creation in general and what the next generation of video game music implementation especially with virtual reality and new user interfaces is going to be like it's been such a pleasure thank you thank you for coming on yeah everyone check out the links below to Greg's work go and support him also give us your thoughts in the comments below on the episode we'd love to hear from you go and share more content about these fields with your friends your family your co-workers online let's give people understanding the craziness of video game music implementation and audio visual engineering creation in general democratizing this art and support the artist entrepreneurs and organizations around the world that you believe in simulations links are below support us as well let us keep coming to awesome places like Cambridge to do interviews and go and build the future everyone manifest your dreams into the world we love you very much thank you for tuning in and we will see you soon peace Greg good job brother