 Okay, so let's get started, because I have a lot to cover, and I want to make room for the fun stuff at the end, so my name's Lauren Siegel, I'm on Twitter, Elle Siegel, you can tweet at me, follow me, I tweet about fun stuff. This is just a warning, there's going to be music in this talk, so if that's a problem for you, the doors still open, otherwise feel free to join us. But what is music? So to me, music is sound and rhythm put together in some interesting way, that's music. So we're going to be looking at sound in this talk, and we're not going to be looking at rhythm, but we're going to replace it with another interesting R that we know very well, Ruby, and we're going to be talking about sound and Ruby, and that's going to be this talk. So let's start with sound. In our context, there's sort of two ways to look at sound. The first one is analog, think about analog audio, and so analog audio is all about vibrations. Basically we have a continuous signal, there's somewhere in this room there's electrical wire running to the speakers that you guys are listening to, and that has an electrical signal in it that is oscillating some voltage, plus voltage, minus voltage, plus voltage, minus voltage drawn on the screen over there, and that is driving the speaker to vibrate back and forth a bunch of times per second, which pushes sound waves out that you guys eventually hear. The other side of audio and the side that we care about as programmers is the digital audio, because we live in the digital world, and we don't have the benefits of dealing with these infinity electrical signals, and we have to deal with samples, and so a lot of people, because there's this computer sitting between your speaker system, a lot of people think there's some kind of magic involved in the computer to the speaker's kind of the pipeline, so hopefully this talk will sort of explain what is happening in between your computer and your speakers. Effectively, what we're trying to do is trying to emulate what happens in the electrical analog world, and so what we do is we create, we get samples in time of the signal that we want to emulate, and we try to sort of emulate that signal that we're doing, that we want to create, so it's not a continuous signal, it's just samples of the signal at points in time, and so what we do is we take a bunch of samples, and then we take more samples, and then we take more samples, and more samples, and more samples until we get a really close approximation to that signal that we're trying to create. So that brings us to our very first vocabulary term, which is the sample rate, and the sample rate is really just representing the bunch of numbers, a bunch of numbers that we want to push at our audio, at our speaker system for one given second of audio, so a bunch of numbers representing one second of audio. If we had a sample rate of 20, 20 samples per second, and you wanted to generate some kind of oscillation for that 20 second, this is what it would look like inside a ruby, and so you generate a bunch of numbers, oscillating positive to negative, positive to negative for 20 samples, that would generate one second of audio at 20 samples per second. Now actual sample rates are much higher than 20, they're a couple orders of magnitude higher, we're looking at somewhere between 44,000 to almost 100,000 samples per second, and basically it just means there's more numbers, that's all it is. So really the only thing that sits in between your computer and the speakers is a bunch of numbers, it's just an array of numbers that you're pushing at your sound card. So we sort of have a good, hopefully you guys have an understanding of like, okay, numbers pushing at the sound card, driving the sound card, trying to emulate this signal, how do we actually create those numbers, what are those numbers representing? So to understand that we have to understand the fundamental waves that will create waveforms, and the first fundamental waveform is the sine wave, and so I'm going to preface this with, there's going to be some math in this part, math, and I'm going to try to keep it high level, I'm not going to try to go into details, but there's going to be some funky numbers, so if you don't really understand it, don't worry about it, I want you guys to understand the concepts, not the math. So this is the function for a sine wave, don't worry about the pi and all the symbols, the important ones we care about are the frequency and amplitude. And so those things basically explain how we scale this sine wave, both in the horizontal and vertical axes. And so amplitude makes it taller, frequency makes those things, those peaks closer together, and that brings us to our next set of vocabulary terms, which is amplitude, which controls loudness, how loud the sound is, or how quiet it is, and then frequency, which basically is pitch, how high or how low the pitch is. And then there's this S value, which I sort of skipped over, because it's not that important, it's basically our sample rate, because we are scaling this value to a number of samples per second, we want to scale that mathematical function to have T equals one second equals 44,000 samples. So I mentioned frequency as well, so frequency higher frequency equals higher pitch, lower frequency equals lower pitch, but what's some context behind that, what are the actual numbers? So humans typically have a range somewhere between 20 hertz to 20,000 hertz, that's the typical range for human hearing. So the range for, if you were to pick any number between that 20,000 to 20,000, you would have some kind of sound that would be audible to most humans. But to give you more context, the middle A note that we sort of, if you've ever played guitar, you've tuned a guitar, this is sort of the frequency that we tune our instruments around. It's the middle A key, and this is sort of the pitch that I'm speaking at right now, very close to the pitch that you hear music at. Music goes between around like 200, 2000 hertz, that kind of range. And so, and obviously there's higher and lower, but most of the sound we hear is around that range. So pick a number between there and you'll probably hear something that sounds like music, or sounds. So if we take those numbers, we take the 440, we take the 44,000 samples per second, we take an amplitude of one, we get this sine function, which looks like this. And basically, it's just a sine function, not too magical. If we got to 44,000 on the X scale, we would see that basically there are 440 oscillations in this sine wave. And that's all it means is that sine wave is oscillating top to bottom 440 times within that one second of time. So the next fundamental wave form is the square wave. And the square wave looks pretty much like what it sounds like. It looks like a square. And so basically what you're doing is you're oscillating from the negative maxima to the positive maxima to the negative maxima and back again. Again, 440 times per second or whatever your frequency is. Don't worry about the math, it's not that important. And basically, if you connected the lines, those lines going up and down don't actually exist. But if you connected them, it would look like a square. And so that's where the name square wave comes from. Then we have the triangle wave, which again looks exactly like what it sounds like. It looks like a triangle. It goes up and down and up and down and just oscillates. So anything that oscillates is going to create some kind of sound. And then we have the sawtooth wave, which again looks like what it sounds like. Instead, it's very similar to the triangle wave instead of going up. It just goes down, down, down. Then we have the noise wave, which seems weird to call it a wave. It's really just noise. And it really is just randomness. And it may seem weird to bring this up, but you'll see why I brought this up in the talk later on. So this is another kind of wave you can create. Probably the easiest one you can create. So rather than talking about this stuff, let's actually hear some noise. So I have this audio tool open up here that I'm going to load up. That is really, really small. So I might have to bump down my settings here. I apologize to the people doing visuals right now because I'm going to mess with the presentation. All right, so hopefully you can see that over here. That's better. Okay, cool. So I'm going to load this up. All right, so I'm going to play one note here. So that's a sine wave. Sounds like a sine wave. Basically, what happens if I can play a sine wave like that, I could go up one octave and you're going to see the sort of the peaks getting closer together. So you can see how that happens. So they get closer together as the pitch goes up. That's basically it. Next, we have a square wave. And this is going to be a little bit more noisy because square waves are a little noisy in general. So you see the same thing happens there. Ignore the little noise that you see at the top and the bottom here for now. I'll explain why that's there in a bit. But that's the square wave. Then we have the triangle wave. So that's a triangle wave. And then we have the saw wave, which again is a little noisier but cool. And then we have noise, which is going to look like randomness and going to sound like it too. That's noise. And then you can take these sound waves and you can add them together. So we could take one sine wave and add it to another sine wave. And that really just does the sort of addition of two waves together. And you'll see how they sort of interoperate. Let me just move that. So you can see the interplay of these two waves added together. And they sort of because they have different frequencies, they sort of create this really interesting intermix. Every sound is basically every sound that you're going to hear in the real world of my voice, some guitar instruments, any instrument is not just one frequency. It's multiple waves put together. So this kind of thing over here is really just like that's what sound really looks like. It's the addition of multiple waves put together. So let's jump back into our talk. And let's keep going. So we talked about audio synthesis, which is basically creating audio from nothing. And we're going to talk a little bit about analysis of audio. And basically analysis of audio is all about transforming audio, visualizing it, looking at it. There's one important tool that is used when you're analyzing audio. And I'm not going to talk about it too much, but it is called the Fourier Transform. And the Fourier Transform is probably like the most important tool that you can use when you're when you're doing audio processing. I'm not going to talk about the math. It's actually, it's pretty complex. It's literally complex. It's complex math. So I'm not going to talk about too much about it. It would be way too much to talk about here. I do want to talk about the sort of fundamental principles behind it because they're pretty easy to understand. And hopefully it'll sort of make things more clear. The way I like to think of the Fourier Transform is you have this waveform on the left of time domain. This sort of this is a square wave going up and down. And the way I like to think of it is you take this square wave at a point in time, and you turn it sideways. And you're looking at this two dimensional view now of this wave. And you're looking at the side view of the wave. And it's sort of every single frequency because I mentioned every wave form has multiple sort of frequencies involved in it. You're sort of looking at every single frequency involved in the creation of that wave on the left. So you're sort of turning it sideways and looking at frequency mode. That's the way I like to think about it and it makes it easier for me. If you want to learn more, there's a website called the4a-transform.com. It's really has a lot of good information about the Fourier Transform. Really well named domain. The thing I like the most is there's a quote from the author on that site, which really is a good fundamental explanation of what this theory is. It basically says that every single wave form that is created, that you see, that the square wave that we saw, the noise waves that we saw, my voice, every single sound that you hear can be recreated using nothing but sine waves. So nothing but sine waves. So all the other waves that we saw, the square wave, can be recreated using nothing but the addition of sine waves. That's a really powerful concept because it means not only can you add a bunch of sine waves to recreate any single sound, it also means you can take any single sound and decompose it into the set of sine waves that created it. And that's really powerful because every sine wave has one frequency. So you can see how that works. So here's an example of that. Here's a wave form on the right that we're going to try to approximate using adding a bunch of sine waves together. So we add in one sine wave on the left. It doesn't look anything like it. Let's add another sine wave. OK, now it's starting to get closer. We'll add another one. OK, getting closer. And then finally, we add the fourth sine wave and look at that. It's exactly like the wave on the right. And it turns out that this wave on the right was actually created using the exact summation of those four waves. So I told you guys to ignore the noise in the square wave. The reason why was because Ableton Live is actually creating those square waves using the addition of sine waves. And this is why over here that you see the noise is because we're actually adding sine waves. And if you don't approximate to infinity, you're going to see a little bit of noise at the top and bottom where those oscillations happen in the sine waves. So basically what you're doing is you're adding a bunch of sine waves together and you're approximating a square wave like that. So what are some applications of analysis? The first one I like to call passive analysis. Passive analysis of waveforms. And that's basically pitch detection, tuning, visualization. Instrument tuners are a really good example of that. You sort of get the frequency mode, sort of look at it, figure out if you're on the right tune. Otherwise, you change your instrument or you retune your instrument. The second one is wave shaping. And that is when you actively change the waveform. So we go into frequency mode, we take this waveform, we turn it into frequency mode, we have all the frequencies that are involved in this audio sample. And then let's say we want to get rid of the low frequencies. So we delete the low frequencies, we get rid of them. And then we take those remaining frequencies and we turn them back into time domain. And now we have this sound that has everything but the low frequencies involved. And that's all that equalization is, is going into frequency mode, manipulating those frequency waves, going back into time domain, and now you have a manipulated sound. Auto tuning works very similarly. It uses pitch tracking to figure out what the actual sound is. And then it sort of manipulates those frequencies that don't match and then slightly moves the ones that do and then brings it back into time domain to create an auto-tune sound. So I'm not going to show a 4-day transformer auto-tune stuff like that. That would be awesome but I don't have enough time. Instead I'm just going to show you guys sort of what this looks like. So I have this graph over here. Let me just load up the sine wave that we saw before. Okay cool. And so I'm going to play one note here. I'm going to play it really quietly so it doesn't bother you guys. But basically what happens is I'm playing a sine wave note and you'll notice again sine waves have one frequency involved in them. So you'll see that one peak represents that one frequency of the sine wave and that represents about 900 Hertz. It's the A4 note and it's about 900 Hertz. And so I could go up an octave and then you'll see how the pitch changes in the frequency. So that's basically what it looks like. Let's look at a square wave because I showed you guys a square wave here and I said it was a bunch of sine waves added together. So let's look at what those waves look like in sort of frequency domain mode. So this is the square wave. Let me just... So that is that. I'm gonna not keep it on too loud. Okay there we go. So that was the square wave. You can see that it's actually the addition of a bunch of different frequencies. And those frequencies were created using sine waves. And so that's really all that's really what it looks like. And so if I wanted to get rid of one of those frequencies I could sort of delete one of those go back into time domain. I'd get a completely different sound. Cool. So now it's time to get practical. So we learned how audio is represented. We got a little bit of information about that. We learned how to create that stuff, those numbers that we have to push our audio card. And then we looked at sort of how you could process that audio. And now it's time to actually figure out how we can use that information to write audio with Ruby. So I'm going to show you guys three audio libraries that you can use today to create sound using Ruby. The first one is called Ruby Audio. This basically lets you write audio directly to wavefiles.wav files on disk. It's basically a libsoundfile wrapper. And it looks a little something like this if you were to write us a sine wave to disk. Don't worry too much about the code here. Basically we're creating a buffer that contains 4,800,000 samples. We have 48,000 samples per second. For 10 seconds, 4,800,000 samples. And then we fill it with a sine wave that we saw before, that little function we saw before. And then we just write it up to disk at the bottom there. Nothing too crazy. This is a really fun way to sort of write out music, play it, test around with writing functions out and see what they sound like. But it's a really slow feedback loop. You have to write it out, you have to play it back, it takes 10 seconds, blah, blah. What we really want to do is we run it right directly to our sound card so we can get instant real-time feedback. And so when I was doing this talk I wanted to get instant feedback on the stuff that I was playing with. So I wrote a library called Easy Audio, which lets you basically write directly to your sound card and hear that stuff in real-time. It's basically a port audio wrapper, which is port audio itself is an audio library that is cross platform, lets you basically it abstracts out the OS layer when dealing with audio. And there are FFI bindings in Ruby for this as well, so you can get the port audio gem and just use them directly. But the low-level API is really difficult to use and it's not fun. So I made Easy Audio and it's basically way easier to use and you can just play a sine wave with that one line, you can open a stream, pass in a lambda that processes one frame of audio at a time, play a sine wave, you could do fun things like taking a triangle wave, starting at a frequency of 220, increase the frequency over time, and let me just play that for you real quick. So this is the increased frequency one, I'm going to play it here. So that's what that sounds like. Let's move on. Cool, so we can play with audio, we can do fun stuff live, but there's one problem with writing directly to your sound card is the fact that you don't get it to take advantage of all the other fun stuff that your system, all the other audio tools that you have on your computer, synthesizers, filters like delay and gain and all that fun stuff, so you can't really do anything if you're writing directly to your sound card. What we really want to do is we want to write to our audio tool like Ableton Live that I was using before, so that Ableton Live can also add in its ability to synthesize and sample and do all those other, that other fun stuff. So we want to write an audio plug-in, which means we want to write a VST, that's typically the most common plug-in that you'll find out in the wild. There's different kinds of audio plug-in formats, but VSTs are definitely by four of the most popular. Any audio tool that you'll use, if it supports plug-ins, will most likely support VSTs. So I wrote a VST that will embed Ruby, so that I can actually write Ruby in Ableton Live and actually play around with sound in Live, get the benefit of all the tools and visualizations inside of Live. So basically what I did was I embedded Ruby inside of a C++ application. Don't worry too much about the code here. The important part is it's basically embedded Ruby calling out to an external DRB process. So I'm calling out to another process that actually processes the audio locally on the system. And then every single frame where the VSTs asking me to process a bunch of frames for samples, I basically call out to that remote Ruby object sitting on my local machine. I return back with the samples and then it returns back to the audio system. Let's see it. So this is where it gets fun. This is where we can all have our fun here. I'm going to load up the VST code here. So I'm going to go into easy VST and I'm going to type rake. That's going to drop me into an interactive shell which basically has the music server loaded up, the server side of my component here. And I'm going to load up a audio, a little set that I created with that VST setup at the bottom here on the left. And so that I have the VST loaded up and I have the server side component loaded up. Let me just show you some of the code here. This is the server. I created a bunch of instruments that process sort of one frame of music. So here's the sine wave function. So that's a sine wave function. That's a square wave function. That's a triangle wave function, etc. And then there's just a little bit of a music server that actually processes that audio. So what I can do with this music server is I can set a function. So let's play a sine wave and then I can go into live here and I can actually play one note. Sorry about the pitch there. So there's a pitch, there's some sine waves action going on there. This is a sine wave that's created with Ruby. Same kind of deal with the octaves. So I can do that. I can create sine waves. I can create square waves. Warning, this is going to be a little bit noisy. So that's a square wave. You'll note that we're actually using the exact square wave function. We're not approximating with sine waves. There's no noise at the top and the bottom. We can do other fun stuff. So we can create, we can add sine waves together. This is three sine waves added together with different frequencies. And so you can see you get the same thing. And we can do, we could do really fun stuff. Like we can take the frequency right down here. I don't know if you could see that. So this phased constant right here is really this crazy function that actually oscillates the frequency over time with another sine function. So it's actually oscillating the frequency over time. So it's going to sound a little bit crazy. Bear with me here. Let us... Cool one did that. So you do fun stuff like that. You can play around with Ruby's function. Create interesting functions, random things. Do all this weird stuff. And you get the feedback live in Ruby. It's really fun to play around with all this stuff. So you can do all that fun stuff. We can also do real things with sound. We can actually create real sounds. So I'm going to create a stabby lambda here and make a function that calls a sine wave and then multiplies the amplitude by a fall off. So I'm going to use a linear fall off here. And this linear fall off is basically going to make the amplitude reduce over time linearly. So I'm going to play that for you guys. So we can do that kind of stuff. And then we can actually make it fall off exponentially. So we can make that sound like this. Let me just bump that up a bit. So you can do fun stuff that sounds like maybe like a string instrument. So we're getting close to instrumentation here. Cool. So we're getting closer to making music. Let's create some cooler instruments. Let's create a bass drum. So what's a bass drum if not just a sine wave times the exponential fall off thing. So let's go back to that. I don't have to change code. It actually is the same thing because, so this doesn't sound like a bass drum, but that does. So a bass drum is really just a sine wave that has a really low frequency. And so that's our bass drum. So let's create a snare drum. Okay. So snare drum is really just noise. So let me bump that up. Cool. So we have a snare drum. We have a bass drum. We have a sine wave playing like a string instrument. We have a bunch of cool stuff. And so now you guys have the tools to create music with Ruby. But it would be unfair me to get you guys all the way here and not actually create music with Ruby. So I created a song for you guys to show off what is possible. And I hope you guys enjoy this song that I created using nothing but the Ruby samples that I talked about in this talk and I showed you guys. So hopefully it's not too loud. There we go. That's sound using nothing but Ruby. So I'm just going to close off this talk with a little summary here. Basically audio is not scary. It's just a bunch of numbers that you're throwing your sound card. You're just telling it to vibrate a speaker system. That's all you're doing. You're just throwing a bunch of numbers at a sound card. So we learned how sound is represented. It's just a bunch of numbers. We also learned what those numbers actually should be. Like how to actually create those numbers using sine waves and different functions that oscillate. Like sine waves, square waves, solid waves, triangle waves. We also looked at how you can use Fourier transforms to sort of process and analyze audio as well as do funky things with those sine waves. And then finally we took that knowledge and put it all together and actually applied it to Ruby using three libraries. One that lets you write out two wave files. One that lets you write directly your sound card. And then one that actually acts as a VST plugin that you can use to plug into other sound tools. And then finally we made actual real music with Ruby. So I blog about stuff, various things. Lately I've been blogging about music so if you want to read more about all the music stuff that I've been digging into you can go to my blog. I have a couple posts up there. I'm going to be adding a couple posts later on with all the stuff that I talked about in this talk and more. All the code is available on GitHub so you can go check out Easy Audio and Easy VST and play around with them and do fun stuff with them. Again I'm on Twitter you can tweet at me, follow me, ask me questions, etc. And again I just want to say thank you. I have two passions. One of them is programming and the other one is music so I really enjoyed the fact that I was able to pitch a talk at RubyConf that included both my passions and actually got accepted. So thank you guys for actually coming here and letting me play music for you guys and letting me sort of share my two passions. So thanks.