 Cool this one. So good afternoon all of you Yeah, so we are going to present a bit of a cheeky thing and try to attempt creativity in a computer. So So I'm harsh Minjamur I In my day job I design algorithms for mechatronic systems and by training I am a controls engineer with some Further experience in mathematics and currently I work in a startup that makes control units for engines and motors Music is just a hobby Hello I'm Vikran Patil. I'm software consultant trainer I'm curious about various musical instruments and I try to understand music through mathematics through playing I take advanced programming courses at people Academy I have worked at DRDO. I have worked at a company called strand life sciences Okay, so How do we learn music So for that matter, I would like to ask how do we learn swimming? How do we learn cycling? So, you know For these we get some guidelines some rules But those rules we can't use we have to practice it. Isn't it? We practice and we learn we understand if If I know swimming and you ask me, how do you do that? I can't tell you isn't it? so these activities are called as right brain activities and Something that you explain is a linguistic activity so While doing these things What we are trying to understand is whether we actually Apply logic apply some rules or We memorize some patterns So let me give some small demo. Okay So look at this So this is some notation. Okay This notation is represented as language It's a rule. It's a rule to play some music and If I tell you okay read first line and can you can you identify what is next? You can't from the words, but if I played So I've chosen this tune deliberately because I know most of you know it I play one day Your mind starts playing inside Maatram. Isn't it? So what is it? So it's it's nothing but memorizing pattern if you remember In childhood our mathematics teacher asked us to remember tables multiplication tables. Why? We already know how to do arithmetic, right? We can compute it anytime when we want, but why because Memorizing will make it fast Huh, so I would like to give one example from my learning. I started learning music quite very late after 25 years and By then my logic was very strong so and You know on top of that I used to play flute So I used to take notations from my teacher and I used to follow the notations while my fellow students My classmates they were younger very small kids They would follow my teacher directly as it is without knowing notations So they were trying to imitate audio patterns Isn't it now instead of one day Maatram if I would have taken some other tune What was the possibility that you would recognize? What is the next thing coming up? No possibility right so anything that repeats it's stored as memory and that's what we call as pattern And we are trying to model this into mathematics and see if we can imitate Indian classical music Hurj will explain for the details Thank you so if you look at music as a whole the Structures therein tend to be far more complicated. So for our experiment We will assume in Indian classical music because there's already quite some structure built into it. Yeah, so we'll quickly look at Three important structures and then we'll show you how you can put it into a matrix quite easily and Teach a computer to play the flute instead of Vikrant here. Yeah, so let's say there is some Raga and We will restrict ourselves to seven notes. So the commonly known notes from Sa to Ni and They'll repeat in the lower and upper octaves. Yeah, so Some Raga's don't have certain notes So if you look at the string at the top the fourth note ma and the last note Ni are absent in this Raga Yeah, so That is something very easy to encode in a computer Then a little bit more complicated is some notes tend to be more important than others So in this particular Raga, which we will play for you in a minute You will notice that the first and the third note are have a darker shade So they are more important in this Raga and they will come So they will occur more often. They will occur in certain patterns and so on Yeah, so missing notes important notes relatively easy to encode What about sequences of notes? so if I take the same set of notes and Play them in a different style today as against yesterday. I'm sure they will appear as different Raga's all together so the sequence of notes which we will call trajectories is quite important and For us the patterns that we can't talk about Are these trajectories and we will find a clean way to embed them into a mathematical form? yeah, and Finally, there are certain sub sequences of these trajectories which we will call typical sequences they are called puckered in in traditional Indian classical music and their beauty is that it keeps repeating and Once you get to know Raga a little bit more It is these typical sequences that tell you that it is this Raga that is being played and it's also these Typical sequences that make it pleasurable to listen to the Raga Yes of four four structures missing notes important notes trajectories and typical sequences now. Let's look at two extreme examples of Raga's So This is a Raga called Boop So anyone who has learned even a little bit of classical music in their childhood would know that they start out with Boop So Boop is simple in the sense that There are only five notes. So ma and ni the fourth and the seventh notes are missing and The trajectories are also simple. So you can just ascend notes linearly descend notes linearly and There are certain typical sequences or pockets. So Vikran, maybe you can play it Thank you. Yeah, so Five notes you could ascend just as if you were reading the string at the top Descend as if you were just reading the string at the top backwards and then there were some typical sequences So the same four structures This was a relatively simple example. Let's look at a more complicated example Yeah, so there's till a come out also something one would learn maybe already in the first year when you learn music It is Apparently simple. Yeah, there is no missing note at all. There are two important notes. So the second and the fifth one However, it's far more complicated. So when you ascend notes it is far from linear when you descend notes again, it is far from linear and and The typical sequences are also not as simple as they were earlier. So let's just see how it sounds. Yeah, so When you ascend notes Yes, I did not just read the string from left to right and when you are descending notes, it's even more fun Yeah, so you went to the last note jump to the seventh note again went to the note above that and so on and It's typical sequences are also nice Yeah, so we have looked at two extreme examples a simple one and a complicated one which had the same four structures now We are claiming that you can put this into a mathematical model which are without too much difficulty so let's look at how we want to do that so Ultimately everything here is a string of is a sequence of notes, right? So we are going to treat all of it as what are called Markov chains. So We'll forget all the jargon in a sense. It's quite simple so So all the notes that are missing we will simply skip. Yeah, so they have they have no existence in our model Then to each note we will assign a probability of jumping to some other note Yeah, so if you look at this matrix here and look at the first column Then it says that if you are at the note saw Then with high likelihood so with likelihood of point four you will jump to the note ray With zero likelihood you will jump to the note Yeah, so there will simply not get played if you're already playing So to each note we will assign probabilities of jumping to some other note And you can imagine that naturally this will Allow you to embed important notes by giving them higher probabilities Now what about trajectories? So this matrix only tells you how to jump from one note to some other note with a certain probability So the way we will do it is the way students learn when they learn music We will feed in some known compositions Yeah, as sequences of notes and compute these probabilities which with Vikrant will explain in a minute and Actually build this matrix So in a sense the trajectories part of this raga is all in is all captured in this matrix here Yeah, and the typical sequences which anyway occur very Nicely and very often when someone renders a raga will naturally have higher probability of being played Yeah, so something important to note here. This is a probability Matrix, right? So if I start with the notes as you can see in the lower right figure You could end up following the pink trajectory in one case while the orange trajectory in the other case and so on so it might Actually be like someone is playing it differently every day. Yeah, okay, so I'll just summarize So when we feed a sequence of known compositions to our model We are going to construct that matrix in the top right corner that I talked about This is what we will call learning raga. Yeah, so we are claiming that all the important melody structures are actually encoded in that matrix and Once you have that matrix here is how you can actually make a computer play the raga So you start with some initial note, let's say saw size there in practically every raga You go to the matrix of yours and predict some note depending on the probability distribution in the column Yeah, you play that note and note and note which note you've played and Jump to the column for that played note and just repeat this exercise. Yeah, so this is sort of a finite state machine that will keep running all the time and If that matrix like we claim has actually captured all the structure, it should sound as if the raga is being played Yeah, so that is all That is the theory behind what we want to do. Vikrant will now explain how we have actually implemented this Okay, so we have given some input sequence. We have taken some known compositions and fed them to our bot And what it has done is for every note It is trying to compute some histogram. So for example Let's take pa So after per in my input sequence input known composition What are all the notes? Those are possible which come after per I count them So this is the histogram, right? So after per moist coming highest number of times eight times and from this histogram We compute probabilities and this is how we construct probability matrix We'll have probability for every note that comes into that raga Okay. Now once we have learned the rug It's time to play. How do we play it now? We have the model which is captured in the form of probability matrix Now, let's take a simple example. So for example, I am at some node Which has got next three possibilities ray or and me With these three different probabilities point to point five and point three They all must sum to one Okay Now what what we do is we take sticks of length equal to probability stack them This total stick length will be one Now I generate uniformly distributed random number random dot random Between zero to one and wherever that lands up. That is my next sample It is as simple as that. Okay. So now we do this Okay We were showing so much of mathematics and whatnot Just to make sure that we have written it in python So power of python is you know, it's a very simple program You can go over it in ten minutes and find out what all it is doing It has got only those these two three things histogram transition probability Sample and this main function alab That's it. And then we have a bot which can keep generating a sequence from a learned Raga now next thing is we want to play it So let's play it. I'll show you demo. So I have this bot player to this I'll give Input I have this This rug so initially I started with one day mataram and that is in this rug. So we'll give as my input and Yeah, so Those who know classical music and those who have heard Rock this they would know that certainly this sounds like this rock Other than that, is there anything that I can test So, how do we know that certain song is played in some different some particular rug? Isn't it so for example Yemen So it has to have a typical sequence nerega. Isn't it? So similarly every rug has a typical sequence which her to explain we call it as puckered So I did one small experiment So using my bot I created 8 notes 16 notes 24 notes like this till 96 and every experiment thousand times and In that experiment I count how many times puckered comes There'll be multiple sequences for puckered I'll count how many times it appears and this is the statistics so around in 32 nodes we capture one note one puckered and Even mathematically I can say okay, if you give me at least sequence of 32 nodes It will sound like the rug that it has learned Okay, so still we we want to do some more stuff on it. So what is it that we haven't done yet? So first part So what we had claimed was using just simple model. Can it capture structure of raga? So in our program, we have never mentioned that in boop This is verger meaning this where you cannot you cannot play or this note is most important we we reduced it from the data and Our probability matrix is actually capturing Most musical structure melody structure What it doesn't encode is rhythm So all these Indian classical melodies they come with some tall and There is no input that we have taken from tall angle and That is not there and we would like to add that also and Once that is there, then we can actually have jugal bandi between computer and real ustad Next as you saw We gave CSV meaning notations handwritten notations as input which me and Harsh Practically wrote it down in a file and fed it We would like it to learn it directly from music So I give a song and our bot listens to it. It finds out. What are the notes? It builds probability matrix and directly it plays. So this is what we would like to do Okay, these are links Thank you any questions Anybody has any questions? Very seasoned musicians. They actually go for I forgot the term They go for 40 or 50 days of like doing only music Yes, and so even though like they know everything like that With some insight. Yes, so I feel I don't it's my personal view like music is something more than the transition Matrix because that differentiates Like really good musician from a super star. Yes. Yes, so so do you think like computers can ever go there or? Computers can mimic I'm not sure whether they can go there computers can mimic So if you have a master from which our AI can learn it will learn but I'm not sure Whether it can generate something new Right, so you said about how it's difficult to train the model to detect puckers, right? So what if we take a transition matrix and we have the poker in our mind and we simply Increase the probabilities of the particular notation transitions that is the poker so that might have addressed this right and Another question I have is the third thing you showed like how can something learn directly from songs? So if we add a module through which like, you know the frequencies detected the frequencies mapped into something And that's continuously being fetched. Yes. So that is something I hope can yes Yes, can be done can be done. We haven't done it. So this is this is work done in almost three four days. That's it So this open source, right? We can build on top. Yes. Yes, you can build So there is link here you can work this and you can Good work Thank you. Okay. So you have this transition matrix where you have probabilities. Yes Going from one note to another. Yes. Now music is about something else as well, right? So how long you're going to stay in that particular note? Now, how do you capture that? That's how you write the Notation notation. I'll show you so so basically We try to capture it in notation. So you see that pause somewhere, right? dollar That that is representing stay at that node for some longer time and you would have Heard it in the tune that I played there were many pauses Because I captured it like this So the main point is you pause at certain bits of rhythm for example some or Call Kali, isn't it and that is not captured in this bot That is not captured, but pauses are captured Can you hear me? Yes. Yes. So not just the pauses. So Sa, pa are there, right? Now the amount of time that you spend on Sa and the amount of time you can pa are different. I got it So basically you're trying to say mind mind and gamak that all you are trying to say If we can capture it in notation if we can capture it in notation Then then it will come Okay, but you know that is the difference between human voice and computer voice that This can be perfectly automated because we are writing on the notes, right? Yes So if if if I can take this as notation, then that will be captured Anybody any more questions? You said that you will try to make this what smarter by adding some features that it can recognize the music directly So what's your plan for doing something like that? What would approach would you take? Would you analyze the music in like how the western tools do it like the BPMs and everything? Possibly frequency as he said possibly trying to analyze on frequencies. I'm not sure yet what to do about it And I'm sure those who are already doing machine learning. They must be knowing Maybe her she will tell something and at a fundamental level you need Fourier transforms which tell you roughly what is being played and then once you have that you can have a sequence I mean a concept Julia. I know there will be practical difficulties So the concept of Fourier transform is basically Yeah, so Fourier transforms will tell you what frequency is being sung played, etc The practical aspect is that when a human voice or an instrument actually plays it Let's say the lower size say 250 Hertz. It's never really just 250 Hertz. It's several frequencies around it So there'll be some work needed to Decide what is sa. Yeah, it won't be just 250 Hertz But that apart it is only really only that