 Hello. Welcome everybody to this seminar. It is a pleasure for me to introduce to you Rafael Ramírez, who is a professor aggregado in our department. So Rafael did his undergraduate studies in Mexico, and then his PhD in University of Bristol in the United Kingdom, and then he had an appointment as a lecturer at the National University of Singapore, and then he came to Barcelona to join our department. Rafael is currently head of the Music and Machine Learning Research Lab, and he will today talk about the research that he's been carried out within his laboratory with his colleagues. Thank you Rafael. Thank you. Thank you for coming. I will be talking about what we do in our research lab. Our research lab is the Music and Machine Learning Lab, which is part of the Music Technology Group, and then we do research in music topics. Our research is funded by several projects, by one European project, the Maria Maesu program that you are probably aware of, and the Spanish Ministry project called Timul. So thank you for the funding. One thing, I will talk about music learning and how we can use technology to enhance music learning, and then we can probably identify different kinds of people that practice music. So it can be painful. I can't see my mouth, actually. And then we can see all the kind of people that have practiced a little bit more. And when we see person A and person B, we can ask a lot of questions. For example, how do you get the most natural question is how do you get from A to B, or why there are so few Bs? How many people have access to B and A? And the question we ask in our research, in the main question, we ask all these questions, but the main question is how can we use technology to enhance the transition from A to B. So this particular research we are doing in our group is in the context of our European project called Tell Me. It's a research and innovation action, and then we have different partners. We have technical partners, Genoa and ourselves, some pedagogical partners, the partners Royal College of Music for the music part, and then some two companies or two industrial partners that deal with all the issues of the project. So what is the motivation for doing this? Well, it's nice. Why do you want to really study how we learn or how to improve our learning in music? In the literature there are plenty of works reporting the benefits of learning music, not only in terms of being able to play an instrument, but there are other cognitive skills that you developed when you learn an instrument. Social skills, of course, motor skills, hearing skills, you have less probability of having diseases that relate to older people with the brain, and then it's a nice thing to do not only for the music part, but for the benefit it provides. Second observation is that there are these limited access to formal music education. If you take the whole population, only a small percentage have access to have music classes, and then where still the people that already have access, they are lucky enough to have access, there is a big, large rate of abandonment. So many people all around us, when they were children, they used to play trumpet or saxophone, and they don't play any more at all, they just abandoned. So the aim is to create or provide assistive and interactive music education technologies that complement traditional teaching methods, not replacing, that would be arrogant, to say we're going to, with technology, get rid of the long tradition of how to teach an instrument, but then complement them, and in order to widen access of people learning music and lower the abandonment rate of people already studying music. And also we want to investigate how do we learn music. So the project, as I see as an overview, is student-centered. We have the students surrounded by sensors, the most natural sensors are microphones, of course, with music, and cameras for gestures, musical gestures. And then we have also the physiological sensors, such as EMG to detect muscle activity, EEG to detect brain activity, as well as other physical sensors like accelerometers or gyroscopes to detect movement. And then these sensors are connected to a computer, and this computer on the one hand has access to a database of expert performances, people that are very good at doing what they do, and have recorded a multimodal data in the sense that they have recorded with all the sensors. And then this is stored in a multimodal database called RepoVis, which has been developed by the group in many other projects. And then we can have, apart from the recorded data, we can have things that we compute from the data. For example, the velocity of the ball, if it's a violin, or the spectrogram, we can compute from the audio, from the movement sensors, and then provide this synchronization with all the data, multimodal data. On the other hand, the computer provides real-time feedback to the student. And this database, I cannot see my mouse. Let me see if I can go here. So just to finish with this, you have synchronized data in which you have the audio, you have all the scriptors you can compute or you record it, and then video and motion capture data. On the other hand, we have the feedback system in which we provide different kinds of feedbacks, and there are plenty of things we can provide. We have been working with students at the Royal College of Music who are high-level students, they are almost professionals, and then most of the interfaces of feedback they want is in the score. They are based, the whole location has been based on scores, on music scores, and then we provide annotations on the score if they play on time, if they play on time, they play out of tune or gesture information, and most importantly, after they play, we can summarize the mistakes or the issues they could work or improve on, and then we can give a summary in order for them to just focus on those ones. So also this is the score, but we also provide gestures, information about the gestures, and then the idea is to provide a velocity position of the ball, this is a Barling Center project, and then useful information for them to have, like how close or far away they are from the target performance, and they can compare with the target performances in the database. And then we have a social network that students can plan ahead the plan for studying what they want to achieve and how to achieve that, and also they interact with the database with the system and they interact with each other. Students can upload data into the database and the other people can comment on the data and also they can communicate among themselves. So just to give you a very brief idea of what kind of data we are capturing, we have been capturing data from experts, and with all the sensors I saw at the beginning, and we have cameras, all kinds of cameras, we have high quality cameras, like quality cameras, we have Kinect which is a 3D camera, we have a normal camera, we have microphones, ambient microphones, contact microphones that you can see in the next picture, and then we have also EMG, you cannot see them because they are below the suit, and then with this we record from very basic exercises, so just sounds up to scales to concert. So we have recorded material from the very beginning to the most advanced level. So this is very nice, and then how do we do this? There are a lot of places we can do analytics, we are a machine learning group, we apply machine learning, so this is like an ideal playground for a machine learning person. There are plenty of places you can apply machine learning, for example, you can analyze the system feedback that you provide and try to personalize according to the performance of the student, of different types of systems, which ones have more impact on the learning process. We also can mine the user interaction, how they interact with the system, how they interact with each other, how they plan and detect patterns of good practices or patterns of activities that lead to progress in learning. And of course the multimodal database is a very nice place to do machine learning because we have data from different sources and it's interesting to know how these different sources, the information in these sources correlate. So for example, one issue in the project is we cannot record, recording is expensive. So we cannot record every possible piece, every possible exercise that any student would like to practice. So if it is not there, what can we do? Well, the idea is we could at least try to generalize from the recordings we have to record we don't have. So we apply, for example, in pieces expressing performance modeling, we take a particular musician, an expert, we know how he played several pieces, we extract patterns and then we get a new piece, we can just generate or simulate how he or she would have played the piece. What is expressing music performance? We have been working this topic for a long time, not only related to music learning, but expressing performance is studying what the musician adds to the composition of the music. So what we listen is two things. One is the specification of the piece, specified by the composer, plus probably more interestingly what the musician adds to this piece. If we just render automatically the scores, music wouldn't be an interesting area or activity. It would sound very, very flat and robotic. The interesting parts come from the human that interprets the piece and applies transformations to make it sound more expressive. So how do we model it? On the one hand we have the scores of the pieces, then we encode this into some machine representation and we extract some structure from the pieces. On the other hand we have the performance, the audio, the recording, and then could be audio or could be the gestures as well. We obtain the recordings and then we extract symbolic description by applying signal processing techniques. So we basically get a kind of expressive score. We just detect onsets, offsets of notes and energy and so on. And then with the symbolic description and the machine representation of the score, we compute the difference. We know what is actually the musician adding to the performance. And together with the analysis of the structure of the piece, the score and the audio, we apply machine learning, we apply many kind of techniques. We apply machine learning to obtain an expressive performance model which we can analyze just to understand what's happening when and how the musicians are introducing expressive transformations or to analyze the model which is not great. So we go back and change something in the description of the scores. And once we are happy with the expressive performance model, we can take a new score and apply the model and get an expressive performance in the same style as the performance we are modeling. This work is in our group. I've been doing this for a while. Sergio Gerardo has done his PhD in expressive performance in jazz guitar and Fabio Ortega is a PhD student now working on violin expressive performance modeling. So we model durations. When you have a score, you don't play exactly the duration you are specifying the score. You lengthen or shorten the duration of the notes. Then the onsets, you advance them or you delay them. The loudness of course, dynamics is a very important issue in expressiveness. How do you emphasize parts of the piece by playing it louder? Then we also model gestures and intranote features like the envelope. We have been doing that, how the note sounds on itself, notes apart, and how notes are put together in articulation. So we take a score. Then we, of course, we focus on one note, but the note is not enough to tell properties about the note. The most important is the context in which the note appears, in the musical context, in melodic or harmonic context in which it appears. So we take features of the note itself, but also features about the environment or the context in which the note appears. We characterize each note by a set of descriptors with this. If it is a polyphonic piece, we will take not only on the melodic line, but we'll take what's happening in the other melodic lines that are concurrently happening. Then we take the score and the audio, and we compare, as I was saying before, we compare, for example, duration, let's say, we take the duration, and we calculate the duration actually that was played in the audio, and we calculate a ratio of lengthening or shortening, and then this is, for example, one expressive transformation, the duration, and we place, for each note, how much it was transformed in some dimension. And then once we have this, we apply machine learning algorithms, we have applied the typical ones, and also we applied very successfully, one algorithm that is not used very often is inductive logic programming, which just induces logic programs, and the good thing about it is that you can introduce in the learning process background knowledge. So you can just put in first order logic, kind of, you background knowledge whatever you know of the music, and this will be incorporated into the learning. So the most efficient algorithmic models that we have obtained have been done with inductive logic programming, which is normally not very much studied. Then we obtain, after applying machine learning algorithms, we obtain the expressive models, then when we have the models, as I said, we can get a new score that we haven't used for training, then apply them all and get an expressive score. It would be like a MIDI with duration, which are not exactly as stated in the score, and an energy and so on. And then we can apply a synthesis engine to synthesize this. Just an example, I show you, this is work in the past we did with saxophone, so this is induced or generated by the system that has been learned from a professional saxophonist. Oops, let me show you. Of course, it has problems. It's not perfect, but it's much more expressive than what we would get from the straightforward synthesis of the score. So another area we are working on in the concept of the project is normally you can see the score, and then synthesis is taking the score and of course synthesizing the sound of the score, of the music, and the way back would be performance information retrieval or automatic transcription. So we take the sound and we would like to know what is the score, but we can think of this as when we generate music, it's not directly that way. The score produces some gestures. With whatever instrument you use, you use gestures to generate the sound. The gestures generate the sound, and we can see the other way. The sound is coming from the sound trying to predict what kind of gestures produce the sound and the same from the gestures to the score. This work has been done by Alfonso Perez. We are post-doc in the group. And the interest of this, if you say, well, it's very nice, why do you want the gestures, is because normally gestures are difficult to capture even with cameras. If you don't want to be intrusive, you have only cameras. So how do you calculate the position of the bow and so on and the velocity you are applying, the force you are applying in the bow. So this also is how you can apply machine learning. You take the sound, you have a lot of recordings with the sensors. You have the sound. And the idea would be that without needing any cameras or sensor or movement detection, you can, for only the sound, you could predict or predict the gestures you are applying to generate that sound. So that would be great if that was possible in general because you don't need cameras, you just need a microphone and with the microphone you are able to tell how much pressure you are using, how fast you are moving the bow. This has problems because it works very well for one instrument in one room but if you want to have different violins in different rooms then but you have trained it doesn't apply that much so Alfonso Perez is working on this. Then another topic that we are working on is because we have the EEG data of some of the recordings we are generating recordings with EEG data is how do we, how this brain activity data we take with EEG correlates with learning. So then we have the musician, we record the audio and we also record the EEG data of the person learning and then from the audio we can compute some descriptors of how good or bad the person is doing the ones we have been working so far are those ones because they are complete beginners so then it's just how accurate you can maintain the pitch of a note how accurate you can maintain the volume or the loudness of a note and then from the EEG we compute some typical bands in frequency bands and then we try to correlate them. The preliminary results are as I didn't say Angel Blanco is doing his PhD in this topic and then the results are preliminary but of course there is correlation but what's happening in the upper brain activity correlates with the performance of the musician or the student and for example if we record at the beginning of the trials we do a lot of trials at the middle of the trials and at the end of the trials you can see for example in beginners that there is more alpha activity along the way which means that's also a response to what is in the literature you have more free resources at the beginning you are very concentrated using all the resources and gradually you start to liberate resources while advanced people that play the violin but advanced students they don't have this issue actually there is a strange thing because it's the opposite which shouldn't be, they get more concentrated at the end and at the beginning and then this correlates with the measures of goodness. Also one of the motivations as I mentioned at the beginning was to widen access to music education so allow more people to have the benefits of playing an instrument and then one group of people that clearly don't have access are people with motor disabilities people with motor disabilities unfortunately are not able to control fine movements which are required to play a musical instrument so another area of research in our group is musical instrument learning for people with motor disability so the idea is very simple in a normal situation we have the person that the musical instrument like guitar, violin or piano is perfectly adapted to our hands because it was designed to fit our physical abilities but when you have a person with disabilities motor disabilities the instrument becomes a problem you cannot really play the instrument because you don't have the skills that were supposed to be required to play the instrument so what we do is build that orange box that we build an interface that adapts well to the person with disabilities and disabilities is a general term so this has to be treated in different cases and then the interface fits very well how it triggers sounds in a very efficient way so we have identified three broad levels of motor disability one is the motor disability like for example for slide cerebral palsy or you have control of your body but not complete control or fine control fine movement control but still you are not able to play traditional instruments then we have severe motor disability in which you only maintain high movement control but nothing else or probably a bit of head control and extreme motor disabilities when you don't have even control of your eyes this has been done mainly that was the PhD thesis of Zacharias and other people and a master's student also working in this area so for the first one what we do is if they can do they still have some movement which is not fine but still have some movement the idea is to see what is the best possible interface that adapts to what they actually can do so then the typical process is we have a set of sensors a lot of sensors we have for example the proximity sensors in which we detect the proximity slides on potentiometers touch things, buttons we have EEG, we have eye tracking we apply all the sensors we have in the lab basically and to see which ones are the better suited for that particular person and once we identify what are the capacities of the person related to what sensors he can control then we provide some design we build the interface we have a guitar controller and then we give it to them and also we normally monitor how they learn it's not just a toy this is a musical instrument which has the same properties as a guitar so you start being very clumsy and gradually you get better and better we quantify this so for example this was a German person who always wanted to play the guitar but he had several pulses and we built this interface in which he could strum with one roundy thing and then he could control the chords because he could control only movement in one dimension so I don't know very quickly but he has the chords oops sorry he has the chords and he can control with that dimension and then he can, he's not strumming because you can automatize the strumming and he's playing a concert in his center so the idea is that we potentiate what he can do and very often what happens I don't know if you saw he was using his right hand he's using his left hand and for most of his life he was using only his left hand the other one never used it the moment he started to play then to get more accuracy he started using his other hand so this is not only intellectual therapy but it also helps in motor abilities so then the second one is severe motor disability in which you can only control the head of the eyes movement and this was an interface built by Bambacusis and then this is IHARP and then you can just by looking at the screen triggering sounds and also chords you can specify chords or arpeggios at the beginning display and then once you are happy with your arpeggios then you can control the key or the chord you are doing with the arpeggios and also play the notes you have volume and so on in this paper we it's not just not a toy again we evaluated the expressiveness of the instrument from two perspectives from the player perspective like when you play how expressive you can how much expressiveness you can add to a piece and from the audience perspective you can see to how this expressiveness is perceived so for example this is Zacharias and we have been using this for example he has this is a child learning music with the instrument he has cerebral palsy and he can only interact with the eye tracker in the school he only uses eye tracking he cannot do anything else and then he is trying to learn his learning and I think looking at this we say well it's kind of entertaining we have another person that uses he was a musician he has an illness that doesn't allow him to move the body anymore and he is using the eye harp to play music and you can do much more better than I mean you can use it as an expressing instrument for example this was playing music the point of this is it's been a digital interface you can adapt it to the users as well as if you notice this is a more complex interface than the one that the child was using before this has like semitones in between has more resolution and has other features that you can individualize or personalize for particular users or make it as simple as possible the analogy would be having a piano in which you can change the size of the keys you can change the amount of notes you provide and so on and then the moment you look at the key will sound so you can do that depending on the expertise and you can also the idea is that someone starts with a very basic interface and gradually adds more and more functionality for extreme motor disability this is related to brain computer brain computer music interfaces or brain computer interfaces the idea is to since you cannot move anything not even the eyes or control the movement of the eyes the only way and you still have a healthy brain then the idea is to put a brain computer interface and interact with the computer through your brain activity so we have done several work several interfaces based on this mostly some of them based on P300 potentials I don't know if anyone I just explained what they are but if you are if you randomize some stimuli and you're focusing in one when that happens your brain triggers a spike in activation and this is normally 300 milliseconds later after the event has has appeared that's why it's called P300 and then by focusing your attention then if you have you are detecting the brain activity you can detect where your attention is where you are focusing your attention so then we have for example this interface it was like a guitar you could change chords and you were focusing in only one I wanted to change to A minor for example on the top and then I just focus on that one they were flashing flashing around it so each time that flashed the A minor flash then my brain produced like a peak after 300 milliseconds and after so many repetitions the system is quite certain that you are focusing on that one and it will change the chord to that chord and so on and then once you change you focus on another one so you can navigate a chord space by just focusing which one is the next chord you want to play we have done also like with P300 like the same idea as in the Iharp in which you have a sequencer and you want to trigger sounds each roundy thing is a note so you just focus on that one and they are flashing and then you will after a while the system detects that you are focusing on that one and it will trigger that sound we have also more like switches this is only I think this is video I will just show you this was a hack day and there was an instrument in which there was a sound and these are that kind of switches so you have each row of two circles has two possible sounds and by focusing on one of them you change that switch and then the arpeggio changes in texture so probably let me see so suppose she already changed the first one and the task was to change one by one and then by focusing on the second one each time there is a flash and also the sound you can also P300 also generated by sound events auditory events so then when that sound is produced then you have that spike and that spike is detected in main repetitions and then you change so you could change gradually the texture of the sound by changing the one you wanted so another approach we have taken for brain computer music interfaces is not only based on P300 but we have built emotion driven brain computer interfaces so we detect the EEG we transform this into an emotional indicator of the person and then with this emotional state we trigger what way we want in this case we mix the research in expressing music performance with brain computer interfaces and then the music is changed accordingly to your emotional state so what may I show you before I for example he's asked you can map any emotional state to any music parameter but here he was trying to relax to make the sound the music sounds sound southern so he's asked to low down to make it sad and to do it to put it down it's a kind of active listening you're listening but of course the emotional state you generate will change what you are listening this seems like a bit of very nice but why do you want to do that and we have applied this to as I will tell you in practical contexts but the way we do it now is not just general expressivity it's not just how people pay expressively and that's it we have recorded the same pieces a number of pieces each of them with a different emotional intention we have as the person played this same piece angry we play it happy play it sad and play it relaxed so we have we build four different models for each for each performer we have four different models and then we applied machine learning to get the model and when we have the model we interpolate the models we have different distinct models with some parameters we just interpolate these parameters to generate intermediate performances and once we have this we obtain the emotional state of the person we estimate the emotional state and without the emotional state we place the which plays in this arousal valence space the person is and the music will be adapted to that emotional state of the intention so we have applied this as treating depression we have we conducted a study in a elderly home with elderly people with depression diagnosed by the psychology of the center as depressed and then we did this kind of neurofeedback system in which they could control the music with emotional state in order to try to improve depression this neurofeedback has been applied not with music but in many in many areas including depression so we just included music in the equation so everyone knows what depression is the only thing to say is that people with predisposition to depression they very often show an asymmetrical activity in the two lobes so then they have normally more activity on the right lobe compared to the left while people with no depression we have a balance in activity so they show more left frontal alpha activity alpha is inactivation so they show less activity in the left frontal alpha in the left so the idea was very simple so we try to encourage with the system to have symmetric activity but there is a normal way to treat depression in neurofeedback and then what we do is we have a neurofeedback system the person is sitting there listening to music then we obtain the HG data the HG data is transformed into some indicator of the emotional state and how excited or relaxed the person is and how positive or negative the feeling is and we place a coordinate in there and then we have trained an expressive performance model as I just mentioned before with different recordings and then we can apply this to MIDI music which is easier because we can really change the durations the articulation of the notes or we can do that also with audio music and we change only dynamics and timing and then that sounds emotional component so this probably reinforces the emotion and then it's a feedback system and we are we are asking the people to do is to try to to make this music sound happier which very very roughly would be a bit faster and a bit louder but not exactly so they sat there for 15 minutes trying to do that and just just trying putting the state of mind in which that happened and then we repeated that 10 times in a long 10 weeks or I don't know about like 15 weeks or something like that so there were 10 sessions 2 per week no less than that on the 10 sessions and then we applied a pre and post evaluation of a standard depression test there was improvement of 17% of everyone improved and then most importantly we have the easy data while we're doing that so we can analyze what happened within within one session and from the beginning to the end of the session and then there was improvement a significant decrease in relative alpha in the left frontal lobe from the first session to the last session from the beginning of the first session to the beginning of the last session there was a statistically significant difference in balance in how positive negative they felt and also within one session we analyzed what happened and how excited it's a bit changing but it's not correlated with time but the balance is a strong correlation they get feel better along the session so we also have worked with autistic children and this is trying to extend before was active listening and here is just passive listening we use music just as a tool to try to help them identify emotions in faces in people faces so what we do is we show them faces from a academic database for studying emotions we show faces with different emotions we ask them what emotions they think they are representing they fail initially a lot autistic children because they have problems recognizing emotions and then as a second condition we introduce music when they see the faces we play music that is important so then we ask them again and then they get better of course because they have the music that helps them to identify the emotion but then we remove the music again to see if there was some residual effect and then in one session they could improve or not and we repeat this as well in many sessions and we see if they improve from the condition fresh condition no music to the last condition condition and then also we do it from the first condition of the first session to the first condition of the last session and then they improve they increase from 46% as a group they fail most more than half of the time they were wrong in the emotion and at the end they were 75% of verbal accuracy but also we have the EG we work on the EG and amazingly this is a pilot study because there were very few children but initially the emotional state decoded by our system is completely uncorrelated to the stimuli so they see a happy face they don't feel happy it's not completely random but interestingly we didn't expect this at the end of the study the emotional state decoded by the system is started to align with the stimuli we face so this is not if it is true I mean this a pilot study but it's not only training identification there is no kind of empathy with the stimuli represented the last one we have this is the last thing I want to mention because I'm running out of time we are currently working in a project using music as well but this is in case music therapy in the hospital esperanza in the palliative care unit of the hospital esperanza they have a terminally ill terminally ill patients with cancer so there is nothing else to do the expectancy normally the average of living they have two weeks so very short in the last steps of their lives and this hospital has a program for music therapy so music therapists go and do music therapy to these people it's not of course eliminating anything but it's just improving the quality of life at the end of their lives so music therapy is entertaining and many people think it's just like you play to them and so on music therapists get really angry at this because they see actual changes in physiological measurements but not physiological but they can see improvement in them but they normally are not scientific oriented they are musicians so they never quantify what they are doing and what we are doing is taking the EG and quantifying along several sessions because as I said the life expectancy is very very short so we do it only once and we have two groups we have 20 people in the study group in which people come there are like three musicians they play music there is someone recording the exact timing of events during the session and then the second group the same emotional state they go and then provide company to other patients we know the music they just talk to them and we record also the EG signal of both groups and then this is a ongoing project so we have not definite results but from what we have been analyzing there is a significant improvement also in bailings of course they feel much better we record at the beginning and everywhere but we compare the beginning to the end of the session and they feel what we detect is much more a positive state of emotional state than the beginning and basically we cannot do much more as I said because we don't have this repetition of sessions I think you said 50 minutes and I'm 48 so I will stop here thank you very interesting talk any questions so I think I thanks Rafael so maybe you can say a few words about these inductive logic programming methods that you are using and why are they successful compared to other ones yeah I don't know the inductive programming is not used very much because it's based on first order logic so people don't like it and it's not you have to know about a bit how it works and there are plenty of algorithms that work on this kind of formalism but the idea is some training data as usual and then apart from the training data you want you can provide some background data which is just background knowledge so you can in first order logic you can have like a sequence of clauses which specify whatever you want knowledge and then when you try to learn the model that is actually trying to do is generate models logically imply the background knowledge together with training data we logically imply the model so it's a logic induction in logic and then why it has been working here very well because music has a lot of background knowledge you can put many things that you know about the music about the structure and so on so these instead of having to decode it you can have it like a formalism that put it into the training data you can have it a separate knowledge that can be introduced in the learning process and also the models most of the machine learning algorithms that we use and decision trees and so on are propositional in the sense that they could be specified with propositional logic propositional logic is just simple so then when you get these models from inductor programming these are intrinsically more expressive because the model is a logic a first-order logic formula so I don't want to be more technical but so this is a very difficult problem to combine this type of combinatorial search in this structure and spaces with more traditional machine learning so maybe you can say how do you do it combine them combining these yeah I mean we haven't the approaches in which you can combine inductive logic programming with more traditional machine learning algorithms what we have done is applied directly inductive logic programming and this the issue you mentioned is complicated because the search space for hypothesis is huge you have a much richer language in which to specify your hypothesis so this is of course a bigger space so that's it's not kind of a kind of clicking and that's it you have to impose restrictions how do you explore this search space so you have to apply restriction how the length of your of your clauses you can keep adding preconditions to your rules or you can specify what kind of the shape of this precondition should be not explore things that you don't want to explore because probably you know what makes it more complicated but then you get relational models as opposed to propositional models probably we can talk later about the technicalities and what kind of algorithms because you can there are two basic algorithms that you apply one is top down and bottom up you can start with a hypothesis that covers the whole space your instances and then specialize this into more specific hypotheses that are set and then have a rule or a clause for this one that only applies and then try to remove preconditions from the rule to make it more general until you cover a more significant set of training data questions thank you for the interesting talk I was wondering two questions one of them is quite small is for the treatment of hard sessions where for example if they had one session each month their condition could have been improved because for example it's Christmas and they get this from family that was a pilot study and several things that could be improved but it was twice a week it was two sessions a week and during five weeks so there was no holidays in between or anything but certainly it changed completely the states for example in the process some of one participant died for example he was a friend of them they were living in the same elderly house so they were close and then the family don't come to visit them sometimes so they are more depressed because they didn't come the day before they could have come and bring presents many things can happen so we couldn't control on those ones it was just from the outside and trying to quantify this because it would have been impossible to quantify all the all the things that happened in between thank you and the other question was when you said you had the model that you extracted at the beginning of the talk you had the recordings the detailed models of the expert performance and you could extract some you tried to extract some models of of the performance of the scores directly from the music when you had enough data of the of the performance right I was wondering whether you that could be possible with learners because for example I guess in the casuistic is quite big not everyone has the same problems for example someone might have problem reading the script but I'm guessing the problems with people learning music are going to be somewhat similar in different stages of the learning process the things we want to analyze but it's not related really to the models we are not we are not interested at the moment at least to build models of the students like models expressive models or how the style how they play but we can record the interaction with the system so how what they are playing and we can record exactly where they fail or where they do it well so depending on where they fail the system is working progress should provide feedback on what activities they should practice next so for example this is very relevant with the Royal College of Music students they are students as I said that are like semi-professionals they are the best of the best and then they always complain they don't have time they can record themselves but they will never have the time to listen to the recording to see what they did well or what they did wrong because they don't have time so what they are really interested in or the system to with some intelligence give them a summary of places where they have to focus more and this would be in the performance but they they can be given not at the level of a particular piece but in a more lower level at the level of of what exercises of techniques techniques they should focus on to improve on these particular problems they have so that would be like kind of bringing the feedback to each student because we have so much data what they have been doing how often they do it and how well they do it so we can then infer what should be doing to improve but they are not doing well so I guess that the situation on this very question for example is is very different when the music is not score based but very very different improvisation you mean improvisation or real-time compositions yes well real-time composition it's probably another level but improvisation for example or not even necessarily improvisation but variations on a theme even if the particular melodic aspects are present variations on a theme could be many and not and we would probably understand when there is something wrong or I don't know I mean identification of certain issues may be significantly more difficult I guess so we have in the project in this project we are focusing on classical music in which there is a score and so on but in the past we have worked on other music for example I mentioned Sergio Heraldo's thesis was on jazz guitar we have recordings in which the score had like 30 notes but the person recorded 100 notes where did it come from? I mean it's almost between improvisation of course it has some similarity with the theme but there are so many things that you do in between that is in the limit of improvisation so the problem is very different but we explored it by trying to build models of ornamentation in which context you add notes which content you remove which kind of notes you do and so on so we studied that and then also we studied in the past not recent past free improvisation saxophonists were playing whatever they wanted and then the idea was to construct a score of what they played of course you have to quantify and then see how in this score for example dynamics change so there was no fixed score but once they play we can analyze what they played and see the score and we were identifying who was playing for example so we took a model of different saxophonists and then just by these variations we were trying to we were telling we were predicting who was of a new of a new recording who played it just based on the style not on the on the notes thank you I have a question when you study the expressivity features or characteristics that you can modify to make it better better so do you actually have an idea which things are better so because you can have a recording that is not successful that has modifications on duration and pitch or whatever so do you have this kind of information as well to actually select the one that will be better or it's just for one individual and try to replicate how this individual will play yeah I mean you are saying how do we identify the set of features we feed to the algorithm to produce a model no? I mean if you have recordings so some of the recordings could be successful because they are I mean they are variations that are actually good they are variations that are very good yeah I mean we don't give features in the sense that related to particular pieces we give features related to the context in which notes appear so the music material and what we have done in the past is to extract as many features as we think are musically important like the relationship with the previous notes where the note lies within the bar if it's a strength a metrical strength a strong note and so on things that when you play you consider how you play the music and then we get many and then we normally do feature selection and then we use the set to get rid of the one that actually in the data set in our data set they don't provide much information and then this the problem probably you are mentioning is that what happens if you for example train a guitar and then you try to you give a a score which is blues or rock or a different genre this would be a problem because you are training on patterns of a specific genre and then trying to predict a piece which is not not related to it's not representative it's not represented in the data set and so we try to to focus to have a data set which kind of covers whatever music we want to to synthesize or study later more questions for people with disabilities so they were actually they actually did know music before or they were we have different we have different kinds for example the German guy the one with several Polsi he had several Polsi very strong several Polsi and he always wanted to play the guitar but he had never been able to so he didn't have any musical knowledge he was very bright but physically he couldn't control movement so he started from scratch and then the child I showed so this is someone who always wanted to play music because his brothers play music and he cannot so he didn't have he had never played any instrument but the one I played not the but the video of the performance with the Iharp that's a medical doctor who was a guitarist in the past but then he developed an illness to play anymore so he had all the music and knowledge he just wanted a means to express what he wanted so it's like a normal instrument a piano you give it to a beginner they will start doing very basic things you give it to someone that already played piano he will make more use of it so before we close there will be some snacks Mexican snacks apparently waiting for you but Rafael will tell whether they are evaluation so thank you very much Rafael for your wonderful talk thank you thank you