 Hello and welcome back to the FEM channel. Our next speaker is Kai Kunze. He's a professor at the Graduate School of Media Design at Kaiyo University Japan. His talk will be about Fresson waves. From what I've understood is that it's about shivers and goosebumps in media performances that are usually spontaneous, but somehow he and his team managed to induce them artificially. But I'm looking forward to this talk as I don't really understand it right now. If you want to ask questions in the hack and REC at the channel RC3-FEM in the rocket chat at the channel FEM and on Twitter and the FETIVERSE at the hashtag RC3-FEM without the dash, you can ask questions that will be answered in the Q&A session afterwards. You can hear the translation by listening to the language in the web player from Native of Translated Ender. And now I'm looking forward to a very interesting talk. Hello and welcome to my talk Fresson waves, augmenting aesthetic chills in classical music performances. This is conceptual early research work from a collaboration of a lot of artists, designers, researchers, and I'm just a speaker to introduce it to you a little bit. My name is Kai and in the next 20 minutes, I will talk to you a little bit about what is Fresson, give you a bit of motivation and background information, why we are interested in this feeling. And then I will talk about how can we recognize, induce and also share Fresson. And then at the end, I'll talk about some conclusion and a little bit of outlook. So the question is, what is Fresson? You might not have heard the term. I actually haven't heard Fresson before. We started the research two and a half years, three years ago, but I definitely knew the feeling. So if you're listening attentively to a musical piece, sometimes you might get goosebumps or some shiver down your spine. And that is usually triggered from the music. Fresson is from the French shivers and sorry, my pronunciation with the German accent. So you part for that, I hope. And it's this psychophysiological phenomenon that we feel when we get these goosebumps or shivers that are triggered from music, but also other experiences. And you might wonder why goosebumps, how can goosebumps be related to a positive feeling? And there is actually no need to answer a lot of theories. One that I particularly like is that Fresson is often induced over music or over some kind of stimulus that is repetitive, that has a certain pattern, and that at one point, the pattern breaks and that surprises you. So this triggers your autonomous nervous system. So the fight of light response. You get the surprise, you wonder, alertness goes up, and you realize that there's no danger and you will relax and feel these aesthetic chills. So the talk I will give today is an exploration of the feeling of Fresson with technology. So how could we detect, induce or transmit it using especially variable sensors and actuators? And to be a little bit acclimatic, I can already tell you that this is still in process. So this is really exploratory work. However, you might also wonder, why do you care about this? Why do you want to do this? And one reason, of course, is because we can and because it's fun. And I think that's definitely kind of one aspect of the research. However, also another reason is, so our lab in Yokohama works in human factors research, so HCI, human-computer interaction. And we lately revisited a lot of work also from cyber netics, also non-linear dynamics in terms of research, and also in terms of art and performance. We are very much inspired by Stelaq's work on extending and augmenting our body. And there's this realization. If you work on research that, you know, knowledge is not merely functional. There's always some kind of enjoyment in understanding a concept. And I think also this community will really understand that type of feeling. And this sense of wonder and this feeling we also want to explore. We want to understand ourselves better in terms of cognition, perception, but also in terms of our feeling. And actually last year, I gave also a talk on boiling mind on an effective feedback loop that we played with and started researching on. And to some extent, this Frisson wave talk is just a continuation of this. And overall, we're just also looking for more creative ways to use physiological data or other variable computing sensing that is not related to surveillance. So extended to this, we also wonder, what does it mean to be live? It's easier if you think about transmitting audio video easy in quotation marks, because, yeah, there are some experts that know a lot about that. And I see also the effort that goes into the remote experience and other conferences or conferences. However, we still don't know how to transmit an atmosphere or a feeling that's much more difficult. I think the Congress is a very nice example for that, because it moved from Berlin to Hamburg to Leipzig. But every time I visited, I kind of felt at home. I felt, oh, yeah, these are, you know, kind of the people I like. These are the culture, the community I belong to, even though it's at different places. And we wonder, you know, kind of how can we transmit that this types of feeling and two efforts that we get inspired from from this work is one is NeuroLife. That's a project, an EU project with co-investigator Jamie Ward, and also a cybernetic being project here in Japan, headed by Kota Minami-Sawa, that deals with things like parallel agency and similar. And both of them are actually also collaborators in the work that I will present today. So this is the high level overview of why we are interested in Fresson, but now getting back to the aesthetic chills. And first, the question is, how could we go about and try to detect or recognize them? Looking into related work, of course, we see Fresson or aesthetic chills, of course, affect our physiology. And the first thing that you notice is, of course, the pylori erection. So the goosebumps that you can get on your arm. So the hairs go up. So we could try to detect that. However, that might be a little bit difficult, because some people might not have so much hair on them and so on. So then looking into other physiological changes, respiratory rate is going up. For the sweat glands, electro-dermal activity, you will see more peaks. That's a stress and excitement indicator. And heart rate goes up, blood pressure goes up, and usually heart rate variability related features go down. Because also, if you saw last year's talk, we already built a system to record electro-dermal activity, so the sweating on the hand, as well as heart rate, we just thought we'll move along and use that. Luckily, we also did redesign of the wristbands in the meantime. So they look a little bit nicer now. And you see also live demo on my background right now. So you see EDA and heart rate behind. And if I press here, you should also see some noise on the sensor. The visualization, by the way, is done by Kirill Rakozen. So thanks for the work. And then moving forward, so we used these wristbands to set up a control experiment to detect aesthetic jail events. We just added a trigger so to add some self-reporting to it. So in this case, we really used the user as a self-reporter to classify or to label the Frisor events. That has, of course, also some limitations. So we hope that that's good enough to capture it. And we used some music pieces also from related work and did some counterbalancing and run this lab study just in kind of controlled space, so with headphones and so on. We finished this, but then we also wondered, you know, how does it look like in real life in the wild experiments? So we also organized a concert with 18 audience members for one of our musical program and the setup was the same. So everybody got a wristband and a trigger. We also added a food center for the pianist, so using EDA from the food actually works also relatively well and then recorded here the data and hope that people would report their Frisor, their aesthetic chills. Here's now one video, a short minute video that shows you the recording. How about the analysis? I have to say, I'm sorry, this is still ongoing, so we don't really have a lot of results yet. Then of course, there were a lot of issues with the live recording. If you're interested in doing something similar, contact some of the technical staff or also me, we can give you hints. I'm doing this now over 15 or 20 years and always something is going wrong depending on the setting and so on. Now I also know more about the classical music concerts. However, we got some useful data. The problem there was we could also train a machine learning model because we really wanted to detect it real time and it seemed to work really well. We're just still not sure if it really works or not. So we want to be very careful about that. So we get higher curiosities back, but given the limited amount of users we had or so on, we want to look into that a little bit more. However, the analysis as well as the data sets will be publicly available. If you want to get them a little bit earlier, just also contact me. Then moving on, this is the progress on detection. How does it look like for triggering or inducing Frisor? There's also a lot of cool related work. I just show or highlight two of them. One is work by Shoko Fukushima at all and they're using the electrostatic effect on the arm to control pillow erection and they use it to increase the surprise feeling of somebody. So you put your arm inside and they can control the pillow erection. Other work is from HA at all where they're using three peltier elements on your back, on your spine, and they activate them upwards to also induce Frisor or aesthetic chills. The problem with those two setups, it's quite hard to get them into a concert hall and some people might not really have much hair on their arms or there might be limitations for it. So then for first iteration, we decided to go for a neck prototype because the neck is also a part of some of the Frisor responses. So you get either chills down the spine or up the neck or also your hair might stand up. So we thought it's a good start and we used first peltier elements or thermal modules and also vibrotactile feedback. And later iterations we moved just to thermal feedback to activators on the back of the neck around on the upper side of the trapezius muscle and they would activate with slight cold feedback. So for an initial test, it seemed to work. So this is just with 10 participants, around 30 minutes per participant. We had two music pieces that based on related work, so Chopin and Gustav Holst, we counterbalanced the conditions or music pieces with neck band, without neck band, with neck band, with activation and without activation. And from an initial test, we can say that it seems that slight cold feedback really provides more instances of reported Frisor. So there's a slight positive feedback, but you know, still quite little participants and we'll have to continue and see also with a little bit of redesign. So we wanted to change the order and placement of the peltier elements for the continuation work as well. Now moving to the last part, so we talked about detection induction. And now let's talk about sharing or transmitting Frisor. Here, the idea would be, you know, you are listening to a musical piece, a classical piece, and one person gets a Frisor that is detected over the wrist band and then it's distributed, it ripples through the neighbors, they get activated over the neck band and hopefully also free feel Frisor again around the same time, just after the red circled person felt the aesthetic chills. So in this case, then, you know, we would have all of the audience members need to wear sensors and activators. We would need a friction detection and also then the activation based on that. For that, we also organized another concert, in this case, 50 audience members, the program was around one and a half hours and the setup was, as you see here, so performers on the top and then we had two sections, one so 25 users would wear just the wrist band as a kind of control group and the second group, 25 users would wear wrist band and neck band. So it would get actually the detection and also the activation. So, you know, 50 plus wrist bands needed charging and 25 neck bands where manufacturing, this is a picture from the actual concert with the neck band section and here is how this should work. So, you know, you have first one person you detect the Frisor and then you ripple it out to the neighbors, then the next person might feel Frisor we detected over the wrist band and then ripple it out to the other people that haven't gotten activation yet and so on. So you have then a wave of Frisor, hopefully moving through the audience members. This is another setup, Jan He, who also did a lot of the organization parts or so on at the piano and here's then a small video that summarizes the work and at the end you see also the servers, the recording server, the activation server and the detection server and also the activation server. The question you might have now, did it work? Hmm, not completely sure. Again here, work in progress. So analysis is ongoing and we can't also really say because yeah, we had this control group and we could see more Frisian events in the sharing group, but how to interpret that that's really, really difficult. We are also working on the design of the wrist bands as well as the neck band and especially for the neck band we got a couple of users, I think five or six or seven that really didn't like the neck band, not the activation. So the slight cold activation was okay, but just because it was a little bit too tight and a little bit too uncomfortable. So we're working on a redesign. We have the next concert in control, recorded all of the data and make it also publicly available. Soon enough, look also a little bit more what we can find out about what happened. This brings me to the end of the presentation. I hope you enjoyed it. I just wanted to thank a couple of people first and foremost Jan He, who organized this, who also introduced us to Frisian and also the team. So this was for the second concert, the extended team. Thanks a lot for everybody who was involved here. Then also all of the names. So these are the people that did the actual work, that not just doing the presenting like I do right now. I hope I haven't missed anybody. So also thanks to George, Ding Ding, Danny and so on and all of the other people involved, GY planning, the studio Apollo, also the pianists and interactive performers. So thanks a lot. That brings me to the end of the presentation. As I said, we have a third concert probably in April next year in Yokohama, Tokyo area. So if you're interested, let me know. Also, if you have a general interest in affect or similar phenomenon, also just write me an email. It would be good if you mentioned Frisor, also the remote experience in the subject. So I can just filter that out and something completely different. We also have a conference next year, March submission deadline is January 7th, augmented humans in Japan and Germany and cyberspace that deals with maybe similar work. So thanks a lot for listening and I hope I told you something interested in the last 20 minutes. Bye. Hello and welcome back to the fan channel. Thank you, Kai for the very interesting talk and Kai should actually be with us to answer a few questions. Hello, Kai. Hello. And we actually do have a few questions already. And the first one sounds a bit more like a comment, but I'll tell you anyway. So one viewer noted that there's a technical cybernetics and systems course here at the Tureimino and wanted to know if you were aware of this already. Actually, I wasn't, but that sounds quite fun. I'm getting more and more interested in cybernetics as well and I think it's useful to revisit some of the ideas around feedback loops, as I said at the beginning. So that's cool if you're already looking into that. And I think especially if people go into HCI fields, I think it's quite useful to get a little bit of that background. Nice. Okay, let's go to the next question. It's about neural networks, which type of the neural network was used? Have you considered neural different neural differential equations such as echo state network as all reservoir computing, which are good when modeling stiff time consciousness processes? That's actually a really good question and actually also a good hint for what to do next. I just tried to look up. I saw the question already also in the chat. I tried to look up what we used. And I think at the beginning we just used support vector machines, so not neural networks. And I know now we are using some neural network, but I don't know the configuration and I couldn't check. I'll get back to the person who asked the question. What was interesting for me was that the data looks already quite good. The sensor data looks actually quite good. And I would assume that in most cases, any classifier will do a decent job for the lab experiments for the other works. I think that sounds quite interesting. I also want to go more towards nonlinear dynamics work as well in terms of estimating Friseur or different feelings, but that's a really good hint. But that would be more a question also for Jaouen Han, co-author of the paper that is also linked. She is our data analyst and so on and knows what she used. So the first classifier was a support vector machine, fairly basic. And I think recently we use a neural network, but I thought it's just very straightforward PyTorch, long training, but nothing special, nothing fancy so far. Okay, nice. I guess she will probably also hear from the questions then. And then let's go to the next question. Have you considered or tested the effects of adversarial stimulus such as attempting to cause Frisson waves in in boring situations instead of like interesting ones? That's also quite a good or interesting question. I mean, there were some audience members also that mentioned that the neckband was actually a little bit uncomfortable. So I'm not really sure if we crossed the Frisson with them. And I don't know what would happen if you I think you probably would just make them the situation uncomfortable anyways. I'm not sure what would happen then. If you're stimulating cold feedback, actually you might get a fear response in these cases. If you in a boring situation, I'm not sure if you if you hmm. Yeah, I actually I don't know. It's definitely an interesting idea to use it in boring situations. Can you get somebody to change their their feeling and get to a more excited state? We're playing often we played a little bit with thermal feedback and it was always interesting if you change the thermal feedback. So instead of if you see something hot in VR, you give cold stimulus or so on, it really is a little bit confusing and interesting. I haven't thought about that in the Frisson situation. And if it works for boring work, but that's definitely cool or interesting. So if somebody wants to play with that, I would be up for also giving a little bit of help or ideas in that direction. Okay, thank you for for answering these questions. Unfortunately, they don't seem to be any more of them. So that's it. Thank you very much for the very interesting talk. That's a topic I haven't really thinking that much about. But thank you very much. It was very interesting. Thanks a lot also for having me and it's always fun and I always enjoy the feedback. Yeah, there's also again the live demonstration behind me. So you saw my excitement level kind of increasing or decreasing with the questions. Oh, wow, that's that's pretty cool. Okay, yeah, thanks a lot. Then see you and here on the FEM channel. The next thing happening will at 11 p.m. the lightning talk or not on the FEM channel, but one of the next things happening at 3 p.m. will be at 11 p.m. the lightning talks at remote Ryan Wall stage. And here on on our channel actually at 12 a.m. or midnight, there will be the next terrorist news show. And until then, until then, bye.