 So hello everyone and thanks for coming and staying after a long, hard day of FASDEM. So this talk will be about a kind of software ecosystem with various different small graphical software, mostly for art creation. So interactive music, art show, installations, museums, this kind of stuff. So and I will first start with a small video, which is of an installation that took place in France a few last year. So the voiceover is in French, but it's not really relevant. It's, for instance, it's a big, big carousel. I'm sorry. Up. OK. And so, for instance, in this installation, people were playing musical instruments that were all synchronized together. And in order to make a song, so this is the kind of work that we are trying to make and make tools that allow such things to exist. We have, like, there were 23 instruments. Well, here we have the overall thing with a big computer center that makes everything run, various MIDI devices, various sensors, so everything coming in, light shows, and how to synchronize all of this, how to make it work, how to make nice shows. So, yeah, so basically what I said. And of course, this is free software, which is cross-platform as much as possible. The main software is built with Qt, and then there's a lot of libraries, which only depend on standard C++ and nothing else. And it works with a lot of various environments, so pure data for, you know, processing, open frameworks, Unity, 3D, quite a bunch. So, the first thing I will present is Libosia, which is the basic C++ library that will allow to define basically what we want to do. So, Libosia is a distributed object model. That is, it has objects on various software and which communicate through the network to, well, make everything work. So, one computer can tell another computer, increase the volume, set the light to red, whatever. And the particularity of Libosia is that it has been ported to a bunch of these environments. So, maybe it's too small here, but so Java, processing, pure data max, well. And it is based on open sound control. So, for tools who don't know open sound control, it was touted as a replacement to MIDI, which is the standard for devices such as this, which send notes and play sounds. And open sound control is much more general. And as such, we based ourself on it. So, and I will make a few demonstrations of the various possibilities of Libosia. So, I will start with a pure data patch. So, I don't know if any of you know pure data. Does it work? Okay, looks great. Okay, so this is pure data. And so pure data is basically a fairly old software nowadays. And where you just put boxes on the screen and you draw cables between the box, and this makes sound. So, we have a set of objects in pure data that you can use to create basically to give names to the parameters that you are going to control. That is, for instance, but again, it's very small and sits in some old software made in CLTK. There's no way to zoom, so I'm sorry. But basically, you can control notes and things like this and do, for instance, send triggers up, like this. And, yep. So, for instance, you draw cables like this and you will hear very beautiful music, hopefully. I'll trace pretty terrible, but it's to give a simple example. And so in this pure data patch, we basically define a set of parameters to gain, frequency, stuff like this. And the point of Asia is to make these parameters accessible to any other software or hardware. And as such, we use multiple protocols. So, for instance, we can use QML applications which will, well, allow to control the patch. Okay. If it works at least, but it doesn't. Great. Fantastic demo effect. I think that the wifi killed my test. Okay. Okay, let's see, with the HTML example. Okay, so here, I have an HTML webpage where I can control the parameters of my pure data patch. So here, we can see, for instance, if you look, these parts will change when I change this slider. And so that's a very strong point when compared to previous solutions which required some kind of OSC bridges, like with Node.js between the media application and the HTML webpage. So nowadays, you can basically control a lot of such applications directly from the webpage or a lot of other things. And so, yeah, just to look at another example. So, we can look at the QML example. So QML is language for graphical interfaces. And where is it? Okay. Too many windows. Sorry for the noise. Okay, my computer is hating me apparently. Because, okay, a simple text editor won't start so I don't know what to do. Well, okay. So these are just slides for people who are not here so that they can see basically what happens. And yeah, so in terms of interoperability, I already came here three years ago to make a presentation with another protocol which was called Minwi and which was entirely based on OSC. So OSC is nice that in that it only requires UDP but in some case, UDP is not enough when you need reliability. When you need, for instance, something to love, tough messages in a small amount of time, it may not be most efficient. So with multiple companies, we tried to find alternatives and so we worked for this new protocol called OSC query which adds a query layer with WebSockets and JSON on top of OSC. And so we're a lot who made many different implementations which all work with the original OSC stuff and you can find here the specification. So yeah, so to give an example of the software that are compatible with OSC queries, so there is, for instance, VDMX which is fairly used video mapping software and video, well, VJ. Vezer which is an OSC timeline so you can send OSC message in time and Milumin which is specifically made for video mapping and so projecting video projections and stuff like this. So all these software generally don't work on their own, they require communication with other software and up to now the question was how to make this communication work really ably and well, the answer was this protocol. So yeah, what I said, JSON over WebSockets, it supports WebSockets directly and has various features that I can show in a web browser. So here, yeah, here for instance I am connecting to the pure data patch which is making these other variable little buses and I can see thanks to Firefox's ability to display JSON neatly, all the parameters of my patch. So for instance, I have a say modulation ratio which is a floating point value, the current note which is two floating point values and currently 38 and so this kind of stuff, yeah, the gain which is interesting, it's in decibels so there is a unit system you can say, okay, this is in decibels, this is RGB colors, this is Regen and polar positioning. So you can add a lot of metadata to your objects which allow this software to make interesting choices. For instance, if you have an RGB color, you can send it HSV values which will, of course, convert into the correct color spaces. So this is basically the general idea of Libosia and on top of I see it also adds some kind of publish and subscript mechanism which allows to say, okay, I just want to listen to one or two address amongst all the possible address that are available and the interesting thing is that all the current implementations work with zero conf so it just works basically, you see all the available devices and you connect to it. So next, I will move on to another small example. If I can kill everything, that is, okay. So this next example will be less sonic and more graphical. So if you don't know processing, it's a graphical, well, it's a textual language based on Java which allows to make, well, nice visuals very easily. So here I have a, I took a simple example that comes by default with processing so credits to its author, David Pena and I've modified it to add Libosia binding. So here for instance, I don't know. Can you read the text here or is it too hard? Okay. Okay. And so for instance, VAPI looks like create a node which is called wage which is a float and set it between zero and 50. So again, if we go to a web browser, okay. So it's not the same IP. Okay, so here we will see the nodes that we created in Java and well, basically the same thing. And the question arises, what happens when you have like multiple applications like this, like the one which made sound and the one which makes graphics and how do you control them when you have a lot of them? So generally you make what is called control surfaces and, okay, example do. Yeah. Here there is a software which is called, which is fantastic. So if I is basically preventing my stuff to work for some reason, okay, fantastic. Okay. Well, that's the best demonstration effect I ever had. Okay. Cool. Let's see what did go wrong. Sorry for the annoying thing. Okay, so apparently I have some network entrances and since everything is working through the network, no, no. But it's using Avahi and I don't know, like Avahi didn't like that I connected to the local Wi-Fi apparently. So, I'm sorry. Avahi is not. Yeah, yeah, yeah. Yeah, yeah, yeah. Oh, it's blocked on the local network. Oh, okay, so thanks for the explanation. Okay, just restart the network and, okay, goodbye. Can't see them at all. Okay. Well, okay, I'll just move on without the example I think. Sorry for the noise. Okay. Well, I just move on like this. So, the idea is to have this kind of graphical stuff with parameters that you can control and, well, the problem is if you want, yeah, to map these parameters with say, well, to control them. If it's only code like this, you don't have any kind of graphical interface you can. So, you have to, well, control them by outside and so we have a remote software which can see automatically when Bonjour works over the network what is available and you can just basically drag and drop all the parameters and they will show up as the correct parameters. That is if it's a float, it will show a slider. If it's a boolean, it will show as a toggle, this kind of things. And then you can of course have multiple devices in the same environment. So, this is the remote which is also free software and available on the OSIA repository. And then there is the main meat of the thing which is OSIA score. And okay, I just stop the network manager. And I just try to stop the network to see what it gives. Okay. So, here, yeah. I don't have any kind of local host. Okay. Okay, I don't even have a local host working. So, do you mind if I read with my computer because I think like there's something which is seriously, so which seriously went wrong at some. Okay, it's the HDMI cable which, yeah, it's HDMI cable which did that sound. So, yeah, that's fairly weird. Yeah, just so best presentation ever. I'm sorry. You tell us that it's a lot of work to get it right. Well, yeah, and I spent a lot of time today making sure that it will work right right now and rehearsing my demos. So, yeah. So, there should be okay. And okay, I don't know why but the most basic IP address doesn't work. So, well, I just move on and. So, so OCA score. So, basically when I came here like three years ago and I had this slide and well, it changed quite a bit since because basically the main difference in then is that we added an audio engine to the software. And so, while it used to only send OSC messages to the outside world, now it also supports some playback. So, it can be used by musicians and so this and the formal model is based on directed acyclic graphs. So, like your data, like Max and like a lot of software that hide it behind a nice user interface. So, and sound and I will just stop trying to make my original demos work and just make mine on the fly because of course, that will work fine. Okay, so this is how the software looks. So, there are three parts to the user interface. One is the remote devices like MIDI devices, OSC devices, this kind of thing. One is the score. So, basically where you write the progression in time of your objects. And one is the inspector that is. What kind of object are we seeing? So, for instance, this is a small dot which just says that there is something at that point in the timeline. And this is the way to say, okay, something will happen for five seconds, more or less. So, if any of you know usual sequencers, well, it works mostly the same. And, yeah, up. So, basically, the big difference since all the version is that nowadays you can go into music. Yeah, okay, say this for instance. And have a nice sound. Okay, I'm sure you all heard that. Woo. Yeah, so since the screen is crowded, I'll just move this. Okay, so, ta-da, it works. Finally. So, the big difference between iScore and other sequencers is that you can say that things will depend not only on a fixed time, but on various events that can happen. So, for instance, here I can say, okay, this sound will last until something happens. And then we move on to something else. So here, if you can see, it continues because I clicked here and it told it, okay, stop here and continue, which is fairly useful in show situations. For instance, if you have dense search that are too fast and there is a fixed soundtrack, well, you can say, okay, this part will move on a bit faster than the rest. And so, there is this main feature. And another very important thing is conditions. So you can say, for instance, a part will run like this. Okay, and yeah, let's move on. According to a condition that is, if something is true, if someone, well, pressed the button, then we can move on to the next part. Else, we won't play it at all. So, uh-uh, right, something I can say just for the sake of the example that it will never be true. And these parts won't execute at all, which is basically standard logic. So it's a if condition, but, which is graphical. So there's a whole visual language that has been made for the sake of the software, and which can be used to determine various outcomes in musical scores in this kind of stuff. So, and another new thing is the support for FOST. So I don't know if any of you know FOST. Okay, so FOST is a programming language that was developed at Grenoble in France by Doug Ram, and which is especially made for high performance sound, basically. So I may have some example online. Yeah, so FOST code looks like this. So this is based on functional languages. So you basically combine operators, and it's very close from point-free programming if you know what that is. And so you can easily write sound algorithms that will perform very efficiently because FOST comes with a compiler that leverages LLVM to produce fairly optimal native code for your 20 CPU. So for instance, in score we can say, okay, I have my preset library with various FOST sounds, and I can take, say, campus, yeah, this one is nice. Okay, nope, no, no, okay, okay. So, and at this point, for instance, I am going to play an instrument, and of course I want to keep playing it, so I will just say, don't go, well, keep waiting until I tell you to stop playing. So when I am here, we hear it just a bit. Well, that's fairly subtle, I'm sorry. But basically, yeah, it's like VST plugins, but they are compiled for your specific CPU model and architecture when you add them, so it's quite resource efficient if you need to have a lot. And it also works on so many architectures like ARM and x86 and well, all kind of stuff. So for instance, I can also take my original crash sound and add it some kind of reverb. So, yeah, there is reverb. And here I will say, okay, let's take the output of my crash sound and wire it to the input of the reverb. So, this looks pretty much like pure data. And, okay, so I don't know if you hear the sound of the crash feeling more and more. And I will just now keep the reverb sound. So here we only have the reverb part of the sound and I will just kill this small synthesizer here. Okay, goodbye. So yeah, so these are the main new features in SCORE, which are basically some processing and a graph of the values. So I will try to see if I can have something that works with, that this is wrong also. So processing, that makes that weird noise. Okay, so this is my processing patch and can I see it? I would really, really like to see it. Device, okay, so things start working so I don't know what I restarted, but okay. Okay, and no. So it doesn't actually see it and this is fairly distressing. Well, okay, localhost maybe. Okay, well, no processing then. Great. Okay, well I can still send it manually OSC addresses. So we just heard the ending of the reverb queue. Five, six, two, eight, I think. Yep. So even if this doesn't work, we can still do it basically with the old OSC way, which is sending parameters directly to the OSC port. Okay, and there you can say, okay, I want to create a new OSC address, which we will call wait because that's its name here. And a child, okay, wait. Load, which will be between one and 200. So this is all the part that we have to do manually because the other protocol apparently doesn't want to work today. So where is the big thing? Okay, so the sphere is here and if everything works correctly, eight, okay, so this work and I can send OSC messages like this to well control the weight of my processing thing. And I can say, okay, after this sound, let's take this weight and it will go from, it's from let's say zero to 200. And, okay, let's move on. And here we have a simple automation which runs and does some kind of a nice flow and yeah. So this is the first thing that we can do, but we can also, for instance, try to map sound parameters to, well, to the controls. So for instance, we can say at some point we will start playing a drum loop. So I will add a loop and in that loop we will play a sound. So I will just need to find a sound to play in a loop. So maybe not a snare because you will hate me. Yeah, a small hat, for instance, yeah. Okay, so this should be quite short. So here you will see that when you, so here we just play a small thing and basically this allows to do stuff just like most looping environments because you can end up with the same kind of behaviors that you have in life with multiple loops that are all synchronized together and this kind of stuff, but I don't have time to build them. So we want our loop to keep running and actually I think that I have some better drum loops than this one. Okay, yeah, drum 12. Okay, this one is much better and we will use the sound of drum loop to change the graphical object. So, we want this to last like three seconds. And, okay, so this is fairly basic drum loop and now we'll say, okay, let's take the amplitude of that sound. So like this, library process, audio envelope. So we had the, we take the output of our sound, we put it in a plugin which will compute, you know, the root min square and the peak volume of the sound wave and we'll say, okay, this will actually control this parameter. So I can just say, okay, the weight goes here. And here it disappears. Why? Because the envelope of the sound is fairly small and we need actually to make it, well, to multiply by some value to get something that shows up on screen. So instead, we will first add something that allows us to do some computations. So, I went far, envelope and custom mapping, okay. And I take the output of my first sound here and the output here. And here we can basically do some inline computations of various kinds of mathematical formulas. So I can just say 10 plus x times 100 which will just well apply to the formula. And it also use a fairly optimal basic, I would say, engine, mathematical engine which does just in time compilation of the formula. So it's quite efficient actually. And I will just wait. Okay. Okay, and here it's much smaller. I really need to multiply it by much more than 100 actually it seems. So, 1000, 1000 we start to see stuff happening. And 10,000, and 10,000 it actually starts looking like a otaker visual clips well, just a bit. And okay. So, and the thing, so as you could see, and this is also something that didn't work three years ago is that you can basically edit everything while it's running and it doesn't crash which was not always true. Okay. And so you can do this and you can also since we have this kind of things, we can also control the colors. So for instance, I can say, okay, let's make a color gradient like this, so automation color. Okay, and we will add another node for the color of the of the processing thing. And child color, I think it's called, which is the curve four. And we will set it as a RGB color, I think I'm not sure of the actual unit used by processing, but okay. Okay, so yeah, it's not a RGB. It must be RGB eight, maybe. RGB eight. Okay. Okay, so I don't know if you see the color changing a bit of the, maybe I should set it to another color to make it more visible. Okay, it actually doesn't work. Great. It's certainly not the right name. Okay, ARGB, ARGB eight. So some software work between zero and one for colors and some others work between zero and 255. And I don't remember which is which. So yeah, fantastic. Just check the processing code. Yeah, I don't know if it's okay. Well, okay, what else to show? So basically all of, so this is a three-part inspector, so you can see the user library with some things contributed by the user software. And the internal capabilities, which are all of this. So for instance, another useful stuff for art is step sequencers, so like this. So here, for instance, it will start doing sequences on the previous parameter up until I say it to stop. And we have to tell it that it has to be, for instance, between zero and 300. Okay, and here you see that the weight changes basically. And then it moved on to the next part, so I have to start again. Okay, so basically like this and it can change fairly, fairly quickly. Let's say like this, for instance. Okay, so for instance, stuff like this. And well, it will just make some kind of nice motion, things like this. And so the last thing that I wanted to show, so this was basically a previous example, so first we did it. This current example and the last one is another example which won't work because of the network issues, which is a video player, which is also available through OSIquery and which use MPV for playback, which allows to play fairly smoothly a lot of HAP videos. So do you know the HAP codec anyone? Okay, so for instance, HAP is a codec which basically use GPU frames and codec so it basically can very easily be scrubbed because it really doesn't need to do any decompression. It's a video card which does decompression and I can just show you what it looks like. Oh, first time. So video player and for instance, I can launch five instances and it will play back five. So these are 720p videos and they are all GPU decoded and well can be controlled over the network so they can be scrubbed. So yeah, the big interest of HAP is that you can basically scrub extremely easily so there is no delay when you tell it to scrub, it just goes at the right frame unlike X264 or five, which have some time of decoding because it has all this B-frame problem which you don't have with HAP, every frame is a full frame, so. And again, so like earlier, we've worked with the VDMX guys who also made the HAP codec and well to get it to work. So yeah, and it should have looked like that with score which controls multiple videos and tells them to scrub at various moments in time. So, and for the remaining things, we miss scripting with LLVM, so there is an ongoing extension on the internet. Do I have internet now? So a plugin which basically, so most of the software is written in C++ and right now we make releases that are well targeting the baseline computer of our users. So with this, we can also use LLVM which we already used for first to basically compile at runtime the some of the execution part of our engine to make it more performance on the user's machine and allow easily to distribute add-ons, so you just send the source code on the user's machine and it compiles the VC++ code and executes it immediately. So this currently works on Linux but not really yet on Mac and Windows, they're almost there. So the video playback currently is out of process so we'd like to move it in process and have more powerful shader capabilities on it so like transforms and if possible combining multiple videos. There is a work in progress part to web assembly so the main user interfaces they build currently to start but the execution still fails because due to the specter mitigations last year all browsers have disabled web assembly threads and well threads are currently in debt to make it work so we are stuck at this point and fix them back like today, okay now I know that there can be nice network problems and all the links so of course everything is open source and well you can just grab and well take it so it's on Mac, Windows and Linux so yeah and there have been quite a lot of artistic installations done with it up to today so and the carousel so I'd like to thank Pascal Balthazar who is the main user of the software and who did a lot of bug reports, Pierre Couchard and we don't see, there isn't his name but Thibaut Keller so there's a community of users who keep making artwork with it and making bug reports and sometimes even contributing code Antoine Villere who made the pure data part so yeah some people are working on making this actually usable for installations and well multiple things out there in the wild run using OCR score so maybe you seen some I don't know but yeah that's the overall thing so thanks and I'm really sorry of all the technical problems it sucks so thanks everyone and thank you No question at all, no second thoughts I too want to take a small beer too We have one question, I'm coming but slowly Hi thanks for your presentation perhaps because of all the network problems perhaps you can elaborate a bit more about how it works Yeah yeah well it's it should work simply that is Libosia basically creates a web socket server in the application and so there are two parts there is the zero consparte bonjour which doesn't use TCP and UDP it really passes through DNS requests which is a whole kind of form and once this is done well there is a simple web socket connection once the connection is established we receive the same JSON blob that there was for instance here we can see the video one which okay no I'll just check well okay so like we saw earlier we just get a big JSON blob and then we create on our side the OSC namespace basically so yeah that's all there is to it and what do we need to do to integrate it well mostly so there are a lot of example on the website so I'll just show you the simple C example for instance so the library itself is returning I'd say post-modern C++ but you have a C API if you want to make it work in other environments so for instance you can go to documentation so but are there dependencies? only the library it's a I mean the library itself like the C example you just showed yeah does it depend on? yeah it depends on the library but you can also implement it yourself but we found out that in Vienna you need OSC clients and server and WebSocket clients and server so yeah so a simple C example for instance which would expose a few nodes over the network no okay I don't know why it wants to so now this is a C++ example it's not the one I wanted documentation, tutorial C so there are a few the same thing made multiple times so yes this one is pure C and you just okay say okay I want to create a server I will create nodes in the server which will be for instance of type float which will have the value 3404 and then you can just from your server continue to push value so it really works like many OSC so far and on the other hand you also have all the client API which is okay I want to connect to this device which is at this address WebSocket, localhost, blah blah blah and you will automatically here get all the parameters of the client so at this point you can just say okay I want to find the original node and well maybe the connection didn't work and you don't have it but if it did work well you can be assured that you have the same it's a mirror of what exists on the server basically so yeah it really makes a copy of it and then you can push on the other software and it will send the value to the original one and an interesting feature is that you can choose if you want for your parameters to be sent through UDP through OSC or through TCP so for instance if you have a say play message that starts a video you really don't want it to be lost due to UDP packets getting lost because it's always a fan so you can say okay this one I wanted to it's safety critical I wanted to always go through TCP because at least I know that if I have some kind of internet connection it will get through so yeah that's mostly the a gist of it and yeah in terms of conceptual model it's just a tree of parameters with properties and all right and so if I implement this in my own application then it will show up in this this score application yeah in score and also in all the others that I've shown I've shown in the slides so for instance it will show up in yeah Vezer which well is fairly similar to score video mx they all support the protocol and they will all see it so we spent a lot of time with these companies making sure that interoperability worked fine so yeah and but I was wondering about the GUI application because it looks amazing which one the the the score application you were demonstrating oh yeah however it seems you're talking about an interactive installation and the the application seems time-based oh yeah but there's a small trick which is just that if you want to make an installation so here for instance we have so item lines so like a Cubase or whatever but this is not a necessity at all and we can instead have a loop so process and structure and loop okay ah you inspector okay mm-hmm here we go and here for instance it will just keep running forever basically not like this like this and here I will just set a small duration and for instance fade this okay okay I've lost audio it seems okay this works and yeah so here it it will just keep doing the same thing over and over again so if you have if your installation basically needs to keep running you just do this and well it will it has it's loop logic basically in here because that's how you will do it in C for instance you just make a big loop while while we can receive events then we process events and that's really the same thing but it's graphical basically so yeah that's how you will do installations and yeah it works thanks well thanks you any other question well thank you so again thanks everyone and thank you everyone for the dev room