 After almost a little bit of baboon, after all the graphics talks in the past few days, I was in all the attention to the X-Wizards system and again to wondering why I'm using the microphone, I'm just doing a live demonstration of the system just to show that it's real and not vaporware because everyone likes light signals. First, now to understand what I'm located from and why I'm using this fine decoration, that stands from a research project originally done by my employer, but I'm talking just now about this research project at the time, to make this into something useful otherwise because code written further to the paper is something different from code written than actually useful. Okay, what I'm running here currently is this small application I wrote. It can check my voice through this microphone I'm playing right now because you have to use the voice in the only connection and what I can for example do is activate a few of the effects. I don't think I need to explain this too much. Okay, this is fairly audible. It's low-mass filter and well, it's not much left of my voice after that to just make on. What I also can do is, for example, test the audio mixing. So if I just start the second audio application, pick this one. Well, then my voice still is all mixed together. And by this small application that I posted by Tray, it's the audio manager that I wrote to. The window manager is determined how multiple clients currently accessing the audio device are multi-packed onto the output. It can do a number of effects as well. For example, clocked-term client volume settings is quite easy. Or it can shut off the client completely or just kill this. I can do other funny effects. This is one of the primary applications I'm talking through right now. So I'll just spend all of the years with those two speakers. So my voice should be there. I guess I'm moving up to the left and right. That's just a reverse here on the overhead to my desktop speakers. But I think I will just start with the microphone for now. Because I actually prefer strolling around while we're talking about this. Okay, so now this is the extension. What you would probably expect, what I would like to note for first is that audio processing is performed only inside the X-server. So some data is not passed back from the X-server to the client application doing effect processing and not pass back to the audio manager doing the multi-packed mix down to the client audio device. It's processed under control of the client but inside the X-server. For obvious reasons, I still don't want to do any latency by passing sample data around but I want to process it inside the X-server. What you would probably expect is that I'm building a sort of filter graph or a filter chain because this is basically every other audio system does. I'm not doing that with this filter graph. The process tends to lead to a plug-in inflation. You just have to keep adding more plug-ins and it never ends. And of course I don't want the whole of plug-ins to be X-server. I don't know what that does. And I've done something much more simple and I will explain that on the following slide. So, first of all, I've written not only one extension but these are two eight extensions that are active here at the moment. First one at full time, the other one at full audio. I could have implemented this single extension. Of course I did not just do for the sake of presenting the clarity of concept that these are really distinct concepts. The time extension code has a lot to do with the single extension. If you don't know what that is, don't care. It's not one of the most important extension systems. I use this for work to provide the time services for processing my play. The only extension that's all the work that I fully play, the capture and this processing that I demonstrate. So, first we'll just give you a short overview of what I need of this time extension code for my timing for the process. This extension introduces two new server-sized boxes and two new server-sized objects. Clots and schedulers. Clots are what you would intuitively expect them to be. It provides time information. You can start a service period. It runs independently from any client that provides tips or something. And the second more interesting concept is that a client says scheduler is always attached to a clock of one or more scheduler, zero or more schedulers are not attached to a clock. And it permits the client to submit operations to a scheduler which, unlike normal X requests, are not computed immediately but they are deferred to a later point of time. And it's the best time sense that you find when or when not these operations should be included. It's highly time-sensitive because they want to have operations piled up to the time you can serve a total of a specific error that can't handle or request. The operation that I can perform for scheduler is also processing. I'll return to that. But there's only that. Actually, I can also do some render operations. Yeah, render operations pertain to render extensions which also have other operations that are most interesting. I don't need them for audio. I'm just mentioning them because they are in the implementation and I need them for other work, for video work that I've also created. But I don't want to talk about that. Just mention that there's not a purpose of those various generics that you could do more than that, more than just for this project. Okay. The audio extension also includes a number of new service sites to resource the process. The first is a sample buffer which, as the name suggests, is used to start the data inside the server. It's a one-dimensional data structure so it takes not to store pictures and start the buffer to store samples. It's one-dimensional, so it accesses to a single index value and it stores samples, values for each index value or for each index value. The value stored on scale-up means that you must do multi-channel audio or stereo audio. You need two sample filters, one providing data for the left channel, one providing data for the right channel. I've done this decomposition because it creates a single file on the application interface and, well, otherwise, it wasn't the active part of the process. I tried this in the beginning and it ended up sitting by this way. The sample buffer only stores sample values but it's no temporal dimension of each sample. This means that they do not store any samples or anything. So the server just stores samples without knowing which, according to which sample rate they should be set. This is as later as using a different mechanism. I'm just mentioning this here. I'll return to this later. The extension supports several mechanisms to get data into and out of sample problems. Of course, you can upload, download samples simply through the expert code. Much like you can upload pictures, some elementary functions and fields and arithmetic functions. These are really simple functions which are tritical, tritial, and a little bit. For this example, you might need to accumulate a little bit of a range of samples from one sample buffer, a range of samples from another sample buffer to the multiple pairways and then plate or accumulate it into a boot to the target sample buffer. So all these arithmetic operations are of the similar quality like the operation I just showed on the right. So they are really simple operations to perform for the only extension. Another type of operation I can perform is a convolution operator, which it allows me to implement finite and constant responses. This convolution operator is a little bit more generic than what you would find in textbooks and digital science processing is a little bit generalized with this great convolution operator. This convolution operator is also used for re-sampling. I've already said a sample buffer doesn't store any associated sample rates, so a client has to track whether the sample in a given sample buffer has to be interpreted as 20 kHz or 44 kHz. The server doesn't care. It doesn't know. So if a client wants to transfer sample data from one buffer that the client knows to be interpreted as 20 kHz into a sample buffer that the client knows is to be interpreted as 48 kHz, the expert simply has to issue a request to transfer the data and to re-sample at the same time. And this is usually done using convolution filters. So the server also never does any interest in sample rate conversion. This also delivers in the design because unlike for images where you can use, for example, built-in interpolation as a safe default that gives good results, there is no such safe default for audio that you could really use that very much depends on the application scenarios. So I decided to pull this out of the server and provided that the client just had to do this and to issue an important request. The sample buffers can be used for static sample buffers in that they store sample values for static range I think, but they also support the sliding window concept for streaming audio. We'll try to illustrate the concept here a bit better, I hope, than with the RISC alone. What I can do is if a sample buffer represents sample values for the index value and I can change the window of interest of the sample buffer to the state code 311 and as a result these sample values they drop out of the window and their values are lost. And on our file samples they shift into the window they initialize to 0 but you lay the load you can later go and replace other values into the sample buffer that you should using the mechanism on the previous slide. You can always extend all the bounds of the sample buffer it's a well defined thematic it's basically the same as if you withdraw out of the basement or something the data is lost or keep it back there you can adjust it to blackness it's well defined and it's always possible. Now, this sliding window concept or this window shifting concept I would include that for standard reading the principle of the sample buffer the concept allows random interest to any position in the sample buffer at any time so I'm not using a simple 3-4 buffer this is important for the loss and recovery of simplicity and this sliding window concept provides a good recovery between the producers and consumers of sample data of course the roles between producer and consumer may change sometimes the clients the producer and the server the consumer are the other way around depending on if you're capturing or playing back and this allows me to decoupling of that note that excess within this range is uncreated by the shift so this overlapping range of all the new window that excess is always safe and so if anyone does control excess in this range nothing happens to them so you're always safe if you shift the window at the rate to make sure that all excesses of all concurrent excesses fall into this overlapping range due to safe on clients may have this little shift operation has to be performed by clients the server does this not automatically it's a client request there's a client request form on a sample buffer and well there are multiple options of course the client may have to use this and make sure that the window is always played correctly you can do large window with infrequent lazy shift or small window with aggressive updates it all depends on the client and for the simplification requirements so the server is simply done here so now that we can do let's be constored then process or compose a sample base inside the server we probably want to do something else with them and that is probably played as a capture this is achieved through to other server side objects with this PCM device or well should be quite obvious what this is something that's capable of capturing or playing PCM data and the second is the PCM context this is essentially a configuration of a PCM device so a set of parameters that can be applied to such a PCM device contains information such as sample rate may contain other information such as a local playback latency and other things and it also references typically it references one more sample process and points the position inside the sample process which serves as the point where playback or capture begins and once the function is activated it will just start reading or writing values into the sample process beginning from this position notes the client has to do window change to make sure that playback never slides off of this window otherwise he will read whatever through this to create a layout and read back zeros which equals filers in the case of your loss of superdensity the important connection point to the time to the time it's written at the beginning is that each PCM context can also serve as a time source for scheduling operation so this means I can attach a schedule to such a PCM project submit operations to the schedule and these operations the synchronous to playback or capture of sample data to PCM device and of course this is what the client application that I demonstrated at the beginning actually does submit a request to the PCM schedule and realizes the effects in this way for example for the reverb effect I just added multiple copies of multiple small doses behind the main using modified Q and A is very easy to do to find out the cost response buildup or the low cost buildup the implementation tries very hard to satisfy all of this timing request actually it's usually it kills most of the client's request in a dedicated real time spreadsheet to make sure the timing requirements are really met of the return to this so now just the Q and A now just the Q width or the depth of audio mixing model the way it works is roughly shown in this figure the audio manager goes and sets up playbacks for the PCM device from a sample buffer created by the audio manager or multiple sample buffers created by the audio manager if you want to use stereo audio obviously and all the other audio clients that want to play back something to create their own sample buffer solution they provide the data they would actually like to play back but instead of letting the PCM device reach from all of these sample buffers the audio manager has a responsibility to machine the compositing operation that transfers the sample data from these play back client sample buffers into the main playback sample buffer and this model offers a lot of flexibility because the audio manager can basically do what he wants with the sample data to make more clients or it can do for example positional audio as my demonstration beginning by attenuating all that the right channel accordingly a lot of options something that I did not demonstrate but that serves as a possibility to transfer the device switching if I just attach another audio device and add it to the user to track the audio manager it can just pull the PCM device under the client the sound playback the audio device and the nice thing is it requires actually no cooperation from the external audio service that knows or doesn't care that we just switch the main playback device the audio manager can do this to provide the words that the new device has appeared something like that now a few rather random affection notes on how this works in the outside the extension itself is really tiny it's about 8000 microsecond 8200 something and this report all of that I have just presented this also includes the ability to send a time render request so if you factor this out you are well under 8000 microsecond I try very hard to implement it without any we choose into the 4x architecture simply to make it easier for anyone who wants to avoid protecting the person you can basically just combine it down with those dynamically but there is one cable to the digital press because I just have to start a dedicated audio processing press usually with real-time or otherwise they can't met a usable time requirement so you just have to re-link the server against the threat statement although on Linux systems you can do a neat LD preload strip so if you're LD preload to the correct pthread library it will also work yes otherwise I'm actually not really touching port code I'm not really touching the threading model but the ex-server is a dedicated audio thread really only done audio processing and nothing else or the drawing is not controlled from this the response back to the main ex-press the need never arises about the 3rd time audio the server tries very hard to execute client request in the client frame this usually means that the client operations are in the threat performed with real-time priorities so it is open to windows for clients to execute code in a limitless way in real-time threat of course I need a little bit of access control for that first of course to try to get and also to thread client from locking up the server completely with those ones the process that doesn't have can really run real-time just make this work for me and the approach that I've chosen here is that I approximate the cost of the request for the client or that would have to be executed and according to the response model schedule them to real-time threat or deny schedule them to real-time threat or maybe dropping them and this is very easily possible in this case because remember the operations that I'm doing are really simple they are always on the scale of multiple accumulations do a fixed set of multiplications do a fixed set of additions they can actually compute the number of instructions you need to do this very precisely and you can actually compute the number of cycles you're going to spend even if it blows your cash completely because it's just not in there you can compute that fairly accurately that's a lot of unlawful experiments with that if you're overestimating the typical case of a factor of 2 it doesn't matter because it just means if you're overestimating a factor of 2 it only means that you can get 50% of the CPU processing part that you've spent in the real-time threat you don't want to spend 100% of the CPU processing part in there anyway so overestimating 30,000 you really want to do this with a low 10% to detect a potential or unintentional overload case so we're now at uprising this approach to it's very important here if you have any personal ideas we'll just reuse a lot with the infrastructure you actually end up implementing or re-implementing it and you also have to implement a security for example if you go into the XO and just enable the Ignace is something that you get security for your attention automatically for free so you don't have to worry about that and what you also get on the video synchronization if you want to do that you can actually pre-schedule with the audio as it works as it also works reliably across the network I've tried to make the audio model mimic the X image and video processing model quite closely so well the consumer artists are quite intentional try to maintain the quality of the server and ship all the processes to the client but they should not neglect to tell you that in fact gets very complex because you have to do a lot of planning ahead and maybe if you decide to pay a lot of dollars anymore want to do something different change the mind roll back to do something different and well I can get so that we can see through this mechanism it's quite suitable for this for audio processing I can play for example to ship memory and in that case they found the purpose of the data there is only full by the real time threads running through the X image or any other instance to ship memory to communicate with the real time threads so it's quite suitable now we have some graphics drivers that are isolated because it's exactly the graphics driver that we have to be able to achieve the latency so maybe I take your curiosity that's the case you can try it out it's all there and pretend to put in the manual it's not done yet but I will do it Monday night late so otherwise if there are any questions I'm here for interrogation I don't know if we may still consider the time for that and otherwise I would just thank you for your attention