 We can have a nice afternoon together. And if you didn't expect WebAssembly, you're probably in the wrong room. So we do some WebAssembly today. We will probably not be very in-depth on how a for loop would be transpiled down to almost assembly code or something. We will look at it from a more use case specific point of view. So we will do some actual use cases with WebAssembly today together. We will use some Rust and some C, but I'll get into that right afterwards. First, let me quickly introduce myself. So my name is Patrick. I am from Switzerland. There are some guys from Switzerland here, right? From my own company. Hello. I started programming back in 2005. I did an apprenticeship. I stumbled kind of into PHP. I didn't know at the time what programming was. So there was that thing called PHP. Somebody on Stack Overflow told me, put it in a file and put it on a server, and then it will just run. And actually, it did that. So I was quite fascinated at that time. And later on, I wanted to do something in the front end. So now, when the user clicks something, something should happen. That would be nice. So I stumbled into JavaScript with jQuery. Had to support EIE 5.5 and 6 at that time. It weren't nice times. So if I think about our browser problems today, they're much nicer than they were back then. So everybody who had to support IE 5.5, 6 knows what I'm probably talking about. Yeah, so I have grown to become or I became a JavaScript enthusiast. Yay. I do work for a company called Sulkem. Sorry for these funny dots under you. That's just the German language, right? So that's the last name of the guy that founded the company. So we are a Swiss-based company. We were founded in 1968. And we do mostly software consulting and business, like agile business transformation. We started off in Switzerland. But nowadays, we are kind of global. We are operating in eight countries. So don't worry if you can't read the names. So the basics are we are in Middle Europe or Central Europe. It's called a bit of Eastern Europe. And two years ago, we started right here, Sulke Asia. That's why we are here today, or why I had the opportunity for my company to come actually here to Asia, which is very cool. So we have Sulke Singapore. We started two years ago with zero people, right? And now we are at 36 employees. So we are growing quite fast, which is very cool. And actually, we want to grow more. So we would like to reach a target of 100 employees at the end of next year. Let's see if we can reach that goal. There's also Sulke Hong Kong, which is a bit smaller. It's still a bit fresh. So it's 1 and 1 half years ago when it was founded. And it's currently at 20 employees. So a bit smaller than here, but still a nice little team. So if you're interested, if you want to know more about what we do and how we do it and where we are all from, we are these guys with their shirts. Now, if you took one, you have one of these shirts too now, right? But the guys that wear it now, where are they? Behind, back there. All these guys are working for Sulke. If you have any questions, if you want to get in touch with us, just come talk to us. We actually like to talk. So let's get into it. Oops, voila. So our goals today are that we can all kind of say, what is WebAssembly all about? How can I achieve new native performance in the browser? How do I send the code to the browser? How do I interact with it? That's basically the most important part. So how can I send data back and forth between my JavaScript and my C, Java, Rust, Go, whatever language code I want to have running in WebAssembly? And in the end, we're also going to have some fun, like do some exercises that you can just fool around a bit and see what it's actually capable of. That's basically the schedule around here. I thought we might want to have two minor breaks, so we are not completely down in here, and we can get some oxygen from the outside. We will first look at some basics, a bit of theory, and then we will get into our first exercises. And then we will have a look into more in-depth use cases. So how does it really, really work if we're going to pass strings, if we want to work with 64-bit integers and stuff like that? And in the end, we will have an experimenting exercise with OpenCV. I will explain that a bit in a second. Exactly. So if you wonder if you never have seen that, this kind of presentation is with the framework Reveal.js. I guess most of you have already seen that. I think a lot of speakers actually use that here at the conference, so it might not be that new to you. So let's jump right into it. What is WebAssembly? So from now on, I'm probably going to call it often Vazem because it's shorter than WebAssembly, but it means the same thing. We will see Vazem throughout the whole workshop, so don't get it. I hope it don't get you confused or something. The cool thing is why WebAssembly also really took off in the last couple of months or years is because price support has increased a lot. So you see maybe two years ago, you would have to start Chrome with certain flags that WebAssembly is actually available to you. But nowadays, it's readily available in all the browsers that you actually want to use. Obviously, since IE11 is not being actively developed anymore, IE11 can't handle it, but Edge, Safari, Firefox, Chrome, they all have a good implementation of WebAssembly 1.0. We will talk a bit about that, what is going to come in the next couple of months or years. So this is the first WebAssembly version. So what is Vazem app actually? So what do we have to think of when we talk about WebAssembly? Basically, it's bytecode for the browser. We can actually send a binary to the browser, and it will run in a VM with near native performance. So whatever we can compile to that format, we'll be able to run in the browser right away. No need for any backends, no need for any other calls. You get your binary, you tell the browser, run that in that WebAssembly VM, and it will just work. We will see that later on. It's actually a first, I think, or one of the first times that all the browser vendors, including Apple, were really able to sit at the table and say, this is the format we all support. There are no prefixes. There is nothing else in it, so they really all have the same VM, well, the same implementation of it. That's really cool, in my opinion. But you might ask, what languages can actually be compiled to Vazem, right? Can I use Java? Can I use Go language? So I included a list of a GitHub repository where somebody actually curated or has a curated list of languages that are supported currently. So this is basically just a list of languages that have a compiler that can compile to Vazem. So you see here there are hatched chickens and eggs. So the ones with the eggs, they are not there yet, but the ones with the hatched chickens are very good supported. So you see, the C languages, like C and C++, they are very good supported. We have Go, which is good supported. We have Lua, and I think BrainFuck, right? So if you want to write BrainFuck, go ahead. And Rust, that's actually what we will be using a bit throughout this workshop. Don't worry, if you don't know Rust, that, well, you don't need any in-depth knowledge. So we will have very basic small functions in WebAssembly and fool around with them, right? So you don't have to know how the memory management of Rust actually works, how all the details of all the types are. We will just have a very basic set, so don't worry about that. I hope it will not be overwhelming at all. So let's look at a small compilation example that we kind of understand what happens behind the scenes. So this is a little C code. Can you all see that? Is the contrast okay enough? Is it good? Okay. So this is a little C function that does factorial, right? You probably all know that if we make, if we have the factorial of five, you would multiply five times four times three times two times one, which would then be 120. So a very easy function that is recursive. So we call ourselves with N minus one, or yeah, N minus one. So we would first compile that in an intermediate format. So from there, it will be easier to compile it to actual WebAssembly. So you still see some of you might be a bit more familiar with code like that. So you can kind of figure out what that would do, or maybe you can at least kind of draw some lines between here and the C code we saw before, right? You might think like, okay, I see kind of there is the integer and we do something with that. And in the end, it will look like that. So obviously we are all able to read that right away, right? We are programmers, so we just see that like the matrix and we see, oh yeah, obviously this is factorial. Yeah, no, I just wanted you to show this is the actual format that will be sent over the line. So that you saw it. There's also a text representation of that whole thing. So this is what it looks like when we are using the ID later on when we'll program a bit. If you click on the Vasem binary, you will see that code. Actually, this is not what will be sent to the browser. This is just the format for us that we can actually read at least to a certain extent, you can kind of see what's happening. So what we actually do here, this is not what we will send to the browser. We will actually send the binary to the browser, not this string representation. This is just for us. So we can see the bit or we can at least read it a bit. So let's come to the limitations, right? So we have seen we can compile, cool. So can we compile anything? Does everything work? Do we have like any native function you could think about? Unfortunately, no. So let's first talk about memory. Memory management is a huge thing, right? We probably all struggled with it once at least. What WebAssembly has is flat linear memory. If that doesn't really ring a bell with you, this means pretty much you have one array and every element of the array is one byte. So if you would have one kilobyte of memory, you will have an array with 1,024 elements. Every element would be one byte. So fairly easy, right? You can just navigate on that array. If you wanna do some pointer magic, you can just increase and decrease your pointer and read any byte that you wanna have until you get some out of memory exception or index out of bounds exception. What it doesn't have, and this is a very important thing, this is also why you saw Java is not quite there yet. It's not a hatched egg. It's still kind of in that breathing state. We don't have garbage collection. So this means we have a memory array and if you are with C or C++, you are used it, right? You free your own memory, so that works. But languages that use garbage collection don't work out of the box. So that's a big thing we will talk about later. But it's work in progress. I will come to that point. So if you thought like you can send a binary to the browser and it can just run as a whole web application as of now, this is not really the case. We don't have access to all the browser stuff. So for example, DOM is missing. So if we have our C program, we can't go like document query element and then we will just get a DOM node. Currently, this is not supported, but this is one of the bigger issues that is being discussed and is proposed for one of the upcoming versions of Vasen. Then this is quite a high limitation if you want to do a lot of hardcore computing where you want to have the leverage of the GPU so you can actually be fast or have fast math. Currently, there is no GPU access and it's not really planned because it's a very hard discussion. There is so much to discuss about how you actually access the different graphics cards with all their different architectures and different programming languages. So this is something that is being discussed but it's not like as a big priority currently. So I'm sorry if you want to have your blockchain miner in the browser, not yet unfortunately because CPU mining is not that good, right? But also what's maybe a bit of a better actual use case that we have, heavy machine learning is often done on the GPU. So you can't do that right now. So when we do machine learning stuff, we are limited to CPU calculations. Then one of the biggest limitations that we will stumble across right from the beginning is there aren't many types in WebAssembly. That doesn't mean like your C code can't use all its types, right? C can use any type it wants. I mean, it has direct memory access, it just does what it wants. But the interface between browser and WebAssembly basically has these four types or these five types. We have integers, 32 and 64 bits. We have floats, 32 and 64 bits. And we have function refs. And this is pretty much all we have to, we have that we can actually use for the first couple of exercises. We will basically be sending integers forth and back. So that's kind of a limitation but this is being worked on too. So what we do when we wanna exchange other types, so we are back to pointer arithmetic, right? So what we would do is if we take a Rust example, we would allocate memory for a string in Rust, would write that string into that memory, then take the pointer, the pointer is an integer, right? So we send that integer through the interface, get it to JavaScript. JavaScript on its side has access from above to the WebAssembly memory. So what it does is it takes a pointer, throw it into that array, reads the string until the first zero because C strings, for example, are zero, null terminated. So it reads until it finds the first zero, does interpret every byte as a UTF-8 byte and voila, we have our string in JavaScript. It's not very cool, but we will do it later on so you will see how it does. So this is what we kinda have to deal with now but we will see that later, so don't worry about it too much. But what is coming, right? We wanna know, okay, these are the limitations but what should be coming? And one of the first things is also the first things I mentioned at the limitations. Actually, since garbage collection is not supported right now, all the languages that use a garbage collector kinda have to do their own garbage collector, then compile it down to Wasm and then send it with your actual binary to the browser. So the browser instantiates your Wasm code, plus that sent garbage collector that will then work in WebAssembly. So Rust does it that way, for example. Obviously, not very good for binary size and the whole efficiency. So what they are working on now is that you have some basic garbage collection in WebAssembly right away so you don't have to send your own garbage collector. We don't wanna be writing our own garbage collectors, right? Then another big thing is host by links. This is actually cool, so this would mean we can actually take DOM notes, send them to WebAssembly, or even create the WebAssembly, the DOM notes in WebAssembly, send them forth and back, transform them, update them, delete them. So the host by links proposal is very cool and would actually allow us to do a lot of things. Then what I didn't mention with the limitations, multi-threading, so threats should be coming soon. What they were thinking is that you can have an abstraction over threats, so not every browser has to deal with every, or your WebAssembly code doesn't have to deal with on which operating system it runs, right? And which way it does create a threat. So what they were thinking is you already have threats in the web with web workers, right? So you can already kinda have threats, so they would leverage what is existing and would take that to WebAssembly so you could actually have threats in the form or in the same way that web workers would work, or kite in the same way, not exactly, same way. Then one of the biggest problems, I think everyone in here has struggled with that, right? Exception handling. Every language has its own concept. We have a lot of various exception handling mechanisms. And we will see that later on, WebAssembly doesn't have one yet, so if you see code fails, this can be very hard to debug and kinda find out what actually happened because the exception is not like going up to JavaScript and we can just have a try catch around it. This is not working currently, but it should be in probably more of the far future, I would say, because if you go to the GitHub repository of WebAssembly and follow that issue, you will see there's a lot of discussions and this goes way beyond what I understand, so I don't even try to explain that to you here. Reference types is very closely coupled to the host bindings approach, so this would mean that we actually leverage that interface to have that any ref type so we can pass in various kind of types and references. This would be closely coupled to the host bindings, so both of them would kinda have to be solved for it to work properly and to a cool extent. And there's a lot more, so there's a cool list of that. I think, where is it? Oh yeah, that guy Colin Eberhardt, he has a very good list about that. If you're gonna have a look at that, he has really every proposal there is in which phase it is, what he thinks about it, what he thinks is the actual hard part about it. So if you wanna read more about that and you wanna really go into all the details, all the nitty-gritty details, check that out. Really cool blog post in my opinion. You will have access to my presentation later on so you will have all these links. Cool. So that's a very short and brief introduction to web assembly, the theory, before we go into our first exercises. So do you have any questions at that moment? Still feeling okay-ish? A bit tired from lunch probably, right? Cool. So let's move on, let's go to the actual exercises, the fun stuff, right? Cool. Where are we? Here we are. So let's try out the basics of web assembly. Well, I just kinda talked about we wanna build and run some code. That's what we are here for. At least I hope that's what you are here for, right? So we're gonna take Rust for a minute. So I will do probably the shortest Rust introduction you will ever get and it will not be in any way even closely to be complete. So we'll just look at a very, very easy function declaration, this one. Fairly easy function, I think what it does, everybody of you sees what it does, it adds one to a given integer. So this is hardcore math, right? We will be so fast with that and JavaScript will be so slow with that. No, obviously not. But let's look at the Rust function. So what is it? So the first thing we notice is that little flag up at the top of our function. It says no mangle, I hope you can all read that from the contrast. What it does is basically just tell the compiler, hey, please don't change the name of that function and don't remove it because you see that nobody actually calls it, right? If you would use the Rust compiler without that, he would just remove the function and the binary you would get is zero bytes because he says, yeah, you don't call that function. So I'm gonna optimize it away, good job. So with no mangle, we can actually tell the compiler, please don't do that. We need to be able to call that function by name, so please don't change the name and please don't remove it because, yeah. So with public extern C, we say Rust, that this function will be called from externally and it will be called by a C-like language. So what Rust does is make sure that a C-compiled language can actually call that external function. And this is what we will have when we call it over the WebAssembly interface. You would actually not have to write that C, that's kind of the default of Rust, but I did include it here, so you have the whole example so you see what's actually going on. So this is the function declaration. I think that's actually pretty similar to what you would see with TypeScript or maybe you are used to CoffeeScript or something. The only thing that you see here is the 32-bit integer, so we get one parameter in called x and we will return one 32-bit integer. I think simple, right? And this is the actual block, the actual function. So what you notice here, there is no return statement. If you want to, you can always add a return but Rust doesn't need you to. Rust will just return the last expression you have in your block. So if you're feeling lazy, today is the day. You don't need semicolons. You don't need return statements with Rust. If you want to do them either way, they will not hurt you, so whatever you like. Cool. But since I don't want you to have to install the Rust compiler, right? We have Linux, Mac OS, Windows. Certainly it would work on some of the machines and on the sun we would just have some weird issues. I'm gonna use the WebAssembly Studio, an awesome project by the Mozilla group or the Mozilla company. It is an online IDE that allows us to actually write Rust or C or I think there's WebAssembly script but we are gonna use Rust today. So we can write our code in there and directly compile and run it from there which is amazing in my opinion. So if I can ask you all to try to go to that URL, when I tried it last time, somebody was trying it with Edge and it wasn't working with Edge but it should work with Firefox, Safari and Chrome so I hope there's nobody wanting to hardcore use Edge. So you can try it. Yeah. Is everybody in the Wi-Fi? The Wi-Fi password would be over there otherwise. So Lasalle chairs comes with the Wi-Fi and chairs convention would be the password. When you go in there, you will get that little assistant over here. Please select the simple Rust project, the empty Rust project. We will go step by step until we kind of reach that lower hello world Rust project. Please don't open that yet because it will have pretty much everything in there but we will kind of try to go step by step until we kind of, until we finally understand what the hello world project actually does. So please go into the empty Rust project and let's see what we have there. Oh, so. Since I wanna try to code a bit too here, I have to do it like that. Okay. So if you open the empty Rust project, you will see that source folder over here. Here are the three files that we are actually interested in. The HTML, JavaScript and the Rust. So first of all, if you go to the Rust file, you will see that this is absolutely one to one the function that we just looked at, right? So nothing too fancy here. You have just heard about that function. Cool. If you click build up here, like there is a build command. If you hit that, you will see there is a output on the bottom. Task build is running and then build complete. What you will notice is now we have an out folder and there's the Rust file. So this is our binary. As mentioned before, if you click on that, you will actually see these S expressions that we were talking about earlier. If you really want to watch or I want to take a look at the binary, you can right click on that file name. And somewhere down here, you would have view as binary. And this helps us a lot, right? Cool. But if you want to look at it, this is what actually will be sent to the browser. Just so you know, this is not the real format. Nice. But we see here, we have a function that is exported called add one and there's some integer 32 param and the result that is integer 32. So what we would expect, right? Nothing too fancy around here. So let's look at the Java script. What do we need to actually get that running? And I will be reformatting that quickly a little bit. So it's easier to read for us. Or at least I hope it is easier to read for us. So let's have a look at that. Yeah. Is it big enough? Can you read it? Otherwise just scream and say, cool. I hope everybody in here knows fetch, right? So we just fetch a file out main wasm. So this is it, right? We know that file. Then what we do is we create an array buffer out of it and that will be handed directly to web assembly. So if you go to your browser console nowadays, you will see you have that web assembly object, right? On your window object. So every browser nowadays has that. You can just use it like that. What we can do is we take this array of bytes and instantiate the web assembly instance out of it. Cool. A promise and it will return us this result. On that result, we have the instance and this is finally, finally our wasm interface, right? So here we are. You see over here, there's an exports on that object. So this is where all the exporter functions of our wasm, of our running wasm will be located at. And as you probably guessed, right? We are gonna call that add one function. Yay. What we're gonna do is we expect that to be a return value and we will add that to that element called with the ID container. If you look at the HTML, I think couldn't be much more easy than that, right? So you have a span that is called container. What a surprise. And a script that we fetch, right? Main.js, the one we see over here. Nothing too fancy going on. So the HTML will take the JavaScript, JavaScript will fetch the wasm file, will instantiate it, call that one function on it and we'll show this to the browser. So let's hit build again. And there's one important thing that you have to know. If you don't save it, you see it. Here I have that, I have that circle. If you don't save it and you click build, it will not take in your changes. Your changes are only in the compile step if you save it. So don't forget, just press control or command s, whatever operating system you are on and this circle up here will disappear. Now when you hit build and run, voila, this should be it. I hope you all see that. Can you try that out? If you have build and run, we will send that Rust code. I think it's done on the server. We'll compile it, make that wasm file, load it into the browser, give it 41. The Rust will actually add that plus one and then we will output it to our browser again. Does this not work for anybody or is everybody happy? Everybody sees 42, the answer to all of our questions. Perfect, nice to see. So this is easy, right? So let's go into this a bit more. So that was maybe too easy. So yeah, okay. Let's have a quick look at the instantiation. I already explained that we have to do an array buffer at webassembly.instantiate expect us to have that array buffer because it is a binary file, right? We can't really do anything else with it. We can't kind of make a chasen out of it or something. So an array buffer is probably the appropriate format of that whole thing. There's a new method, instantiate streaming it's called. So we don't have to deal with the array buffer. This is all done internally. So here we can just add the fetch promise that we get back into that instantiate streaming. It will be handled by WebAssembly or the WebAssembly API and we don't have to care about the array buffer anymore. This is cool and a little bit more performant when you have very big binaries that can make a difference. But unfortunately we have a new IE here. So Safari is not yet on that but they should have it hopefully soon. As of yesterday I think it wasn't in there but I hope it will be soon. Oh, that's the wrong arrow. But now let's talk about, I mean it's all fun, right? We call the function, we get an integer back not very interesting. We wanna kind of have this ability to have a callback, right? Tell Rust, as soon as you're done call us back because we are JavaScript developers there's only one way to pass values and it's callbacks. Cool, there is the import object. You see on both of these methods the instantiate end instantiate streaming. There would be a second parameter that I didn't tell you about earlier. So next to our array of bytes we could also input a import object. Let's have a look at how that will look like. That import object can have functions on it and we can actually pass that into Rust at that point and Rust will see these functions and will actually be able to call them. So this is getting cooler, right? So we get some functionality done on here. So how is that actually done? Let's look at the JavaScript for that. First you notice just, I named it import object a very simple JavaScript object. Then there is one property that we have to set it's nth. This is just expected so all the imported functions into Rust should be on that nth property. Otherwise, I think during the build step the IDE will complain and say, hey your import object doesn't have an nth property so please add that. Otherwise you won't be able to actually call your functions. So nth is just another object and in here finally our functions are situated. So cool things. And as we know, I want to debug, right? So I need console log. So what I do is I call it log results as one parameter and I will just call console log in it. So the cool thing is here, right? I'm back at the JavaScript function. So my scope is actually what you would expect with that JavaScript function. So we are not in some fancy-pancy scoping or something. When Rust calls us we are exactly at that scope you would expect. So this is cool. We can use all these window methods. We could use other variables and functions that would be in our scope like you would expect it to work. This is fairly cool. And in the end you see WebAssembly instantiate. We have to add that import object in there and we're done. On the Rust side, so the compiler will obviously complain and say, no, no, no, you call a function that isn't existing at compile time. So we have to tell the Rust compiler, please trust us we will have that function at runtime. You don't see it at compile time but that doesn't mean it won't work. We will promise you we will add that function at runtime. So this is the external block. With Rust we can say that and say there's a function it will be called log result. And since we can't pass strings yet, we will just pass an integer. So we will say you can put in one integer and it's a 32 bit integer. We will obviously look in this a bit more afterwards so that we can have our proper hello world project. But now for now we have to stick to integers. What will it look like then in Rust? So if you take the same function as before you see there is no return type anymore. Obviously we call a function, right? We don't return an integer anymore. And we have that unsafe block before we call the function. Why do we need that unsafe block? Rust tries to be safe like he tries to, or he, the compiler, tries to make as many assertions as possible and tries to help you as much as possible so you won't screw it up, right? You won't shoot yourself in the foot. So what it does it will tell you, well this is an external block, I can't help you on that. So you have to wrap your call in an unsafe block so I actually know this is expected to be unsafe. I can't have all my assertions on there. So as soon as we have that unsafe block we won't get any upsetting messages from our Rust compiler anymore. He will just wave it through and say, okay, you seem to know what you do. I hope you do. Let's just go ahead and compile that. Cool. So I have to quickly check something. There should be a QR code somewhere around here. I think I skipped it. I'm very sorry that was around here. Yes, I'm sorry. So this presentation is actually on the web. So all the exercises have these code snippets in there. So please follow my presentation. If you open either bit.ly slash web assembly minus workshop or scan the QR code you will get to my index page. This is the page that you saw before. From there you can open these various presentations. So every chapter, every chapter is an own presentation. So keep that index open. So we'll always have the possibility to jump into any chapter you want to. Did everybody get the link? Everybody on the page? Perfect. So now we are at three, chapter three. So if you open that you will get to this slide over here. So please take a look at these arrows. So usually what I do is I have these kind of topics and we have an exercise we will go down vertically. So you can navigate with your arrows or you can click on these arrows. So whenever we are in an exercise you have to navigate vertically to find the code examples that you will be using for that current exercise. So now we are at here. Call JS functions from Rust. If you hit escape you will get into this overview. So this is actually the last exercise for that chapter. You can go into here and then you see all these code blocks and in the end you have the link again for web assembly studio. So let's try to do that. Let's try to call a JavaScript function from Rust. So you have all the code examples that you use to create this exercise. Go to web assembly studio, try to recreate that. And if you have any questions I'll be around. Just come and ask me or I will come to you. Let's take like 10, 15 minutes. Let's see how long you need for that and then discuss the exercise. Any questions for that? Cool, happy hacking. Are you the outlaw? So far clear? I think it's quite easy, right? It's not too much to ask. The audio? Yeah, it's a little bit quick. I probably can't do anything about that, right? Which is so sad, so sad. I hope so. Is it okay if I take a picture of all of you working or is there anybody who wouldn't want to have? Yes. How can I get the console for the C-Weth inside this? Oh, oh yeah. Let me, let me. Just got a very good question. So how do you see the console of that? Your actual browser console is the console for that IDE. So if you open up your console, you will see whatever you do with console log and stuff like that. So you should see it in here. Build and run? Yeah. Doesn't do anything, huh? Console log one. Yeah, I mean, that's easy. Huh, this is weird. Want me to try here? Well, yeah, you can, but I'm just... Oh, did you save? Did you forget to save? Oh, yeah. That's so stupid. I stumbled over that a lot of times. Don't worry, don't worry. Sorry. Yeah, that looks simple. Essentially, it's bite. Oh, oh, look at that. There's bite and it should be bite. See? There's an S. Let's see, let's try it like that. Okay. Always save? Oh, no. So stupid. That looks better, right? Yeah, perfect. And you'll also see it over here, I think. Yeah. You'll also apply that if you look at console. What, these? Okay, cool. What? So this is the returns of... Oh, yeah, look. What he tells you is, I mean, he expects to have an I32, but he found a function. So you just have to remove the return type, right? Because we are a void function now. Right, yeah, yeah. So if you remove that, save. Always save. So yeah, that's annoying. Awesome. We are not used anymore to having to click save, right? Perfect, here you go. Nice. Sure. I'm supposed to call it like instant log result? No, the idea is that you actually call this from your Rust code, right? So here you are in JavaScript. So you can remove that over here, but go into your Rust file and you should be able to call it from here. So what you can do is remove that return type from your function, right? Say this is not returning anything anymore and then you can call it here. Log result, right? Exactly. And you have to have that unsafe block around it. Unsafe. Yeah, that's... Can you look at the slides, I think, if you go down a bit? Yeah, here you see that. Because the compiler will expect you to only call functions that are safe. So, cool. Let's try it. Yeah. So... Did you say? Yeah, I say, but say. Cool. Show it. Oh, there's another. What happens? Oh, we still have the return type. So... Either you do that or you remove the return type, right? Because we don't necessarily have to return anything. But this is okay, too, yeah? Try it out. Safe? Probably. This is when you're not on the file, then Command-S will trigger the Chrome Command-S, right? Oh. Perfect. So, does it work? Oh, perfect. Cool. Yeah. Do you not need to no-mangle, like, external stuff? No, because this is actually from Rust, right? Right. It expects you to say these are external functions, so it will not optimize that block away. Yeah. Yeah. Yeah? If Rust compiles it to C, why don't we use C directly instead? We can. So, Rust does not compile to C, right? Rust compiles to wasn't binary. We will use the C types to move between JavaScript and Rust. So, Rust itself uses its Rust compiler and will be compiled to its own binary. It will not be C. But we will use the C types to move between JavaScript because JavaScript knows C kind of strings. We will explain it when we get to that because we kind of have to tell JavaScript. We have to read it from memory ourselves, right? So, we have to know what format is that string. And the C string is just bytes with a zero at the end. So, this is easy to read for us. So, does it mean I can use C? With Rust C. C with Rust sub. You mean, like, inter? C, real C, and Rust C. Rust and C. These types. I don't know Rust enough for that. That could work, but I haven't ever tried it. And Rust was created solely for the purpose of WebAssembly? No. Rust is actually one of... It should be an easier approachable language than C, but should be pretty similarly low level. And they just, when they came out, it was like, oh, there's WebAssembly. We will do a first citizen compile target for Rust. So, Rust is just a kind of coincidence that Rust is so heavy on WebAssembly. So, I think, yes. Tell me. So, add one is not a function. Show me your Rust code. This part? That looks good. Only change here is I just added this here, right? Yeah, exactly. Yeah, do that. Is there some other sort of failure than results? This is weird. I don't see it right away. Import object is the right reference. PUC, I will return a rating. Now, I will longer return a rating. Exactly. That's why I just removed the fucking start. Perfect. This is what you are expected to do. Yeah, so I'm not sure what's... Yeah, I mean, it looks good. Can I show me the Rust again? And I added some stuff here just to show you it's really calling for the Rust version. Oh, yeah. Oh, no, I mean, the template string is fairly easy. Can I go to the Rust again? Oh, you need the GNOME angle to be right above the function. So, the compiler removed your function now. That's the issue. Yeah. How can I show the Apo3 working? Oh, this is a good question. I'm not sure. What is this? Split is split. Can you open, can you click on it? Oh, it's just another, okay. I'm sorry? We increased the volume on the speakers. Thank you. Um, I am very, very sorry, I don't know. Or maybe just reload it. But yeah, I mean, then you will lose your code, obviously. I don't know, I'm sorry. Cool, did it work for everyone? Anyone having problems with it? We can otherwise look at it in the break if you want to because we are already almost there, but let's quickly go through this. Oh, so, I've been higher as well. So, let's quickly go to the WebAssembly project and just do what we said. So, I think a lot of you noticed you can actually just copy paste the code from here. It will work. Or it should work. So, what we need, I'm sorry, this is a bit, can anybody read that, right? Cool, so, we have the import object over here. Just add it like that. There is no return value in the end for this. So, a lot of people noticed it themselves. You can remove that here. Don't have to, but in the end, we want to call log result, right? We don't have to return anything anymore from the add one function. Yeah, so, let's take the rust external block over here. Oops, I like that. Make sure that there's no mangle is directly above your function. If you take this away and like move it up to here, the compiler will not care about it anymore because it's only meant to be on top of a function. So, the compiler actually knows this function will be staying in there forever. Okay, so now we only have to call this log result with X plus, let's multiply it for once. And this is important here. When I don't return anything anymore, I need to remove that type, right? If I leave that in, the compiler will tell me, no, no, no, so if we're gonna look at that, you see now rust is always returning the last expression. So what he says is I expect you to return an integer 32 bit, but what you did is you returned a function call since log result is a function call. So these messages down here, I think the rust compiler is actually quite a nice compiler that helps you with rather nice messages. So always take a look at the output down here. Usually it helps. So let's remove that. And the last thing we have to do at that unsafe block. So this is all. Let's quickly try it out if it really works. Well, build and run. So yeah, I think a lot of you already struggled with that command S all the time. I'm sorry, we are not used to that anymore, right? That we have to save manually. Okay, you see 82. And also what I said before, actually in your browser console, you see, this is no trick. We really put in the console log function that we think we put in. Cool. So that's it for the first part. That was easy, right? Oh, you can't see the quiz already. Okay, if you wanna go check that out, WebAssembly on MDN, I mean, as always, Mozilla developer network documentation, which is fantastic, and WebAssembly Studio is actually an open source project. You can go, I mean, maybe you've seen that, there's a GitHub issues button, right? You can go there, you can check it out, you can build it on your own machine and run it locally, very cool. So you don't need internet for it to work. I think to learn and teach like this, actually a very cool project. So this would already be our first short break. So you can kind of maybe go pee and drink something and we will see this again in 15 minutes. So, 10 minutes after two, so that I'm clear. It was able to manage this great working. Yeah. But we just wanted to know like the trust code is getting compiled to the graph. Exactly, yeah. What exactly is doing this? Oh, okay. Is this something we'll be explaining? No, I think I brushed over that a bit. This is the actual Rust compiler. So what the WebAssembly, what the Studio down here does, I will explain that after a short break. There's a built TypeScript file. I should have explained that. I will do that after the break. The main.exe file. Yes. Can we directly open the browser also because like we have some HTML content over there. Is it possible, this main.restimal file, I downloaded from there. So if I download, this all this file comes. JS, RS, and main.restimal. So. Okay. Is everyone still here? Doesn't look that way. Well, let's hope they will get back soon. Oh, I see people coming. Cool. Are there any unanswered questions right now? There's something I didn't really explain all that well. So I will try to explain that now. Some people asked me, how does this actually get compiled, right? I didn't really tell you that. So over there, there's a built TS, that ominous file. And this is how this is actually being built. And let's quickly look into it. So we use gulp. Still people using gulp today? A few, a few, that's cool. I haven't used gulp for a while. So this was a kind of a reminder for me that I don't know it anymore. So we have one task here built and you see there is this service. If you go look at WebAssembly Studio GitHub repository, you will find they have a little service here which will make a call to their backend. So what's happening here is, we get the file main.rust and we send it to the backend to be compiled. We say, look, this is a rust file and we want you to build it for WebAssembly. What we will get back is that object called data. We will see it later on. And then we will create here project.newfileout, mainwasm, this one, and we'll set data to what we got back from the backend. So the JavaScript code doesn't have to compile now, right? But the rust code has to compile and we can't really compile rust in the browser. Probably would be possible somehow, right? But it's easier now. So what they do is send it to the backend, let it be compiled on the backend and get the binary back and create a file with it. So we have to find an actual binary file. We will look into that a little bit later. I was always also getting the question, like rust is not a WebAssembly-only language. Rust wasn't invented with kind of WebAssembly in mind. I think it's more of a coincidence that when they actually came out with rust, that WebAssembly was right there. So they said, let's make WebAssembly one of the default, or one of the easy to access compilation targets. So this is basically everything it is. Rust, you could also obviously compile rust just to an executable and then run it on any machine you want. That they support this, it's just cool and it was very easy for me to make these teaching cases here, right? So that's why I was using rust. So we don't compile rust to C or anything here. We compile rust to WebAssembly binary. Just a little clarification about that. Okay, so let's go to our in-depth exercises then. WebAssembly in-depth. So I mean, it's all fun, right? We pass some integers, but that's not really mind-blowing. Like why would we pass integers in these functions? It doesn't really help us a lot. And the use cases, well, maybe we have that extremely hard to calculate math function, okay? Maybe that could be something that is cool. But let's be honest, what we wanna do is we wanna pass arbitrary data, right? We wanna be able to really have an API where we can pass whatever we want. So how do we handle these kind of complex types? Let's start with not the most complex type, right? String. Currently, some of you might have tried that. If you wanna pass a string, it will fail miserably. You will get fancy exception messages that this is not working. So Wasm currently only knows these integer and float types as I mentioned before, integer 32 and 64 bit and float 32 64 bits. So what we have to do to actually be able to pass strings is in Rust, we will allocate memory on our linear memory array, write a string into there, take the pointer, return it because the pointer is an integer 32 bit, right? So we can take that pointer, return it from our function to JavaScript. So JavaScript now has an index, right? It's like 1,347 or something, whatever. That just points to an element in the memory array. So we go to the WebAssembly part in JavaScript, take the memory out of there, well, the memory array and say, I wanna read the byte at that position. And then I'm gonna increase and read every byte until I find a byte that is zero because we are gonna deal with the C string implementation. That means we have UTF-8 strings. Every byte is a UTF-8 byte and it will be finished by a zero. As soon as we read a zero, we know we're done. If we read over it, it will get that classical index out of bounds exception and stuff like that. So let's not do that. Yeah, so how does that look? So now we get into Rust a little bit more. Bear with me here for a quick while, it's not that complicated. So you see, we have some imports, we have another return type and we do some fancy stuff here with C strings. So let's break it down part by part. So in Rust, usually stuff is immutable. So when we have mutable things, we will prepend MUT for mutable. For all of you that know C or C++, that asterisk means we have a pointer. We will not return the string, we will return the pointer to the memory location where that string is in. And the C char is kind of an alias by Rust that is the C type. So since C types are native, they vary a bit from architecture to architecture. So Rust made like one alias for that char character type in C that is called C underscore char. So this means nothing else that we are handling a C type in here. So in the memory, there will be that character type of C. Then we have this higher level construct of Rust, the C string object. So this is for Rust that has a lot of these helpers on it so we can actually instantiate it, do things with it like you know it from other high level languages, right? We have just these string helpers. So what we can do, we take that in here, make new. So we have that C string object now. This means the basis down below is what I just mentioned before. We have a string that is now null terminated. So there will be a zero in the end. When we write that to the memory, there will be a zero at the end. This is important for us because otherwise we would probably fail. Then as soon as we have that C string object, we have to make sure that we get to that pointer. So the first thing is we do, the first thing we do is unwrap it. What that means is it will take like kind of unboxing, it will take away that C string stuff and we will be left with that C character type. This is what we want. Then with into raw, we will convert it to a pointer. Instead of the actual type, we will have a pointer in our hand. This is what we want. Also a very relevant side effect of that, Rust will not garbage collected, right? We have to care about that. If we don't use that variable, Rust will eventually go ahead and say, oh, I don't need that string anymore and kill it. And if we are in JavaScript trying to read that, yeah, this will not work, right? This will throw an exception, obviously. So into raw, we'll do that for us. We'll have a pointer and Rust will not garbage collected anymore. But memory leaks, I hear you say. So now Rust is not garbage collecting it anymore. It is not on the JavaScript, not on the JavaScript heap. So that means JavaScript is not garbage collecting it either. So what happens is it will stay in that memory. Not good, not good. But obviously we can fix that. And please bear with me here. This is a hack, I know that, but it will do what we want. So we can have another function that is called the allocate string. It will take exactly that pointer that we just returned, right? A pointer to a C character. And what it will do is it will instantiate a variable from that string and not use that variable anymore. So Rust will garbage collected eventually. It's not nice, you know? So we're just creating a stupid variable that is good for nothing. But with that kind of hack, we give the garbage collection back to Rust. We say, hey, here take that string. And when you don't use it anymore, free the memory. That's what we do here. This is just a helper function. This is not what we wanna look into it most. But this will help us not make memory leaks so we will never have to care about weird memory issues. Yeah, you saw up here. There's again our lovely little unsafe block. This one here, right? We need that because we are reading an arbitrary point you could pass in any integer there. Rust can't validate that. So it doesn't know, is that an actual pointer to our memory or will it overflow the memory? Is it type safe? You say this is a C character, right? But you just give me a pointer. It could be anything. So everybody who had to work with C++ at some point probably knows the issues around dangling pointers and stuff like that. So we have to tell Rust, please just trust us. We are engineers, we know what we are doing, probably. Cool, so we have the Rust part. We make a string, we return it as a pointer and we have an function that we can call to deallocate that string again. So the Rust side is already but the JavaScript side is not yet there. So you see there's a little function here. It looks very nice and fluffy at all. That copy C string function. I mean, how hard can it be to read from memory, right? Oh, oh yeah, that's it. So I don't expect you to fully understand that. If you want to go ahead, take a look at it. What it basically does is we have a generator function up here and it's just looping over the memory with that while and you see it will stop when it hits a zero, what I said before, right? We start at a certain pointer, we loop over each byte in the array and when we will hit a zero, we say, yeah, we're done, we have the whole string. If there is no zero, we will loop and loop and loop and loop and we will eventually crash. So let's hope that doesn't happen. So what we do here, you integrate array, so unsigned integer array, then we have to decode it. So now we have an array of bytes. So we basically have a copy of that memory snippet where the string is and this is relevant to get. We now have a copy. So JavaScript has that string in its own heap now as a copy. It's still in the WebAssembly part, that's why we have to free the memory but JavaScript copied that string now. What we need to do is that text decoder UTF-8, we know each of these bytes is a UTF-8 character. So we have to tell JavaScript, hey, read these bytes as UTF-8 characters and voila, we should get a string in the end, hopefully. And what you see down here, this is where we call it. So we take that original pointer, you see that up here, we save this pointer that we get in, we save it initially and then in the end, as soon as we have copied that string, we call that funny little deallocation method here. So Rust will garbage collect that string that is on the Vasen memory. So we don't have to care about that anymore. And in the end, we return that string. So this is now a UTF-8 decoded string, cool. And this is what we will do now, yay. So yeah, we are in chapter four, handling complex types, you can go down here. All the code you need is basically in here. Go ahead, try it out. And in the end, if everything works, we are very close to having a proper hello world function. Yeah, I think we take 15 minutes for that, I would say, because this is a little bit harder. This example, can I just show? Yeah. Can we check on this? Oh yeah, can I try something? Then now to be honest, let's omit that last thing here and remove the other statement, like just follow through, right? Oh, this is right, like this. Okay, so let me quickly try without the semicolons. I had some issues when I would add semicolons, so you never know, just to be sure. And then zero and then one again. Is this what we expect to see? What do we expect to see? Fibonacci should always add up the last two numbers, right? Yeah. So in this case, I injected three, so it's supposed to be returning just three, because two plus one. Yeah, this is what it does, right? Three, two, one, zero, one. That's kind of weird here, but the first, can we kind of take that out for a second just to see what the actual result is? Oh, can you comment that? Sorry, thanks. I'm so bad with foreign, foreign. Yes, exactly. That's cool. Exactly. Oh, X, it has to be X. So we're doing something wrong. Maybe it's because of the algorithm here. I think it's smaller equals one, return X. So it should never be zero, right? Because we kind of make sure of that. Well, now it can be, right? If X is two, X minus two is zero. So I think the algorithm is wrong. Let me quickly check. Can we increase that to a two? I'm not sure if that's right. I'm not very good at math, I have to admit. So what happens now when we take five? So four plus three, is that right? I'm not very sure. I think, yeah, I think what the basic problem was, I think you should probably omit semicolons and return statements as much as you can, because I had some weird issues with semicolon, but I'm too bad at rest to tell you what the actual problem of it is. So I think what you do works. Now you probably have to just figure out what the correct mathematical algorithm is. And what about the elsewhere? Probably, I think else is okay. I just took that out to be sure. Can you, right? Yeah. Yeah, I think the semicolon problem is the issue. I'm not 100% sure. Tell me. We can define multiple methods like this, right? Yes. So like earlier, also in the same, same... Yeah, yeah, cool. Cool, yeah, just go. Oh, the use has actually have to be on the top. Oh yeah, sorry. I should have mentioned that, I guess. Cool, let's try it out. So I am getting some result is not defined. So you have to copy string method? Yeah, exactly. You copied it right over, right? So this is also on top or here fine? It's fine, right? Global organization. No, you have to be in here, right? Because here you don't have access to the actual web assembly instance. So it has to be in here, exactly. Yeah? Yeah, absolutely. Cool, so we are in the right scope now? Yeah? Yeah. So run it. Here you go. Thanks. Strings, tell me. Print the message, should we deallocate the string already? Or like will we... The allocation happens in the copy string method. You see it here? Because the thing is, we copied, right? So JavaScript will care for its own string that we will actually print out. It will care about that, right? We have garbage collection. We just have to make sure that after reading it that the web assembly memory is freed. So this string up here, this will be garbage collected by JavaScript. That's all okay. Okay, awesome. Perfect. If I don't have anything in the local scope, can you check in the global scope? Is there any global scope variable thing like that? In Rust, you mean? Yeah, yeah, like here it was earlier outside, right? This all were outside, so it was not able to identify. So there is no concept of global variables type? So you are here in JavaScript, right? So your global variable is window in here. So we are in basic JavaScript. You have the window object. It's available to you anytime in JS, but Rust doesn't have something like that. Rust doesn't know a window object. So due to the window object is not there, that's why it expect everything to be in the method? No, no, this is just because you get the actual web assembly instance in here in that method. So here you access results instance experts, right? You can't access that above that, right? Because this is a promise. You are in the promise then block. So you have your asynchronous, right? So you don't have any access to that instance from the outside. That's why we have to be in there. Thank you. Yes? I think I'm going to ask again if I understand what but not necessarily why. Yeah, tell me. Why are we doing stuff for that point if we can't return a string from here? Yeah, because the web assembly interface defines four types. So Rust-nose types, strings-nose types, strings, sorry, JavaScript-nose types. But in between the interface where they have to go through only-nose integer and float, that's our base problem. This is a bit ridiculous, isn't it? It is a bit, yeah. Is that going to change over time? Yeah, this is one of the things I showed with any ref and the host bindings. They should leverage that. But currently, this is the limitation. I didn't realize it was up here still. Yeah, right. So we're doing this, right? Yeah. So, yeah, we are point-to-magic. Yeah. So I was just trying to change the code a little bit. Oh, yeah. Cool. Nice. Xscript-sexy. So I want to, yeah, just de-allocate string thing that we were getting from the module, right? Oh. I don't love the idea of passing around the whole module. Like, it just felt a bit weird. That's cool, yeah? Yeah, the thing you need to make sure is you need to have the memory. The memory module here. Exactly. Because we need the memory buffer, right? That's a ray that we're going to read from. So we have to make sure that this is in there. What we're getting back from here is not just the exported functions. This isn't like exporting the public. Exactly. Memory is something that WebAssembly exports for you. So you can actually access the memory array of WebAssembly. Absolutely. You get that for free, kind of. So there's a whole bunch of, like, exports. It's not a lot. I think it's memory and your function. I think it's really. So in theory, just go memory. Yeah, yeah, exactly. So you can actually pass it in, yeah? Uh-huh. Now, and also it's naming those with, like, I guess, these ROS conventions to use underscore. I really, like, hate mixing these when I get them. It's the same thing. Oh, yeah. We have to kind of make sure here now we have a name clash. Have to make sure. Yeah, and that one, absolutely. Yeah, that's it. So try it out. And I'm supposed to define this as a utility function? The allocate string is not a function. Do you have a typo in there or something? Could that be in here? Oh, you don't have it in here. You need to de-allocate. Yeah, I thought that was native. No, no, it's really only actually memory that is native. I think it's pretty much on the top, if you go, yeah. That one? Yeah, exactly that one. You need that. And the no-mangle. Don't forget the no-mangle. I think we've got that, don't we? Do I need it? You need it for each function? Absolutely. So I can keep. That's why that wasn't working. I thought that was just a one-man. Oh, yeah, no, you need it on each function, absolutely. That's just a ROS thing, so. Yeah, yeah, now you can do the namings that you want to. Cool. Yeah, sometimes it's a bit quirky. You have to be in here. You click in here, and then, yeah, now it's good. Cool. Cross fingers. I see this too. It's not defined. I missed one. I missed one. Did I? Well, oh, yeah, the actual call. So that's cool. It could read until there, right? So it seems like now, yeah, there you go. Cool. Yes? So you mentioned that the wasm, the execution of the wasm, is actually running on the VR. So how does the browser keep its synchronous? Or is it not synchronous? I mean, my JavaScript calls a wasm, then do something else in JavaScript again. Yeah, this is not synchronous, yeah. Oh, it's not synchronous. Yeah. So you have two separate run times. All right. Yeah, that's it. Well, but ROS is synchronous. So when you call add one, this will be directly executed. So your JavaScript code won't run any further. So, you know, it's not like you have promises around your WebAssembly calls. It's just like, it's two run times, but when you say call the add one function, it will directly return synchronous. So this is synchronous. Oh, because the JavaScript is waiting for the ROS to finish? Yeah, exactly. It's like the call stack. So the ROS call is on the call stack. And the next thing JavaScript can do is actually access it. OK, I hope that most of you have gotten this to work. This is kind of a cool little thing when you realize, OK, now I read something from memory in JavaScript. I've never done that before. Like this is, let's say, the most low level thing I've done in JavaScript. So let's try it out. So I'm going to try that quickly, too. Let's remove what I have here and just add that in, right? Add the second function in we saw here, the allocation, this one. So some of the people in here ask me, you need that no-mangle in front of every function. Otherwise, the compiler removes your function. So if you forget one of these no-mangles, that's when you probably get some issues like, this is not a function and stuff like that. So you need it in front of every function. So this is all the ROS code we have, right? We have two methods, welcome message and the allocate string. And I will remove that import object for now. So it's a bit an easier overview for you guys. And then paste that in. So make sure that this JavaScript stuff is in the then in here, right? You need access to that instance that you just created. Since this is asynchronous, we can't paste that code up here, right? Because results is not accessible from up here. Because this will be only available after we loaded everything. So we have to have that code in here. Let's remove these two lines. Oh, wrong. Cool. So we instantiate, we get the module results instance exports, and then we call get welcome message. Then we copy that C string. Oh, the C string function. So let's copy that as a whole. So we don't have to worry about that. And when we build and run this, it should say hello from Rust. Oh, or oh, perfect. Hello from Rust. Cool. So what we did is we passed pointers, yeah, pointer arithmetic. We read from that memory. So you might have seen this here. We just access module.memory. WebAssembly is actually exporting this memory object for us. So this is how we can access that buffer, so the actual array where we can work on. We need that. So this is the memory from Rust in WebAssembly that we access here right away. And yeah, as soon as you see hello from Rust down here, you have actually saved a C string into a memory and read it from JavaScript and passed it as UTF-8. But this is a bit hard, right? I mean, with a string, it's easy. Well, easy. We can just read until zeros. But try to pass now arbitrary objects with arrays in it. That's where it's going to get really heavy. So we don't want to do that. And we don't need to do that. There's a good tooling for us. I only wanted to show you this, so you would actually know what happens underneath it that you understand. This has to be done because we only can pass integers and floats. But we're not going to do that anymore now. Now we're going to have one abstraction between us and WebAssembly, so it's going to be easier for us to stay sane in our head. So every current language that you can compile to WebAssembly has its own kind of helping tool for that. In Rust, it's called wasm-binchen. For C and C++, you might have heard of that. It's called mscripten. People that know asm.js know mscripten is around for quite some time now. But yeah. It allows us to have high-level interaction, so various types. And it does all this host bindings kind of for us. So we can now pass in and out of WebAssembly whatever we feel like. And we will see that this can really be a lot. What's also cool is wasm-binchen can actually generate typings for you, for TypeScript. So that's, in my opinion, a huge plus here that that tool does very well. We will not look into that because of time constraints. We won't have time to explore all the options. But you could, theoretically, generate typings from your Rust code. So in TypeScript, you would actually know what this function is returning and what you can pass into it at any time, which would be nice. The downside of this, now we have a wrapper between WebAssembly and Rust, or the WebAssembly interface, and Rust, and JavaScript, and the WebAssembly interface. Because when we send some arbitrary object on the JavaScript side, we kind of have to transform that and save it into an array. And from the Rust side, the same, right? We have to know how is this kind of stored, but we don't want to see it. So we generate code now on the JavaScript side and on the wasm side, Rust side. And we will only interact with wasm now through that wasm-binchen. How that looks, we will see it right away. So what now happens is when we compile a wasm file, it will actually compile into a JavaScript file and a wasm file. On the JavaScript side, we have the bindings so that we actually know how we read these arbitrary types. And on the wasm side, we have the compiled Rust code plus also these bindings. So JavaScript and Rust know what the other part is going to save into memory and read from memory. So we have to kind of abstract that for us. So we have basically, we have three steps to do. We have to make sure that Rust is compiled with wasm-binchen. So this is actually a compiler extension in that sort. Then we need to make sure that the compiler knows wasm-binchen. It will compile it with wasm-binchen and we will need to make sure that these JS bindings that I talked about are loaded into the browser before we access WebAssembly because they have to be there. They have to provide us all these wrappers to send all these arbitrary types in and receive them out of WebAssembly. How does that look like? On the Rust side, it's kinda easy. There is this line up here that helps us with that. Oh, I'm sorry, this one. I will not go into that too much. This is Rust and we say Rust that we need some macros to be executed for that. I think that's a detail we just omit for that workshop so we don't have to go into Rust, macros, and all that stuff. We're just gonna leave that out for a moment. This will activate wasm-binchen for the compiler so we can actually use all of it. Then we say there's an external crate. A crate is like an NPM package for Rust. So there's a wasm-binchen crate. This will actually load to the library that we will then be using. And here we say that we need this prelude that will help us with all the helper functions that we will be using now. The actual Rust code doesn't look all that different. You see up here, this is slightly different. Instead of no-mangle, wasm-binchen introduces a new flag. It's called wasm-binchen. This has the compiler for one no-mangle. Please don't remove it. But on the other hand, it will also make sure that all these types are working correctly. Then you see we can just return a regular Rust string as you would expect it. That's cool. So this is what we do here. But what we get in is a string reference. I will get into why this is the case. This is a little macro from Rust that is just helping us. You can have these braces here and all the parameters you will add afterwards will be replaced into these spots with the braces. So this is just a fancy print line, right? Nothing else. This will return a string that is hello, comma, and then whatever you pass into it as a name. So the Rust function isn't that all that complicated. This is all that we have to do on the Rust side. So far, so cool. Yeah. These macros, by the way, you always see them because of the exclamation mark. Rust has this, yeah, Rust does this that every macro has to have this exclamation mark in its name. But as said, we're not gonna look into more of these macros. We just use it for creating that string. That's all it has to do. So this is the case here. That's relevant. JavaScript strings are UTF-16 and they have some limitations to that. So how pairs are matched up. I am way too stupid to explain this. So the Rust guys, they have done a cool documentation. Why it's hard to go from this UTF-16 to that UTF-8 of Rust without getting these nice little black boxes with question marks in it. So this is why they give you that reference. They have to do some stuff that it is guaranteed that that string will never have unrecognizable characters in it. So this is why you get a reference of a string in the end. There's a base type behind that. It's called JSString. That's what they use to actually convert all that stuff. If you wanna read up on that, these are good starting points, but this would kind of be too much for that workshop now. So we're just gonna accept the fact that when we have something in JavaScript that we have to, when we get a string or give it back, we'll have this reference type here. So there's one last thing. If you recall the diagram from up above, this one, when I showed you that build.ts file, you saw that the wasm file is only compiled, the rust file is only compiled into one wasm file. And we have to change that now, right? Because we have these JavaScript bindings. So what WebAssembly Studio is providing us, this is what it is now, right? Service.compile file, but WebAssembly Studio actually has compile file with bindings, very cool. So what that does is, back here we have the same call. We say we enter main rust. It is rust code. You have to generate wasm code. And it's the same options, but the return type is slightly different. Because if you look at it, before we had this data object, and we would just say a wasm set data. But what we do now is that data object has actually two properties on it now. One is wasm, this is the binary, and one is wasm bindings.js. Bindchen, sorry. So we create two files here. So this is the basic difference. We have one out main.wasm file, and we have one main underscore bindings.js. So now we created both. So we compile that rust into JavaScript and into rust, wasm, sorry. The only thing that we need to make sure now is that it's actually loaded into HTML before our other script is actually executed. I mean, it's kind of trivial, right? So we just load that main bindings.js that we just created in the build step, and that's all we need to do. So how does the JavaScript side look like? So this is different now. You remember that before we had to call fetch and then array buffer, and then WebAssembly.instantiate with that buffer. All of this is now done by wasm bindchen. As I told you, we do not interact anymore directly with WebAssembly. We will interact only through wasm bindchen. This is our interaction point now to actually get this code running. So the string is the same, right? We just say, hey, take that file and instantiate. And what we get when that promise returns, the instance is ready. WebAssembly started up and we can just call it. You see that this function is then calling that greet method. And this is the second thing that is new up here. Except instead of having to go through instance.exports and then calling.greet on it, wasm bindchen is providing us the exports as these properties on that global object. So wasm bindchen is on our window object. This is a global object that we now have and we can get all the exports from there. So if you wanna make it easier and not have these destructuring things, you could also go ahead and just say, wasm underscore bindchen dot greet, right? This is the same. So here we just have that fancy destructuring. So we take greet out of it as a standalone function. But you could also, oh, well, not like this, but like this. You could just call it like that. This is just, as every window object property, as you know, you have access to it. It's global. You can just go ahead, grab whatever you want to and execute it. Yeah, so maybe if you go to the exercise, delete your existing JavaScript and Rust code and take the ones from these examples. So you don't have any clashes with names or something. So you don't end up trying to track down some naming issue or anything. So this is all the code you need for that. Yeah, exactly. That's what I already said. That main bindings.js, this is actually the code. There's the script that does add this vasenbindchen property to the window object and we will have access to that now. Yeah, so there's three things we have to do. We try to pass these strings now without having to read on the memory, which would be cool, right? So make sure that it's composed of vasenbindchen and import these bindings, and then you should be all set for that. So let's try that out. I would say we take another 10 to 15 minutes, depending on how many questions there are. That's a reference of a string. Well, string reference. But that's that specific JS style reference that I mentioned because of that UTF-16 to UTF-8 conversion. This is what it does. Yeah, it's a little bit weird. And then are these built into kind of a compiler? This is kind of rust macros for the compiler, yeah, exactly. And then, so instead of having like a package JSON, we have dependencies in it and then doing, yeah, it's just rust pump with that component. Yeah, rust has, it's called rust pack, I guess. Yeah. That's that whole dependency resolution stuff. Yeah. Sorry? Oh, I'm sorry. Ah, yeah, I'm sorry. If you, I will show it. If you look at these bindings, you will see that it does pretty much what we did earlier. So you see that this old generator code now, what you see here, they do the cache, getU, int8 from memory, they go to that buffer, they take that array, they have the text encoder, you see they pretty much do it the same way as we did before. Is it not so sad? No, no, no. What? So in this NPM package, create is an extents package. It's very simple. You see, we're maven dependency. See, that's a library. Do you need it? Okay. Rust in your set, do you use Rust in your company? Rust, I am not a heavy user of Rust now, but I think we have Rust developers, but no, I wouldn't know of any production project we have with Rust. I think there are smaller services when each company has a kind of set of languages that they use. So I think we have customers that have small services written in Rust and go for, go. Yeah, Rust is very easy to teach the basics, but there will be one more exercise and then we'll be done with the basics, then we get onto the fun stuff. So let's discuss that. Okay, so let's discuss this exercise quickly. This will be the second to last basic exercise. So bear with me for one more and then we will have the fun exercise where we will actually do a more real world example, right? Because this helps us understand it, but it doesn't really reflect what we will use it for afterwards. So let's quickly go through it. We import the crates that we need. We add that one function here to actually get that hello world message. Yeah, then we extend our build step like this. So this isn't that build.ts that I mentioned before, right? So we take this one out and oops, like this, replace it with the new one. So in here now you see that we do the main wasm and that main bindings.js. Cool, so we got the rust all set up. We get the build step to do what we want. So we need to import the script in our main HTML. That's around here, this one. And then there is just the JavaScript left. So let's take that in as well. So if we build and run that now, we should see hello.js.conversia. Perfect, that's what we wanted to see. Nice, okay. So you see now here we have that main wasm, the binary and that bindings. If you take a look at that bindings file, so this is all generated, so it's a bit ugly code. But if you check what they do, you see that they also use the text encoder that we used before. They go to the memory buffer as we did before. They do this unsigned integer array and decode stuff and then you see the function pass string to wasm, which allocates memory on the rust side and then writes the string in the buffer and everything like that. So they pretty much do what we did before. So that's what wasm and bind chain is now providing us, right? So we don't have to care about that anymore. It's not nice code since it's generated, but now we don't have to care for ourselves anymore, right? We don't have to maintain that memory stuff and read everything out of there. Wasm, bind chain will take care of this. So that's actually what we wanted, very cool. So there's just one thing left and then we have all the basics covered or yeah, all major basics covered before we were able to call JavaScript functions out of rust. Can we also pass strings back to JavaScript, right? We did not pass strings back, but call a JavaScript native function with a string. We haven't proven that yet. So this is a very small exercise. It uses wasm, bind chain again. So you know that little external block that we used earlier for our log results function. We can leverage that a bit and add that wasm, bind chain up there. What that will do is wasm, bind chain on the JavaScript side will search for that function and try to inject that on the instantiation phase of your web assembly. So when we don't say anything, wasm, bind chain will go to the global scope, window and will search that function. So this will inject window.alert. So now we have the window alert function passed into it. We don't need to do anything on the JavaScript side since this will all be done in our binding strs that will be generated by our compile step. So the JavaScript is not different. This is a thing that we only have to do on the Rust side and wasm, bind chain will inject the correct functions. Now you think like, yeah, but what if I wanna have some specific functions? So they have it covered. If you say nothing, there's no flag. It will search for window.alert. If you say, hey, hey, hey, the chain's name space is console, wasm, bind chain will go for window.console.log. So this is how we would inject console.log. And if you wanna have it twice, you know the console log statement of JavaScript obviously takes in arbitrary types. Rust doesn't really know that, right? Rust doesn't know that pass me anything log statement, pass me an array or an object log statement. So if you wanna have a second console log injected with just another typing, you see? Here we say unsigned int 32-bit and here we say string reference. You can even go further and say, okay, the chain's name space is console and the function name is log, but in Rust that function name will be console.log underscore U32. So wasm, bind chain will not look at your function name anymore. It will only check these two variables if you set them and just from there will search the correct function to pass into our WebAssembly code. So wasm, bind chain will set up that import object. It will add the correct functions into that import object and will pass them in. So we don't have to care on the JavaScript side anymore because that JavaScript bindings file will do it all for us. Is that kinda clear? Yeah. Okay, so the question was if you wanna console log math.log, you mean the JavaScript math.log. So what it would do is another line here and it would say the JavaScript name space is math, right? So let's just write it. If it will go like that, oh, this is written in like this. So this one now, wasm, bind chain would now go to window.math.log. So that would now be the logarithmic function, right? Math.logarithmic. So now I could go ahead and say console log underscore U32 and call this log function. Well, this is a bit, the naming is not very nice, right? Because it looks like it's a log statement but this is math.logarithmic. So we would probably go ahead and say, okay, the js underscore, let me quickly get that into here. So I can actually write that. js name is log math.log and here we would probably say like math underscore log and integer would be a 32-bit integer probably. So this is how we would inject the math function, right? So math.log would now be available for us at math underscore log in Rust. And then we can go ahead and say, okay, and we need a third function. Voila, oops, okay, good. And that is the console log function again and this time we wanna add a 32-bit integer into it. So we would say this is now console underscore log underscore I32 like this. So now we have both methods in there, right? So we now injected the math logarithm function and we injected console log but under completely different names. So they are not kinda messing up our brains. Does that make sense? Okay, so let's try to do that. This is a short exercise, just try to do whatever function you would like. Pass it in, see if you can call it, if you can alert hellos. Like this example that I think I did here, you could do a alert hello world or something and have fun with it. Yeah, let's take five to 10 minutes, try that out and that will be the last basic exercise. After that, we will do some fun image processing that we actually see whatever assembly is capable of which is hard to learn the basics first so we understand what happens under the hood. Be there in a second, yeah. Just quickly, we've got this little bit here and I was curious what it's kinda like waiting until that's ready before it calls the function that uses. Yeah? The grid? Yeah. We're importing grid from here. Yeah, this is only setting up the WebAssembly instance, right? Because wasn't bind gen already knows what you are exporting in Rust because it was there at compile time, right? So it already knows you are exporting a grid function. That's why it provides you that output, but. I can't call it straight away though. No. And so this is, what is this doing to the grid function? What this is doing is it loads the binary and then it does WebAssembly.instantiate. It does have to set up the WebAssembly instance, right? All that just is what we're doing. Exactly, exactly. This is all it does now. Yeah, right. Does it return anything? I think, I don't think so. The instance? No, I think this is all handled in that global object. I don't think it has a return tab anymore but I'm not 100% sure. Makes sense. Yeah, okay. Yeah. So, my question is, do you know what will happen if we try to load a function that might be on the find, like something in Navigator, which differs between different browser? Good question. So, when you look at the JS that it's actually putting out, did you build that now? Not yet. Try to build that quickly because you see what it actually does and then you can kind of see what you would expect to input there. So, now search for one of the methods you imported. Like what did it do? Console the log or something? What do we have? The alert. Search for the alert function because you should see it here now. Oh. Oh, yeah. So, you see what it does. It's fairly weird. So, you see, actually it creates a new function for you and in there it will call the alert. So, it kind of wraps your function. So, if that would be undefined you would get, undefined is not a function, right? Yeah. That would be what happens. Okay. So, In that specific case. Put it from error in JavaScript. Yes, it will throw an error in JavaScript. So, it will even let me run wasm file. Exactly, exactly. Yeah. Well, no, it would probably run the wasm because this will be called from Rust, right? We will call it from Rust, but then on the JavaScript side it will fail because it tries to call undefined. Yeah, so wasm will pass the correct params Exactly, exactly. Cool. Cool. Okay, let's already get into that because this is just a last quick exercise. Let's just do the basic example, right? I think most of you already have it running. If not, don't worry. So, if we have that external block, right? This is all we do. We just have to call it from somewhere. So, I mean we can make a new function, right? So, we can say, well, let's be honest with our function names, right? Let's call it alert. Let's call it alert hello world. This gets in a string and will return absolutely nothing. So, what we will do is we will call alert like this, right? So, we can call that alert hello world and this should actually call our alert function. So, we should get a stupid alert message, right? So, let's just try that. So, instead of greeting here, we just call that new alert hello world like this. And since we don't have a return type, this line down here doesn't make any sense. So, now we call it like that. And this should now throw an error because I did leave something out that is relevant. So, let's just try and build and run it and see what Rust says to that. Oh, oh. So, it said, I am expecting you to add this reference string but you give me a correct complete string. So, what I have to do, the issue is down here, I have the format macro which is returning a string. But as you see alert is expecting this. So, what I do is I add that and say, put the reference of that in and Rust will be totally happy with that. So, let's see. Yeah, you see up here, it's not the best contrast. So, the native alert of my browser fired. So, if we go here like confirm, you know the confirm right with the buttons on it. So, let's just call the confirm now. What happens? You see I got these buttons up here. So, these are really the native methods, right? We can call them now. Cool. So, this was all for the basics. I hope it was okay for you. It was done for the basics but now we wanna see what an actual real-world use case looks like. So, how do we do, for example, image processing? But I would advise that we make a short break of 10 minutes and start with the last exercise at half past three. Cool. Soon, but we will already start. Otherwise, we'll get into a hurry for our last exercise and we don't want that. So, go over to the lowest, the last one, chapter five, real-world use case. This is what we will look at now. So, what do I think of real-world use case? What we will use is OpenCV. Does anybody in here know OpenCV? A bit. Are you actively using OpenCV? No, but it's really cool. So, we won't do hardcore functions of OpenCV. So, I will explain every function that we need. For those who don't know OpenCV, it's one of the biggest libraries for computer vision. It is written in C++. So, as you may think, this CV in OpenCV is computer vision. And the cool thing is, it's C and C++, right? And if you remember correct, if you remember from my first couple of slides, C and C++ are pretty much done with WebAssembly so you can compile them to WebAssembly very easily. So, let's do that then. Luckily, there is somebody at OpenCV who already did this. So, the standard build of OpenCV can be triggered to compile to Basm instead of native architecture code. We won't compile it ourselves because you would have to install C-Lang compiler and MScripten and you would compile it for five to 10 minutes. Since that is kind of a not so nice thing to do, I did the pre-compiling for us. So, we already have a compiled OpenCV library and it's eight megabytes big. So, it's not optimal to send it to the browser often so we'll try to cache it. Oh yeah, I forgot to mention MScripten. So, we used Vasen-bind-gen for Rust, right? So, that helped us with those bindings from Rust to JavaScript. On C and C++ side, we have MScripten and you can basically, if you put it very simple, that's not entirely correct, but if you put it simple, it's comparable to Vasen-bind-gen. It does also do JavaScript bindings. It will help us to have this high-level interaction with all the types we want to. So, this is gonna help us a lot and this is what OpenCV internally uses when you compile it to WebAssembly. So, let's do image processing, shall we? The first thing you have to do is check out stat little and give your repository. Shouldn't be too big. This is the setup for our last two. Two exercises that we will do. When you have it, when you downloaded it, execute npm install and npm start and then you should have a local running web server on port 8080 or if your port 8080 is actually taken, it will start on 8081, 8082, whatever is free. It will show you in the console. You will not have to restart that web server because it's just a plain static web server. We have no build come, we have no build step, we have no post pre-processing, nothing like that. So, this is really a stupid web server. So, you don't have to restart it or anything. Just keep it up and running and the only thing you have to do is refresh your browser when you actually change things. Okay, try to check that out. I hope everybody has an editor with them. So, like your favorite editor on you. It's not like you need a super-duper editor so it will be okay with pretty much any editor. Did everybody manage to clone the Git repository? Everybody has it? Perfect. Cool. As soon as you have it, run npm install and npm start and then you should be able to visit localhost 8080 or whatever it says on the console. If you don't get a page shown, I've seen that on some Windows machines that can happen then just add slash index.html to the URL. Sometimes HTTP server doesn't automatically redirect you to the index.html page. I haven't really found how to properly solve this. So, if this happens, just add slash index.html in the end and you should be good. If everything works all right, you can open the console and after some seconds you should see that console message over there, runtime ready. What that means is WebAssembly started with OpenCV in it and OpenCV runtime is ready for you. We don't do anything yet with it. So, don't worry if you don't see anything. Cool. So, do you see that message? Who sees that message? Okay, like half of you, right? Okay, does anybody have like an error message or are you just not there yet? Error messages. I'll come and see quickly. The message port closed. No closed one. Oh, do you have some sort of plug-in up there? That looks like a plug-in issue because you see it's not from the main chess. Open it in an incognito window for once and see if you get that message there too. No, okay, so it's one of your plug-ins. Don't worry about that. Who had an issue? Two? Nobody? Oh, cool. So, who sees the console log message now? We have some of you, okay, not too many. Let's wait for another minute. Anybody have exceptions or something? Are you stuck? Or opposite? Yeah, it seems to be. This looks good, huh? Yeah. Can you try to add slash index.html? Slash index. Yeah, exactly. Okay, that's the issue that I was mentioning, so I don't know why. So if you go to the console now, you should see that runtime ready. No, the browser console, sorry. There it is. Yeah, it's the browser console where that console statement should be at. Not the terminal console. Okay, I hope that it works for you then. Otherwise, when we are at the exercise, I will come to the people that still have issues. Good. So what do we do from here, right? That's not very interesting. A very short introduction to the project structure. We have three main files that are relevant. In the binary folder, you have that opencv.js file. This is the compiled opencv file. Maybe don't open it. There's a six megabyte base 64 encoded string with the complete VASM binary in it. It might kill your editor, depending on what you use. You can open it. You will see a lot of funny letters. But yeah, maybe don't. And then we have the index.hml and the main.js. So a nice little web project, no build step, nothing. So I kind of like it already. Cool. Yeah, I think I told you that. So the difference to what we did before, this one is not having a VASM underscore bind gen on the window object. What we will get is a cv object. So whenever you want to interact with opencv, we will go over that cv dot whatever it is, then whatever function we want to use. Cool. So let's do our first exercise. First, when we were to have real time image processing, we need some sort of video stream, right? So let's go into WebRTC. Who knows WebRTC? Okay, not too many. So I'm going to quickly go into it. With WebRTC, we can basically get streams of audio, video, or both in the browser. The idea, the basic idea where it came from is that you can have video chats or audio chats in your browser because that stream that you have, you can send it over the network and then you can basically make your own Skype if you want to. And that is pretty much integrated in every browser nowadays, probably except for IE11. I wouldn't know. This is just a random guess. We won't use that for sending it over the net. We will just fetch the webcam camera stream. So we have a stream that we can sample on, right? That we can do image processing. This is actually not all that hard. What you do is navigate to dot media devices. This is where it's located at. And then you can say, get user media and you have an options object where you can say, what do you want? If you look at the documentation, these don't have to be boolean. You actually can add a config object. So you can say, I want the front facing camera. I want the back facing camera. I want it in landscape mode. I want whatever. We are just gonna be very simple here. We say, you have one webcam stream probably on your notebook, I guess. Just give us that. That's all we need. It's a promise we can say then and in here we will get the stream. That's all we want. Cool. But what do we do with that stream now? We can get the video tracks. Since we are a very simple webcam, there will be one video track. So we're just gonna say, give us that first video track and not care about it anymore. We will use that video track later on. You see it down here. We will have the width and the height of our video stream on there. We don't use it from the start. So don't worry about it. These are the settings that we get from there. When you look at the JavaScript object, you see that on the top two lines, I have two elements that I get from the HTML. You can look at them. It's a video element and a canvas element. The first element we are gonna use is the video element. So what we can do now, the video element has a property named source object. You can just set this stream onto the source object and what will happen is you will see yourself. Yay. So let's try to do that. Get the video stream added to the input object, a input video element and see if you can see yourself or if something goes wrong. Does anybody still have problems with running the project? Like basically running it. Let's just take five minutes or 10 minutes for that, depending on how long it takes you, but it shouldn't be too hard. Yeah. You have to check out a little Git repo there. It's in the fifth chapter of the, maybe go back to the index. I think it's at just exactly there. And it's the fifth one. And then I think pretty in the first and the second, one more or another, you can go one to the right and then down. This is the GitHub repo that you have to check out. There, this one. Cool. Yes, there you are. Got it? Oh, I see faces all over the place. That's cool. Very good. Did it work? Yeah. Yeah, another face down here. I like cameras. It's working for you. Nope. Can't use the media, I'm only fine. You don't have to get used to it. We're in Chrome. No, it's- Can I just try it out in one second? Can you make a hard refresh? You should have media devices on there. Did you, oh, can you check your 086 settings if Chrome doesn't have access to your webcam? So, where is it? System security, probably, I guess, or Eingabe. Oh, you have it in English, right? That's like accessibility. Oh, no, this is just if you can use the audio stuff. Can you try that in Safari, quickly? Oh, there's the camera. You have to click on the lock on the bottom left. This is weird. Chrome should ask you for that, I think. Can you try it with Safari? It should work. Yeah? Last example. Uh-huh. This color pops up. Ranging out of memory. Probably because of other memory. Do you have some sort of loop that is- Not really, just copy-paste. We can look at that later on. I have to move on. I see a lot of faces on the screens, so that seems to be working all right. Does everybody have some major problems with it, or does it work for everybody? Working? Cool. Okay, so that's not really what we want to do, right? That's just what we needed to do to get an input stream. So now we want to have OpenCV, right? Now we have that input stream, and now we want to make sure that we call our C library that we just cross-compiled or we compile down to Wazen. Now we want to really call it and see what happens. So how do we do that? Luckily, OpenCV has a thing called video capture. That can capture frames from a video element, very convenient. So we can say where we have our video, and then we can say videoCapture.read, and it will read a OpenCV matrix out of that image, and then we can work on that matrix, which is cool. So this is how it looks. You need to have a global variable cap, and then you can instantiate it with our input video element. We can just say cv.videoCapture, and we can instantiate that with new, and this is how we create our first OpenCV object, right? So OpenCV is now instantiating that videoCapture thing for us. So this is already a very complex type, right? So we can already share that between WebAssembly and JavaScript, which is kind of fascinating already. So this is how we set it up. Now OpenCV knows there's a video stream going on, and we just have to pass that into OpenCV now that we can actually fetch. What we need is the output canvas needs to have a width and a height. We wanna paint that picture that we get from OpenCV onto a canvas now. So whatever we do on OpenCV, we do some transformations and whatever. In the end, we can't put that back into the video element. We have to render that image that we just did onto a canvas. So I made that HTML element already available to you. It's called output canvas, and you have to make sure that the size of the canvas matches the size of our stream. Otherwise, OpenCV will complain and say, well, you try to paint a matrix onto a canvas, and they are not the same size, this doesn't work. So it's important that we set width and height of our canvas. Then we need two matrices. One is the source matrix. This is what we get out of the video stream, and the other is the destination matrix. In between, we will do some conversion on that image, right? So we read a matrix, transform it into a destination matrix, paint that destination matrix onto our canvas. This is the steps we're gonna do. This is just how you instantiate them. Also make sure that you do that and that start streaming function. We don't wanna create a new matrix with every frame, right? This is not fun because that would be heavy on the memory. We can always reuse the last matrix, right? We don't have to do a new matrix all the time. Yeah, also here, it's important that height and width are really correct. And behind, well, in the end of that line, you can see these two values of OpenCV. What they mean is, you have heard that already, 8-bit unsigned int is 8u, and c4 means how many channels we have. When we read from our webcam, we get four channels. Basically, it's our GB and a alpha channel. So this is what we get in. And now, for our first exercise, we're gonna do a gray scale. So we reduce from four channels to one channel. That is only gray. We will remove all the colors. That is our first easy exercise that we're gonna try. Yeah. So we have four things that we have to do. We saw it earlier. We set up the video capture at the matrices. That is already cool. Now, we have to have some sort of painting loop, right? So we have to read from that video capture. Then we convert to gray scale, and then we draw to the canvas, and then we call us again. And then we have our basic animation loop that will fetch from the video stream, gray scale, all the things, draw onto the canvas. So how does that look like? The video capture is actually super easy. You say, dot read and hand in the source matrix. So what it will do, it will transform that image to a matrix. So now our source matrix has an image in it. Cool. Then we convert the color. So OpenCD has a method called convert color. So we say, we go from the source matrix, we will write into the destination matrix, and this is the color transformation that you have to apply. We go from RGBA to gray scale. And last but not least, we make an EM show. It's called IM Show. We have to input the canvas element here. So OpenCD knows about that canvas element, and we say which matrix should be printed or drawn onto that canvas. And since the destination matrix is now our gray scale matrix, this is what we wanna do. And in the end, loop it. Who knows request animation frame in here? Cool, cool, okay. So for the ones that don't know it, we can ask the browser to call us in 60 frames per second if possible. This is that function. We can hand in the function and the browser will schedule us and will try to call us every, what is it, 13 milliseconds. So we have 60 frames per second. If that doesn't work, if we are too slow, he will just schedule us as fast as he can without blocking his rendering loop. So this is just a way that the browser makes sure we are not spamming user interaction between browser and JavaScript. So in the end of our function, we can say, okay, we are done now. Please reschedule us. We wanna draw again. And then the browser will let us draw again. Yeah, this is it. So you have all the code you need for the exercise. Put it together and see if you get a gray scale image of yourself on the right side. Let's take 10 minutes for that. Exactly, it has to be yet. You can set that up here. You don't have to do it in the loop. Well, you can do it in the loop, but you don't have to. That's work you need to do once, right? And then it's good, yeah. So I just have to do the capture process, is that right? Yeah, I would do the initialization of these things. You can also do up here, right? You only have to do that once. So this is the transition, I don't have to do it. This is not just capturing one of them, I just start capturing it. Yeah, this is, we do initialize all the opens. TV objects that we use, yeah? And this is the function we wanna repeatedly call now, right? So this actually do the actual reading frame? Exactly, this is what we do for each frame, right? That we can. So this part should be in here. You should call it from here once, but that function should stand somewhere on its own. Like outside of your context, for example. That's why we save these things into the global scope. Okay, okay. Fun, if only inside here would not be metalized, this should still work, right? Yeah, it will work, it will work. So we can leave it in there if you want to. The only relevant thing is that inside the function, you have to call that window request animation frame again, right? Because like this, you will only be called once. Okay. And you have to tell the browser, yeah. Okay. Okay, that should work. I'm just gonna size this much. Oh, size of implement, the size should be the same as the video. So let me see. Nice work, okay, okay. Oh, you have to do that after you set the stream, because before that, it will not... Oh no, no, no, it should work. It should work, you're right. Sorry. Because I'm getting the value from the video track. The video track, yeah. That should be all right, yes, you're right. Um... Can you try to set the width and height of the input video as well? The same way you do it up there, just for the input video? Uh, you need to move these down? Yeah, no, no, just copy it, duplicate. Um, right below them, it's... Here? No, no, right below, right here. So, here. And now, um, do it on the input video instead of the output canvas. So both are set to the same size. You know, the input video is the video element, right? Try to do it there. Yeah, yes. Okay, try that. Yeah. Oh, so there are two lines of codes missing, I am very sorry. Uh, I made that example up here that you have to set the canvas width and height. I forgot to mention, here, you have to do the same on the input video element, otherwise OpenCV will complain and say your video is not the same as your canvas. So, you have to do these two lines for the input video as well. You can do it on the same spot. Sorry, I forgot that. Why is it black? What happens? Oh, your own function. Oh, you need to do this part in one specific function that can call itself again, right? You can do that outside, whatever you want. You can say like process video or something. Exactly, and then you can, you need to put that into here as well. The function, right, you have to, exactly like that. Perfect. That should work. Yeah, there you go. Yeah. Let me quickly see. So, the width and the height, you can get from these settings. Settings has width and height on it, so maybe do all of these things up here, like until cap read, like this whole block you don't, you only have to do once, so you don't need to do it. And the video capture too, you can do that up there, this one, yeah. So you only need this in your actual function, yeah. And then you can go for settings.width, yeah, and settings.height, exactly, perfect. Okay, so that looks good. Now, we just have to make sure that the window request animation frame is inside your function because you want, at the end, when you are done, right, you want it to be called again. Cool. Try it. Oh, doesn't allow you to do that. Okay. Okay, almost. Do we get an error message? Nothing. So, can you show me the function again? That looks good. What do we have? Source. Fortress. This actually looks good. I don't see your issue. Sit it back to defaulting. Yeah, that's good. Width, height, you're doing it the right way. Huh. You can't currently see what the issue is. Can you reload your browser? Huh. Can you try it? Are you using Chrome or Firefox? Can you try it in Chrome just for a quick try? Not that it should work for Firefox as well. I just want to make sure that this is not some weird bug. Okay, cool. We are having an actual error. But I don't see it currently. So, input video is the capture. You have video height, video width, settings width, settings height. You have the matrix. Bless you. No, you don't have to do anything on the HTML. Or you shouldn't. You shouldn't have to. That looks all good. Cap, source, test. Process, video. Can we go down again? Like, is this right? CV color RGBA 2 gray? Oh, oh, it's 2 gray scale, I think, isn't it? Can you check on the... Or did you copy it? This one now, it's that one. Huh, that's weird. Can you... In my opinion, you're doing the right thing, yeah? I don't see it currently. I can't really know that, unfortunately. You see it in the browser though. If you go to the page where you have it open, and you enter CV.color. Oh, you have to write it in small. RGBA. Hmm. This is weird. I'll have to continue. There's a branch with the solution. We can then look at it, what happens. Okay, so I hope most of you have gotten it to work somehow. I included a solutions branch. So if you check out the branch solution gray scale, it will do the code that you need, and it should work for you all. So what do you have to do? Oh, this is a bit small. Can you read that? Is it good? So what you needed to do is... I did the specific pixel width. You could omit the pixel because the browser is thinking that you need pixels when you don't say anything. So we still hate... And we set that on the input video and the output canvas. We set it on the source object. This play that I added in is basically for iOS Safari, because Safari is very restrictive about auto-playing videos. So you don't need that specifically. I added it in that it works on most browsers. Then we have the matrix settings. As you've seen them, the video capture is set up and then we say, go ahead, process video. So what you see here, I have added a little something in a frame counter. So you would go with read, convert color, show, and then request animation frame again. So let's just look at it, right? Let's go for localhost8080. Voila! This is how it should look like. And you see up here, we are getting 60 frames per second. Well, 59 frames per second. This is what it should look like. So that's a very easy example. I mean, grayscaling isn't supposed to take a lot of processor power, but we can do something else that does. So let's look at that now. Let's go to that... We are in C, so we have a lot of troubles with memory leaks. Now, since we start when the web page loads up and we end when the web page is closed, we don't need to care as much, but usually all these objects, the matrices, have a delete function on them. If you don't need them anymore, you would actually have to call delete because it's C, right? We have to free memory. It won't be garbage collected. But in our example, we won't have to care about that as much. So let's go to the fun part, right? So we have a stream. We have OpenCV. And OpenCV can handle classifiers. If you have been at the talk yesterday, I think it was, about classifiers that can detect is it a cat or a dog, we will use a classifier that can detect is it a face or is it not a face. It's not the best classifier, so you will be able to fool it somehow, but it is an OK classifier. So that's the first thing we need. Classifier. Then we need to run that classifier with each frame. We already have set up that loop now, right? And if it didn't work for you, check out the solution branch, then you have that loop set up. Then we get a vector, which is basically an array, right? So we can go through this vector with faces and draw these as rectangles on our video stream. The classifier that we are going to use is called HAAR Cascade classifier. That's a pre-trained classifier. I haven't done that. This is from OpenCV. They provide that. There's like a standard classifier for you to choose. So we don't have to train that. We don't want to train it here. We don't have time for that. So we can just take that pre-trained one. Then we load that to an XML and we give it to OpenCV. You don't have to do that because it's a bit cumbersome. We have to fetch it, make a byte array, input into OpenCV. I already did that for you. And I did it on a branch called ExerciseFaceDetection. So when you start this exercise, check out ExerciseFaceDetection. So what we have to care about now is we run that classifier. The classifier is designed on a grayscale image. So you already know how that works. But the end result, let me show you the end result. So there's also a solution face detection. Please don't go right there. That won't be fun. But we will do it now that you see what we are trying to achieve at least. So let's go on there. So here you see. Oh, you see there is actually something up there so he recognizes my face over up here. That's cool. Oh, he's not so sure. Oh, now he's pretty sure that this is also a face. So you see it's not the best face detection algorithm. But it's okay. It's okay. So we don't want this image to be grayscale, right? That wouldn't be fun. So what we have to care for is that we have a new matrix that is not grayscale because that grayscale one we use for the classifier. So we will have a third matrix that is just to show to the user that is looking cool, right? That has all four channels on it. So we basically need a copy of our source matrix for that. So this is how we can copy a matrix. Nothing very fancy around here. This matrix for the grayscale we don't have to initialize with hate and width because we are not going to put it on the canvas or read it from the camera. So we will get the source matrix in here and it will get the size of that source matrix just inherently. So this is basically what we do instead of from source to destination, we go from that copied destination thing to gray, that new matrix that we just initialized up here. Just one more matrix. Then we need to instantiate a vector where our algorithm will put the faces in so we know what faces he found. And this is how the classifier is then run. You can look at the bottom, this is where you see how the classifier is instantiated but it's a bit cumbersome so don't worry about that. We have to load the XML files and everything. So this is just a new classifier with that XML file that we just loaded. Detect multiskiles so it will scale the image in multiple factors which faces are real faces on all scaling factors so he is pretty sure that this is a face. The values down here or at the end here these are values that probably people that have actual experience in image processing can tell you more about. I don't know them very well. These are just like default values to have a basic face detection algorithm running. So when we do that here we see that we put in that faces vector. After this one ran, that faces vector has faces on it. So we just need to look through that vector and paint these faces. Easy as that. This will look something like this. Since this is a C object now, that vector is not a JavaScript, well not a real JavaScript array. You can't do .forEach .map or .filter so you have to do a classical .forLoop over it. Get the size with .size and get the face with that index axis. A face has four things on it. X, Y, where it is and height. So we only have to draw a rectangle with openCV this is called cv.rectangle this is how it will look like. We draw it on the destination matrix. So we will make a rectangle on the destination matrix. We need two points. Here I gave you the code for one point. This is top left. How you get to the bottom right is your exercise and this is the color. RGBA. So this will be red. You can do whatever color you want to there. One small note. I scaled down the input to 20 pixels. This is where the algorithm works best. Usually you wouldn't do that visibly to the user. You would take the matrix, scale it down internally, give it to the classifier and then scale up the rectangles again and the output destination matrix would be full scale again. So the user has best possible resolution. But this is just more code. So what I did is just I cropped the input matrix to 320 pixels. That's not very nice. But it is fast and performant. Yeah. So you can go to the exercise face detection branch and try to get that up and running. And you also can try to fiddle a bit with parameters and colors and you can also do circles if you want to draw a circle around the face. Whatever you like to do. If you want to check it out I deployed the solution on the cloud. So you can go on to here and you can try it out so you see what you should be aiming for. Yeah. Tell me. Yes. Oh look. The problem is you try to read the matrix here. But you're setting the width and height here. So this is after it reads. So here it will throw an exception because it has no width and height. Okay. After I read it. Exactly. That's can you go back to your, this is the problem you see. You are reading and while reading he says there is no there's no size on it. So you have to do that up here somewhere. Before we have to define them only we have to read it. Yeah. Here you need a dollar in front of the, yeah exactly. And there too. Cool. So this should work. Oh wait, wait a second. Can you go back process video. Where is that function? Oh yeah. You have to put this part into that function. Right. This is the one that loops all the time. You don't want to do all that, right? All the time. You just want to do D3. So maybe make a new function down here or something and put these in. No, one, one yeah exactly. All this needs to be in there. Exactly. And you can put that in. Perfect. And now we just have to call it once from up here so that it actually initializes the loop. And this should then work. Oh, what did it say? Huh. Oh, you mean because of caching? I don't think so. Let's see again your code. I think there's just, probably just something very small. A window. Ah no, settings sorry. Settings. Not window. You have to take the stream. I think I will do the quiz now because people are kind of leaving already. So we can get the chocolate. You already showed it. I already showed it, right? Well. Oh yeah, I can. And Oh no, where is it? What does it say? Can I do I am sorry, maybe. No, no, no, you don't need to restart the web server. That's just a static web server. He does nothing. Can you go into the main chairs quickly? If you go to sources. Oh yeah. Yeah, this is definitely the newest code, right? Okay, so let's go back to the code. We missed one height or one width. Input video. It's twice. It's height. Oh yeah. Yeah, up here it's width not height. Copy and paste. Don't worry. Cool. So now it should work. Reload and that should work. Here you go. Cool. So I see that already some of you are leaving. So before all of you are gone, there are two things I want to say. We will quickly look at the solution so you can work on that on your own if you want to. And I will do a little quiz to get some Swiss chocolate. So the three best ones will get a big chocolate. Let's actually look at the solution. Just brush over it very quickly. So you see the only change that we have to do is in process video. We will copy to that new destination matrix. We make that new grayscale matrix. Then we run that multi-scale classifier and then we loop over all faces. The bottom right point, we can just say face.x plus width and face.y plus height. So we get the bottom right or from your point of view the bottom right point. That makes more sense. And the rest is the same, right? We have that rectangle and we still do image show. So if you checked it out, I think we are not getting the best frame rates. There are people that optimised this better. So if you look at it, I think before we, oh, that's good. So before we were able to get up to 15 frames per seconds sometimes, yeah, you see now 14, 15 frames per second. Not very good. So if we would be optimising that thing and website people did that, you see, we would get they get 30 frames per second for WebAssembly. They did the comparison to asm.js and native. That's the same kind of algorithm we are using just, we are not using it very optimised we just do a very simple example. So if you do it with asm.js it is around 20 frames per second and native JavaScript is at two frames. So this is where actually new native performance makes a very big difference. If you want to be somewhere close to real time or something in that direction you should totally go for WebAssembly in that direction. Cool. Yeah, I think that's probably my sum up of everything. You don't need to use WebAssembly to have minor code improvements on your JavaScript code base or something like that. WebAssembly is meant to run heavy stuff. So, use it for that. Don't use it to get your Angular up faster or something like that. That's not what it will be good for. Cool. So all the resources we used, all the stuff we can read up on. I mean, you have to link to my presentation now. Yeah, let's do a little quiz in the end. I think you have already seen that there is this Kahoot thingy, right? So, go to kahoot.it and enter that pin and your name and then we can have a fun little quiz and after that we will be done. Okay, we have naughty nicknames. Oh, okay. Cancel. Nice. Cool. Are we all? Is there somebody who wants to try? We're all. Good. So, get ready. I fucked up with the logo. Sorry. What's the short form of WebAssembly? You have 10 seconds time. So, the earlier you click, the more points you will be rewarded. Ooh, I think all already answered that. Cool. Yeah, and obviously some might have clicked wrong, right? But almost everybody knew that. Cool. So, we have somebody in front. Johnny. I hope I pronounced that right. I'm sorry if I didn't. Cool. What's the next question? What was our IDE call used for our basic examples? I see. Way too easy questions for you guys. Way too easy. Oh, wow. Okay. Somebody was very fast on that. Okay. Move up to the next question. How do we tell the Rust compiler to not remove our function? So easy questions. I mean, so easy. Cool. Yeah. Who is still up front? Let's go on to the next question. What's the name of the computer vision library we used? Yeah. Who's the fastest presser? That's cool. So, somebody clicked wrong. Didn't take my clicker first. So, Kahoo seems to be very fast. That's cool. Let's go to the last question. How many employees does Sulka Singapore have currently? I mean, there has to be one to decide, right? So, is it 34? Is it 35? Is it 36 or 37? Oh, that's a nice curve there. Cool. Okay. That was a fun question for the end so that we definitely have a kind of people on top. So, congrats to these three people. So, come up front in the end. You can collect your Toblerone here. I hope you have fun doing that workshop and that you like WebAssembly now as much as I do. And yeah, have a nice rest of the conference. Bye, guys.