 I am in a weird bend diagram of people who know a lot of what's there across the web assembly. So let me know if you know anyone else who fits in the middle there. And I'm a demo enthusiast. You're going to see a little bit of a demo, but I don't have a ton of stuff to show you today. So I like to start off by saying what I'm going to talk about, start with what is assembly, then talk about what this was and talk about the nightmare, which is what we ran into with distributed applications, and then open telemetry, implementation, and talk about what's next for us and what's next for what we've seen for our observability. Now, before we get into this, I'm actually pretty curious, who has heard of bug assembly before. Nice. That's a pretty good amount. Who has heard of bug assembly in the context of running on the server, not in a web assembly? Kind of a smaller amount of people. So today, we're going to be talking about what is assembly on the server side, not specifically running in the browser. So it's like, okay, I need to open that. Is that fun? All right. Cool. So we have this kind of in this way of introducing bug assembly as neither web assembly because what it started as is a way to run native code like a C program in the browser. Needed to be small, needed to be efficient, needed to be fast, and we want to be able to reuse a bunch of code that we have in current C libraries. But when you look at those aspects for deployment target, there's a lot of things that are even staying on the server side. We want our services to be rapidly scalable. We want them to be small. And when I say small, a typical web assembly model is 20 kilobytes to 200, 300 kilobytes. And we want it to be secure. We want it to run in a denied by default sandbox that gives us security in a way that a lot of popular platforms don't give you today. There aren't very many, you know, we need to avoid things in an always in a container. There are lots of things that you can kind of pack on top of that like set com, like security policies. But when the assembly starts that in the beginning, everything is denied by default. It's an open W-increasing standard, so this is supported by the same organization, and kind of just forward HTML, CSS, JavaScript, you know, the development of the spec is still supported by them. And notably is polyglot. So the whole native web or assembly is, you know, first model for web, but then the assembly is, it's not really like the assembly language that you would write. You write code in the language of your choice, and you compile it to an assembly. So, and sitting at analogy is like the way that Java works. And I know that, you know, some people may take events for that analogy, but like you write code in Java, you compile it to Java by code, and you run that on the Java VM, right? For web assembly, write code in the language of your choice. The language's list is constantly growing. You compile it to a web assembly, and you run it on a web assembly run time. So the web assembly part itself is completely architecture and operating system like that. But even with all the promise, all the great things that I'm saying, there are still three large gaps in running web assembly along the server side. The language support is really great for some languages, Rust, C, Grain, you know, some things like assembly script, but it's still pretty limited. Things like Java, scripting languages, some other languages don't really work quite well with it. But it's still quickly growing. I came from Plot. It was a day a little bit earlier today, and there's constant language talks about, like Python support, Ruby support, things like that. Networking was a huge problem for a while, but it's still, you know, even though you can technically open a socket with web assembly, it's still pretty rough around the edges. And when you interact with a web assembly module from outside, the interaction model is like importing and exporting functions. So it's still pretty rough. When you get down to trying to write a web assembly module and you want to write a hello world, right? But you add to your feeling great. You write hello world. As soon as you have to take a string and pass a pointer and a link to read an array of bytes to write a string, you go like, all right, obviously, but this is not the thing that I would be writing. But everything else is so complicated. You know, all of this is leading up to what I and my co-maintenors have been working on in the Wasm Cloud for the past two to three years. The way that we see the modern computing environment is things are getting smaller and smaller, and the more and more complexity is abstracted away from the developer, you know, starting from everything tightly coupled to a PC, to the public cloud, the virtual machines, and then what much of the world runs on today, which is containers, we're further and further abstracting things away and reducing developer responsibility on these aspects. And of course there's some nuance to that. But the status quo is you're not worrying about the underlying kernel when you're running a doc container. You still do have to kind of compile for Linux, and if you want to cross-compile like the new AWS traffic lines, since it's coming out, and you weren't doing build-nets before, you do have to compile the harm for those, right? The way that we see WebAssembly is that it creates a completely platform-like non-stake target, so that now developers, they write their code, compile it to WebAssembly, they can run it anywhere, and that reduces their responsibility and management burden for people who are actually doing DevOps or other platform work. Now, Wasm Cloud, the project, this is an application run plan. It's focused on distributed computing. So running these WebAssembly modules wherever and seamlessly connecting them, we have to do with a technology called NAS. You may have heard that you may have used it as an SD broker. We're also abstracting away non-functional requirements for what we call capabilities. So when you think of an application you want to write, like a to-do application, your functional requirements are you want to create, read, update, delete, to-do, right? That's the code that, you know, business logic, the code that you as a developer want to get down and writing. It's not the non-functional requirements that you need for your after work, but it's not code that you're ultimately writing. Your HTTP server library, make sure that you're getting with us your key value connector, the library that you're sending key value requests back and forth. These are open source projects that you import and then you have to manage, but you haven't actually done it. And so, you know, this is what and this actually works really fine for a long time and it's not that complex. But when you see things like vulnerabilities coming out of open source libraries, when a vulnerability happens in a library that isn't your own, you know, in the source, you still have to go and update that and see, recompile your container, redeploy as a product from all those things. There are automated tools that certainly make that easier, but wouldn't it be better if I like that in the beginning? Now, I've talked about Web Summit a little bit and I'm going to talk about Wasm Cloud and our five different layers of architecture and I'm just going to preface this that my goal of this talk is not to completely just sell you on the product that's working on. This is all open source in any ways, but I want to lay the foundation because when I talk about the way that we've implemented open elementary and tracing across our entire stack, it's important because there are a lot of complicated layers. So the base level, we're running Web Summit. We run Rebson Lee, developers write it, so we're taking advantage of all that platforming mastic on the aspect of Wasm. The Wasm Cloud application runtime is written in Elixir and Rust and it kind of deals with the orchestration of each Web Summit modules and connecting them to those non-functional requirements that we call capabilities. And capabilities is kind of a loaded term. People talk about capabilities in a purely academic sense, like access to a single file or access to a single socket as a capability. We get it in a more broad contract sense. You are coding your application in terms of an abstract contract, something like a simple store or a mastic broker or an event source system. But you don't actually mention a vendor. You don't mention the runtime implementation because you actually have to take that. On your local machine, you can code it with Redis. You can deploy it abroad in whatever enterprise cloud, whatever you use that key value store instead. It doesn't have to be a Redis one. It can be like an object store that implements key value functionality. This is really important because the move from development to production is still a hugely painful story. I previously worked at a really large company, a large U.S. financial institution and the often needed developers are waiting 15 minutes in the development loop because they make a change in their container. They push to get up, Jenkins picks it up, it builds, it deploys the dev, and then you see if your code actually changed. That's a huge pain in the butt on Season On, so I know that people just made it. The composable app here is these are our web assembly modules where you implement the functionality of your app. This is what developers are writing and what's really important for us to experience is needs to be crisp and painless experience. I'll show you a good example to hopefully know this one. The last part is the lattice network. This is what seemingly joins all the compute together no matter where it's physically running and then this is all enabled by NAS under the hood. So messages, RPC calls are passed back and forth with NAS. It doesn't matter where it's actually running. Now Cosmonic, company I work for, building a distributed platform on top of the Wasm Cloud. We actually diluted our whole product. I'm giving another talk about that on Wednesday at the main conference, which is fun. Please let me know if you're interested in that and hope to see you there. But the whole goal is a painless experience for people developing applications. And when you think about tracing and open supplementary and metrics and logging, those all fit into the non-functional requirement piece of an application. So you don't actually need tracing to run your to-do app. But if something goes wrong, sure would be nice to know where that thing went wrong and how long things are taking to your system. So when we're thinking about this from the perspective of the user, it's a non-functional requirement and we don't want people to have to deal with that. So a simple Wasm Cloud application, I said it to do the app and another popular thing to do is a key value counter. So receive HTTP requests, increment value in key value store, returning HTTP value. Pretty basic. The key value counter, this is your WebSimly module. You write the code like get HTTP requests, return HTTP response, increment value in key value store. The actual code for this is going to look pretty similar to what you may have seen the light function as a service offering because it's a reactive piece of compute as a single function like I said, receive HTTP requests, return HTTP response and even if you haven't seen the rest before, hopefully this isn't too difficult to grok you're getting a key value center no mention of if this is Redis or you know what app you have and you're incrementing it in a key value store returning the new values as bytes. So we talk about tracing for users of the open source or users of our product. We don't want them to have to come in here and had all these like macros derived traits and things to just add a span or make sure that they know what's happening with their workloads. So lots of background I'll keep going, but this is all just a service of showing you the developer experience for Watson Cloud and some of the what it looks like to write a web assembly app that's not too wacky. Now, I like to talk about what we call the distributed systems nightmare. We send a complicated request across multiple different systems we expect something to happen it does not happen, we go find the log and what does the law say? Right? What? Where is that actually happening? There's nothing it doesn't give me information and so to give a concrete example one of the things that we do is a platform Cosmonic Space Team is we open wormholes for people you can think of this like an interest point when people have like a second fit amount like a network space they need to be able to get HTTP requests you open a wormhole interest point into their network space. So whenever we do one of these things, we have our API that sends a network request to a web assembly module that sends another network request to another web assembly module to actually do the work another web assembly module picks up that event, we use an event source system but I'm trying to go down that route and it actually projects that so that we know the state of the entire system and when you look at something like this which to be fair is actually not even super complicated there are a lot of places that get error or timeout it could have come from right and all of these things could be running on the same machine all these things could be running on different means, different containers you have no idea where something went wrong and so you know trying to get some mileage out of some means that I enjoy lately right but getting the timeout or getting the over responders is so unhelpful and you know spent us took us multiple hours debugging which time has happened just to nail down the core issue. What we want is just this, what we just want when we send an operation, if it fails, where did it fail and you know sounds simple right, sounds like we would have designed this from the beginning but we didn't and here we are. So implementing this as I mentioned the Cosmomic platform runs on top of the on top of Wazel Cloud which is our open source experience and so when we implement it for our platform we actually go to implement it in the open source and that's the same thing which is a great consequence of us downloading our own products because we need that back right at the end. So there are three different kind of components that I talk about for a long time, the capability providers of the actual implementation of services and then the web assembly modules so you know I just took some great shots on a few pages. In Elitzer traces are stable and for us traces are stable web simply and it's because web assembly modules can actually interact with an outside environment without exclusive permission so we can make that work request for traces to a URL or even read a file on disk without exclusive permission not to mention that the web assembly modules are what our developers are right we don't want them, we want them to automatically be able to have tracing if they want it and not have them themselves. So just a little bit of a closer view into the host architecture we have our Wazel Cloud OTP Elitzer OTP host that's the thing about web assembly, we have implementations of services called providers, those are Rust binary so that's actually great support for our tracing and Rust so the host itself can be traced the actual provider we can trace the code that developers are writing is a web assembly module and can actually export that so what we actually did we have a really big wrapper which is in a Berlin or Elitzer process which is super lightweight and not an OS process that actually kind of inspired when Gover teams are today so very lightweight that wraps the web assembly module listens for our PC calls and traces it on the way in and the way out so when we implement this in the open source developers actually get that tracing for free so as soon as we implemented it developers can flip the hotel switch provide an exporter and they get all their workloads traced which is great going back to the a few slides this piece of code this is something a developer would write they upgrade to a lot of version 50 whatever and then it's automatic none of this code actually is changed which is our home goal our home goal with how computing should be is that to change your non functional requirements your logic shouldn't have to change should have just included that again you know I was talking about a specific trace what we're creating and what we're creating and first point to somebody's face right this is the trace so we implemented it across all of our different services you can see like what does this one have six different services each change in color here is a network hop and all the way down the stack we have an internal provider built in SAR and it kind of sees everything and you can see the entire flow actually down to the bottom that one is probably the small but that one has actually collapsed this trace has a depth of 81 states so when something happens you go over yourself and you see this you get angry you can come back all the way down you can see that this request failed because of a no responders issue of maps which means there wasn't a service listening on the other side super helpful immediately to get down to the source of the issue I know I'm preaching to the choir here because this is open observability day but we're just so really excited about it we didn't have this from the beginning and now I wonder how we ever lived without it right not to mention that this is exactly what people who are developing apps we wasn't about to get automatically out of the box so when we talk about the amazing trace the kind of inspiration for this talk there is this is the amazing trace when somebody comes to our platform and clicks login we're in open developer preview anybody can come and sign up and try it out we provisioned them a new and that includes like giving them a key value bucket giving them a segment to network space setting a lot of things up for them so that when they hit big ground and this is a heck of a trace we're using Honeycomb Honeycomb has been great I'm not saying that we are doing everything perfectly as much as I'm here to talk I'm also here to learn so please feel free to offer suggestions on how to make our own trace and things better but when you look at this like we're overflowing the CSS of this actual product just the provision a new user Honeycomb you know and I like to chat but this is the kind of thing that if a new person comes on to try our platform and they say hey my account didn't like work like I didn't get an account and I signed up who's going to want to go to find where this is we don't need to raise it so we had a lot of I had something that really goes to the say here and different services and it has a depth of 342 traces big not seconds ideal but it's what we do when we're provisioning a new person so like I said I'm going to try to get some mileage out of a company means while I'm here I'm going to get some mileage out of some means I think this is probably the last year that I'm going to be able to use this one before it becomes overused but talk a lot of good things and was it all great on the implementation side no, it was actually kind of painful when we got down to doing that and part of that is because we're retroactively implementing something in our system and let this be a lesson that I'm going to continue to preach the wire here you can start with those are ability it's a little easier than doing it later on so the kind of difficulties that we had in implementation I saw people who were speaking here from Krippana this is not a fault of you I think I actually, we probably missed it but we're using tempo at the time because it's free, it has an open source container it's really easy to just start up and get working and at the time it didn't support the scout portal traces so like if I could do something and it fails I had to go back in my logs find the trace ID and search it through that I couldn't actually search through the list of things that happened recently but someone actually came in from our community and said hey that thing actually does have this newer version and so I probably would come back and update these slides before coming here now, on the elixir side it's actually kind of a surprise it's all one kind of module on the REST side we needed tracing, tracing futures to deal with asynchoade boundaries and tracing subscriber to deal with like wall padding and using this finding the correct indication of the correct spell to use across the different libraries is actually pretty difficult and when you think about preserving traces when you go into asynblocks or tasks that are pushed off in the background it's kind of unintuitive and it takes very good effort to do that correctly another pain point is linking spans and traces together across network boundaries is difficult across language boundaries is difficult and doing both at the same time requires a lot of consistent checking especially across elixir which is a dynamically typed language like free crawl and rust which is a statically typed and very scripted language on the compiler so linking them together like that kb counter app that I showed you a little earlier getting htdp request trace starts as a network request over nas to go to the web assembly module and then exit the web assembly module to go to the key value provider which is written in rust so what two or three network hops and two language boundaries even just as simple as the case is complicated of course we took on the complexity and it works it was just something that we had to put a little consistent effort into now things are great but there's no tracing within web assembly modules at this time and I think that's next keeping the developer experience pure for developers so that when they come in and they want tracing they provide an export and done but there are legitimate reasons within a single piece of code to want to add as full span as we do it at the 5.0 level developers should have that control initially if that's what they want to do open telemetry logging metrics are things that are kind of like in alpha or not implemented stage in elixir and rust and so that's next for us as soon as those things come out we're already tracing saturday logging metrics that too and just to note I know that some products have kind of done this recently like they release a version of their product and you run your workload using it or it's a new terminal they're actually tracing what you do and that can make people pretty uncomfortable and it makes me personally uncomfortable we don't actually trace customers for cosmetic we have the ability to supply an export URL the next for us is allowing people to bring their own exporter so that when they run their workloads they can have their own individual things traced and exported but it's something that we actually see like we're launching a compute for them but we don't enable tracing we don't watch over their workloads things like that I think it's really important we love to set up a learning this is probably the capability and the income but alerting for failed traces and of course sampling where at the beginning of our product phase we're already generating like 200 billion events now 20 million events which is a ton alright that is about all that so for some references all of the Wasm cloud is open source feel free to join our Slack if you're interested in what we can do on the server side check it out we're pretty proud of our developer experience and the capability providers are like implementations where we have traces and this is kind of a final note kind of a shameless plug Cloud Made of Wasm Day is today and my actual whole company is here we're really excited about Wasm we're hosting an event tonight with one of our contemporaries and tomorrow kind of a web assembly crash course if anyone is interested in these things it's free to attend we will be there and happy to come and ask any questions about Wasm I think I have a few like 2-3 minutes so any questions? thank you first it's really amazing to see where also probably two of the same standards that goes typical language course servers build into a system such as Wasm is pretty interesting for me any questions? thank you comment on the large tracing we'll talk about that later it's funny I'm curious about your experience with the hotel box and specifically when you were learning how to do tracing with Rust what's the hotel let's be a little like 1-10 you don't want to use a hotel anymore and 10 is everything's perfect so how would you rate the experience and what are some things that could have improved if you had the ability to just wave a magic wand and make them yeah so every language does and with Rust what I commonly do I've been a Rust developer for a couple of years is I go to the crate or library for the same thing and I feel like tracing on crace.io comes up right away click on it and what I expect personally is that in that crate documentation is a quick start guide the things that you need to know and where to go from there maybe not central to the github but at least if all the quick start and how this works isn't going to be all contained better than linking out some more else the fact that the functionality we needed to span three different crates is not ideal but not awful there are other things that do that probably say what was one was bad and ten was a beautiful place I think the whole experience probably left us all in like a six or seven the discoverability of what we need to do specifically around like a single wave which is if you have rarely been used for us as kind of a pain point for the ecosystem that was the big thing because we would implement what we saw tracing shoot off an indication and like you get disconnected traces and wanting like the right function to call for that was difficult great, any other question I can probably do a fast question I got up one minute more question from Isaac for such less traces how much what you're paying for honeycomb to ingest what was mistaken I don't know I don't know but ok so what are the questions like what's your plan for such big trades do you what's your future plan yeah so we implemented tracing right great but that whole like 342 that like it's obviously not very affordable is better if we get a lot more observability but for our whole platform we need a little bit more specific we need to be able to point to places better than dropping the span and like each place where it could fail and tagging from there um you know other than that it's down to feedback from the community people who use wasm cloud and are used to using tracing so this is some really you know you guys have it I believe what I'm used to it's really up to that one more question this is to get an opinion I mentioned at the start that like people don't want to have to write traces but we've also always intentionally written lots and metrics I'm just curious about like why people think they don't want to use this because they think traces are something people don't want to write them so I'll maybe is it okay to answer this question about this topic yeah I kind of want to clarify what I meant I don't think that people don't want to write traces I think they don't want to have to set up the entire system and do the things that we did like setting up the specific the right invocation for traces to be exported and connected and all of that I think people want tracing on their functions but if they're going to do it then it should just be a single line like how they do an error it should just be like a new span and calling it this, attaching these fields things like that so now that we have that that's not a capability for them to be able to use but like I said that's on kind of our what's next list the logs that they're using should be attached to the traces and not just kind of freeform which is up to like a hotel logging but is that going to help clarify what I meant and to be fair if we had automatic traces yeah whatever it's going to end up with traces like we have a huge thing because if we just trace every function call it's not useful information sounds like some balance it's a high resolution thank you very much