 Welcome to wasm plus kubernetes beyond containers. We're going to talk a little bit about how we can kind of leverage some of these technologies together To build a working solution to build web assembly services on the back end that are real and running Uh and solve some business problems along the way. My name is sean isam I am a senior manager at adobe working on optimizing our cloud infrastructure and this is callen Collin's also a senior engineer at adobe working on some of our Uh new web assembly based stack. Yeah Web services. Yes Okay, so I always like to start with a quote or a thesis statement. Um, here's a somewhat provocative one People don't really want innovation. This is a quote from justin eathridge People talk about innovation a whole lot But what they are really usually looking for is cheap wins in novelty If you truly innovate and change the way that people have to do things expect mostly negative feedback If you believe in what you're doing and know it will really improve things then brace yourself for a long battle Show of hands in the room. Who can who can empathize with this in this room? I'm sure we've all been here with uh technologies before So, you know, I think you know still want to keep this to be a very positive talk But a lot of this is going to be some of the lessons we've learned along the way getting into the state We're in today where we can kind of get some of these technologies in our operating Wasm is the tool I personally believe it is the tool to write better software faster But innovating is is hard, especially in a large enterprise organization Uh, so we'll just talk briefly, you know, I'm going to start with a little bit of history I don't want to bore you too much with some of our background on kubernetes This is wasm day not kubernetes day right now But uh understand a little bit of the history, you know, some of our current setup How we're thinking about this some of the business problems we're going to solve Collins going to have a pretty awesome demo of some of the stuff working in production Uh, and then you know, summarizing why we think this is worth it History so all right, so what do we mean when we mean running things in the cloud? Obviously a lot of us probably remember pre container days VM backed services These are the the the dark days Before containers magically fixed everything for us There's a group at adobe as a startup that they actually acquired called behance That was really focused on solving a couple of different problems. First was cicd This is something that other people can probably empathize as well But build knights right trying to move all that code out to production Trying to get all these systems working together. That's a challenging task. And so You know kind of standardizing around this is early late 2014 early 2015 standardizing around a cicd framework That could be used to help automate and solve some of those pain points And then deploying apache mesos to run containers First as like system d units. I don't know if anyone remembers those days But uh, you know, that's uh, you know, that was kind of the leading way I'd say but this is a little bit for I think ecs was even released. So this this is the the bad old days of containers Why why did we do this right? So if you think about a developer, this is a developer But again, a lot of you can probably empathize with this as well Um, what are the things that a developer has to think about? There's a lot of different things and that landscape is changing all over the time with technologies And you know, really like this causes problems when you're trying to do things in a large organization Do you want, you know, 20 30 50 100 different teams different orgs different geos reinventing the wheel all over time? I don't think so and so thus in 2015 ethos was born late Yeah late 2015 early 2016 and so the idea behind the ethos project was can we run multi tenant web services at adobe? Like can we magically get all these different service teams inter operating putting their stuff together? Containers was the tool of choice based off of mesos based off our ccd framework And you know kind of automatically scale, you know Get those get the cost efficiency from the economies of scale of running those together so You know running mesos manually is hard. We standardized around dc dc us 2017 migrated everything over to that You know dc us was a great platform for us for for many years But there's a lot of things we had to build ourselves that did not come out of the box And you know, obviously you see the direction the community has gone. So sorry late 2018 Early 2019 we started building around retooling the entire platform around kubernetes When we think about web services, there's kind of three different models that exist in kubernetes. There's ethos cas which container as a service and this is kind of where we took that paved path for And we replicated that experience in kubernetes, but obviously infrastructure is code with you know, the vast amount of kubernetes experience that the community has We built this new pass base system too where you can provision namespace on a cluster and you know Get some basic network policies and get right access in that namespace and hook up your own cicd tool Or do whatever you want to pull your own infrastructure in there And then fairly new is using argon helm product called flex Which is how we can kind of get the best of both worlds Where we can give you a cicd suite that actually give you right access as well so you can create your own cloud native resources Ethos is a huge project powers. I would say at this point most of adobe's web services Over 5,000 production services lots of containers lots of cpus This is kind of interesting because it's going to be a central theme of this talk is this concept of multi tenancy Right and so you talk to a lot of other large companies. You know, there's 500 clusters a thousand clusters There's an infinite number of kubernetes clusters that are running. This has scaled up a lot in the last You know year or so I would say like we this account was under 100 for a while because of our multi tenancy That has scaled out, but we're still at 285 clusters running, you know those kinds of statistics. So I think that's pretty notable We're so along that journey. Where has that gotten us today? I'd say our kubernetes has been very successful given those numbers We still have you know, kind of that unified s re operations model We have a security model that other teams would have to onboard to by default We are still mostly a 12 factor app based system mostly a multi tenant system As that cluster account has gotten larger. I would say that over time we have You know pivoted a little a little bit away from that You know for specific larger workloads But the idea is that you know, we can provide this basic platform for hosting services We can be cloud agnostic. We can be platform agnostic. We can run containers as a service We can run I mean streaming workloads anything you can name and essentially provide this infrastructure as a service With an additional set of value add components on top of it Okay, so again, this is wasm day. Let's start talking about what wasm can do for us So as we look to how multi tenancy works in kubernetes today Is operationally very simple allows us to get you know economies of scale with our people with our cluster operations This promise of you know, just make everything multi-tenant. You've got large applications You've got small applications. They're just going to fill up all the space automatically and get rid of all of your overhead That's not really the case. It's very efficient in theory But you know when you start mixing applications different architectures different scheduling constraints different taints and tolerations different zone affinities You know minimum replica counts that are more than the nodes in the cluster You can imagine how over time things scale out and things the efficiency level drops So, you know, I'm not here to convince you that you should get rid of kubernetes Obviously a lot of people are running kubernetes But when you run this kind of multi-tenant setup while it's operationally very excellent It can be very expensive to run and so when we're looking at what's next and for for us today That's why the assembly You know, we want to be able to leverage our existing investment kubernetes We've got a lot of it We know how to work with it. We've got all these different other resources that you know These hundreds to thousands of teams within adobe have built for these thousands to tens of thousands of services that are already operating in kubernetes So we need a model where we can still leverage that investment beat but be able to integrate in What's new so kind of the question is What if there's a better way? Can we make some of these things work together using web assembly and so You know how i'm personally thinking about this is a model more towards fast. I would not say pure fast, but Actually, let's go to the next slide There's a lot of services that are running on our platform that look kind of like this, you know, we It's a like it's a multi-tenant platform lots of different types of services But the standard service if I was to define what a service looks like it's probably written in java spring brute Using a lot of internal libraries They will pull some utilization statistics about 28 cp utilization about 55 ram utilization So that's good, but not great, right? And so what that really means is that like we've got a large java service that's serving web traffic And it's going to take incoming requests and crunch some numbers on them And then maybe like fire off a call to an external dependency pull some data from a database Wait for that other external dependency to do something and then you know With that transform data send it back to the user maybe you know asynchronously go store that You know send that to another service stored in another data system And it's kind of like data processing flow I would say is is a very standard workflow that's replicated across Hundreds if not thousands of services and so if you look at a compute profile of those services, right? Like we're as strong as our weakest link which for this one I might have pulled the wrong example, but uh the idea often that is cpu, right? Or memory bound or iops bound we've got some other constraining component and there's just idling cpu there That we would like to be able to do something with So, uh How could web assembly be the answer to this then? Let's give a bit of background on how web assembly at adobe has historically worked We have a lot of experience with this on the browser side and actually predating web assembly If anybody remembers portable native client from google Asm.js, you know kind of predecessors of web assembly These are all things that have been integrated into adobe products over the last 10 years starting with the lightroom web beta All the way at the production version acrobat web Moving towards the you know our flagship experience of photoshop Which has been running in beta and browser for almost a year now In some next generation products where wasm is really a key enabler in allowing us to to build that rich client experience In a web browser So we have all this experience working with web assembly code already mostly, you know Taking native code mostly interoperating interoperating with javascript But uh, it's something that that you know We have the experience to leverage in other places because it meets all these same requirements that we're looking for the back end It's a very high-performance system. It is uh secure by default for example, right? We don't have to worry about multi-tenant isolation So This is the experiment. How do we actually run this in kubernetes? And so we're utilizing wasm cloud right now You see on the left here. This is kind of a legacy setup. I would say This is uh, and there's some links there and you know, I'll send out the slides and everything But if you take the cosmonic kubernetes applier, this is a script that allows you to kind of bootstrap a wasm cloud installation within a cluster We started by doing this within a namespace and so this is within the context of a client application and that's important because Namespaces are isolated from each other right like when a client service on boards to our system when we run a multi-tenant system If they need to vpc appear back to one of their own cloud resources They get a netpall that allows their namespace and only their namespace to talk to that inside a multi-tenant cluster And so this was a nice way of kind of like keeping that, you know namespace boundary isolation for external resources Um, but as you can imagine this probably didn't hit our efficiency goals as much as we would like because we're replicating that core Wasm cloud infrastructure for every single app that needs to work this way And so what we're moving towards is this more multi-tenant model This is utilizing some very hot off the press scripts Some coming out of the wasm io workshop that the cosmonic folks did But essentially the idea here, which is this diagram on the right is that within the namespace boundary, which is are these blue boxes We've got uh, you know, we've got the automation and provision services allow us to get ingress to talk to a web assembly module We've got uh wasm cloud providers running the back end these can be like htp providers that are serving this htp traffic and then They communicate through gnats, you know over this lattice this green box back to a pool of common wasm cloud hosts And so with that we have core compute capacity that's per visit to just do work and do work in a fass kind of way And so we uh, it's very efficient We can run all of these different workloads side by side because they're secure by default We can run the actual Execution of that code outside of the namespace boundary and then just communicating back and forth back up to providers that are running in namespace You know using a gnats topic Which is secured by default as well In order to then communicate with those external dependencies So with this it's it's kind of you know the best of both worlds If there is a best of both worlds to be had between kubernetes and web assembly because We can have other things that are still running in that namespace that we need to talk to you that we can access We can talk to all these external providers But we're still able to drive down that CPU cost by those greater economies of scale by actually isolating out the core client compute not having those services Just spending all of their time idling Um a lot of this is going to involve getting code working For web assembly in the first place and taking a service and building it for web assembly So call in has a awesome amount of experience and demo here that he's going to show you Thanks, Sean So, um, yeah, I I have to preface this whole thing because I didn't do it this last time I for you who are in valency remember you might remember I gave a talk last year So i'm going to preface this with i'm going to read it okay, um My goal is not to make any sort of criticism or indictment of decisions that have been made This technology has a very bright future. Everyone involved should be very proud of what they've accomplished Um, i'm just trying to highlight the work that still needs to be done Um in the next phase of focusing on developer experience. Okay, so Afterwards, okay, if uh, I don't mean to offend So this is you know, we've already hit a lot of these highlights. This is this bailey This is from her youtube video with luke wagner So a lot of work great work's been done and uh, and it's really it's really great. It's been great to witness um so Last year it was background removal. It was a really simple service. I think the hardest part for me was Taking a spring boot app and learning rust So that was that was the hardest part there It just worked really flawlessly and the the reason was that it had very few dependencies. So I would I could take a signature and I would I could Remove the background and I think I ran it on maybe fastly and cloud flare was in cloud and fermion spin worked great This year Is the content authenticity proxy? so Here we have the the holy father balenciaga So right this is a very timely thing and adobe's really made a big investment in ai generation Um, so this kind of thing. So is this a real image? Was it was it ai generated? Was it modified in photoshop that kind of thing? Okay, so it sounds simple right i'm gonna i mean i have a proxy This is meant to be running kind of on the on the edge Close to the users so cdn edge think of that kind of thing Um, what we call cdn edge compute right? User will pass a url to the service the service will download it Check it. Uh, it will resize the image now. There's a cache. Maybe it's been cached Transcode it so make make resize it Record that in the content authenticity manifest This is something adobe's been working on the content authenticity initiative That's a embedded data in an image on the history of that image and then Return the image and Push the transcoded image into cache Okay, so as i said before last year worked really well because the only the only crate that was a dependency really was image And this year very different. Okay, so Fast image resize because this is supposed to be fast that needs sim sim d And the fast image resize crate before i started did not support wasm sim wasm. So there's some sim d work Um, the c2 pa sdk needed to work with web assembly wazzy specifically Uh, oh which required open the ssl xo x 509 ring and chrono. So, uh, I did what any Uh Full hardy developer did I forked everything? Um, so and the the worst part was when I when I came to the content authenticity initiative team they said, um Uh, oh it already supports wasm right of course it it doesn't support wasm It supports wasm in the browser right and that's kind of the problem is the wasm 32 unknown unknown target Which I used last year for background removal Wasn't going to work because it was going to pick up wasm bind gen and j s s and so I had to Go through the cargo gut toml and everywhere That required any of these crates in the code and do lots of little configs. So, you know As you can see, um, so that was that was a that was a real challenge And actually so I actually did get it all to compile but the open ssl Not quite there yet. So it actually kind of fell down, but this actually this this highlights um Kind of a positive thing is that I I got it working in wasm cloud because I moved that open ssl into a provider So once open ssl is ready for um for wazi Uh, it can go into the into the web assembly boundary um Okay, so the simd which is kind of the successful part. So I led I I lowered your expectations and now here's the actual success so um This was what the real challenge here was getting wazi simd to actually be faster than wazi Or sorry web assembly not simd, right? So, uh, and this is kind of a bigger point Which is web assembly would not be possible or if it was it would be much more complicated If cpu architectures were not so homogenized, right? Um back maybe back 20 years ago when we still had big endian mips or power pc This would be a different story, right if we had to compile something and maybe it'll run big end And maybe to run little endian or or you know take anything else, right? There's a real uniformity now in cpu architecture um But the problem is simd very very, uh You know simd instructions vary greatly between neon and ssc4 and avx 512, right? So Uh, this is the list I took it from the inscription docs Um And uh, I had to avoid these and of course I didn't have to avoid them in my actual code I had to avoid them in the optimized code, right? So here I have The most efficient way to write something And that was 10 times slower than the fastest way to do it Which is five commit five instructions versus 12 instructions So I'm pulling out watt and I'm making sure I'm not getting these very slow shift left shift right Commands, right? And and those are slow not just on x86. They're also slow on neon on arm So that was uh, that was very challenging um And so I think we you know, we've really pounded this today What goes in the wazam box what doesn't go in the wazam box? And I think there's some debate on this kind of thing um You know we I think hopefully you've been convinced if you were on the fence that there's an advantage Things in web assembly today You know so we want things to go in there We just if we want to be performant We don't not everything can kind of go there so What what I would kind of like would be cryptography in web assembly because you know open ssl should work on a very variety of architectures and os's Simdi maybe not so I kind of what in my example I actually kind of Swap these Like I would prefer it was the other way but in my example simdi's inside the wazam box and cryptography's outside Although cryptography the um the simdi instructions were ended up being four times faster So it was you know, it wasn't a complete waste of time Um, okay. So demo time. All right. Here we go Let's see Okay, so it's Nope Yeah, okay, this is good Sorry This is always terrifying So here we've got adobe firefly And I was like what's a good image because it doesn't understand web assembly firefly, you know, not in the model Web assembly for the firefly am I am a training data? But I was like well crabs and gophers and amsterdam, right? So made some images. I picked one of them And then I put it I put it somewhere I put it in github. I tried s3. It didn't work. I don't know why And uh, it was very slow and so then um here so I Here so this is kind of the output of the of the wazam cloud and i'm just gonna so i'm not going to refresh that Uh, but I will curl my way and see if we can get it Uh Okay, so yeah, it worked. Okay, and then uh If we do cpa tool, so so that's what it did it it grabbed the image it transcoded it and updated its c2pa manifest um I can't type up here when I'm uh, sorry. It's true. You really can't type A tool what is going on Sorry, uh, I'll see if I can hold on See oh c2pa tool Hey, there we go. Wow Um transcode so just seeing here. We've got the transcoding action that was done by the wazam cloud um Service so so that is the demo and the whole idea is this is actually going to hopefully run Um on edge compute and maybe in wazam cloud or if we can get that in multiple places Uh, yeah, so that's that's the demo um See if we can get back to power point All right handing it back over to shawn. Oh, sorry. Actually. Sorry. I have some notes. Um We do use web assembly on the server side a lot at adobe But it runs in headless chrome in fact and including in the projects i'm working on around Uh newer web services around creative cloud. Um, so We have to at least match what we have in the browser for web assembly with wazi Or else it's going to be really hard to convince these people who already think they know web assembly to move over to wazi So That's it. That's that's it and that's not a criticism. Okay. Thanks I don't really have much I don't really have much more to add but uh, basically I just want to summarize, you know So like this whole journey we've been on so we've gone to cloud native. We run kubernetes We run kubernetes fairly well. We've got some business problems. We want to try to solve Web assembly is a tool that helps us write better software faster But you know as we've seen, you know, both on the how we run it Although that experience is improving how we write it what the developer experience looks like like this is hard How do you convince, you know thousands of java developers that are used to writing code? You know that is like dm base that we've wrapped a docker container around that we've put up in the cloud, right? Like that's the model How do we convince people essentially to get that back to a model where it's like We want to free you from those infrastructure concerns We want you to be able to write the software once and run it anywhere And so, um, you know, I think we're on that journey I think we're in kind of like, you know the technology adoption curve We're in the like, you know storming phase of this right now trying to get all of this going but This is worth it, right? It's worth it for the efficiency. It's worth it for the performance It will be worth it for the developer experience Wasm isn't the enabler of all these things, but we just also, you know, like we're not going to stop running kubernetes tomorrow So, uh, I envision a world where these coexist Far into the future and I appreciate all the work that everyone in the community has been doing to make All these things reality That's it. Thank you absolute Hello, absolutely fantastic. Um, definitely love to see and these are really challenging problems So it's cool to see like how y'all actually tackling them. I know y'all have questions So please, uh, if you have a question raise your hand so I can come give you the mic Making my way back here. Just give me a moment If you could please introduce yourself So larry carvalho with robust cloud, um, you know, we had the days of cloud first that we went container first If we are going to wasm first and you said, you know developers There's still some more time before it becomes easy for developers. What is stopping? Uh, the the way to make wasm easier for developers to use Yeah, so so great. So so the point isn't that I mean, um, the it's just we have to get the language supports, right? Um, we we can't have every developer forking major, you know Rust libraries or and let alone rust because rust is the best The best supported language, right? Um, so, you know, when do you get down to python or or, uh You know, our other language is that we you know, we have to have a good story We can't have them say yeah, you can use python, but you can only you know go within the the main packages, right? So yeah I'd also throw out just like core core library support, right and and sorting out dependencies and dependency trees and dependencies of dependencies There's a talk I can't remember which one is but like if anyone's ever seen a dependency graph of like a real microservice It's it looks like a firecrackers. It's an explosion, right? So like all it takes is one thing not working in wasm to blow that whole thing up Um, the provider model where you can, you know, basically go out to native code communicate and distribute systems fashion helps Don't get me wrong. But also like we need more things running in wasm natively Any other questions? All right, then give it up again. Thank you so much