 All right, thanks for joining us, guys. I know it's the last session of the day. Today, we'll be talking to you a little bit about some work we've been doing with OpenWisk and OpenSack Swift to create this AWS Lambda and S3 functionality. So I'm Sharma Rakhami from IBM. This is my colleague, Andy Bodine. And I hope you find this session informative. What we're going to do is first talk a little bit about serverless computing or functional computing models, platforms such as OpenWisk that help deliver and drive those types of functions, as well as how we've integrated OpenSack Swift with OpenWisk to drive a lot of these functions. So at the end, we'll show you as a live demo. And in the next few weeks, we'll make all the code that we have in the demo available to you for download on GitHub. So to start off, I wanted to start with some historical background of how we've migrated and transitioned to development in the cloud. Over the past 10 years, we've seen a steady improvement on how developers can create and deploy applications into production. A lot of developers traditionally worked on bare metal. The time to get those resources took weeks. The cost of the research is very expensive. And more recently, they moved to a new virtual machine model or container model, which helps them deliver and test out their applications a lot quicker. But more recently, what we've seen is that there's this whole notion of functional computing or functional development. And with this shift between paths and functions, the developers are really focusing more on the business and service requirements and to help get those functions quickly and available to other applications or service that may need to consume them. So we're really looking at a microservices deployment model. When we think about microservices and cloud-native applications, there's this notion of this 12-factor application. 12-factor application model really helps one look at and focus on, how do I create my application? What are the steps and points that I need to focus on when I create a microservice application? And so as we move into this serverless area, a lot of the functions and factors of 12-factor app are being handled by the platform itself. So if you look at things such as IoT applications, those types of applications are really consumed through this event model. So we look at event programming. As I mentioned, IoT type of applications, when you have devices or endpoints that trigger some type of data, for example, a change in database or sensor information that you get from a weather sensor or a door jam sensor, these type of things, typically you want to have some kind of action or response to those actions. And so consuming these functional applications make it really easy for those event models to consume those services. So if we look at things like cognitive trending or server-side backends for mobile apps, as developers for those types of applications, you really want to have the ability to quickly spin up resources and from a functionality perspective to help drive the changes or responses to those sensors information. And so over the past few years, there have been different computing services. So for example, Amazon Lambda is an event-driven compute service, which Amazon provides to run your code in response to events, and it takes some kind of action. And this really supports this functional programming model. Similarly, we have OpenWisk. And OpenWisk was announced earlier this year in, I think, February by IBM. And this is a model, an open-source platform where anyone can download the code, deploy it, and get that same type of event-driven platform model. So in a nutshell, OpenWisk is an open-source platform that executes code in response to any type of events. And it also provides this sort of polyglot development model. So you can write your actions in any language or even deploy them as Docker containers. So as I mentioned, OpenWisk supports many growing different types of workloads. From IOT applications, sensor data, you want to invoke some kind of action based on some type of event. We could use OpenWisk to trigger different steps inside of your DevOps process. So within the pipeline, so you may want to send out an alert to a group or maybe trigger a bot to do some type of event. OpenWisk really helps enable you to basically do any type of function or serverless type of computing with ease. Another example is, when you look at big data, you have a lot of things that you may want to do within your analytics pipeline. So as it's going through the pipeline, you may want to do some type of action or some type of analysis, which could be highly specialized. And so OpenWisk allows you to invoke those types of functions really easily in a really scalable manner. So to kind of sum up how you interact with OpenWisk, this is a very simplistic diagram of what the programming model means. And so at the top, you have rules. So as a developer, you define the rules of what happens when something, some event comes in. And so these actions handle the events that are emitted by triggers or sensors that are generated by services or incoming feeds. As I mentioned, the triggers, they're again a class of events. So things and feeds, such as Twitter feeds from social media, data changes, geolocation updates, weather updates, user input, all these type of things can help feed and drive different actions utilizing the OpenWisk platform. And what do I mean by actions? So actions are really a set of code that you define that do some type of event based on that trigger information. And all of these actions can be shrinked together. So again, looking at a functional programming model, you can have a higher level function, invoke other functions that you can shrink together to actually coordinate a meaningful outcome of something that you want to do in accordance to an event. I also mentioned that OpenWisk supports a polyglot development environment. So these actions can be defined and programmed in languages such as JavaScript, Python, Java, Apple's OpenSwift, and other programming languages that you're really accustomed to and familiar with. Again, OpenWisk also supports Docker containers. So you can take any of those languages or even binaries, drop them into a Docker container, register the action with OpenWisk, and have them execute easily and consumable through an API. So as I mentioned, the consumption model, OpenWisk, at the high level, has this notion of an API proxy which will route any incoming request for an action or trigger down to invoke your particular action. So there's a rest interface that can be consumed by browsers, web apps, mobile devices, et cetera. For example, if I want to get customer data for application needs, it's a simple request through OpenWisk to run an action to get my customers from a backend data source. So we all really came here to see how we've integrated this event-driven programming model with OpenStack Swift. And Andy will go over an example of how we've correlated an AWS S3 and Amazon Lambda function to an OpenStack Swift web hook and OpenWisk action. So I'm familiar with middleware. Okay, so a good handful of you. So for those of you that are not familiar with middleware, it's really a way to extend functionality within OpenStack Swift and other OpenStack projects. You can think of this as adding custom code into a pipeline, and in our case, it's gonna be a transformation, oh, sorry, a web hook that can call out to any URL. So in Swift, a majority of the middleware sit at the proxy server. So as requests comes through, for example, if I'm gonna create a container or add an object, they all come through the API server and then trickle down into the various other sub-services like the accounting service is an object storage. So that's this picture you see on your right-hand side. The middleware, as I was mentioning earlier, feed off this WSGI server and sit within this pipeline. So there can be one or more middleware components that make up your API pipeline to process and handle various actions that a user request kind of goes through. So along with that, there's this concept of metadata. And so we, for this example, we need to be able to save a URL and associate that with a container. And so we utilize System Metadata to assimilate the user's actions for when they create a container and associate that with that URL. So we store this as System Metadata, which is user-specific or account-specific, and we pass in a specific key that we have in our code and we'll show you this in a few minutes along with the value or the URL that they provided. And that's really the way that Swift enables us as developers to store information securely. So Andy's gonna go through the demo from a high level of what AWS and Lambda, their example on their website goes through and how we've enabled open-stack-swift and open-wisk. All right, thanks, Sean. Good to go. So as Sean alluded to, if you go to Amazon's Lambda site, there is an example there. It's literally this example. Basically what you're looking at, for those of you familiar with S3 storage, they have this concept of buckets and it's analogous to open-stack-swift containers. So in their example, a user is going to upload an image or a photo up to a source bucket. And then some event is kicked off in Amazon Lambda and a function goes, takes that image and it's going to create a thumbnail of that image. And then upload that thumbnail back to a target bucket. So it's a separate bucket, but they're kind of linked in a way. So that's the example and what the session's all about is how did we replicate that example or do the same thing with open-stack-swift and open-wisk. So what you're looking at is pretty much the exact same things, just different terminologies. So like I said, in Swift, we have containers. So we're going to have a source container. We're going to go through uploading an image to that source container. And like Sean talked about, we've injected or we've defined and added to the middleware pipeline in open-stack-swift this webhook that when you upload an image to that Swift container, this webhook is going to reach out to an open-wisk action and that action is going to do something similar to what Amazon Lambda is doing to create that thumbnail. And then it's going to upload it to a, it's buddy container, it's the target container here. So it's pretty much the exact same thing, just different technologies. So again, how is that implemented? Really all it consists of is a piece of Swift middleware that we've defined and installed into the open-stack-swift runtime and an open-wisk action. And what we're going to show you next, I'm going to get into kind of just a high-level, kind of basic overview of what it takes in terms of code-wise to implement these. And then we're going to get into the demo and hopefully that just comes together for us. So middleware-wise, Sean talked about the WSGI server and the pipeline that you can hook code into. So literally, if you go to open-stack-swift, the developer website, you'll find a piece of code that looks very similar to this. It's an example of how to inject a webhook as a middleware into the proxy server for Swift. So you can see here really what we want to do is we want to map the sys meta, which in open-stack-swift for those of you who are not familiar, you have the storage of the actual objects themselves, but then there's also associated meta data, as Sean had alluded to earlier. So really what we're doing here is when the request comes in, we're going to basically associate or map those, the webhook and the authentication, whatever we need to do in terms of authentication. We're going to grab those out of the incoming request. We're going to map those to the sys meta, sys meta data. We're going to send that down the pipeline and that's really, on, there we are. So we're mapping the meta data here from the incoming request. Then we go ahead and we send that down the pipeline. So this is going to be the WSGI pipeline, the middleware pipeline. We're going to send that down. We're going to let Swift process the request and it's going to generate a response. That's the response we're actually going to send back to the original requester. Past that, we're going to do a couple of things. We're interested in a couple of types of actions on that request. Specifically, for our example, we're primarily considered with if the request method is a put, which means we've uploaded an object to that container. We've added an object to that container or delete, that could be something too, that we might be interested in too, in terms of actions on that container. And then the rest of this is kind of how we hook into the open-wisk action. So we take that, so we're going to generate, sorry, with the request headers for the webhook here, we're going to go ahead and we're going to create a new request. It's going to take that webhook value, which that webhook is the URL to our open-wisk action. We're going to add the authorization so that when the request gets there, it's actually allowed to do what it's trying to do. And then we're going to bundle up this object and that's really how we're going to tell the open-wisk action, one, what the object is. So we're passing in the actual object that we've uploaded. And we're also giving it the necessary Swift URL and also the token so that once it's done generating that thumbnail, it can actually do something on that thumbnail. It can upload that thumbnail to a new container, okay? And then the rest of this is, let's send the request, right? So pretty straightforward. Again, you can find a similar example on the developer site for OpenSec Swift. Let's get into actually one more thing. So once you've defined this middleware, you got to actually put it into Swift. You got to make Swift aware of it, right? The proxy server. Really, all that is is you got to place the webhook.py which is just what we were looking at. You got to put it at the right part of the file system for Swift, okay? And then you're going to update your proxy server comp. This, the pipeline Sean was talking about, that's literally defined here. It's slightly abbreviated, but it's going to show you that this is the webhook filter that we defined and this is where we're hooking it into the pipeline, okay? All right, now let's get into the openWISC side of things, the action, like what does that consist of? So as Sean mentioned, openWISC, one option you have if you don't want to leverage the built-in runtimes, if you have some custom thing, it allows you to bring your own Docker image. So whatever you decide to put in there, you package in a Docker image, container image, and then you can associate that to your webhook in openWISC, okay? So this is really, for those of you who aren't familiar with Docker, if you read this, it's pretty straightforward, especially if you know Linux commands for Ubuntu. Really, we're just, this is just kind of some preliminary stuff. This is the meat of the setup of the image, okay? We're gonna grab node, we're gonna install it, all right? And then the rest of this stuff is just kind of setting up the, when the container is run, what's the context that it's running from, okay? That's what WorkDir does, so it sets up when the container runs, where it's the path in the file system that that's gonna kick off at. And then the rest of this is just, let's copy the actual WISC action node.js code into that container, all right? Then we just give it a port and tell Docker how we want our container to actually run, all right? Now I'm gonna get into the actual, that node.js code, what gets put in right here, this copy server, this is copying the server directory into that path, this working directory path, okay? So, it's really simple, it's really straightforward. For those of you are familiar with the Node.js web framework Express, this should look pretty straightforward. Really what we're saying is, when we get a post on, I think I messed this up, this is a put. But when we hit this pass slash run, so this is inside our open WISC action, when we get a request that comes to this path, we're gonna pull out the Swift object, right? That's that object that we packaged in in the middleware when we made the web hook request. So now right here, we're pulling it out, okay? We're saying, if that request to Swift was a put request, now we gotta do something. So, this Swift client transform image function, I don't have the details of this here, but all it's gonna do is gonna take that image, it's gonna use an image library, create a thumbnail of that image, and then it's gonna use the Swift credentials that are in the payload here, that's why we're passing it in, okay? It's gonna use those credentials, and it's gonna upload that image to the target container, which is what we had, show the example, okay? And then the rest of this is just straightforward, it's sending the result of the WISC action back to, whoever sent the request in this sense, it'd be our OpenStack Swift middleware, all right? Okay, enough slides, let's get into the demo. Hopefully, as everyone says, the demo gods are with us here. All right, get that out of the way here. All right, so, in here. And there's a font. Ah, yes, I'm gonna increase this. It's a shift plus, is that big enough for everyone? Good, get in the back, all right, awesome. You move this more. More, okay, how's that? Good, awesome, let me just make it so it's not way off the page here. All right, so, for this demo, I just have a server running. What it's got in it is, it's got OpenStack Swift in a standalone mode. No other OpenStack component, it's just that. And it happens to be running in a container. I can do a Docker PS here, and that's gonna be a lot of output. Let me make this a little nicer for us to look at here. Okay, so hopefully that's a little easier to see here. This S-I-I-O, that's our actual OpenStack Swift container. So we just have that process, we have that OpenStack Swift in the container here. We can see it's running at port 8080 on this server's IP. All right, we have this S-A-O UI here, which that's gonna be the UI that we just created for this demo. So that's gonna help us show you what we're talking about. The rest of these containers is open-wisc. It's a collection of containers. Don't really need to get into the details of what those are, but that's what the rest of those containers are. All right, so there isn't any setup to do with OpenStack Swift. We've already taken that middleware, put it in place, and it's running inside the container. What we do need to do is create the open-wisc action, all right? Now, we have this script here, let's create thumbnail action, all right? I'm gonna run this and then just kind of talk through what it's doing, all right? So, really quick. This is Docker building, okay? So this is Docker saying, or because we defined our whisk action with a Docker file, Docker went and built that image for us, all right? So that's right up till here, that's the Docker image building. And then this next part, we just run a simple whisk command, say whisk action create, we specify the Docker image because we tagged it with a name, so we associated that image name with this action called thumbnail. If I look at my Docker images, I can type okay. All right, there we go. So we built this thumbnail action image. That's the actual container image that's going to run when open whisk kicks off our action from the webhook in the OpenStack Swift middleware, okay? And if I look here, whisk action list, there's my thumbnail action. So now we have the action set up with a route so that our webhook can actually hit that and when it hits that action in open whisk, it's gonna run our container image. Now, what I'm showing you here is the list of activations. So this is kind of historical data in open whisk of what actions have run. And I'm really just showing you this so we can get, and I really, I'm just gonna get a count of how many times these have run. So that way, 10 minus one in the top line here. So we got nine activations so far. So when we get in the demo, we should see one more activation here, okay? All right, so our whisk activation set up. OpenStack Swift's ready. Name space. Yeah, it's worth mentioning that open whisk has some kind of delineation, it has a separation so that this user that we set up, right? So I have this user called SwiftDemo. All right, and this is actually the key that we're gonna need to associate with that. Remember from the open stack Swift middleware code, we were pulling the webhook header and then the webhook auth. So this is gonna be the actual webhook auth, okay? And the webhook is just gonna be the URL to our open whisk action. And what Sean was saying about name spaces is in open whisk, it's isolation for your actions. So if I was a different user, I could have another thumbnail, another action called thumbnail. They would be totally isolated. So this is within the SwiftDemo user name space, right? All right, now I'm gonna hop to the UI. See how that's going, all right. I just refreshed this page here. So here's my server. That's where this terminal is open, this demo at demo. Demo machine, that's its IP, 192.168.99.101. And if you, I can show you here, right? So the UI is running at port 3000, right? So that's what I'm hitting right here. This is our demo UI. All right, so first things first. We need to, let's see, let's just kind of get a refresher now that we look through the code, the example that we're trying to do. We need to upload an image to a Swift container. So what we're gonna do is we're gonna interact with OpenStack Swift right here, all right? So I'm going to provide the Swift URL. And remember, the container is running at port 8080, all right? I have some demo account user and password here. So test account, tester user. All right, so we've logged in, or we've authenticated with OpenStack Swift. And you can see for the account test, we don't have any containers. So first thing we gotta do, we gotta create that source container that we're gonna upload the image to. Now, when we create this container, we wanna associate it with that open-wisk action, okay? So that's what this is, is this webhook, okay? So open-wisk, I'm just gonna hop back to the terminal real quick. At port 80 here, this guy at nginx, so that's our proxy into the open-wisk containers. So this really is what we're gonna hit here. And that's when we're specifying 192.168.99.101, no port, it's gonna default to 80 here, okay? And so that's really what we're targeting is the nginx container. So we have namespaces, SwiftDemo, we were talking about the SwiftDemo namespace, and we've defined our action thumbnail, okay? So that's the webhook. We need to get the token that we queried from whisk right here. So I'm just gonna copy and paste that in. All right, there it is. And SwiftContainer, we'll just call it demo, all right? Let's go ahead and create that. Boom, done. SwiftContainer's ready. Now, I can click on that and we can see, so this is a container called SwiftContainer called demo. And we can see right here that the webhook has been associated with container, right? Because if you remember from the Swift middleware, the request coming in has the webhook headers in it, right, so the client, the front end, should have access to that, it should know that, and that's how we're able to display that here. It's an empty container, no objects, so let's go ahead and create one. We're gonna get a big image here. Let's go with this one here. All right, this is a pretty big image. So we're gonna upload that. It's locally, so it's quick. But it is big. If you see here, it's a little over two meg here. And if I click on it, you can see I'm not zoomed in. This is the actual size. So this image is actually quite big, right? Just scrolling around. It's pretty large. And what we're gonna do is, well, what should already be happening because we've associated that webhook in the middleware is an action, or a open with, our thumbnail open-wisc action should have already been creating a thumbnail, all right? So let's see if that actually happened. Activation list, I think we had 10. We got one, something happened. Let's see if the right thing happened. All right, I just need to refresh this view here. I'm just logging in again, bear with me. Well, we got a container, and it's a thumbnails container. So our whisk action did something. Let's see if it's actually in there. And there's our thumbnail image. You can see by the size of it, it is significantly smaller. If I click on this, it's obviously much smaller, okay? So right there, that's the demo. That's literally the example. So we put that image in the container. Open-wisc created our target container for us, created that thumbnail, put it there. And all we had to do is upload the original, all right? That is the end of the demo. Let me hop back to the slides here. Let Sean take over again. Thanks Andy. Yeah, so to summarize, we're gonna have all this code uploaded once we clean it all up. But in addition to this, again, what you saw here was a sort of a webhook model or integration into Open-Stock Swift. What we're doing at IVM is trying to extend that functionality to be more event-driven. So the whole idea here is to enable Swift to take a lot of these events and be able to publish them onto message hubs or message buses, things like Kafka, RabbitMQ, Zacker, et cetera. And so at the moment, there's current patch out there. It's hard to see because it's in gray, but at the bottom, if you click on the link, you'll see our current patch for that information. So what we're trying to do is really contribute back to the community, help extend these capabilities of Swift. So some of these things are gonna include notification APIs, cloud inner service notifications, and other things like monitoring. So what we have here is a list of resources, the example that we saw for Lambda, the example we have for Wisk, if you wanna go and download it, deploy it yourself, along with how to create Open-Stock middleware. And again, the demo code link is active, but right now we don't have any code push but we'll be pushing it next week and you'll be able to hold down everything you saw today. If you have a server, it'll deploy, we have scripts to deploy everything as containers and you should be able to run that same, exact same demo on your own system. So we have a few minutes, if there's any questions, we'll take them. If not, thank you for coming and any questions? Is there a mic here or no mic? All right, which alarm? Yes, yes. Okay, so I think it depends on how the proxy is configured. So if the message body is too big, it won't be able to do it, so we might have to chunk the data. Oh, solometer alarm, okay, sorry. Solometer alarm to Open-Wisk. So if you're able to configure the alarm to call an H3 request, yes. And then it really becomes down to how the data gets pulled in. So you may wanna have the action pull the data rather than push it through the payload, right? So you could have an action, talk to your data source if you're warehousing the data, edit some other data source. Does that make sense? Okay, question. So just to clarify, do we see current actions in the community that are moving to this event model? Yeah, so I think that's what we're proposing a blueprint for this. Any other questions? Okay, fails. Yeah, so the question is what happens if the indication to the whisk action fails? So that's a good question. And one of the things we've looked into or proposing is it really comes down to how you implement the middleware. So you could have retry, you could also have it offload to a message queue and do asynchronous requests. But again, it's an HTTP request, therefore it's best effort. So from whisk to swift, back to swift? Yeah. Yeah, okay, so the way we handle it, and it's not, sorry, the only way or the best way, is on the request from the middleware, we package the user credential in the payload. So by the time the action gets it, it reads that out of the payload, so the small snippet, the payload information, it'll suck that credential out and reuse that on the response back. Is that a good answer for the question? Yeah, so these are completely separate processes and instances, so we actually have to tell whisk what to call back to, and so within the action, we have to coordinate passing the callback URL, which is the swift URL, as well as the off-token of the user that's creating the container. And then we feed that, we pass that all through the payload to the action, so it knows where to call back and what authentication credentials to call swift back to, to upload the image. There's another one. Question. The rules, okay, so there's a notion of triggers and actions, so you can think of a trigger as a higher level way to kick off one or more actions. I don't know if we have the chart here. So it defines how the trigger is associated or calls the following actions down the pipe. Does that make sense? Okay, so there is no requirement to use a trigger, so you can, from the API standpoint, for whisk, you can invoke a trigger or an action directly to simplify the demo we just called the action. Okay, sure, okay, yes, that is true, so if that happens, that's why the way we do it is not necessarily the best way to do it. So in that case, we would have to either federate the credential or have some common credentials like integrating with Keystone. Which action? It's a spam, so like an attack on top of the whisk API. Is that the question? Yeah, so. Right. Yeah, there's isolation at the namespace level. Is that what you're asking? If anyone can, so is the question, if someone writes a malicious action that gets executed within the open whisk environment? So, most or all of these actions are run as Docker, in the Docker container. So within the Docker, you have an isolated namespace which has no root privileges. You can, so you mean like call out to and do a DDoS attack from whisk. I'm not sure, we can talk about it offline about how the security model is and how they're trying to isolate and make it more secure. It's appealing. Dan, do you know what it is? Do you have threads? But is that in the open source? The metric collection. Which project? Stolitz. Stolitz? No, I'm sorry. Okay, well, there's a lot of projects going on. No, Stolitz. Do you know about Stolitz? Yeah, this work was kicked off earlier in the year as a proof of concept. So we decided to demonstrate it as an example. I know, we do have like this event model. That's another group in IBM that's working on. We do have communication with them. And some of the lessons that we have here, we have communicated over with them as well. But the purpose of this was not to create another project. It was more to demonstrate the capabilities of Swift and OpenWisk. Any other questions? All right, thank you. Hopefully you had a good summit and a good talk.