 It's not nine yet. Okay. Good morning everyone. Welcome. My name is Nahati. I work for Intel. I work on Swift upstream. Here's with me Ganesh. You want to introduce yourself? Hi, I'm Ganesh. I'm a software developer at Intel. Hi. I work on Swift as well. So today we're going to talk about middle layers and here's the agenda. We are going to have overview of Swift starting with overview of Swift and we will talk about Whiskey application, Whiskey middleware and we'll see sample middleware and sample application and we will also see a sample test for it and then what configurations and setup is needed for to do that and then some of the references that you could go back and take a look at. Also, you could download the slides from the link that's given here. It's a Dropbox link. It's also at the end of the slide. So I think Ganesh will start with Swift overview. Okay. Hello everyone. So let's do a quick overview of Swift. What is Swift? Swift is an object storage system in OpenStack. It was one of the first two systems that got enabled in OpenStack, I believe in 2010. It is a massively scalable, very efficient system. It touts high durability and high availability with eventual consistency as being the other effect. Swift, you don't need any underlying systems to use Swift. You can use a bunch of hard disks, format them in XFS format and that's how Swift stores all its files and you can start using it. The objects are stored in those and the metadata of the object is stored along with it as X attributes. Swift is not a file system and it is not a static web service, but with the help of Middleverse, you can enable a static web service using Swift and we'll talk about it a little bit. So let's talk a little bit about the Swift architecture. It is basically a two tier architecture as you can see here. There is a proxy tier that handles concurrent requests and there is a storage tier which takes care of your objects, accounts and containers. You have accounts that can have multiple containers and a container can have multiple objects and that's how the cache gets it. So if you have an application that has very high concurrent and you can just keep adding more proxy servers with a load balancer and that should satisfy your request and if you need high storage and a lot less writes, you can keep multiplying on the container servers or the account servers or the object servers. Based on where you need, you can add those respective servers. This makes it very efficient to add systems without having to just multiply your systems just so you need an extra storage for a few extra objects and stuff like that. Let's do a quick demo here. Not a demo as such, but how Swift works. Here you can see that there are four nodes. Each node is an object storage node. When the client requests an object to be put in place, you do a put request to the proxy server. The proxy server finds the three particular nodes to which the object has to be stored. It uses a ring implementation to come up with predefined locations where this object has to be stored and the object is written to all those three locations. What you're seeing here is the replication model. It also now supports Erasure Coding. If you prefer not to have three replicas but use Erasure Coding to use reduced space and have it 1.5 times the size of three times, you can use Erasure Coding as well. When you try to retrieve the object, the client does a get request. The proxy finds the closest node from where the object can be retrieved and it passes it back along to the client. This is also equally configurable. You can program Swift to actually reach out to all the nodes and the first node who returns back can pass the object or you can ask it to find the closest node. Swift goes on finding out which is the closest node as you progress and it will use that particular node. It will give it more affinity on future requests of reads. What are the more common use cases on Swift and what will be more effective on Swift? Data backups, for example. Swift has the attribute where you can say you can give a timeline at which point an object has to expire. Basically, when you put the object, you can use this header file and say, you can either say the time delay from now or you can give a specific date. At that point, Swift has an object to consider the backend which will go and wipes all the objects you have. Swift is perfect for large objects. I think the Swift client as such has a limit of 5 gigs of objects, but you can have static SLO and DLO which is dynamic large objects and static large objects and can actually split your objects into smaller chunks and just upload it to Swift concurrently and it uses a manifest file to track all these objects. On your subsequent read, it will concatenate everything and you get a single object back. The more common use case would be the web content and general data storage area where you have very high concurrent reads and for a very temporary amount of time. Basically, Swift gets you an object and you should be done. What are Swift middlewares? The way Swift architecture is written is the code as such has a limited set of functionality so that the code can remain clean and you don't have to worry about becoming big and stuff like that. You start adding features as middlewares along the pipeline and then you create this giant pipeline where you have a vsg server and then you have all these middlewares through and then you have the application of the proxy server or whatever is at the other end which handles the thing. For example, in the all-in-one, if you see, Swift actually has 15 middlewares before the object, before the request actually gets to the proxy server and that's after that proxy server goes on trying to figure out what it has to do with it. Like I said, the code proxy server code is much more cleaner and then you can add features on your middlewares. You can extend features from existing middlewares and stuff like that. It just makes it much more easier to deploy. Let's look at some of the middlewares that we have. If you take an all-in-one deployment that you have today in Swift, you will find these to be some of the middlewares. The first one you see here is the catch error middleware which is a very high-level error checking. That's the first one in the pipeline. As soon as you put a request, a get request gets in. The catch error middleware actually is the first one who gets to see it and what it does is make sure that there is a transaction ID for every single request or if there is something misformed, it actually returns the code back so it doesn't have to go all the way to the proxy app. The next one is a gatekeeper middleware which is used for header filtering. Swift has two sets of headers. There is a user header that you can have and there is a system header. This sort of makes sure that the client never passes a system header that shouldn't go and similarly, on a response back from the server, none of the system headers actually get out. Something that shouldn't go out never goes out. You have a few more that goes along the line. Then you have this temp URL where, say you have an object and you want to have a URL from which you can actually download, directly access this object for a limited period of time. You can ask Swift to give you a URL and I think it's a configurable amount of time that you have and the object can be retrieved directly from the URL without having to go and authenticate with Swift over and over and over again. There is a static web middleware that you can use for your container. Basically, if you enable this as a header file, it's also a header option. If you enable this in Swift, what it does is you can add your index, your error, HTML, you can actually have CSS files and it'll let you browse your objects that's inside the container using a set URL. This is sort of how the pipeline is set up and then you have some more and then eventually it gets to the proxy app. There are a lot of third-party middlewares that's also been developed and in use today. For example, the Swift 3 middleware helps making Swift compatible with Amazon S3. If you have an application that's written in Amazon S3 and if you want to move all your objects over from S3 to Swift, you pop in the middleware and your application doesn't have to change it. It does the same call and Swift translates this to Swift's object storage. It just makes it much easier for you to move and port. SWR is the... It's a standalone... So it's an authentication middleware that you can use so that your Swift can be standalone and you don't necessarily have to use Keystone, for example, to be your authentication server. You don't have to run an extra service. Everything is tracked in the SWR. Swift Inspector is an excellent debugging middleware. It sort of gives you a profile of every single request that goes to the proxy server. It will tell you how long it took for the proxy server to complete the request or which are the containers, which are the actual nodes where the object is supposed to be stored and where it is stored and all these sort of information. And you can do it per transaction ID. Sorry. Sure. Please. So if you're running Swift alone and you don't need... In your particular use case, you don't have any other services from OpenStack and you just need Swift. You don't necessarily have to run Keystone. SWR sort of satisfies that criterion for you. I have not compared SWR than Keystone. Sorry. So Swift Inspector is a very good middleware for debugging and it helps you a lot. And I think with that, Mahati will take over and she'll talk about this year. So a lot of OpenStack projects use middleware and we use Whiskey standard for it. So let's have a look at what Whiskey app is, what Whiskey middleware is, and let's have an example of a little sample example. The code that I show in here, you can... It can be run and it actually works and you can download it from the link that's in the first slide and the end of it. So what's a Whiskey app? A Whiskey is a Python standard and it's used to describe how servers communicate with applications. But what is a Whiskey application object? And in Swift case, the Whiskey application is the proxy server. An application object is a callable and a callable... What is a callable? A callable can be a function or a method or a class within a call method. So it has a standard wherein it accepts to positional arguments. Like you say environment and start response, you can name it anything that you want. It's not necessary that we use the same names, but they are positional and environment is a dictionary and start response is a callable. For instance, environment dictionary has a request method and I'm sure these are the most common ones that we use. So I've put them here. Environment usually consists of CGI variables and Whiskey input... I mean Whiskey variables. Some of them can be empty and some of them cannot. These are the ones that I've listed here are the most commonly used ones. The most method is it can be a get or a put or the most common ones. And path info is what is in the URL. Query string is also... We'll see an example. I'm sure a lot of you must be familiar with it. Content type is, again, the type of data that you're passing in. Whiskey input is your data. So start response is also a callable with two positional arguments which are necessary, which are required. There is also a third argument which is an exception which captures the exceptions. Say you have the application has trapped an exception and it wants to display it back to the browser. So that's what you use. Headers are supposed to be a list of tuples wherein you use header name and header value. Status is the 200 okay. For instance, here is the success status. You can use any other names that you want but there's still positional arguments. So here is a simple Whiskey app. What I'm doing is I am importing a server and I am displaying environment variables and with I am filtering... I'm only displaying query string. I'm calling start response before the body because the client requires headers and status before you return a body. In this case, I am supplying the application JSON as my content type. I am also displaying query string variable from the environment. So let's see, and I'm invoking a server on my local host. And so I took a screenshot of the demo of this. And so my simple app.py has the application that I've shown right now. And the get that you're seeing here right now is that gets called after I give my curl command, which is I'm passing a query string hello. So as you can see, the query string is displayed and content type is what we have displayed, which is application JSON. So this is a sample Whiskey application which is really simple and very, very basic, which is just displaying a query string and we are modifying the content type. I'll use this application in my sample middleware. But what's a middleware before we go to the sample of it? Middleware can act as a server or an application. If a middleware is wrapping another middleware, the one that is wrapped acts as an application. Sorry, the one that is wrapped acts as a server and the one that is being wrapped acts as an application. Whatever I'm wrapping, whatever a middleware is wrapping, it can be a middleware or it can be an application. If it's wrapping a middleware and a middleware right next to each other, then that's what we call as a middleware stack. So a couple of instances where it's used is rerouting a request based on URL by modifying your environment variables or another instance is load balancing by passing your request from one network to the other. And yeah, I mean, it allows multiple applications to run side-by-side as I was just talking. So when a request or a response gets passed, a middleware gets a chance to look at it and modify if that's what the middleware is required to do so. So here's my sample middleware. So what is it doing is if I have a callable method which is taking to position arguments which is environment and start response, if I pass on an example, my third call is foo, it's supposed to throw an error which is 480 number tbot. And it's the content type, I'm modifying it to text plane. I mean, I have given it as text plane. If you remember, if you've noticed in the previous, my VisGi application, the content type is I've given it as application JSON. So I'm returning an error body with an error status if I pass on a request method as foo, otherwise I return my app. I'm in working my server which is localhost and this simple, I mean, I'm wrapping my application into the middleware. I'm calling my app from the middleware. So I took a screenshot of the demo as well. So I pass on my first curl request without, the next two lines that you're seeing here is an output of curl command of get and foo. For instance, here, my query string is hello. I'm not passing a request method. So the default is get. So I'm simply doing a get here. So all is well. It's calling the application. It's not calling the error. So what is my application supposed to display? It's supposed to display my query string and give my content type as application JSON, which is what it's doing. So now I'm passing a request method foo and it's, as expected, as we see, it's returning an error body and it has modified the content type to text plane and it is returning an error status, which is 480 nmt part. Testing the middleware, it can be, it's rather straightforward. Whatever I'm doing right here is I am faking my application and all I really want to test is my middleware. I'm not bothered about my application. I just want to see how my middleware reacts if I supply it with my third call as foo. So I am passing it. I am calling a request URL and I am getting my response and I'm asserting that the status is equivalent to what I supplied. In my previous examples, we saw that we're calling the application, I mean, the middleware is wrapped. So what if we have multiple middlewares? We cannot, I mean, the code will not be maintainable. We cannot keep on wrapping middlewares after one after the other and that's why we have paste deploy, which helps us configure it. It is used to configure Whiskey applications and Whiskey servers. In case of a Whiskey consumer, it looks for, it provides a load app function wherein we supply a configuration file. But in our case, since we are Whiskey providers, it looks for an entry point. And so where is this entry point? We look for an entry point, we have config files and setup files for it. Our config file is styled in iini format, which is the informal standard, which has, which is a tuple of name and, I mean, name value pairs. Our config file defines what's our to use, what applications, what middlewares are we doing. We give all those values in our config file. So here is a, this config file is directly from Swift and I've just chosen sections which are more relevant and which are different, which means something. So the top one is Pipeline, which is wherein we have, I mean, middlewares stacked up and when you are declaring a Pipeline, it's supposed to end with an application. In this case, our application is proxy server. And if you notice, Pipeline is the value that Pipeline type has its main. When you declare it as a main, we are telling you that to treat this as a default application. And the next section is filter, wherein we are saying that it's a middleware, middleware are declared as filters. We'll come to the syntax, why it's Swift hash cell check and how does it recognize anything like that and what configuration is a log level, what does it do and why are we declaring it here. And then in the next section, what we have is, we're declaring our application, which is proxy server and we're also passing on some configuration with it. So this syntax, the Swift hash health check or Swift hash proxy, that is what it looks in entry points for protocols like application protocol, factory and filter factory, wherein we are declaring our applications and our filters, which are middlewares. So here we're telling that when you encounter Swift hash health check, go and look for, go and look in my module called as filter factory sorry, go and look for my function called as filter factory, which is in the module Swift common middleware health check. And likewise for app factories and which is the applications, we have object proxy container and also memcache, which is not here. So the default is also a part of our config file. The default section is supposed to give us the global configuration. Multiple applications might need a similar configuration. So you, whatever is in the default section that is treated as global configuration, which can be overridden by using a variable call as set. So where do we use this global and local configuration? This is an example where this filter factory is also right from Swift, it's a health check which is used for monitoring. If you pass on slash health check, it's gonna give you 200K. And so what it's doing is it's taking the, the health check middleware, we're taking a request and it's returning a filtered request. The middleware is going to filter, that's what the names reflect. What it intakes is the global configuration is what we saw in the default section. It's taken as dictionary and the local configuration is passed as keyword arguments. The app factory that I have here is rather very simple, is just returning with the application. That's merely an example that's not in Swift. I didn't take it from Swift. So we'll take a look at how Keystone, how enabling Keystone is going to, so it's a middleware in Swift, so how enabling it is going to reflect and what is going to change in the logs or how is it going to look like. Ganesh will do that. Let's take a look at how to enable Keystone in Swift. SAIO stands for Swift all in one. So probably that's two Swift, but that's the location from where you can actually find the documentation how to install an SAIO. The method that I used to install Keystone is in that link and in the Keystone sources, I use the branch stable metaka. I recently came to know that they've changed the way the Keystone application is run, so I don't know if you take the trunk, if that steps will be relevant, but if it's not, please reach out to the Keystone team. I'd really love to hear from you. In the SAIO, if you take a look at the default pipeline that you have, you'll actually see something called Tempot, which is what Swift users at least temporarily do for local testing and stuff. And right below that, like Mati said, you have a filter section which talks about how Tempot is set up. So there is a container test, sorry, there's an account test, and the user tester has the password testing and he's an admin. There is a test two account, tester two testing two, and he's also an admin. There is a tester three who's part of the account test, but he's not an admin. So this is sort of set up this way so that you can figure out if you add an admin and if you don't have listing permissions or relay permissions, you can actually verify with the other user. So if you do a Swift stat, you can see it is using the test account and it comes back with a bunch of data, like you have nothing in this. If you look at the Swift proxy logs, you will see that the user test, well in this test, actually the user test has used this password testing tester and he's reached out to the application and it came back. He has the right permissions to reach it so he got it back and it works fine. And how do you enable Keystone? So using the steps as mentioned below, run Keystone. You have to create the Swift service. Swift has a service account in Keystone. Make him the admin in the service project in Keystone and you definitely have to add the Swift endpoints to Keystone to make it aware how it works. And most importantly, you'll need the Keystone middleware, which is one of the plugins the Keystone team from one of the Python packages. And you change your pipeline in such a way so you remove Tempot, you add AuthToken and Keystone Auth. And this is sort of how you define AuthToken. So if you see AuthToken is basically the Keystone middleware's filter factory, you give the Keystone URL, the private and the public URLs. More important is that when you create the Swift user, you have to make sure that your passwords then matches the same way what you're listing here. Otherwise, you probably won't be able to have a successful authentication. And you also have the Keystone app, Keystone Auth middleware, sorry. And with that, when I make a Swift stat request, you sort of see something like this. And if you look at the proxy logs, that's a little muddle. But if you see the first two lines, it's saying it's going to make an authentication. The third line, actually, you can see it's reached out to Keystone in port 35357. You have a response from Keystone that says, here is your token, this is how long it's valid, and all the other details that you need from it. And if you look at the Keystone logs, you will see that the application has actually reached out and it's basically got some keys and tokens. And this is sort of how you enable Keystone with Swift. The references are here. You have the PIP specifications. You have paste deploy. So the updated one is, it's not PIP 333. It's actually, there is an updated specter, which is 3333. If there is a link within the reference as well. The paste deploy, these are really good links wherein there are a lot more examples and detailed explanations for the standards and what's going on. The Swift middleware also has a better example and a little more, which elaborates on how it works within Swift. Right now, we haven't seen a demo of how to enable a middleware. We just, we only saw screenshots of it. We didn't show any code of Swift middleware, but the link right there helps you do that. And yeah, the development middleware, and development auth is an example of middleware that I was talking about. Thanks. Yeah, and you can download slides and test out the sample code from the Dropbox link that's up here. Any questions? Thanks. All right, thank you.