 Welcome everybody to my talk today about developing your own Swift middleware My name is Christian später. I'm a software developer working at in advance and I'm mostly working on Swift and testing and automation Personally, I started using Swift at the end of 2012 and at that time point I was evaluating different object storage solutions for my former company and And one of these solutions was open-stack Swift and I was very happy with open-stack Swift because of the flexibility of Swift itself But unfortunately one key feature was missing that we needed and that was quota support at the end of 2012 so I started talking to the community and different people from them and Finally, I created my own Swift middleware to add account quota support for Swift and finally in March 2013 this account quota patch got merged within Swift and Was released with a grizzly release? so I want to talk to you about how to use middlewares to extend the swift functionality and because Swift is using the web standard gateway interface and Several other packages like paste or deploy. I also want to introduce you to these packages We will continue with developing our own Swift middleware for the Swift proxy server and Of course have a look into testing and packaging your middleware and Finally, I want to conclude my talk with some references for you So why do we want to write some Smith middle whereas? Actually, if you look into the Swift proxy server configuration file You will see and notice that most of the features are already implemented as a middleware For example, we have stuff like logging temporary oils form toast dynamic large objects Authentication middlewares and quota middlewares and these middlewares give you a great flexibility to extend your currently existing functionality and another great benefit of Using middlewares is if you want to add a functionality You don't need to fork the Swift code and to modify the existing Swift code You can just use the existing Swift code your upstream release for example create your own develop your own middleware and use both at the same time and Several people already have done this. So there are existing third-party middlewares outside of Swift for example for authentication and adding additional API layers and There are also some companies that created their own middlewares for their own private use So they're not released to the public but to extend functionality is these companies needed So let's first have a look in general How is the web standard gateway interface working and what's it's all about middlewares? so web standard gateway interface is a simple interface between web servers and web applications and it was defined as a pysen enhancement proposal triple three and Also updated for pysen 3.3 lately And this web standard gateway interface defines and defines an application and a server The application simply has a callable object that has a call method and the server invokes these callable for each client request So the client sends a request to the server and the server simply calls your application These application objects must accept two positional arguments and the first one is a pysen dictionary called Environment and the second one is a callable called start response The start response callable itself takes two arguments one a status for example a 200 ok or 400 found and the second one is a list of tuples And these are the heaters that are sent back to the client for example content type text plane or whatever you have The environment dictionary has some useful informations that are required to write some useful middlewares The first one is a request method for example. You have a put or get post delete copy and options Second one is the actual pass that is sent to the whiskey server For example in the case of swift this contains your account name your container name and your object name We have also something like a query string So for example, you can send a question mark and then a parameter name and the value to this web standard gateway interface And this is also included in your dictionary If you send some heaters to the server these heaters are also included in the environment And they are prefixed with an HTTP underscore and finally you have a file like object called whiskey dot input and This file like object you can read from it and it contains the data that was sent to your server So let's have a look at a simple sample server Everything you need for this to run this is included in Python itself so you don't need to install any additional packages or modules or whatever and What we're doing here is simply starting a server on the local host port 8000 and the lines on the bottom of the page of the slide and this server calls the application defined above and the application itself Takes the environment. I already told you and the start response as arguments and what it does here is that it returns the body and Inside this body we return an element from our dictionary in that case It's a pass info. It's just an example, but it works really nice and Gives you an idea how this everything works So but it's not very useful if we want to extend it a little bit We might either need to modify that code or we can add a middleware to this So what is the middleware a middleware can act both as in server and as an application? So the server is calling the middleware so from the middleware perspective It's just calling it's just getting the request from the server, but the middleware itself is also calling an application So it acts as in server for this application And by doing so you can run multiple applications side by side You're not limited to one middle bear or two or five you can well I don't know but you can run a lot of middleware side by side and This is very useful because you can add stuff like Authentication or you can do reroute requests or do some content processing and if you look at The bottom line we have a client that sends still the request to the server and now the server called the middleware and the application Inside the Swift proxy server configuration file You will finally see a line called pipeline and this is something like that imagine It's a pipeline one middleware called the next middleware and finally we call an application that application In case of Swift as a proxy server So let's have a look at a small example On the bottom part of the screen you see just I just started a pysons session and called my application And on the upper part you see I have a career request prepared I want to set to my local server listening here and I added some parameters also for example. We have a pass in for ABC We have a query string called hello and we send some custom header. So what now happens if I sense this I Modified the application a little bit it will return our data So this is some part of the dictionary of the environment dictionary that the server is responding to the client So if you want to extend our application, we need a middleware. So how does the middleware look like? The middleware itself is a class and this class has some callable and Within this callable we have our actually codes that is executed And if we start our server it no longer takes the application as an argument But the middleware and the middleware calls our application and in case of Swift this would be the proxy server itself So what's happening here inside our call method? We first start with the response. We prepare the response here and then we try to find or to identify the path info and if the path info equals to echo Something will now happen and in this case We simply read the data that was sent from the client to the middleware and to the server and responded So we have something like an echo So coming back to the example from before if I now send Sorry Different response You see That's a server is now returning a different response I send the data hello world to the server and the response also with a hello world because of the past and this was Intercepted by the middleware and no longer the application itself so One drawback here is that if you are adding more and more middlewares You have a really long line finally and you make server Call because each middleware needs to call the next middleware and we have stuff like the server name inside it We have the port name as a they have a port number inside it So you have a lot of configuration stuff in your Python code and you don't want this you want to have a separate configuration file We also want to have some testing of course for a middleware hopefully and We need a well better way to deploy the middleware and the server. So let's have a look into this Actually testing middleware is quite easy if you think about it for a moment the middleware takes an application and If I send a request to the middleware, I expect something from the middleware. So what do we need to do? we are using a Python unit test for this and If you look at the lines below We prepare our middleware and Instead of using a real application as an argument We just use a fake application and you see this a lot for example inside the Swift tests also So in this fake fake application just returns with a for example 200 Okay, we don't test that one here, but we want to simply test that the middleware is working so we prepare a request and in this case a put request so we need an environment dictionary for this We need of course some bodies that we want to send to the middleware and this is also defined there hello world and We want to send it to slack slash echo as a pass so in the second Last line there below we try to get the response back from our middleware and finally We compare this to the data that we have sent to the middleware before Now if you want to deploy your middleware and especially in case of Swift You don't want to ship Well single files and you don't want to have the configuration codes inside your files So there's a very nice package called paste or deploy and paste or deploy loads whiskey applications and servers from a uniform resource identifier that might be for example a pysen egg and paste or deploy uses an any-style configuration file and It's makes it very nice because now you can you separate your code from your configuration There's also a script included paste script and paste script can serve your application directly from your configuration file So you don't need to write any Init files if you don't want to and simply can use a configuration file for this and Paste or deploy is also widely used in OpenStack itself not only in Swift But also I think a nova keystone glance and I would assume a lot many more projects so an example configuration file looks like this we define our application itself first two lines we define our middleware in this case it's only a single middleware and What we now do is we define also pipeline and this pipeline has every middleware that we want to use in it And finally our application And last we define the server part and in this case it should listen on port 8,000 What I also did here is if you look at the filter middleware part. I have something like suffix equals echo response so I Can add some configuration options to these configuration file that are used inside the middleware so I need to modify my little bit middleware a little bit, but not that much and Can use any configuration option that I add here inside my middleware now How does Python know that? We have an egg for example like sample and that there's a middleware included we need a setup script for this and The most important part in this case The two entry points here We have two entry points one for the Application itself and the other one for the middleware and if you look into swift We have also a lot of entry points there for each of the middleware for example And if you want to write your own middleware you need to have something similar You don't need an app factory for swift if you want to write a swift middleware Middleware, but if you want to write a middleware outside of swift you need something like that So and what pace deploy does It makes it very The behavior now it's changes a little bit Okay, so now I can simply start a Server with a configuration file like this and it's now listening on port 8,000 and the behavior is Equal to the previous behavior from the previous example, but if we look at the configuration file serve There's some parts the first part exit command equals serve and demon like true We also have the ability to start this process as a demon in the background and As I told you also you can use the paste strip itself to serve directly from the configuration file So I can just call the configuration file Executor start and now we have a listening demon in the background Which is really nice and useful because you can write very fast some small applications And of course you have your user stuff like status and of course also stop Now we are here to write something like a swift middleware and not basic whiskey application So let's have a look and what we do want to do here and For this talk, I was thinking about a small preview middleware that is useful for example if you want to write I don't know maybe your next image sharing service or something like that So what we want to do here now is for every put and the put is in that case an image that I sent to the swift proxy I want to create a small preview image and start it separate from the original object If I execute a get request and append a query string preview I don't want to get back the original object, but only the small preview object and And finally because I'm not storing two objects I also want to also want to ensure that the preview image is also deleted if I delete the original object There are a few helpers and useful tools that we are going to use inside this middleware one is called split pass and split pass takes the part of your URL and Returns your version your API version your account name container name and object name and you can Modify the or you can add some arguments to split pass to ensure that split pass returns or raises an exception in case For example, you have no object name Well, it doesn't make a lot of sense for our middleware if we want to create a preview image If there's only a request on the container level or account level So we want to exit in that case early and continue within the proxy server And because we need to store our preview image We need to execute a request within the proxy server itself. We need to do a may a sub request and To make this working you need to add this middleware after your authentication middleware because otherwise it won't work and Finally, there's a very nice decorator with Gify and what with Gify does it simplifies a lot of stuff for you and If you remember before we had our core method with an environment and in start response And if we add this ticket decorator to it We only take a request as an argument and yet this request has some attributes for example the query path information or your method environment and your body where you can read from and Finally in your middleware, you just need to return yourself app That is the next application or the next middleware from your pipeline. So let's have a look What happens in real in our middleware? so we first start and try to get the account name the object name and the container name and In case that we don't have an object name. We immediately want to return so What I already said we need to modify the split pass arguments And if you do like I've done here, it will raise an exception if there's no object name found in your pass So and in that case I exit my middleware here and the next middleware in the pipeline Will take over so but let's assume I have an object name and I want to store my preview image in a container that has in different suffix So for example, if my container more original container number was test I now want to store the preview image and test underscore previews so for a get request and The get request that has a parameter preview appended I want to modify the path and Yeah, they will help us that will be served by these with proxy server So I no longer use the original path and now use my preview pass Of course to do so. I need to my preview and I simplified the code a little bit this method called create preview as External one in that case. I will give you the code later on so you can have a look into this It's simply a Python imaging Call that just creates a smaller image But let's assume we were able to create this image and this preview so what now happens is We call a sub request and the sub request itself takes the current environment from our request the new preview pass where I want to store my preview and a new body the preview itself and Then to execute the risk risk request. I just need to call get response So in a case, I want to delete my object. I also want to delete my preview object So in that case, I only need to modify and to execute another sub request in this case Also with a modified preview on the modified path So in that case, we are we delete both objects at the same time This request will silently fail. There's no Preview object. So it will just continue to work. You don't get an exception or something like that in your proxy server Now, let's have a look How this looks like in real I have Small swift all-in-one Little machine running here Currently, there are only two Containers one container is called test and the other one is called test preview and both are empty If I now upload a small image into the test container This is like your normal behavior that you were expecting from your swift proxy from your swift cluster But now let's have a look into the test container Just preview container There's a small object and the first number there is a number of bytes And if I compare this one with the original object that is stored in the other container It's much smaller. It's my preview So these containers are both public so I can just request From anywhere never do that from a browser. I get the small premium edge, which is nice And in this case here, I appended the query path To the URL. So if I remove that part Like this I Will get my original object back So I have I don't need to remember Where are my preview objects start? Of course, you don't are you are not fixed to this you can well You can think a lot of different ways how to handle this. It's just an example So I Already said that there are existing millwares and third-party millwares outside of swift that adding that are adding functionalities to swift itself This is a small list of well, I would say the most popular millwares that are existing We have swift ours with ours is a millware that Stores all the data that you need for an authentication system inside swift itself So you don't need a database or a directory service We have swift 3 that is adding a s3 compatible API to swift We have the CDMI middleware the CDMI middleware CDMI is a cloud data management interface and some applications are using that it's different API and protocol and By using this millware you can use a CDMI like interface also in swift We have swift informant swift informant sends Statistic data to a to a stats D server So you can collect your requests and do some processing on on this Statistic data and if you want to use swift together with the content delivery network We have the swift origin server that you can use for that And for example, there's another project see the meter I think everybody heard about it already and see the meter itself is also using a swift middleware to collect the data that it needs to process process billing and whatever you need and It's included in the cylinder meter project itself now if you want to have a deeper look into what I've shown you to before my examples, I prepared a small Git repository and It's available on our github account and advanced github account and There's a step-by-step guide. I will start with a simple sample whiskey application adding more and more functionality and Finally, you have this preview middleware that I've shown you Complete with all of the tests so you can just have a look at the code and what I've done there and Yeah, it's a step-by-step guide basically There are some other existing documentation available Within Swift self so we have a great overview about existing middlewares and all the options these middleware take There is a guide how to develop your own middleware Which is which also has an example if you want to develop your own Authentication middleware you need to be a little bit carefully and you should strictly follow that guide also and Finally, if you wrote a middleware that is useful to other developers and operators It would be nice to just submit a patch to Garrett and at your project and your middleware to our associated projects And if you want to look a little bit deeper into whiskey and the web standard gateway interface and middlewares in general Have a look at the Python documentation On the link below So I'm done with my talk. Thank you very much for attending and if you have any questions Just let me know now or just write me an email Come to our booth and ask me there. Thank you very much No questions. Okay Thanks