 Hello, everybody. Welcome. Thank you very much for coming today My name is Victoria. Here's my friend Cindy And we are going to take today about an overview of Sakura's persistence of our implementation. I Have I'm both Cindy and I are all Richie former interns and we started working on Sakura for those internships We are now working full-time rehab and we will fortunately we can keep contributing to our this project So how many of you knows what Sakura is? Okay, that's a good number. Awesome. Well, just in case we prepare a brief introduction on the second concepts So I will have these hours in this so she can make this introduction So I'm gonna be talking about What is Sakura? What isn't Sakura? I'm gonna talk about its architecture Evolution and I'm gonna give you some use cases So what is that car? Um, well, Sakura is the messaging and Notification service for open stack it was built for open stack by open stack on the left you can see the mission statement summarize we want a Messaging service that is scalable highly available and efficient So Sakura's main goal is to connect applications running on the cloud And we also want to connect Open stack projects together What is not Zach are Zach are is not like rabbiting Q or Cupid. It doesn't aim to do what these services Stripe to do it they may have some overlapping use cases But soccer is mostly focused on providing messaging for web applications It is not a queue service. It doesn't act like a standard or traditional queue service And it is not an email service. It doesn't work with IMAP and It it's not for email at all So suckers main features is that it's multi-tenant it works with horizon to provide authentication and uses keystone to Keystone's ID to To find queues and messages its component base so you can add different transports and different back-end storages and They they it has different messaging patterns Uh, soccer provides notifications It also has some it has message delivery guarantee it depends on your deployment and how you set it up But for the most part it has a message delivery guarantee and it is horizontally scalable So some use cases So Sacker provides messaging for the cloud, which means that if you have an application and you Deploy your application and want to communicate with different components within your cloud. You're able to do so with saccar It provides self-healing applications if you have a multiple nodes and one goes down If that node were to receive messages Since it would be down it would you would have to reset all the messages and start from scratch with Zach are it picks up where the Messages left off. So it's as if there had been no failure and It's good for data processing. So you can if you have different nodes that you want to do or Distribute your data processing it can determine which nodes are best use and Balance your data processing for you and it is used for inter cloud communication for example heat uses a car now to provide configurations and There's some more use cases within trove and cola. This is the second architecture If you look on the left, we have different and clients that can connect to a transport protocol The transport protocol can be different things In for example HTTP It then communicates with the API which sends messages and data to the storage Everything is modular. So you can add different transport and you can add different storage backends In Juneau we did not have an API The API was kind of embedded into transport Which didn't allow for people to add different transports. So that was fixed in Juneau And that's where the implementation started So this is kind of the evolution of what sacchar has been going through so in Juneau We had MongoDB SQL alchemy Redis and whiskey as a transport and kilo SQL alchemy was removed and it was just kept us management and We also added a web socket and beta and Liberty the Web socket was fully functional and for me toca we hope to add Swift as the storage backend And now I'm gonna hand it over to Victoria so she can talk about persistent transport Okay, so So first of all, I wanted to introduce why we started thinking about having a persistent transport alternative for To communicate with sacchar During the Juneau open stack summit design sessions we started thinking on the different application that could beneficiate from the using sacchar and We noticed that there were many applications that couldn't be implemented With the architecture as it was because the transport mechanism whiskey was not very suitable for these use cases and the Characteristic across all these scenarios is Mainly web applications that need to connect With the server in real time or almost real time Some example of these kind of applications include instant messaging Audio and video streaming and also synchronization of content across browsers Then We started thinking okay, what kind of transport would be good enough for this and We consider different solutions for this Some of the alternatives we have been discussing Was raw TCP which was almost automatically discarded because the complexity of Implementing it in the server side and for the clients to implement the survey in the client side Was too high and we didn't Want to reinvent the wheel there for rotissipi to have an implementation with rotissipi would would require as Us and the clients to implement that how the messages are transmitted I mean the protocol and The security over the connection so it didn't make much sense at that time and our alternative was HDDB long polling this was a good alternative in fact Before WebSocket HDB long polling was the Option by default But it didn't offer the expected Persistent connection that we wanted for these use cases We also checked WAMP. WAMP is the web application messaging protocol This is a support to call for WebSocket But it also includes support for messaging patterns. It provides support for RPC and policy square patterns and we This was another kill for us because our API already provides support for different messaging patterns So we only wanted the transfer and it was not suitable enough but in the case of WebSocket It's it's implements at the WebSocket protocol and It's for us. It was Performing enough and good enough for for our user use case So I don't think that by now WebSockets need an introduction I think it's fairly famous already But let me just highlight some of the features that it has that We consider that make us consider it as the different solution for these cases So this technology provides a fold-up. Let's communication channel over a single TCP socket and the connections are closed When either the server or the client decides to close the connection It is efficient because it doesn't have to establish a connection every time. Do you want to send a data? It shows you just create a connection and You send all the data you have to send and you close the connection when you you are over or When you don't need to send any more messages It is simple because it was designed to be compliant with HTTP The way a WebSocket communicates with this server. It's It's Seen as HTTP get request then the communication gets upgraded and it goes Then you can start sending the WebSocket frames It is firewall friendly it use port 80 for playing WebSockets on port 443 for This secure version of WebSocket. So you don't have to ask your operator to open a determinate port for the use case you want to cover and Last but not less it has a standard WebSocket has been a standardized headbutt ISDF and the WebSocket API is being standardized by WGC but Now we when defining the technology that we wanted to add support for That wasn't enough. We had to define several aspects For our implementation. These aspects are mainly the message format How does the message has to look like? We need to make sure that the messages carries all the information we want and that they are Compatible with the current transport we has We will it's whiskey and we also left the Door open for new transport to come So we wanted to make it the most simple and complete enough to make this happen We also talk about message realization Basically, we want to make sure that the message you send to the suck to suck our server our Standard and that you can validate them. So, you know that what the the Message you are receiving has a word format as it's valid We are we are actually talking about connection protocol and security Well, I don't think I have to say why why this is important but it's something that we also had to to research and make sure that we had a good support for this and We also discuss about which third-party libraries we were going to use for this implementation So first of all, I want to introduce you to the message format we had come up with Basically the message format is pretty simple It has three fields has an action field. They basically has a verb indicating What operation want to perform for instance for the queue creation? You only have to issue a queue create and basically that's That's pretty much it Then you had the headers they carry information and made it metadata for Authentication and control in the server side for instance, you have the expersion ID the client ID and the xx out token Thus headers are used to enable multi-tenancy in suck hard Finally, you have the body that for some operation it Must be there and for some operation doesn't need to be there in the case of queue creation You need to specify the queue name that's Required, but you also want if you want to specify some metadata for the queue You can also do it But for some operation is not needed for instance for I don't know a queue get or a queue list It's like you don't have to specify anything else Then let's get back to message serialization We spent a lot of time discussing about this We started like our first choice was protocol buffers. It is a pretty famous technology is this developed by Google We discarded this like very Fast because it's not so straightforward to implement it You need to have a compiler in the server side and the client side and and we didn't want to adapt on complexity At least not in this first stage of development Then we also Check into message pack Message pack was one of our favorites because it's very similar to Shazone. It's very easy to read and It's very efficient, but the main problem we found with message pack is that the Shavascript library is pretty updated outdated and is not very maintained and We wanted the to have users and you have to have what applications to use our Over the planet so message pad was unfortunate we have to to move forward from that and Cap and proto it is very similar to protocol buffers So we hit almost the same issues. It's not so straightforward to implement and then just We should decided to leave it form for another cycle So we will with Shazone most of the applications in open stack use Shazone as a message realization format So we thought it was good enough. It is readable on the At least for this first implementation is good enough In the next couple of cycles we are going to to do some research on other message Messageization protocols and see if we can implement that Okay, about connection protocol and security 4 to 100 in this case. We didn't have to do much work WebSocket Enables connect and a connection protocol it use for 80 and or power 4 for 3 depending on if you are using plain WebSockets or the secure WebSockets Security WebSocket has a secure version and we also complimented this with the uses of keystone Every time you want to to connect to Sakhar you are going to use an auto can and That makes the connection to Sakhar safe enough and finally the third party labor is We discussed to use Probably the most famous right now for WebSocket is autobahn This is WebSocket implementation This is a standard and it allows you to use either a sank IO or Termnado and No, sorry twist it and We we thought that it was a good possibility We selected out one and we are using the assigned KO version for it For Python 3 at least and we fall back to trolley for Python 2.6 and Python 2.7 It's widely used and it has support for many languages It's very well documented and it has it has a very big team working on it and Releasing new versions. So we we thought it was the best option for this But we also check another options like socket.js and socket IO These libraries are mostly intended to be used in the client side is these are shower script implementations So this port for Python they have Python alternatives, but this they wasn't very good And also we checked do this for PI But it was in early stage development and we didn't want so to go with it to that way. So we should move forward without a ban and Well, basically these are the design decisions we had to face This was discussed and Defined it in the sooner release and the killer release And now we want to introduce you more or less What is how is that you had to do to interact with soccer? How does the WebSocket API looks like? Cindy you want to? Yeah, so Currently, so let's do a review what's currently supported by the second WebSocket API So it's part of the version 2 API of Zachar But it doesn't fully support all the endpoints and this was done because not all of them are needed Zachar has some features that for example like pooling where We you you have access to a pool of storage. So Do you want to talk about sure? So the endpoints we are supporting right now is queues basis climbs and subscriptions The control side didn't have didn't make much sense to a support for that because there are no scenarios in which you you really want a persistent solution to I don't know create poles or create flavors or Do management operations? I cannot think on a Is an area in which you would need that please let me know if someone thinks on some of them So for now we only add support for for these endpoints and more or less We really don't want to bore you with an example of how you are going to interact with each of the endpoints And for all the operation you have but this is an example of what yes This is an example for a simple operation of creation for all the resources We have cue creation message false claim creation a subscription creation You can see that a verb is used for for all of them in the action field and you have several fields in the body Defining if the kid the different parameters you need for each operation most of the Advanced would look the same the way you interact with them and We actually I was going to talk about documentation for this we are working on documentation for this so you can know which Bear are using for each of the Operations We are in a current Total revamp of the documentation for soccer because a lot of things has changed in the last couple of cycles and Well, this support for this API is is Is one of the things we have pending and we are working on that so Fortunately with in some ways from now we are going to see a full documentation for this in in the wiki page and We also So thought it was worth mentioning the future where we have plans for the WebSocket API We are planning to add binary support WebSocket can transport plain text and also can can transport binary data currently Our implementation only allows you to transmit plain text The support for binary that data is going to be implemented in the next cycle and we are working on Designing how is that going to work and that is a task that if everything goes well It's going to be implemented by an original in turn in the next cycle and we expected to have it by my cycle next cycle and sorry The message realization as I was telling earlier We are fine with Jason right now. It works As expected and I think the performance is very good But we want to cut some slack on the length of the message that we are Transmitting right now and maybe gain some more performance in that area So we are probably going to revisit the implementation with particle buffer. So maybe cap and proto We are we have to define it yet, but If any of you have experience with those message realization and you want to give us feedback about that We will totally appreciate that finally I wanted to introduce you A new case we have been working on Recently we had some inquiry from other teams in OpenStack and One of them is horizon They thought that maybe the usage of soccer as a messaging system would be good for them So we have been working on a proof of concept. Unfortunately, we don't have We couldn't make the demo for today We didn't think we were going to have much time to do that so Basically, I want to see how let you know how how this is supposed to work and which is the use case So horizon right now Horizon right now This is just one of the use cases for horizon This is screenshot over here is pretty old. I don't know if it's grizzly or a banner But the it doesn't change the essence of this and horizon right now Has tables to show you information about all the resource you have in stack for instance here you have the instance tab and For every change in it resource you have in your stack Horizon is doing lung pulling is pulling all the time to the service to know if there is a change and This is not like very performant horizon is always trying to find if there is a change for all the resource you have in your stack so we discussed about that and We thought that a liberation here the slide Liberation soccer as is As Cindy mentioned we have support for notifications. So basically The solution we suggested to them is as follows you Need to have a deployment with Cilometer Cilometer gets all the events that happens in the stack every change in any of the resources are notified to Cilometer and Cilometer creates a queue on soccer and Horizon had creates a subscription to that queue in soccer. So every change so every message that comes to that queue in soccer is Notified to horizon. So horizon has to forget Doesn't need to keep pulling the rest of the services soccer will notify on any changes for instance In the example we were seeing the screenshot and in another in the other slide when you create a new instance your instance is going to be in a Building a state and then it's going to sector active if when the resources are located and for that is like There is in certain events that Cilometer emits that horizon would listen to and Doesn't have to do anything else that then you get that event and horizon issues some change in Shavascript code and Tables are updated and you're done some other use cases that are not in the slides but That we also got from the horizon folks are the error handling right now. They have to keep Pulling all the services to know what happened and where those that happen and Usually, I don't know. What do you think about that? But most of the user have been talked to they always complain about the information you got in the errors in Horizon and that's because it's very hard to to keep all the information So horizon Susaka would be like could make out real good tool to to improve the error handling Some use case that when Cindy mentioned No, it yeah The intercloud you want to to introduce those use cases so trove uses trove currently uses SSH to connect to its What is it the Yes, well true There are some Projects new person in open stack. There are mostly that those can be considered as platform as a service not so much as infrastructure as a service and all of them has a similar architecture in which they have a Controller side and they have a gaze Asian in the instance they launch running So when they want to perform an operation the control side has to communicate with the gays ancient Like for instance in the case of throw, which is that there is a service When you want to create a new data store You communicate with the API of throws you perform and throw create new data store and that message right now goes through rabbit MQ to the gaze Asian and then that The data store is created in the guest, but that is not very safe And there's something like happened with Zahara Because you have to pass the predation so private MQ in the case for trove to enable the communication between the two and It is not multi-tenant. So having Message solution like Sakhar in the middle will allow to have all the message isolated and Allow to have a more safe secure environment We recently have a decision. I think was yesterday with Sahara and they decided we're going to to move forward with solution for the communication between the two So that makes another use case for this. Well, I don't know if this transport wallet leave us a car But maybe the most important for for the web sucker area was the rise in one And There's also a session happening concurrently right now and yes after this talk we were lucky not that we had an horizon So car session it was a skull at the same time. So in that session, they are showing the demo for the we were talking about And the wall In the next I think in the next slot there is a going to be another session to to do the same to show the demo again because of this situation So everybody's invited to come. I don't remember which one was but it's in the skill So if the science on it schedule. Yeah. Yeah, so I think we're done Is there any questions there's a cue in it So so you have a product the same problem that the trove people have which is exposing the Zalcar API to a virtual machine It is not pretty much the same because in the case of trove you are sending your credentials to The case instance and there is a feature that we don't didn't mention that it's called sign URLs That enables you to create a shared resource that doesn't need authentication But you have a key on it. So it's a thing. It's a concept is also also available in AWS So you don't have to share any credential with the gay station Before you can authenticate you need to communicate The first problem you have is that the virtual machine should never see the open stack control network. Yes So right now how it is implemented if you deploy Trove with the rest of like all in one single node Then you are sharing your rabbit MQ instance With all the resources so so how do you get around that in soccer with rabbit? You have that problem? Yes, but in soccer you are Using a different queue that the one you are using for the infrastructure You keep using Rabbit MQ for the infrastructure and you suck are only for trove so you don't put in a tricky situation all the Information about tokens or everything you are sharing for your infrastructure. I Don't know if that makes sense Are you are you suggesting that we use a car as a replacement or no I'm just I'm just trying to understand what the Zakaar shows up in the data plane of the virtual machines or In the infrastructure. No, it's in the data plane So you don't see it in the as an infrastructure service is not available. It's like a point for Zakaar. It is not It's usually only get a new role and you are done Any other question there's no more questions then thank you for coming Thank you very much For some reason you have any question after the session and you want to reach us out Our IRC handlers are VKMC and Cindy Pajares I'm going to publish these slides and I'm going to share the URL for that in my Twitter So we're also going to be at the design session happening right now for yes horizon in Zakaar Thank you very much again