 So, welcome to live from Oslo for Akta, or Barcelona, thank you for coming, and just going to get started and jump in. Hopefully this will help you useful for all the folks here and show you some of the stuff that Oslo Group has done during the last six months with some various presentations about what has changed, what has been added that can hopefully help your developer experience be that much better. Various other folks will jump on as I as I go through some slides so that they can present their part. Let's get started. So, what is Oslo for people that don't actually know? It's not just a place in Oslo, I mean in Norway. It is actually a bunch of smart people that have helped out with many different things in OpenStack, probably a very many years of experience in OpenStack. I myself have done five years I think, probably others have similar amount of time. Trying to improve a lot of the high quality components that back OpenStack, for example the messaging layer, the database layer, some workflow management tools, some helper libraries for state machines for example, just a lot of utility code as well. All sort of built out of many years of experience from the various OpenStack projects, isolating it as much as we can into high quality libraries that we can share not with just with OpenStack, but with the rest of the world. This has turned out pretty well I think. We've got a lot of useful tools and components that other projects have started to use as well. So, let's go into what happened for the next, yeah for Newton. What do we have from some of the high level is we have some mutable configuration that was done as a way to make it easier to change configuration on demand. There's a messaging back in that we're going to go into equitability on. I think there's even a whole presentation next in the next hour. Next session here. There's this new concept of policy and code for those who wonder what that is. It may hopefully become a little more obvious in a little bit, but it's a useful layer to connect the sort of the disconnected policy that was in files, move it closer to where it's actually used. Of course there's been a lot of bug fixing, various tweaks to the code itself, various releases of requirement updates. Those are our favorite. Thanks to the release team, this has all been a lot easier than it was previously. Now it's more of just the submit a YAML file and the rest happens pretty much automatically. So thanks to Doug and Theory and I think Dimms who helped out would make that possible. So let's jump into some of the changes here. Is Alexis around? Okay, we're good. So, oh wait, let's jump in. I think let me try turning this off and see what happens here. Hi there. So, useful config, also known as reloadable config. What is it? It's a way to reload your config options when you send a SIG up to a process. This is handy for multiple reasons I'll get into on the next slide. A word of warning, each project and each option must be enabled. This is a huge pain, but it's unavoidable, I'm afraid. So what's it good for? API uptime. If you don't want your API servers being unavailable for a few seconds, when you reload your config, incremental development, developers often need to just play around to hone in on what the right setting something is. And of course, when you're diagnosing a fault, sometimes if you restart the service, the fault goes away and then you can't look at it. So, which options support it? Only a few so far, more are coming. In Oslo log, we've got default debug and default log config append. In Nova, we have live migration completion timeout and the progress timeout as well. In Akata, a few more live migration options are happening and I hope that more Nova options will come through in time. So, the big one here is log config append. This is a sample log conf. It's very small. I hope you can read some of that. These can get much bigger, much more complicated. Each of the things you see in here, so the loggers, the handlers, the formatters, they all have some options. You might have a custom date format in here. You might want to add adjacent formatter. Now, all you have to do is change your login conf, send a message to the processes and they will automatically pick that up. This means that you can change log levels. You can add adjacent logger. If you want to start using fluently, you can do that all without any API downtime. So, this part is more for the developers among you. Hopefully there's a few out there. Enabling mutable config in your project. This is the easy bit. You just add restart method mutate to your process launcher or service launcher. This assumes that you're using Oslo service, because why wouldn't you? But if you're not, then you just need to make sure that you call the new Oslo config mutate, I think it's called mutate config options, rather than the old reload config options method. Enabling options, also pretty easy, just add mutable config. Sometimes, maybe you can see in this code, options are cached. So, the value is taken outside a loop and then it's used inside the loop. But the actual option isn't checked again, because it's cached in this local variable. This is unsafe. If you reload your config, it won't affect your live state. So, pretty simple. Move it inside the loop. Some cases, that won't be quite so obvious, but in this case, it was easy. Yep, simple as that. Now the option is checked every time from the config option and it will take effect. Things can also be cached externally. So, for example, when you set your log level, that's what I would call an external cache for your log level. So, in that case, you need to register a mutate hook and then every time your config is reloaded, all of the mutate hooks will run and you can check for your options changing and reset your state. Oh, I pressed the thing. Make it stop. Yeah, good enough. That's it. Next up is a king who's going to go over some of the new messaging components in the back end that is pretty powerful. It got merged really pretty recently and so it's hopefully developer ready for experimentation and so on. Thank you. Hi. Yes, I'm going to talk about new messaging back end. How did that just happen? Ah, you've got it. That's like demons. We're talking about a new messaging back end. Let me go straight ahead and say no, it is not another broker. We are full up in the broker department. This is different technology and that's scary. Okay, so before I talk about it, it makes sense to talk a little bit about Oslo Messaging so you can kind of understand where its best fit is in the messaging deployment. So there's two patterns offered by Oslo Messaging. Remote procedure call, RPC and notifications with PubSub. These are very popular patterns, but they're very different in terms of implementation. A remote procedure call is a synchronous exchange of data between two entities, right? The client makes a call, the call goes to the server, the server applies and you're done. But they have to be present at the same time in order for this to work, right? So they're tightly coupled in time. If the server isn't there, the client is going to fail, the call is going to fail. Contrast. Sorry. Contrast with notifications. Notifications is more of a synchronous pattern. Notifiers can issue notification messages and they have no concern of when they're consumed. So it's going to be totally asynchronous. The notifications can come, the notifiers can go away and at some later point, listeners come in. This pattern, whenever you have two messaging entities that are separated by time, you must have stored forwarding. You must use queuing. This is perfect and this is the reason why we use a broker. However, if you look at the way we actually implement this on a broker, it's not optimal for RPCs. This could be any broker, by the way, it's not just RabbitMQ, it's certainly CupidD. So if we look at notifications, as I already explained, notifications get put on a queue, the notifiers can go away, the notification of the listeners comes in. It's a very efficient transfer of information. However, RPC, it takes four transfers and two queuing operations, actually four queuing operations to do one single RPC, right? The request comes into the broker, it's queued on the broker, but the broker then acknowledges it, which means the client has no visibility into whether or not that message is actually consumed. Eventually, it travels to the end of the queue, it's pulled off by an RPC server which acknowledges it back to the broker. Server processes, replies, again queued, again act, set across the queue, sucked up by the RPC client and then again act. So that's a lot of work for a broker to do compared to notifications. We have different options. In also messaging, I think since Liberty perhaps, maybe Lataka, I'm not sure, you can configure different back ends for the different patterns. You can have one back end for RPC servers and one for notification listeners. You can use a broker, you can use ZMQ. There's plenty of different type of technologies and what I'm going to talk about is this new technology from the Apache Cupid Group. This is a new messaging infrastructure component. It's not a replacement to a broker. It's different. It complements a broker. Think, it's a router at the message level. It routes on messaging addresses. So think of an IP router, but instead of routing on IP addresses, it learns subscribers, learns the addresses of those subscribers and takes the messages that are addressed to the subscribers and efficiently forwards them through the net. And as you see, it's optimally deployed as an interconnect network. You can use it singly, but you don't leverage any of the high availability and scalability that I'm going to talk about very quickly here. Yes, not a broker. Big difference. It does no queuing. It does some buffering when it's sending messages across, but it never accepts the message. It never acts the message on behalf of the client. The message is sent into the mesh and it hops until it gets to this destination. The acknowledgement from the client when it's done and acknowledges the message or perhaps a knack, it gets hopped back. It will knack a message when it determines that it cannot get further. So if we're hopping along a path and then it turns out the path is broken, it will send back a knack on behalf of the recipient. And that is a true delivery status. That means if you have an acknowledgement, you know the message was consumed. Likewise, if you have a knack, you know you can recover from that. You can retransmit. It's a completely stateless. It doesn't keep any messaging state that needs to be restored after a reboot. So these things can go down, come up, and they learn. Totally dynamic. Same patterns, unicast, multicast, and balanced as a broker. Like I said, everything is dynamic. It basically learns the topology in the network and computes and learns as subscribers come and go. And it uses a shortest path algorithm with link cost. So the failure recovery is a little different, all right? It's just like the IP Internet. It routes around failures. It will use a higher cost path, perhaps, if a lower cost path fails. It's all dynamic. Nobody has to pre-configure these things. So high availability is not achieved via clustering or anything high level. There's no external synchronization system that needs to be deployed. All the routers appear as they talk to each other. There's no master router. There's no split brand. You can have degraded performance as things fall down. But once they heal again, you're back to where you started from. Okay, so here's a quick example. RPC client, RPC service, speaking from the mesh along this red line, this is the optimal path. Should a link fail? It's transparent to the RPC client. The routers readjust to a less optimal path but still functional. Should a router get poofed out of existence? You see actually two things here, because I didn't have an internal router in this picture. It could be transparent to the client if it was, but in this case, also messaging has the ability to fail over. You can have two connections, a backup connection, for instance. So we have two things going on here. We have a backup from the library, a failover from the library, and also a new path computation. There's more opportunity for parallelism, right? And since these things are dynamic, you can add routers to increase your messaging scale and you can remove them when you don't need it without service interruption. And if there are alternate paths and two communicating parties don't intersect the router, you actually do get true parallelism in the message flow. So this is what you can do in Newton. Like I said, we can configure RPCs to use one back and notifications to use a... Continue using notifications for a broker? Your option now is to use RPC via the AMQP transport and take advantage of the message bus. In the future, and this is actually part of the Cupid project, not necessarily all that stuff, in the future you'll be able to use brokers as a service within the mesh. So the notifiers and listeners will connect directly to the broker, but they'll be routed intelligently to the broker because they need to store and forwarding, whereas RPCs can go direct. And yes, meeting downstairs right after this, also messaging documentation explains how to use it, configure it, play with it in DevStack, or if you want to deploy it, information about the dispatch router is up at Cupid, and Andy and I have been working on the driver stuff we're in OpenStack Oslo, PoundK, Juicy, and Smith. Thank you. Yeah, thank you. Thank you again. So the next up is our policy and code editions that are used in Nova and I think are getting deployed elsewhere. So here you go. Thank you. I'm Changmo Guo, ICAS GCB. The Oslo Policy Library provides support for real-based access control, policy enforcement across all OpenStack service. In neutral release, there are two important updates. The first one is the Oslo Policy Library supports policy file in YAML file. The main worker was done by Bronte Nuderson. The decision format is a subset of YAML, so the current policy file in decision format is still supported. When we use YAML format, we can add comments in the policy file, then we can explain what does the rule mean. So that's a benefit. Another important update is the policy and code. The main worker was done by Andrew Lasky. I have reviewed most of the comments. So before Newton, there are some drawbacks. First, the employers must provide all policy rules in policy file. The other drawback is there's no way to discover what to do or check in. For example, in NOAA, for evacuation operation, does the only admin sister can operate? So we reject the policy defaults in code. There are some benefits. The first one is the employers only need to provide the policy file if he wants to override something. If he doesn't want to override the default rules, he doesn't need to provide a policy file. In the Newton release, we can generate a sample file with a tool. Then we can know what exactly the rules are checking in the code. We also provide a tool to generate effective policy to combat default rules and the rules loading from policy file. That means we call it an effective policy rule. The last one, we can list redundant policy defined in file and then we can remove the duplicate one. There are four command line interfaces. The first one is also a sample generator. We need two arguments. First is the application name. Here, for example, it's NOAA. Then we put an auto-pull file, direct the file path to auto-pull. If we omitted the object of the file, the file will be sent to sender arrow, send out. Another command line is policy policy generator. The difference between the sample generator and the policy generator, the policy generator will combat default rules and the rules from the policy file. Then we have also policy list redundant. Here we list duplicate rules in the file. The last one is a test tool, also a policy tracker. There are four arguments. First, the policy file. Then access means access information, usually including user name, project ID. So that means it's optional. Rule argument is also optional. If we provide the rule, it will show all the rules. So how to adopt it in your projects? First, we got, if for instance, then we addressed some different rules. The rule defaults have three arguments. The first is the rule name. For example, here is the admin required. It's the rule name, then the real rule. Here is the rule, admin equals true. Opposite argument is the description. Here is who is considered as admin. The description can be used to generate a sample file. Then we use the browser to address the policy rules like this. And lastly, there are some more information you can reference. The first is the also spec how to explain the policy in code. The other is the described sample generator. The last one is the NOAA adopted the also policy. So you can try this file, get more information. Thank you. All right, next up we have more about 0MQ. So another Oslo messaging one. Here you go. Thank you. Hi, I'm Alex Izometin and I'm going to make a short update of what was done in Newton for 0MQ driver. So as Ken pointed out, the optimal message bus looks like this. So we have RPC client connected directly to RPC server. And that's what you can have with 0MQ. So how to make use of 0MQ driver? So the main thing you have to specify the transport URL with 0MQ prefix in your code, in your config. And this URL points to an AMP service, which we use for matchmaking clients to servers in 0MQ. And generally it's Redis. And other options are, you may specify them optionally and they configure the driver in the ways I will describe in following slides. So on this slide you can see the direct connections configuration. There are two bad things about direct connections. The first thing is that you will have a lot of connections on some centralized services like Newton server or Nova conductor, Nova scheduler. So to optimize this we've made those connections dynamic. And that means that you open connections for request and close it after request. And this way it takes some overhead by connecting and disconnecting, but you have quite optimal number of connections. And the other thing is on the configurations on the left side we have slow fan out because the client need to connect all the services to make fan out. So on the right side you can see the mixed configuration with PubSub usage. So it uses publisher proxy, sorry, to send fan out. And this second configuration is quite optimal from connections number and from the performance perspective. So I'd recommend this for a big scale. And if you don't like dynamic connections and you want to have some predictable number of connections you may use static connections. But then you need to use proxies because static connections are like getting accumulated with direct connections. So we need proxies to optimize this number. And here we also have two possibilities to use simply routing. Then again the client is in charge of doing fan out. And we can use PubSub for fan out but then we have more connections. So that's the two main problems of message queue scaling is performance and connections number. So we have so many configurations to optimize those things to find the balance between those problems. And here I've listed those configurations and highlighted the ones I'd recommend to use on big scale. Here is a short summary of features that were added or something changed in Newton release. And thanks to all contributors to 0Q driver. And if you need more info you can follow those links. Thank you. Thank you. All right. So I guess we can open up for questions. We also wanted to put out a couple of links out there where various meetings are for the Oslo team. Just in case people want to jump in and get active there's the team meeting which we have weekly. I forget the UTC time but it's on Monday. We have the open stock Oslo IRC of course and we have a bunch more sessions tomorrow and Thursday. More of design sessions not presentations. People are welcome to jump in on the various topics. You can click on that and see what those are. Any questions from folks that we can answer about new features or policy and code or any of the other topics that are useful for folks. One quick question about the rotor based messaging. What's the performance in terms of detecting failure? Oh I'm sorry. I've been told I have a big enough mouth. Connection failure. It depends on the type of failure. If you have a link go down for example that's immediate and the rotor will start broadcasting changes in the topology. There's also, well it's running a writing protocol which includes halos. So if the TCP connection doesn't immediately fail but there's a problem downstream past the proxy that will time out. Now that's configurable. I think the default is three seconds without hearing a whole or three halos or something like that but all that is configurable. So there's immediate failure and non-reachability failures. Any other questions? Feel free to ask anything. Oslo related. So if 0MQ... There. If 0MQ and Cupid were to fight who would win? Well I have a fix in on Cupid. No, this is not the Cupid program. You're talking about routing versus the MQ, right? Yeah, they both seem useful for RPC. So as a complete newbie to Oslo messaging I was just wondering what the recommendation is. I might be a little biased. So what I'm going to say is you go with what works for you. I think that a lot of the small configurations using a small broker is fine. Rabbit's been the most tested and most well known so you're probably with that. The stuff that I was talking about, this router stuff is going to be adopted by the high end. If you come to the meeting we have downstairs we'll kind of show how you could grow a geographical mesh and connect different offices, different deployment centers together using it and managing it. I'm going to do a demo of the management stuff too. In terms of ZMQ versus this ZMQ is directly over TCP. It's going to be really, really fast. This is very little latency. Obviously when you go through network you're going to latency for every jump, every hop across. And that will vary with load. Sorry. The laptop for example, just doing a low... A low... Thank you. I'm a software guy. I don't know how to... Latency is in the tens of microseconds across and not under load and it starts to vary a little bit. This router has been totally separate from OpenStack. It's about two years now being used in the field by other projects like OpenShift, Satellite, and there's various upstream people are using it. So we're not exactly where we want to be with performance. The goal is to get it and keep it under 100 microseconds per transfer. But that's going to add up. If you have TCP going directly that's about as fast as you can get. But there may be reasons why you can't get TCP connectivity specifically that geographical change. These routers can serve as IP bridges. They can take two subnets and bridge across them transparently to the client. So you can have two unroutable subnets connected via a router acting as a bridge. You don't need TCP connectivity. So maybe that might change your needs. If I were you, I'd evaluate both. I'd evaluate all three. It's good to have choices and you can figure out where you want to go from there. But certainly, ZMQ is wonderful. Hopefully this will be as good as ZMQ or Season 2 adoption. That's great. Thank you. One more question on the router. What would be your recommended configuration? Where do those routers get deployed? Do I need one per compute node? Do I need one per data center? What we're visioning is depending on the size of your data center, one or two in the data center, two for fault tolerance, and then, you know, interconnects to other data centers using two, four. Come to the meeting, sorry, the presentation. We'll kind of talk about that stuff. I think that looks another useful thing for me is that there's, make sure you, there's two distinguishing drivers. Like there's the RPC one and the notification one. You may want to pick different tech, at least what GoDaddy's doing. We're looking into using Kafka for the notification one and we can use RPC. There's diversity and depends on your requirements, sort of where you want to go with it in reality to see where you go, which driver you want to use, which one may be appropriate, what the rest of the community is using, that kind of stuff. Any other questions? All right, thank you folks for showing up. We have more sessions next soon about the Oslo Messaging MQP driver and we have lots of design sessions tomorrow.