 Hi, everyone. Welcome to Live from Oslo. So Oslo is a set of libraries that were initially part of the Oslo incubator repository. They got pulled out of, you know, over a long period of time from different projects and we've tried to maintain the Oslo libraries with a consistent stable interface for different projects to use. So if you pick any project that you have, Neutron, Nova, Cinder, they use Oslo libraries. And there are various Oslo libraries that have evolved over a long period of time. And we are also creating new libraries starting from scratch and trying to define the interfaces for people to use. So this is just to give you an idea of what we've been up to in the last cycle, we have one new library called Oslo PrivSep. This is a replacement for Oslo Rootrap. If you know, some of the operators are very familiar with how to protect using Oslo Rootrap. Now we have a new one rolling out called Oslo PrivSep that we are trying to get different projects start off. Then we have some new drivers for Oslo messaging. We'll go more into details about the individual drivers that we are working on. And there's a bunch of new features that we are rolling out. This is not an exhaustive list. This is just the top of the head that we came up with. But there's a whole bunch happening. And we have updated documentation as well for all these features. So if you are a developer working on any of the OpenStack projects or if you are bringing a new project into OpenStack, then you should definitely check out what we have in terms of Oslo libraries and try to use the Oslo libraries. So to get us started, I have a panel of team members here. And if you have any questions, you would have to talk to Josh. Hi. So Josh is the PTO for the upcoming release. And I'm the PTO for the previous release. So I'm handing off stuff to Josh as well. So to get us started, we have Dima Okhlaug. He's going to talk a little bit about the 0MQ and the PCA drivers. And then we'll hand off to the Oslo PrivSep and documentation. Okay. Dima. Thank you. First of all, probably when we started development of PCA driver, I heard a lot of questions like, why do we do this? We have combo driver and why do we need another driver? I will answer for those questions now. We have a couple of problems connected with, we don't know with which one, but somehow rabid clusters can be divided for few pieces, for some pieces. And we don't know what is going on because a lot of, we have a lot of obstructions, a lot of dependency libraries like Oslo Core, Oslo Combo Driver, Combo itself, Combo extension for rabid MQ, rabid MQ itself. And when we had troubles with debugging these issues, we asked guys from Pivotals to help us. And they recommended us to update libraries and components tech because it's probably some issues already fixed. And even not, if not, it's much easier to fix in code which is closed upstream. And now we have new implementation based on Pica library. It's a lightweight, poor Python library which is developed as rabid MQ client and don't have any not needed for other MQP brokers obstructions. So now we can implement more efficiently driver because we have full support of modern rabid MQ features and avoid red redundant functionality not needed for MQ. Also, Combo Driver has a lot of legacy code and problem connected with it. We don't know why some code is needed and it's much complicated to implement some new features or do refactoring because we don't know, can we remove this code or we should leave it. This driver is developed from scratch and it's much more easy implement new features and fix bugs and something like that. So now we have two options, Combo Driver and Pica Driver. And why you should or probably not should switch to Pica Driver? There are a few benefits of new driver. It fixes a lot of problem by design, as I said before. For example, hard bits which are already implemented by Pica library. Reconnections problem which previously was used from Combo library and now we have our own implementation in Pica Driver, which we can customize for our needs. It's also much more configurable than Combo. It's not because it's better, but you just have already a lot of possibility for configuration. Also, I managed to make it mature enough. It passes all temp scenarios and also it's much more efficient about 50% on RPC calls, but probably it's not very significant because it's very small middleware and I guess that target application will eat much more resources. So why should we not to use Pica Driver? It's not compatible to Combo Driver. It uses different naming of queues, different exchanges and different configurations. So rolling updates is not available from Combo Driver to Pica Driver and if you want to make update, you need downtime, at least for now. It could utilize RebitMQ a bit more if we made performance tests when we had three nodes RebitMQ cluster and a lot of clients and servers and when RebitMQ was a bottleneck, Pica Driver works slower than Combo about 10% because he provides more strict delivery guarantee and it eats more RebitMQ resources. Also, this driver is new, not widely used, so probably less mature and you can be faced with some new issues when Combo Driver works better. So how do you use Pica Driver? It's very easy. You should do the same as with Combo Driver. Just change transport URL and add Pica prefix as protocol. In this slide also you can find links to configuration guide and if you are interested, please take a look. And my plans for Newton. So currently we have patching process for making configurable serialization. We expect that the message pack will be a little bit faster than JSON serialization. Implement configurable connection factory because now Combo Driver eats a lot of connections, one per listener and also has connection pool for message sending. With Pica library I have already patched on review which allows to use single connection per process. Also, challenge, it's implementing generic solution for cross driver rolling update to allow possibility switch from Combo to Pica or another driver without downtime. And polishing, fixing and as usual. That's it. What I can say about Pica Driver and I have to speak about 0MQ Driver too. So also, during this Mitaka development cycle, we rewrote 0MQ Driver again. Why we do this? Why do we do this? Because we understand that broker-based solution is have a lot of benefits but we also realize that probably the biggest and disadvantages of broker-based messaging is centralized broker and it's limited ability to scale and we need at least ability to make deployment without central broker at least to test. We had previously all the 0MQ Driver but it was not maintained and after code review we decided that it would be easy to rewrite it at all. And now we have new 0MQ Driver. Probably I have already talked about this. In this slide you can find the goals which we made for development of new driver. So it should cover of tempest scenarios. For now it does this but we have only problem with salometer scenarios. That's probably it. We can go forward. So to optimal implementation we need to map also messaging patterns to 0MQ patterns. Was decided to implement call using dealer router, 0MQ message boxes cast over push pool, message boxes and finout over pubs and also we have two options of deployment. First of all peer-to-peer that means that each client will be connected to each server. It's very expensive for connection number and don't really work for big deployments. But it's very fast. Therefore to fix this issue we also have proxy solution that we have proxy on each controllers to combine all connection from this host to this proxy and then have only a few connections to other host. Also this driver for now it has only matchmaker based on radius but it implements it has plugable interface and you can implement your own matchmaking mechanism. This is option of deployment when we have here we have common deployment just at radius for each controller from matchmaker as matchmaker and publisher proxy for cast finout. I will explain it a bit on the next slides. Here also we can find some information about count of connections used and it's actually very big. Here explained differences between between broker based deployment peer-to-peer deployment when each host connected to each other and deployment which proxies I said about before. Here just some calculation how to estimate number of connection created. On this slide you can see what type of the RMQ sockets we are using for calls and direct casts means not finout casts we use the RMQ dealer mailbox for client and the RMQ router for service it's recommended solution and we haven't invented the wheel. On this slide you can see finout pattern we have always here publisher proxy because we can say that in the RMQ publisher implements something like broker so it handles messages delivery so we should always have some endpoint to publish the message. Therefore we have pop-up pattern here between service and publisher proxy and the RMQ dealer router connection to connect clients to publisher proxy. It's the same graph but for options of deployment with proxy for calls and direct casts almost the same but we have one extra router proxy layer and how to try the RMQ driver we can find on these links you have a few options just enable it in open stack use the stack plugin for example if you want to run the stack jobs run simulator just to test it and probably that's it. Known issues too many socket consumption as I said before some issues with services recovery and after starts I don't know details but I think it's connected with with radius matchmaker because it's cache and somehow it can contain not valid data. Cilometer doesn't yet work properly I also don't know details but my best guess that is because cilometer requires sending notification when anybody is in it and this notification should be stored and processed later and general probably disadvantages of 0MQ it's that messaging is not reliable so we have in our plans to have to try implement head beating and retrying and acknowledgement but for now at least for me it's not clear is it needed or not because it's 0MQ and probably if we need a guarantee of delivery probably we should use something else plans for future what's interesting here try to new transport for multicasts means usage different network protocol to send multicasts it should be much more efficient provide alternative not discovery but I don't know who needs this but probably why not and may use curve 0MQ to provide security probably somebody requires security messaging thank you for attention Angus will say it a bit about Oslo right let me just switch over here using right the powers of computers there we go okay good afternoon so I'm Magnus and Kevin also helped me out with this talk here so just quickly I'm going to talk about this new Oslo proof set library and the problem that it's trying to cover what route wrap also used to do why do we do this at all a little bit of a history of route wrap and how we got here and then just talk about proof set and what it's currently up to so open stack components like a lot of client server servers basically have clients coming in off the internet off the network the server mediates access to some sort of important resource so it might be starting virtual machines in over it might be manipulating kernel routing tables in the case of neutron it might be accessing object files from the big shared directory for a Swift object server anything like that so it acts as a layer between the untrusted users and whatever it is it's important so when security people look at something like this they see something more like this so there's you have to assume there's bugs in that code and so the problem that route wrap and proof set but really dealing with is how do we design that defensively how do we plan for the fact there's going to be bugs there and then try to minimize the impact of some sort of exploit there because fundamentally what you're providing is access to the outside world where the outside world can come in to you and then you've got a stand between that and whatever your important resources so yeah the approach taken by lots of similar components around software generally is the principle of least privilege you want to break up your software into different components and give them only the privileges they need to do their job in an open stack case typically the privileges you need are root privileges you need to manipulate a kernel routing table you need to manipulate the the hypervisor so what we're trying to do here is make it so that most of our open stack code does not run as root although some of it can when it needs roots in order to do whatever the important thing is so we have most of our code runs as a normal unprivileged user and that's all of your you know json decoding all of just the manipulating the hdp requests and responses and then eventually you have a small focus bit of code that does your privileged operations and the hope there is you've got a small bit of code so you can audit it more carefully and it's a smaller bit of code so there's just less likely to be bugs there all right so event originally the way open stack did this sort of thing was by sudo you run most of your open stack code as a normal user and when you need to do a particular privileged operation you would run sudo just like a normal admin would in a situation and so as you can imagine gradually over time the sudoers file that was required grew quite large now maintaining that was a problem so the next attempt was nova rootwrap or sorry rootwrap generally and we have one entry in sudoers that runs rootwrap and then rootwrap has a slightly more expressive filters files which allow you to say exactly what should be allowed what shouldn't be allowed and here's an example of the sudoers entry needed for nova rootwrap and then a couple of examples of the sorts of things you have in rootwrap filters files so there's run mount as root and that one runs cat on files parts that match that regex and the idea with rootwrap is you can add more filters that capture the particular checks you want to perform a problem with rootwrap is that it's a python process every command you want to run is now sudo a python process and then the thing I actually wanted to do and python has a fairly slow startup time so particularly for cases like neutron where the neutron router will be doing many many trivial operations to set up addresses and routing tables they're trivial ip root commands and it's doing many of them it's overhead of starting rootwrap again every time became significant so particularly for the neutron case but it's also useful elsewhere was rootwrap demon where you start the rootwrap demon once on first use and then it hangs around and you talk to it and it runs commands repeatedly through it but you're not paying the python startup cost again and again the nice thing about this is it's a very easy transition from the previous rootwrap case now the problems we have at this point are the rootwrap filters are not expressive enough they're acting at a very low level and you've lost a lot of the context okay i wanted to copy a file why was i copying the file where is i copying it from and to how do i know whether i should allow this particular copy operation so these three down the bottom here are actually from the nova default nova rootwrap filters and as you can see there's no information there about what should be allowed or not they're simply allowing cp as root with any arguments dd as root with any arguments chmod with root as any arguments and so unfortunately even though we've got all this framework in place i'm sure you can work out how to exploit those three commands so if you did manage to break into a nova compute server you know buffer overrun in the json decoding library or something unpleasant like that um you would then be running as the nova ap the nova compute user and you would be able to use these rootwrap filters to fairly easily escalate to root from that point so it's not actually giving a lot of defense the command lines are clumsy they're they're long and hard to generate and if you look at the neuter on ip lib file it's 800 lines or more of code that is just there to generate and parse the output of ip root commands and executing commands is slow so again the neutron example does many ip commands what they're doing is fundamentally in most cases a single net link call so it should be a very cheap kernel operation to just add that address or see what the addresses currently are we shouldn't have to pay the startup cost or running sudo and ip again and just to give an example this is some statistics kevin collected simple operations done by neutrons so attaching a router an existing router to a new subnet is 57 separate calls to the ip command now presumably some of those can be cleaned up but that's what's currently happening and then a problem that really came to the front with the liberty and mataka cycles uh was how do you keep the rootwrap filters in sync rootwrap filters are basically a duplicated copy of the commands somewhere in the python code is the logic to generate that command line and then in the rootwrap filters you basically have to have the same command line appearing to allow that command to be run by rootwrap and so keeping the code and the rootwrap filters in sync becomes a real problem particularly when the code is in a library like os brick or os vif and yet the filters are in the end project like nova so managing those two uh is problematic we don't have good processes or that we don't have good tools for that um so to address these problems uh we introduced printsep was the printsep library um the fundamental idea was to move from command lines to function python function calls let's move to something a bit more expressive a bit more uh matching what we actually want to do um it must be easy to use securely the rootwrap filters you could add a new rootwrap filter class but no one ever did which meant lots of people use the same uh regex match or usually just the same command match um and didn't go to the extra work required to say and i only want arguments that match these patterns or match this policy so a lot of them are just cp as any argument so as root and variations of that um and also because we're going through sudo all the time we can't use a lot of the new features so unix traditionally has had this all-or-nothing security model if you're root you can do everything if you're not root you can't do anything um and so new thing new features have been added uh linux capabilities se linux um setcomp uh have been added to linux to try to break up that root user into something more fine-grained we can't use any of those with rootwrap because sudo doesn't allow us to use those and uh anytime you try to set up something like that you're using sudo which is punching through and just going straight back to root uh full root powers so we want something that lets us take advantage of these new features so here's an idea of what printsep looks like from the developer's point of view um it's designed to be very simple if you want to add new functionality to it it is as easy as defining a function and putting a decorator on it um so on the left hand side we have the privileged what we'll run in the privileged portion and the on the right hand side is a regular unprivileged code and you can see it just looks like a function call on the right hand side and it just looks like a function declaration on the left hand side very very simple um and then once per project all the magic is in that decorator so once per project you do something like this which just declares that decorator um you don't need to worry about the details there basically that's just using the the actual oslo proofsep library to create a singleton object which you then use for your decorators um down the bottom here you'll notice there's it's setting the default capabilities so this is the linear capabilities all functions called through this particular one will be run with capsis admin capnet admin and no other privileges no other capabilities so even if it's running as user id zero it still can't for example arbitrarily change permissions on files it can't load new kernel modules um there's a whole lot of octals and things that cannot be done through this even though you're running as user id zero um and for most cases within open stack this is a good fit because they're a fairly single use you know neutron really just does network manipulation um sender really just does ice guzzy sort of you know file system volume operations mounting un-mounting um nov is a bit more complicated because it um has a bit of everything uh but even for the nova case we can have more than one of these contexts which have different privileges for each one all right so proofsep works by having a a it spins off a helper process and the helper process runs sort of like rude wrapped demon does as a persistent process with privileges and there's a simple unique socket between the unprivileged of the regular process and this helper process and then it's passing the function I want to run and the arguments the helper side verifies that that's okay yes that's declared properly yes it had the right decorator yes it was meant to run uh it runs the function returns the result or an exception object if if one of those happened but here cross the socket it's very very simple and sockets were used every every time um uh just to give you an idea uh avoiding the root wrap overhead uh sorry avoiding the exec overhead again and again this was about 20 times faster on my original benchmarks compared to root wrap demon uh since I've lost a little bit of speed since then in some um uh multithreading locking and stuff but I know where those problems are so we need to go and regain some of that speed um but that's the sorts of advantages that are available here uh so from the operator's point of view um when you look at the process table you'll see the extra helper process running and the helper process has to start has to get those root powers originally somehow and that can either happen through sudo or through root wrap you just need some way to start it initially uh and then in the oslo config of nova neutron whatever you have a new section which tells you what the helper program the helper command you're going to run is the user id and group id that should change to after it started and those default to root uh so there's sort of no change there and then the capabilities and that overrides those default capabilities that were set by the project so you will probably never need to change those unless you have a particular unless you know your particular deployment options don't need those defaults they need they can do it less than that um and the most paranoid setup you would run this as a user id that was different to the user id you were running your regular projectors and that also wasn't root um but that's that's if you're being extremely paranoid so current status it was merged into os brick um a week ago two weeks ago so it's in early in the newton cycle the development was done last cycle but they wanted to hold off so that's getting the most testing it can over this cycle before the next release uh now as vif similarly just went in and the neutron change is in active development right now if i've written most of it um and that converts all the rplib calls over to uh net link calls directly um this current status uh disruptive so do you have any questions about that feel free to chase me down during the conference uh at some point thank you now over to doug for the last small amount of time thanks trader yep he's waiting for a presenter okay i'll just talk okay um so we in addition to some of the features that we've been talking about in the earlier part of the presentation one of the benefits for projects adopting oslo libraries is that we also provide features that are not necessarily used at runtime but are used to do things like prepare documentation or prepare sample files and things like that so we've done a couple of different things um in the past two cycles related to that um in liberty i think it was we completely rewrote the configuration generator so uh projects that use oslo config have the ability to generate sample configuration files with all of the configuration options organized into their sections with help text and uh default values and things like that we rewrote the way that project works so that um we uh basically to make it more robust and to make it so that the generated output is uh a little bit easier to manage um so now you actually have the ability to for example generate a different sample configuration file for each uh demon that you produce within your project so uh nova has three or four or five i don't know how many different demons they're running these days um each one could have a separate configuration sample file for only the options that actually apply to that thing as part of that um oh you got it okay i'll run it okay um so as part of that uh the the output itself is the same format but the inputs are a little bit different and you can do a little bit more with it um so next um next um we also extended that so that in addition to generating a sample configuration file we can generate reference documentation for the options in a little bit more easy to read format so you get a nice table i mean flip over to the next uh one more yeah nope sorry this this is the integration to actually embed the the config file in the documentation keep going we don't have time to cover all of this okay so you get a table you get all your configuration options they're organized by the groups where they would need to appear in the configuration file um you get uh bold text and things like that which you can't really do in a plain text file um this is sort of proof of concept work right now at this point so we've integrated this into a few of the different uh projects developers documentation and we're going to be working with the uh documentation team this cycle to see how we can use it for their configuration manual so that we can pull all of the configuration options in automatically as they get changed in different projects right um one of the other things that we're doing uh most of the projects that have adopted uh stevedor for loading plugins for things like drivers will be able to take advantage of automatically generating reference documentation for those drivers as well so you can put a little bit of documentation about the driver right there in the code and then integrated with sphinx which we use for building all of our uh developer documentation and the documentation team is actually adopted and now too for the manuals um you can extract all of that information from the code and it formats it nicely so you get a nice list of what the drivers are maybe some tips about which configuration options you need to go look at in order to enable them or use them uh correctly or optimally and that sort of thing all right so things that we're going to be working on this cycle i mentioned that the config generator and integration with sphinx was proof of concept so we're going to work on refining that a little bit um right now it takes the same inputs that the sample file generator does but that means that you might have less control over which options actually show up on a page and when you're building reference material you typically want to have a little bit of exposition and then some options and some more exposition and that sort of thing um and then could you go back one yeah um and then uh i mentioned that we were going to be working with the documentation team as well so i think that's pretty much it and i think that probably we're out of time so there's reference information about using all of these things in the documentation for the libraries where they live um oh and we're probably also going to be looking at generating sample policy files as well uh with the work that's going on in the policy library right thanks a lot thank you we are out of time so you can just ask questions if you want thank you