 Okay. Hi. This is the Glance project review and update. I'm Brian Rossmaid. I'm the current PTL of Glance. Hi, I'm Nikhil. I am the former PTL of Glance, and I'm co-presenting Brian with this bike onwards project updates as well as some context so far. Okay. So take it away. Right. So what is Glance? What is Glance about? It's a code name for the images service within the open stack realm. So if you have been following project updates until today, you might have observed some changes within the Glance mission statement and so forth. Not since Okada. Not since Okada. Not that recently, but it's changed a little bit. It's changed a little bit, and what we have settled down for matured into a long-term vision for the project now is providing services and associated libraries like the Glance Store as well as the client to be able to store, browse, distribute, share, and manage bootable disk images as well as associated data and metadata that's not exactly metadata but tags and other associated information. And this is most closely related to computing resources, but it could be potentially metadata definitions associated with non-computing resources that are neither NOVA or Cinder or Ironic. So that's Glance and Adjust. Going forward. So just giving a little bit context on what we have done so far. So Glance is a pretty old project we started in Bexar and six years down the line we have a great adoption. Lots of clouds use it. I personally know some of the deployments that are using Glance Standalone. So it's a project used just for storing virtual machine images and maintaining the image metadata. But a very good use case would be using with NOVA and Ironic and storing the snapshot as well as bootable image data. So the good news about contributions to Glance is we get a lot of good number of contributors and we had a good count of 55 for Ocata which was approximately the same for the Ocata releases and we are 34 and counting still for Pike. So yay. Thank you. All right. So I want to give the bad news about contributors. That's a picture from the PTG. So as you may know the summits were split into the project team gatherings where design works being done and then what we're doing here is to try to gather feedback and understand what the use cases are so we can go back and code stuff. But the bad news is that we lost our team diverse affiliation tag in December 2016. So the team diverse affiliation tag is something that's put on by the technical committee to indicate the maturity of a project and the idea is that a more mature product is going to have a more diverse team backing it. I don't know if that's true but the idea is that it's better to have more diversity than less because it's a measure of project health. The Glantz core reviewer team was really severely impacted by the OpenStack Innovation Center closing last month. So many people pictured there are not currently working on Glantz right now including me. So I'm unaffiliated so if anyone needs a good Glantz developer please feel free to contact me at your earliest convenience. So just something about the contributors. As you probably know or may not, each proposed change requires two approvals before it's merged into the code repository. So it's two approvals by members of the core reviewer team. So that means that a lot of people have to be looking at the code. You know it's a kind of standardly accepted thing in software engineering that it takes longer to read and understand code than it does to write it. So just pointing out that the reviewing is an important part of OpenStack and so in your capacity as some of you are operators, some of you are perhaps managers of development teams, it takes time not just for your contributors to write their code but they need to be reviewing stuff also and that's what makes the community go. And a lot of reviewing happens which is what contributes to the stability of the code. So it's an important part of the process. So just in functioning and trying to allocate resources it's important to keep in mind the reviewing bandwidth as well as just regular contributions. So it's sort of hinted at this about us losing the diverse affiliation tag. One of the key important aspects of diversity is that a budget cut at one company isn't going to doom a project. So luckily Lance isn't doomed right now. We do have some people working on it but it's kind of close and there are other OpenStack projects that have been pretty much wiped out when a company decided to unfund it. Another key thing is diverse product viewpoints are really needed to help drive feature development and to keep features from being too narrowly focused to one particular use case. So we're trying to build a big project that's usable and consumable by a lot of different people. So it's really helpful to have a variety of viewpoints as we're focusing on features. And the other thing is if you have developers who don't all report to the same boss it's easy for them to be critical of design decisions or feature things that might not quite make sense in the big picture. So that's another reason for team diversity. Okay. So okay. All right open source software. All right so we need your help so as you can tell we've lost some people but people in this room can contribute too. So if you're a developer you can write contribute code everybody knows that. But there are a lot of other contributions that we need to keep the project healthy that you don't necessarily have to be a developer to do. So for instance with the glance specs these are the design specifications for features that are being implemented. So it's really helpful for people to review those particularly people who use consume the software and have an idea of what the feature might do or have an idea of what their use cases are because it can help make the specs a lot better. So we really like more non-developers to take a look. Then documentation is another thing where the documentations stored mostly in the code repository. As you're reading through the documentation some of it's really good and some of it doesn't. It would be nice to sort of raise the standard. So if you come across typos or you come across paragraph that doesn't make a lot of sense or is confusing, you should feel free to put up a patch. And even if you're not you don't have to be a hundred percent just like with the code. When people put in code believe me it's pointless to try to get it perfect and then submit your patch because no matter what you do somebody's going to have some kind of comment and ask you to put up a copy code. It just means that you have a better attitude toward people making helpful suggestions. So the same thing with the documentation. If you see a paragraph that's a little ambiguous or it's not quite clear, you don't have to fix it perfectly. It's enough to bring it to our attention by putting up a patch that rewrites it. And then several other people will look at it. The documentation goes through the same process. You need two core reviewers to approve it. So people will help you put up a patch. And then there are also glance user stories. So this is kind of an underused thing I think. The product that I'm going to be talking about today is a lot of people focus on you're getting minus twos or minus ones and that makes you feel bad. But the way you think about it is people are coming up with specific suggestions to help you improve whatever it is you've submitted. So that goes for the documentation too. So I'd really like to go into the underused thing I think. The product working group has a repository of user stories for particular use cases that aren't features yet, but that could be features. And what's helpful about that is that you have not just one or two product people, but a whole bunch of product people at different companies who are reviewing these things and clarifying the user stories. And that's really helpful when there's something, so for instance we were talking about the image lifecycle management. So there's a lifecycle of images. You introduce a public image, you want your users to consume it, but then some security patches come out so you want to sort of, you don't want to delete that image because people might still need it, but you sort of want to hide it away and encourage people to use a different image. And it would be nice if there was an automatic way to manage that. And it would be really nice if we had a clear description of where you can contribute. And it's just the same kind of thing. The user story is in the product working group repository. You just submit a patch and it gets reviewed and then gets merged, just the same as code. And then finally for developers in the room, we do have some specs that have been approved that are ready to go, but don't have anybody working on them yet. So you can browse the specs repository and feel free to pick them up. Either email us or ping us. If you have questions and you can get working on that. So there's plenty of stuff to do. So just a little bit more on what we need help with. So we started this initiative of collaborating more with the operators and getting feedback on what the real-world use cases look like, what the real-time pain points look like, and we have been getting some really good feedback through the virtual sessions and on-demand operator syncs and sessions like those. One of those sessions is the good initiative by the summit this time is a forum discussion which is a sort of a fish ball room. If you have been to a previous summit, it's like a big room where lots of collaboration, big collaboration could happen. So we have right after this one and we would love to get some feedback there. It's meeting room 103 and at 5.30 downstairs. So it's immediately following this session? Yeah. Okay, so what you're here for is the project update. All right, so this is what we're doing. So as you know, Okada was released back in February. We're working on Pyke now. So I want to tell you what happened to Okada, some things to look for if you haven't deployed it yet. Feel free to give us feedback if you have deployed it and have something to say. So part of the reason why the summit was moved to this time instead of being at the same time as the release was people would have a chance to maybe deploy the release and give us some feedback on what to work on. So some things to look for. Key thing is image visibility changes. So prior to Okada there were two visibilities for images and glance. There was public and private. So the public images the ability to make an image public is protected by policy it's restricted by default to administrators. The idea of a public image is that it's one that the cloud provider makes available to users to boot their VMs from. So it's supposed to have been tested it's supposed to be a quality image and possibly and well most likely. The cloud provider provides support for that. So if there's a problem with the image you can complain and get that addressed. So in addition to that there was the ability to share images in the V2 API that's been since I think Folsom maybe. Yeah. So Folsom or Grizzly you've been able to share images in the V2 API. But the visibility was always private. So this posed kind of a problem because as you're looking at the visibility of an image private you don't know if it's really private that nobody else other than you can see it or whether it's been shared and other people can see it too. So that was the reason why we introduced the shared visibility and then there was the man for a kind of so image sharing glance you need to know who you want to share the image with because you have to provide an identifier for that person. So that's not a big deal if you want to keep like a customer list or something if you're trying to run an image marketplace but if you're an open source project and you just want to make an image available for people to use of some cool operating system or something it's kind of a hassle to find out all these people and then constantly share the image with them. So with a community image you can make the image available to everyone in the cloud. Now that's also protected by policy because right it could be abused but the other interesting thing about community images is that they don't show up in the default image list. So it's basically anti-spam provision. So if you want to find community images you have to discover them by making an appropriate API call. So all that stuff in the documentation a quick introduction to it in the API reference that's listed here if you want to take a look and then associated with the visibility changes or the actual community image sharing themselves. So it's protected like I said it's protected by policy just like publicized image. So the default is that it's on so that any end user can create a community image. So be aware of that if you're a deployer so that you can change that if you don't like that setting. So like I said it was a feature that had been asked for and we delivered. And the other big thing that happened in Okada was we made some things, so the first two are things that you can see in either the image response or the way you deal with the API. This last thing that we did was we made some changes to the API. So we've changed the way database migrations happen in glance. So previously we were using a technology called Sequel Alchemy Migrate and now we're using a technology called LEMBEC. And the reason we've made the change is it's going to allow us to do zero downtime database upgrades, which is nice for you. Now in the Okada release we've made the change to LEMBEC. We've rewritten the database migration scripts, but the zero downtime migrations are not supported, they're experimental because we don't, well what we're working on in Pyke is to get real time testing in the gate to make sure the upgrades work seamlessly. But we're pretty sure it works so encourage people to try it out. That's a link to the glass documentation that explains the database management. I guess the only thing that's visible not to users but to operators is that the identifiers for each migration has changed. So previously they were just a numeral, and now it's a combination of the release name and a numeral but that's all explained in the documentation. So please take a look. And like I was saying earlier if you find something ambiguous or something unclear, please put up a patch or let us know in some other way that there's something we could fix. Oh, this is me too. Okay, so I was going to get the good stuff we're going to be doing in Pyke. Well I just want to bring your attention to a couple of recent OSSNs that Glance has been involved in. So an OSSN is an open stack security note. So they always have really horrifying sounding titles but aren't always quite so bad. So 0065 users of Glance may be able to replace active image data. So just so that nobody freaks out, right? This is in a non-default configuration where show multiple locations is enabled. So it's not the default. But if you do use the show multiple location setting you should take a careful look at this OSSN. Now I do want to stress the check sum of the image can't be changed. So even if someone is able to use the exploit which is described in the OSSN, so you can read it there. Even if they swap out the data the check sum cannot be changed. So a consumer who's actually checking the check sum would detect it and know that there was a problem with the image. But consumers who aren't so careful could be fooled possibly. So anyway, so just want to bring that to your attention. And it's got a discussion and it's, yeah, anyway it's worth reading the security note just to be up on top of things. And the other one is we haven't, well, we're addressing this one in Pyke. So I do want to bring it to your attention as something that you should pay attention to. Basically there, so when you do an image create in Glance you can specify a UUID. And what will happen is that Glance will apply that UUID if it's not currently in use. But if it is being used it will reject the request and you have to either supply a new one or you just do probably most people just do the default image create and let Glance assign you a random UUID. But since you have the ability to do that, you can write here's how the attack would work. You would observe the set of public images that are some deployer has and note their UUIDs. And as I was saying earlier, there's an image life cycle with most clouds where somebody takes an image out of circulation. So if it's taken out of circulation by somebody deleting it, you still can't reuse that UUID because the way Glance works is we do soft deletes. So even though the image is deleted its UUID is still present in the images table and so since there's a constraint the database is going to reject it if you try to reuse it. But as people are using clouds images are created, images are deleted, tables get bigger and bigger and eventually people want to purge the database. And in fact Glance provides a tool called Glance Manage that allows you to conveniently purge the database. So what I'm saying is don't use that tool unless you know exactly what your situation is because if you do purge the database then those soft deleted UUIDs disappear and it's possible to reuse a UUID. So in order to make this attack actually happen, several things have to be the case and the attacker would have to be in a position to flood the database with a bunch of images and then delete them to the extent that you would be forced to purge the database. And you'd have to do that in a situation where you didn't notice that this dude was creating all these images and deleting them also. So anyway it's an actual attack it could happen not, I guess I shouldn't say it's not very likely because now that everybody knows how to do it you could give it a shot. But anyway the main point is don't use Glance Manage what we're going to do is fix this in Pyke and our proposal is this and I want to float this out there because we've thought of several ways to do this. I mean the whole point of having a database purge is that you eventually want to compress these tables because they'll just get bigger and bigger but at the same time we need to track all the UUIDs and so you can see these are competing constraints. So actually Erno came up with this idea so I should say this is Erno one of our Glance Core reviewers. What we're going to do is not modify the Glance Purge so that in the main images table any of the images that were public when they were deleted will not have their UUID removed. So they'll remain as soft deleted images and then that way it'll be impossible for somebody to reuse the UUID of a public image. We're thinking that'll be good because the percentage of public images is small relative to the total number of images in a cloud I believe that's my hypothesis. Oh, okay. Then private images. Okay. So that's a good data point. So anyway, what I'm going to do is are you on the operators list? Excellent. So what I'm going to do is we're going to propose this as a spec light in Glance and explain this in a little more detail and then I'm going to put a point I'm mentioning here. I'm going to put out a note to the operators list so please make sure you look at the spec and then give your comments because there are a lot of people that trim the database but won't expose them to this. Another thing we had let me just throw this out as long as we've got a few minutes. Another thing we had thought about was introducing a policy to restrict who can specify a UUID when an image is created because now it's available to anybody. So we didn't like that one quite so much because there are some use cases that I want to have the same UID but somebody who wanted to do that could loosen up the policy. But anyway, so let us know and we can have the discussion on the mailing list and on the patch for this because it's like I said I don't want to overemphasize this because I don't think it's a big problem at the moment but it could become a problem and we need to deal with it and we want to deal with it in a solid way. So thank you. You were going to talk about Pike, I think. Sure. I can just mention some of the developments that we have initiated in Pike and this one of the features image import refactor which has been long discussed across projects across with Glantz, DC, Nova and lots of different considerations in mind and we have come up with significantly detailed yet yet a good plan about refactoring the Glantz API for which is called UploadNow but a more sophisticated way of uploading image data so that it can actually be an import into your cloud. So if you wanted to support the hybrid cloud use case or if you wanted to use public cloud with different vendors which are not even open-stack you would need a sophisticated back-end mechanism to morph your image in a way that the cloud supports and the operator would be able to support you accordingly. So that's a short background on what the feature is and we have tried to prioritize this and reprioritize this over the cycles and finally the stars have called us to get that MVP done in Pyke and thanks to a good number of developers behind us we have a few patches up and then we are planning to implement that MVP API for uploading and activating the images not all the features that the import would support would be implemented in Pyke but you would get a flavor of what the new API would look like and this would give us some opportunity to basically get some real-time feedback with the operators in this cycle as well as to prioritize this in the next cycle as well. Yeah, the main difference between import and just the image upload is that the import will give the operator a chance to examine the image run tests, do whatever you want basically before it actually becomes active and is consumable by people and it's anyway you can read the spec and print it out it's probably 120 pages I'm not kidding it's got several FAQs because it got so long so we describe in great detail what the use cases are and how to do it you may have heard possibly even me speak about glance tasks so we had a first try at this six cycles ago maybe but the problem was that the API we came up with was extremely flexible which was great from the operator point was great from the point of view of some people and not great from the point of view of complete interoperability between clouds basically so that's why it's kind of complicated in one sense it's a simple thing that we want to accomplish in the other it's kind of complicated because what we've tried to do is take into account the competing demands of public clouds and private clouds of various sizes but come up with a unified API so that you'd be able to use this no matter what cloud you contacted even though the cloud didn't support all possible import methods so I think we've well we'll see if we've actually right that's I'm sorry that's the other the other key thing was it has to be discoverable so the other the other main problem with the task based import was that there's no way to discover really what the calls were except to read the documentation which apparently nobody likes to do so this way there'll be a call you can make that gives you the exactly what it is that you have to supply in order to successfully import there's a call that because for each of the clouds right doesn't make sense to allow somebody to import a disk that's too big to fit on any of your VMs right depending on what your configuration is so there and individual clouds may support some formats and not support others so we've taken all that into accounts all that information will be discoverable and then you'll be able to import stuff so I'm just I don't know I feel like I have to justify because it's like hey you just import you know just upload the image but it's you know a little bit more complicated anyway yeah I guess something for you in pike so I guess like a point to add that is we needed to have a feature parity for an important feature in version one of the API that's copy from URL and we don't support that in version two by default so this new API would with the discoverability would give you give the operators and the users leverage to identify if a particular site supports copy from if it's saved for them to support copy from and you know if if your dashboard or you know if your client needs to use that then that that feature would be there yeah and by the way I'm sorry to interrupt you with the OSSN there's an OSSN that's out about copy from so you might want to take a look at it just if you've exposed copy from to your customers there's some things you should take into account as you depends on how your clouds configured right like everything does so for a lot of people it's not a problem at all but for some people it might be a vulnerability so please keep an eye on the OSSNs this feature is a pretty detailed one and it's like a really giant beast so as an operator I would really recommend you guys to keep an eye out when this implemented and you'll see that release notice highlighted for your convenience as well okay you can talk about two other things we're trying to accomplish the community goals for Pyke so we I mean ordinarily we probably wouldn't put this up here but since we're having personal problems with Glantz we're sort of pulling back on what we can accomplish so the community goals were Python 3.5 support so again end users aren't going to see anything from this except that right it'll be nice for operators that you can actually use Glantz if you're running Python 3.5 it won't have to run 2.7 anymore and then the control plane and API endpoint deployment via whiskey we're just going to make sure that Glantz can be run as a whiskey application and the idea behind that is just that there are some servers that are very much optimized for this kind of thing so we might as well some people want to take advantage of that so it looked like for a while it wasn't going to go but Matt Trenish isn't here is he but we have a volunteer to pick this up so I think we're actually going to be able to get it done in Pyke too so something to look forward to okay alright and just the release themes for Pyke so scalability and resiliency I mean the feedback we've been getting is that Glantz is operating pretty well so we don't want to mess with it too much so it's not that we don't care about that stuff it's just focused right now so with interoperability that's the image import gives us a major focus there and then security is associated with image import also because it gives you a way to screen images before you let them loose and then with modularity since we're refactoring the image import process and then manageability I guess it's part of it too yeah I guess it's like a little bit of overlap with rolling upgrades to be able to manage images as well rolling upgrades will help with that can I get a comment on that sure I would kind of challenge that not a focus can I comment on that so thank you for pointing that out I was just about to say which I forgot that image import refactor that we're planning to implement it is user experience as an end goal but we want to do an MVP so that people so that it can be experimented so we won't be able to provide that by default but it would be still available for consumption and then we can evaluate it so that's why it's not a focus right away but yeah for sure in the future cycles that's show up purple eventually so in Queens what we're looking at is resiliency so we'll have the MVP of the image import in Pike and then continue to work on in Queens to make it better so same thing with manageability right completion of the rolling upgrades work modularity just will be doing more refactoring to make this stuff more solid interoperability will drop to a minor focus because we're hoping most of the user facing stuff will be done by then and then security is we never want to say security is not a focus so we'll be keeping an eye on that too as we're going through and okay and then in Queens so we're asked to project into Queens so this is our thoughts will be some further improvements to image import so that's not surprise and then we'll continue with the enhancements to rolling upgrades the major things we want to accomplish depending on the personal yes depending on right so if you want to make sure this stuff happens please recruit developers to help out glance and then with the themes for Rocky we're figuring by then you know just general growth and clouds will catch up with glance and so we better address scalability a bit but we're also we also have some ideas about image caching so but you know projecting out this far even though Rocky doesn't seem that far away is right depends on how things go right because if people are using more boot from volume stuff then maybe caching images is not such a big priority we'll see but right now it seems like there is some work that could be done on the glance image cache and then what do we have modularity would be a major focus because we'll be factoring stuff that we've done and then we've got user experience showing up in Rocky and I'm trying to remember yeah I mean we're asked to rank all these things and so to a certain extent I guess like the anticipation is like sorry to interrupt but I guess the anticipation is that imagine port would be mature enough that we would be able to provide a final product by Rocky and then say oh that's there you go okay and then one thing that's not listed here but I know is a concern from I do the we've been doing these operator surveys some of you may have seen them and maybe even filled them out somebody at the end we always ask like an open question any other comments you have and somebody every time has been putting you really need quotas in glance you need better quota handling in glance so that is so you may be aware there's a general quotas effort underway in open stack that was approved recently and I just want to yes so anyway so we will be joining the club of having good quotas but it's a matter of right person power as to when that's going to happen so it wasn't listed as a particular thing but I do think that's going to be something that's sort of in the atmosphere that we'll want to pull in at some point and it says that about time yeah okay and then we were I wasn't supposed to okay that says finish time is up so time is up so here are some questions but if you go to the next presentation you can actually do that so if you go to room 103 so just go downstairs how do you use glance we're asking that in two ways like how do you use glance but also how do you use glance so we want to get some feedback from operators about what you're doing and I'll explain some of the survey results we've had but there will also be some glance people available if you have particular questions about how you want to do something so thank you very much sorry we're like running over but thanks for attending anyway ping us in IRC or on you could tweet at us if you really feel like it or just send something in the mailing list if you have a question about glance or anything thank you