 Thank you everybody for joining us for another OpenShift Commons briefing in the All Things Data series. Today we have John Cope and Jeff Vance from the Red Hat CTO office and they are the key drivers for object bucket provisioning in Kubernetes. So we're very excited to have you here. Please take it away. Hi my name is John Cope and along with my colleague Jeff Vance we've been working on object bucket provisioning for quite a while now and it started off as a small project and has grown into a much bigger community supported project that we're going to be talking about here today. So this initially started off as a in-house solution to the lack of an object storage API in Kubernetes. At the time there wasn't and there still isn't an API that allows a normal interaction with object storage for provisioning of buckets for any vendors. And as a result managers often have to go out of band or out of Kubernetes to manage their bucket quotas their storage quotas and to manage cost. They have to handle user policy out of band attaching users and roles to buckets through the vendor's interface. It adds an extra layer of complexity to the system and it creates this inconsistent workflow for users who have to figure out on their own how to inject connection and credential information into their pods or their workloads. This creates, we feel, some security holes when a user or an administrator has to go outside of Kubernetes, get their credentials and then inject them into the workload manually. And of course this at scale becomes incredibly difficult to manage. And I just wanted to mention because I just remembered it now. People do use service catalog or did for provisioning buckets, but A, that's been deprecated and B, it was a fairly cumbersome method. There's a lot of moving pieces in service catalog and so it wasn't very popular. Okay, Jeff. Yeah, that's a good point, Jeff. Thanks. Admin user stories that we had imagined as a guide for our target design implementation here were to be it to allow administrators to control resources, that is storage resources, bucket size, the amount of buckets, the amount of users who can connect to a specific bucket and who those users are in order to control costs and represent their security policies at the Kubernetes layer. And we also want to normalize the interactions that administrators and users would have with object store vendors through Kubernetes. From the user perspective, we want to simplify the onus on them to describe their workloads in their YAML specifications. And just as they can do a PVC or a persistent volume definition in a YAML along with their pod, we want to make that possible for bucket provisioning, so that they can say, don't start my workload until this bucket is ready, create this bucket, and then when it's ready, give me the connection information and inject it into the workload in this way. So we want to automate the entire process and keep them from having to go out of band, either through the CLI or a web GUI to create their bucket and get their credential. And we also want to make it possible to some extent to make these workload specifications portable, assuming that the backing object store vendor supports the same protocol. So if they're moving from, say, AWS S3 to an on-prem cluster running CIF, their workload should still run without any modifications. So to touch on some of the goals of the project, we want to create a control plane within Kubernetes and OpenShift for buckets. I call it control here because what we, I want to explicitly say we're not replacing or abstracting the actual object store vendor protocols, that is S3, GCS, Azure Blob. So users will still be using those SDKs. We want to normalize the experience of provisioning the buckets. We want to provide vendors a easy onboarding experience when they're writing their provisioners. So this will be much like CSI, written against the GRPC interface with a handful of methods that a vendor would write and abstract away all of the Kubernetes controller framework so that provisional authors don't have to be Kubernetes experts when they're writing these provisioners. And to touch on non-goals as well, we do not want to define, as I said, a native object store protocol. That is something we're hoping to enable with this project and it's something that has been talked about as a stretch goal following this project, but currently that's not something we're trying to solve. We're not trying to make it possible to lean on the first point to shift the workloads between object store protocols. So if you're shifting from Azure to AWS S3 and you're using the Azure SDK, we're not trying to make it possible to magically drop that into AWS to make that work. That's not something that this project is trying to solve. And lastly, we're not trying to orchestrate the infrastructure of object stores. That is the spinning up of actual physical storage and the deploying of, say, SEF object onto nodes with storage. I'd recommend if you're looking for something like that, checking out Rook.io. That does a very good job of orchestrating that kind of stuff and they do have support for SEF. Some of the history of this project, so as I said earlier, this start off as a kind of a small in-house solution has grown since. As a result of its origins, we opted for a library that would be imported into projects and that library would wrap the Kubernetes controller logic and we would ask vendors just to write to an interface that we had defined. This had some pretty significant limitations that weren't a problem for our in-house solutions. But as we started to see a bigger community interest in this, the broader Kubernetes community, we started to see some of the real limitations here. Namely, one, every time a library update occurred, so a bug was fixed or a feature was added, those provisioners had to be rebuilt because it was a dependency, a code dependency of those provisioners. The design itself used config maps and secrets to represent a single bucket and at scale, that becomes pretty unmanageable because you've doubled the amount of objects you have to manage per bucket. It was Kubernetes specific, whereas a CSI-like model with a GRPC interface can actually be implemented for any container orchestration platform. Of course, as a result of being written in Go, it required that anyone that dependent on it write their provisioners in Go, which again wasn't a big issue for us. At the time, we're only writing two projects that already had or were already based in Go. Of course, with Go, especially in older Go, dependency management was a real problem prior to Go module projects. We ended up spending a lot more time and increasing amount of time deconflicting dependencies. Lastly, we didn't have support for it at the Kubernetes community at the time. Coming to today with the enhancement proposal, we've got a pull request open for against Kubernetes. It's led by Jeff and myself and we have support for it within SixTorch and are working closely with some of the chairs on SixTorch to really get this moving through. It proposes a native Kubernetes community maintained API. What that means is that the conversation has been around making this an API that is not external to the core Kubernetes API. It would be party core. It would be referenceable from within, say, pods and deployments and things like that. The benefit of that would be pretty large. For one, it would significantly cut down the amount of extra API objects required to manage each bucket. We would have a single bucket object per bucket rather than a secret and config map. That bucket could be referenced by a pod. Some of these I've already mentioned before. We separated Kubernetes orchestration from the provisioner code via this gRPC interface. Provisioners could write their or provisioner authors, rather, could write the provisioners in any language that's supported by gRPC, which there's a list of 20 or more. They're not tied to go. They don't have to learn to go to write a provisioner. This significantly simplifies upgrades. The cozy sidecar, just like in CSI, would run in its own container. Whatever an update, a bug fix, a feature would come out for it. It wouldn't be a manner of having to reinjust it, de-conflict dependencies, rebuild your provisioner, and then deploy that. It would just be updating your sidecar container version, and you're good to go. Probably most importantly, as I had said already, we have support within SixTorch for this. They're pretty significant support. In fact, we had been invited to talk about this at last year's KubeCon. We've gotten a lot of support from the technical chairs on this, and are holding weekly meetings to move this along and prove on the API so we can get the cap merged, and begin designing the lower-level implementations. To dive into the design itself a little bit, we're looking at introducing three new API objects, a bucket, bucket content, and bucket class. The bucket is similar to a PVC. It's a user-created object. It triggers the provisioning of a new bucket by the sidecar and the provisioner authored driver. The bucket content is analogous to a persistent volume. It represents a cluster-scoped administrative sign of the bucket containing bucket metadata that you don't necessarily want users to be privy to. Bucket classes, very similar to storage classes, provide a object store tailored API to represent a set of parameters that users would reference from their bucket object. As an MVP, we want to support three separate cases for buckets, Greenfield being the creation of new buckets on demand by users, so those are brand new empty buckets. Brownfield, we want to automate the credentialing of users for buckets that were created out of band, such that when they create a bucket object, they still get back a set of new credentials tied to that bucket. Of course, Static, the case being that a driver has not yet been written for that object store or rather a provisioner does not exist for that object store, and administrators would have to manually create the bucket content objects, but Cozy would still handle automation such that when a bucket is created, a set of credentials, which would be defined by the administrator in secret, would still be copied to that user's namespace and cleaned up when they delete their bucket object. What we see this doing for our customers is enabling a couple of use cases at least. Hybrid cloud being a big push for the red hat right now with these portable APIs, it would make it a lot easier for users to pick their workloads up out of one cloud provider and drop them in another, assuming of course that the protocol, the object store protocol remains the same. So the earlier example I gave was picking up from AWS and dropping it on a prem to a SF object. That should be a seamless action that doesn't require any changes to the spec. There has been some talking around backup and disaster recovery operations and object storage really enables orchestration of periodic backup and disaster recovery stuff. Namely, the fact that it provides versioned immutable data storage makes it a real prime solution to store the backup information. By normalizing the object storage provisioning, we enhance those efforts and make them a lot more portable across cloud providers. And lastly, it provides a nice streamlined way for applications that are very reliant on object storage now, namely AIML and serverless apps to be written and to be deployed on red hat products or on cloud providers without having to become an expert in that object storage. And I just want to say thank you for listening to the webinar. Again, my name is John Cope. My colleague is Jeff Vance. We would love for you to reach out to us if you have any questions or comments if you have user stories that you think we should know about. We'd love to hear from you. We hold weekly meetings on the review that are public. If you join the Kubernetes 6 storage Google group, you'll automatically get invites for those. And please feel free to jump over on the actual poll request and take a look at the document itself. We'd love to get some feedback, some criticisms. If there's something we missed and we need to know about, we'd love to hear that too. Thank you. Thank you, John. And Jeff, does anybody else have any other questions? I see that Rob has been putting into the chat some questions around how infrastructure-specific APIs are merged with Kubernetes. Jeff, Rob, do you want to take this conversation out of the chat and talk about it live? Yes, Karina, that's a good idea because I don't really understand what he means by an infrastructure API. There is a question regarding these specific APIs. Each infrastructure specifically, let's say AWS, Azure, or that's what you mean. Yeah, they have the specific APIs with the Kubernetes integrated. And I don't understand how this is actually integrated with the open shift. So like John mentioned, and you're right, Google Cloud, Microsoft Cloud, Amazon Cloud, they all have a different SDK and therefore different APIs that you use. If you were outside of Kubernetes and you're trying to create buckets, list buckets, add to buckets, that would delete them. And they're not the same. That is the problem with buckets. There's no POSIX. There's no IEEE standard for buckets. And that's why Kubernetes hasn't addressed it yet. That's what John mentioned about the data plane or those protocols. We are not standardizing that in this enhancement here. There is discussion, very light discussion in that area, but nothing solid yet. What we're trying to do is manage the, what we call the control plane that you can use Qt control to create resources that will be a request for a bucket. And that there is abstraction of buckets and that there's a single consistent way of managing buckets through Kubernetes. Quotas could be applied and the high level bucket is a first class Kubernetes resource now. So that's what we're trying to do. But because there's not a consistent data plane, you can't take an application written using AWS SDK for your buckets. Because that app knows it's using the AWS bucket API. You can't take that app and just stick it on Google Cloud and expect it to work just because the control plane is the same. In other words, you create this claim, this request for a bucket with, which is the bucket CRD that John mentioned. That part's the same. But underneath, you'd be referencing a different bucket class that would have a different driver via Google driver, not an Amazon driver and so forth. So that app can't be portable across different bucket stores. Does that explain it better? Yeah, it does explain better, but I still don't understand how, so you're saying we're putting those drivers from each infrastructure specific and merging with the AWS? No, I don't believe so. So what we're describing in the same vein as CSI would be that AWS or someone supporting AWS would write their AWS driver that would be likely stored within a six storage managed repository or project, but maintained by the authors of that driver and perhaps other people within six storage. The drivers themselves would not be part of core Kubernetes code. The knowledge about the infrastructure would be abstracted through the bucket content and the bucket class, such that, and this is a, as Jeff pointed out, we can't lean on a universal interface like POSIX to have a more specifically defined bucket content API. So we offer a bit of leeway with data blobs that administrators will still have to define and drivers will have to ingest. We, as part of the design, expect drivers to very clearly document the parameters they require to operate or allow to be configured by administrators and then they will use that to operate on whatever vendor or object store they're written for. Thank you. Yes, thank you very much. That was a great discussion. We've had some more discussion in the chat. Erin, do you want to jump in really fast and mention why this is replacing the object bucket claim? Sure. So the object bucket claim, which we introduced last year, which I think a lot of headers are using, was based on lib bucket provisioner. So using the idea of external provisioners to create that storage. And since then, the community has wanted to pivot more towards a CSI implementation because that's how all data will be accessed within the Kubernetes infrastructure. And with that, the community wanted to go with a different naming convention. There was a pretty strong opinion about that, given that if they could go back in time, they would have changed the PVPVC nomenclature. So hence, that's why the name is different. So the object bucket claim and bucket was used before. These are, it's going to be somewhat of the same, but not as tracking as closely to the PVPVC, which is unfortunate in our case. I mean, we liked being able to relate that as an admin understanding. That's the way you consume other types of storage, but we'll take what we can get. I hope that helps clarify it. Yeah, to add what Erin said, I mean, there's a chat conversation on this. You know, the library design had an OVC, which was an object bucket claim. That's what you see in OpenShift documentation and Rook documentation. And people are familiar with that because it sounds like a PVC. But when you really think about it, it's fairly different. When you have a pod, your pod will represent, your pod references a claim that represents a PVC. And then that PVC, in turn, will reference a storage class. And that's how you get to the external provision or the driver or the dynamic provisioning of your underneath storage. But the important part is the pod referenced the claim. And we don't have that with an OVC. The pod, you have an OVC and it's separate from the pod. There's no place in the pod where it references that OVC. So the analogy falls short with the library design. And the SIG storage, and most importantly, as Erin said, SIG storage would not, they did not like that naming. In fact, they said that if they were starting all over, they wouldn't have called it a PVC. They don't think it, plain is the right word. You don't call it a pod claim because you want to abstract a workload. And so they, there was fairly strong encouragement from SIG storage for us to follow the naming conventions of the new snapshot feature. And that's what we've done in this, in this cap. Thank you. I mean, that's really interesting. I'm so glad I'm here to listen to this. And it looks like others are as well, going from the chat. So thank you all for joining us and, you know, giving us an update on object bucket provisioning and all the great work that you're doing upstream. With that quick announcement for everybody, we'll have another session next week, same time for the All Things Data series. And thank you everybody for joining us again.