 Hello everyone, so let's give people a couple minutes to join here. If you're already here, feel free to add your name already to the list to the agenda. Yeah, should I am notes here. All right, yes, five minutes past the hour. So let's get started here. First agenda right and there's always is introduction to newcomer so this is obviously something that has totally voluntary but if somebody is new here and just wants to see how to improve for free to do so. Let's move on. Yes, definitely want to just quickly, maybe, because it just said you will be great to have some of the background. Can you hear me. Yeah. Hi, I'm Stefan. I'm a product manager at Spotify. I'm responsible for the recently open sourced project called backstage that we're looking to donate to the CNCF, and I'm here to kind of listen in and you know, understand the CNCF game, I guess. Welcome to have you. Thank you. Do you want to like share a couple minutes to share what backstage is about for people who might not have heard about it yet. Sure, I can do that. You want me to do it now or later on, whatever you feel more comfortable with will have some time today to go over it sound like a big presentation we can scale a deeper presentation that just in case some people are not familiar with it's just a very brief. Yeah, I think a couple minutes at the end. Yeah, I'll just put it just put it in front. So, then let's start with some organizational topics we have still our local discussion, which was now shut down. And then the vote was closed. We should have a winner now which is 21 21 is the kangaroo that seems we have a logo. Let me know couldn't have joined us today. And now with the next cube kind of we meet in person which is hopefully in order. We can then have our own socks. So I also put on the agenda what I want to do more is also have the working groups also present back on the work that they're doing. For those interested in any of the existing working groups they have usually separate meetings where they dive deeper into some topics. And then they give them more time that we can share and that they can share some of their progress here. And also if you want to move discussions to the main sake for free to do so as well. I think with the air gap working group, they were working also with the timing, which I know especially for European folks was quite a bit hard because I think it was Friday 8pm, which is a bit of a tough point there. Okay, so I think we have Mark here right. Yeah. I can give everybody a quick update on the work we've been working on on the operator working group. I don't have a presentation or anything but I'm happy to kind of walk through what we've been doing and where we are. What the conversation is around right now in the current operator working group document. So we've been meeting biweekly starting to get a consensus on a few points of trying to define what a Kubernetes operator is. We've narrowed down a few things I think we have, you know, general consensus. Yeah, definitely linked to the back here. So, here's the doc for everybody. You can pull it up on the screen share that'd be great. So we've narrowed down a few things like you know what what an operator is, you know, it extends the Kubernetes API to provide application domain specific knowledge and an in cluster reconciliation loop. You can definitely see in this document there's a lot of conversation going on still. First recently we've been discussing you know who the target of this document is is it specifically, you know, written through the app developer or is it to the cluster operator at the end, somebody else, or some combination of those those audiences. And there's been some interesting kind of technical conversation around trying to define. Is there a requirement that an operator modifies in cluster resources or does what you know does does a does a controller that's that's modifying AWS or GCP resources does that count as an operator. So trying to like think through the scenarios of what people are calling operators today, and making sure that our definition isn't excluding anything but it's putting a more true definable definition of an operator on it. The calls every every other week you know the documents been going around on the mailing list also there's still like I mentioned a lot of unresolved items in here. We still, I think they did dive into the operator life cycle and whether that's even included as part of the operator definition or not. It's down in this document a little bit more. We're definitely anybody. It's every other Tuesday, starting next week. And we welcome anybody to join or participate in the mailing list. I know there's a few people I've seen in the in the participants list here today that that do join that operator working group call every other week also. That's kind of a quick update as to that for the operator working group. Yeah, thank you. I think the keys. So if you haven't read it, I would definitely give it a week. And the next step here really is to get ahead of the comments and to try the last call and think it's it's good that we're starting to close down on a certain number of definitions that have been sort of hanging around for a while. That question on whether what did I really find interesting is the discussion about doesn't need to modify in terms of resources because we have a couple of examples where we have operators that actually don't modify in terms of resources, which is an interesting use case because in this case Kubernetes is really more the platform for building the automation where the one that that's running is. Make everybody give it give it a read and if you're not joining the meeting and so working with operators would be good. So yeah, because you're here I think it's also good to get some of redhead. This chart up here is the, I think modified one here but to get some some of the right to the piece. Michael, Michael is red hat representative who's participating. Here's a red hatter already. Is there someone from the helm universe participating in the calls. Matt, Matt, Matt does join the calls pretty regularly and he's he's pretty active in it so yeah. All right, so there's some balance in the universe. We've we've the conversation has has, you know, you know, the love the call last week we actually did include does the installation of an operator. Should that be defined as part of the operator definition. I don't know if we reached a conclusion on that but you know that was definitely the, you know, part of that the helm contribution, making sure that we're thinking about the entire life cycle of an operator. Yeah, I like to be the key goal would still be able to get with this document to a level where we can update the current community document patient about the operator pattern. This was the most specific one I guess the goal that we eventually have. So, okay, if you haven't read it yet. I would recommend having a look at it. I think we have nobody joining today from the from the air gap working group. So overall the air get one is an interesting one so they started to look at air gap installations of communities. Because if you can install when he is in an air gap way. There were several meetings there. I talked to them to the working group leads, and in case you are interested. Please declare your interest to it seems like right now people are like, kind of dropping off the meeting so it was a lot of popularity in the beginning. I think people have other work to do as well. But this was the highest ranked request when we started to discuss the last year at CUBE, what people wanted to work on was for this air gap. This is a situation that we wanted to get involved in. We should also get more involved with the telecom user group here as well. Because it came a lot from the telecom industry, who wants to use it for edge installation with very often older air gap installations here. Okay, last item here on the agenda that I have been here. I have an overview follow up we discussed topics that people want us to work on are interested in is like okay what's like next what's on the agenda what should we start working on. And initially I put together this Google Doc was questions that did not result in a lot of feedback honestly it's just only one question there. One one response there. What I didn't want to have here is the link I took a number of questions that we thought are a good starting point for having a discussion with people and just asking them what they want us to focus on. And I just want to take today's just go over those issues and fix the title. Sorry. So one is, do we have like a simple unified way for replication definition. This was something we had always discussed as part of the charter in the beginning, like this ongoing question how do I define what my application in Cuba need to says. And argue okay help can do it but how do you handle them secrets how do you handle third party dependency and services how do you handle these kinds of things. I would at least everybody in here to get started as to provide their feedback on this one here. And the way all these questions are structured. Right now is the general question what do they think is an issue. It's a slight issue. It's a big issue for that we can eventually have a ranking more than the most the most critical topics. And then there is obviously the open and answer for people are using and we deliberately went for open ended here. Don't say okay just people just have like a set of choices that they can make and this is a little bit more work to go through this and then categorize it but honestly I don't expect like hundreds and hundreds of responses. The next topic to that we put on data on the delivery survey is how do you manage dependencies and also especially certain product dependencies or dependencies that go beyond your existing applications like a server is how we even model dependencies which is what it ties into delivery like this service depends on another service how can you express this even how to handle this which ties into the next one like delivering complex applications. Ties ability to do the operating discussion but in general like how can you ship applications of certain complexity with dependencies and how do you model is that they can easily be installed I think this is something that we from our experience put in there is something we see more and more than people are shipping applications in Kubernetes that are more and more complex than they used to be. And still struggling with how to ideally do this. There's other things coming in here obviously as well like how do you for example if you have to define like certain service routes or configure certain things that have to handle third party. That are installed in the cluster whether they are there or not and how you want to. And again, next question here packaging applications to get a. Kind of like off the shelf application experience. This is also something we see like an interesting point how people are handling this today. And something we start to. Yes, that's like one of them. I'm working on the title here. Topic here is, again, how do you kind of get this like off the shelf experience of installing an application. This is just the idea so you have like your application how do you package it how do you ship it like in the old days where you had like your installer and you ship the binary and then install the application how do you really do this. Like on Kubernetes, especially with a lot of the external dependencies like where do you for example put your container images. And then you get them in the registry that is a lot of companies. Also from from our personal experience we see very often issues that people are not downloading from public registry so they have to export it we imported into their own registry which kind of makes the experience for actually using them a bit hard. And that that's what we put the question here how much of an issue. This is for people and how they're doing it today. Then going. Yeah, this is one like how we install applications with specific infrastructure requirements. In some multiple different clusters. Again, same issue you have for example for your application and dependency on a service mesh or certain configurations should be provided for a service mesh. Obviously there is SMI. But there's also other dependencies and different clusters might have different capabilities. So a lot of these questions are the ground interaction that you have that you built the application once and have to install it into multiple clusters, rather than building an application just just for yourself. Again, how do you do it and then the whole topic around chain of custody. Since you also start to see more. Do you even know what you're installing, or how do you even know what you're installing and where these things are coming from and how they are validated. Again, we will be discussing this for like boost that first version year months. If I'm getting a 2000 lines. Manifest file, like what was everything in there, where do the things come from and what are the dependencies that are pulling into my application and how can I be responsible for production environment actually validated what people are installing or what I'm trying to install here. And that actually fulfills my security requirements and other requirements. And then some of the automations operations topics so that I can build certain things in the more. Yeah. Caroline, we will put off the shop application pieces in there. Caroline just mentioned, she's defined concept applications. So let me just fix my type over here. This is okay operators are right now really custom built the question how can I get to more like reusable operational concepts that I want to use and maybe also how, how can I handle situations. I think that we, for example, totally learned when I have to suddenly have run multiple operators at the same time because I'm using multiple infrastructure components. And how do I connect them together say I'm like using a design run using an elastic and using my own application and I'm using a reddit so how can I get like this whole application stack, properly work together with its independent operational components. So this is a set of questions that we have available right now this is a starting point so the reason why I brought it up today is just want to walk you through it would be great to get your feedback and obviously response on the question if you think that there is something missing in those questions that we should ask in people as well. So we can obviously add a couple more questions if you think like okay this question is a bit off topic, and we should refine it please. Let us know as well. So then now opening up for for comments on this one. So, I think this is this survey is great. Who is the survey getting sent to. We have to work with we have to work with the CNDF and reach out, reach out to the end user community and also for the members some have like they and maybe we can have maybe some feedback from the open to comments community and like others have access to their questions as well. I know that we try to get it by the CNDF, but the response rate is usually not that high. What I also would like to do that for new members coming in or putting this on the landing guitar page for for people like if you're here for the first time. You should familiarize yourself with. And also like being able to constantly edit but I think once you're ready, I'd like to reach out to the CNDF or we could get that survey out there. Yeah, if you put it on the GitHub page with a link to it. I think one of the things we can do is we can, may I share it at the on through the Commons mailing lists because there's about 20,000 people there. I don't know if I'm going to answer. But also we could tweet it and socialize it. You know, your own social channels to to get some feedback. Yeah, that that will be great. Obviously we have to CNDF mailing lists as well if they're delivering mailing lists. We can also share it in some of the communities mailing this people to give us feedback. I think there's a number of them so interestingly what I've seen in in the past is that if you get like 100, 200 replies to actually pretty good because people get a lot of service that they would have to fill out and especially if you're not offering them the free t-shirts sometimes less might be able to answer them but we can definitely work on this for me the most important thing is that we all feel comfortable with the questions. So what I would ask everybody just please provide feedback on this one if you think this makes sense or doesn't make sense. So if you miss an important question that you think should be in there or is it something that's unclear. So my plan would be let's give it to a week of a review period. And then we use the CNDF and other channels to distribute it to get some feedback. Sounds good for everyone. We just learned that they're even virtual something right now in Zoom which is kind of interesting. Good. Yeah, so yeah please provide feedback. I'll share the links in our city agenda. The link will also be available in the chat later on as well. So then I think we have through the official agenda for today. So Stefan, we want to share a bit more about your project for a couple of minutes. I can definitely do that. I can't share my video unfortunately because my camera doesn't work apparently. But I'd be happy to share my screen and run through a short. Oh, I have to. I just want to give my colleague Remy a chance to introduce himself he was he was here as well I think as a first joiner areas. Luckily my camera does work so I can. I can sneak in here but thank you everybody for having us I see some familiar faces here. It's great to be here at the SIG. My name is Remy and I'm the newly minted head of open source here at Spotify. We're starting to get more involved with the CNCF both through backstage as well as just generally spinning up the open source program and getting more involved in the community. We appreciate the time today as we sort of go through the sandbox proposal process and we're getting excited about engaging more fully and being around. Welcome. Thank you. Thank you. So I'll share my screen here. Can you all see my screen. Yeah, so backstage what is that. So backstage is essentially an open platform for building what we call like loosely developer portals. I didn't plan to run a demo. I mean maybe we want to come back and do like a longer demo at some point. Yeah, we can do it in the next meeting if you want to do a demo. Sure. Well, I'll just walk you through sort of what it is and what the problem we're trying to solve this and then we can come back later for a formal demo. So, yeah, what is the problem that backstage is trying to solve. Like the infrastructure landscape is pretty complex. I mean if you look at like the CNCF alone it's huge. There's a lot of different open source projects and we have certainly built like our fair share of like internal infrastructure projects internally at Spotify. And that might not be like a problem in itself but if you look at like the, I mean this is just an example of like the overall complexity of the landscape and when you, the problem that we have kind of at Spotify with that we did have was that as we were growing our infrastructure and adding more and more tools and more and more like infrastructure teams building, you know, distinct pieces of our infrastructure. You know, like the overall, you know, the overall thing or the ecosystem of tools started to become super complicated. To a point where actually like our engineers were starting to slow down because they couldn't either like find the tools or the tools that we were building. They didn't really connect with each other. There was no like, you know, connecting tissue tissue between them or like you know every tool was built like for one specific purpose, which is, you know, that's kind of how you build I guess a successful open source project you solve one problem you solve it really well. The problem is though that when you sort of try to put all these pieces together and ask engineers to kind of use all of these tools. And to become kind of experts or at least operators of very many of these different infrastructure things. And we don't necessarily think that an engineer at Spotify should really have to be an expert in infrastructure and underlying tooling to be productive at Spotify. So, what did we do then. The solution like you know, air quotes I guess is here to, you know, what we opted for was to being build one single consistent UI for all of our infrastructure, and that is backstage. So rather than having multiple, multiple kind of infrastructure teams building, you know, their tools as distinct islands of infrastructure. We opted for a model where we would have one central team, my team to build sort of a platform layer, where these different infrastructure teams cannot plug in their infrastructure and made it into a consistent experience for for the end users who are the engineers and developers at Spotify. So essentially, what started to happen, or what we encouraged and was kind of almost a great kind of an app store marketplace for for these tools. And we were seeing metrics as such as like onboarding time which was getting longer and longer for new engineers as we were growing as a company. We started to go down quite dramatically. And we're now at the point where we were like, have to releasing backstage and kind of consolidating and centralizing all of our technical, like all the developer tools as well as technical documentation into one place. So we've actually cut that developer like onboarding time with 55% we measure that like time until you've sent your 10th pull request, which is, you know, not a perfect metric of productivity by any means, but at least something to throw a proxy for, you know, the complexity here that we have been able to reduce. So when we, as we were kind of, you know, talking to other infrastructure teams inside companies similar to Spotify. They kind of or in demo backstage they were like pretty amazed by how many different things we had been able to sort of put into one central place and sort of create a more consistent developer experience and that's when we started to think about like, is this something that, you know, is this a problem that we've solved that kind of isn't unique just to Spotify but maybe, you know, something that other companies are struggling with. And after doing you doing even more kind of, you know, user research or talking to even more teams we kind of decided that, yeah, this seems to be like a industries wide problem. We think that we have a pretty interesting solution to it. So, long term what we are hoping now with releasing backstage as open source is kind of start building a more standardized engineering, you know, UI layer essentially for different infrastructure tools. And with the same kind of plugable architecture so that, you know, developers and open source project maintainers from various project can, can sort of make it easy for people to use their products by building an integration, like a building a plug in as we call it in backstage. And we're hoping that we can create kind of a big community around this. So that's why we're here to like, you know, we're really excited about really eager to kind of get more people contributing to this because we think that if we can create kind of an open source marketplace for like developer experience kind of layer things very wishy-washy. And where I multiple companies kind of, you know, in joining in the community and building these different plugins so that, you know, in a couple of months, you walk up to backstage and we're using Jenkins, we're using Kubernetes, we're using, you know, Grafana, we're using CircleCI, whatever you're using, like there is a plug in for that. And you can just, you know, configure your version of backstage to match the infrastructure that you have. That's kind of the long term ambition we have with the project. And the reason why we're kind of starting to engage with the CNCF because we think we really want to create a good community around this because we don't think that Spotify can necessarily achieve this kind of bigger vision ourselves. I think that's it. Hopefully you have an overview of what we're trying to do. We're off to a good start. Like we have had some excellent response from the community, like more than 200 companies have reached out to us and wanted like a personal demo and very many of them are already trying it out and like, you know, running proof of concepts internally and starting to build plugins and starting to kick the tires of it. So we're pretty happy about the first like couple of months of progress and excited for the future. Any questions? I do have a question. I'm looking for my notes. So you use terms like portal and user interface. Do you consider it mostly kind of at that layer at the UI layer? Because when you talk about developer productivity, you can start to think about, you also talked about platform team. And so is backstage predominantly about creating kind of a, as you said, portal or plugable UI over infrastructure and that is not part of backstage. The way we look at it philosophically at Spotify is that we want engineers to kind of have three interfaces that they primarily interact with. Like first is their code editor where they, you know, write code. The second is like GitHub or similar. And the third, like for everything that doesn't, you know, like that doesn't like fit into the first two workflows, you know, we want there to be backstage. Like you shouldn't have to go hunt around like, you know, looking for, you know, debugging your data pipelines or looking at your deployments or, you know, checking out what security issues your software has right now. That should all be one system. And that is backstage. That's kind of how we think about. But backstage is pretty unopinionated when it comes to sort of the lower levels. So at Spotify we now have over 130 different plugins. And they span from, you know, all different parts of like the software development stack, like there's experimentation, machine learning, you know, following your code from, you know, pushing it to production and like, you know, following it out to like with you, you know, monitoring and everything basically. So backstage is pretty unopinionated. And then we kind of let the different teams own, you know, a plugin and the use case. So for example, we have our CI team. They own and operate like a Jenkins fleet under the hood, but we don't expose that to our engineers. They build a plugin in backstage instead and sort of attaches that to the service definitions and the service catalog that we have. I think it might be related to, you know, the previous conversation about applications and, you know, we have a, we kind of have a model for how we model our software and all that is represented in backstage. So a team, as you can see here, for example, this like the overview page of things my team owns. All of those things are integrated into backstage and then we have the tooling is kind of connected on top of that service catalog, if that makes sense. Yep. Yeah, that's helpful. Thank you. Looks cool. There's nothing like really cool. There's nothing like Kubernetes specific or anything like that. Like we, for example, managed, you know, our transition from our homegrown version of Kubernetes into Kubernetes or GKE actually, you know, without the users kind of changing the interface in backstage. So the infrastructure team responsible for that migration can sort of, you know, abstract the way the differences between, you know, different container systems or even virtual machines or data centers when we were running there, kind of behind this glue layer, if you will. So the way I can imagine this, like, sorry, so the way I can imagine this is like I provide some type of service and I install it and I write a plugin for backstage and then it's available to other people like I have my CI pipeline, I have my machine running type of environment and I provide it and then a developer can say I just want to use this for my project. Is this how it works? Depends on who you mean the developers. Like a, there are different players here, like there's someone, you know, in the platform or infrastructure organization that kind of deploys backstage and provides, you know, a developer portal that has a lot of different tools and those developers. They build the plugins and kind of integrate different tools and environments into backstage. And then you have on the other side, you have the users, right, like the engineers, developers who are building, you know, they're just users on backstage and they use it, you know, for doing any kind of support they need during, you know, product development in general. Does that answer your question? Yeah, I think I have a better understanding. I think it would really be good like to see the next time obviously we can schedule a demo where we can like walk through the workflow for the different stakeholders, what they're doing and that would even help to clarify it more. Absolutely. I do think it's an interesting project to look at. There's a look at that. There was the other initiatives going on. I just wanted to put them into more context. I was in the operator side. We have operator half, which is very focused obviously on operators and how you can install them, manage them, having the life-sector management in there. This is definitely going in a bit of a different direction. The CNCF was or is already working on a different project where they expose, right now it's mostly hand charts, but the plan is to go further with the artifact hub, in case you haven't heard about it, that might also be interesting to look at. Just also from, okay, how do you relate to these other projects? And if there's an overlap, which is not an issue, so it can obviously be an overlap anyway, but just that people are put into context. But I can see it. So a question from anybody else here in the room? Okay, maybe then we really scheduled this for a demo in two weeks from now if this is fine for you, then we can like really see the inaction and there's usually when more questions come up. Absolutely. One short note is just walking people through, but I think it's just helpful for people that have some sort of understanding here. I mean, if you're, you can't wait for the two weeks. There's a microsite that we have a backstage.io where we have like a demo section, a couple of YouTube videos showing how this is used internally at Spotify. Yeah, I feel free to also post this in the Slack channel. So if people are interested, there's usually more people on the Slack channel as well that are also participating in the meetings, so feel free to also post it there. All right, this actually gets us on top of the 45 minutes for this meeting today. So thanks everybody for joining again for the next time. Please have a look at the questionnaire so we can send it out and schedule a schedule sending it out via the different channels which we have. So looking forward everybody's feedback and also we'll put them backstage demo on the agenda unless somebody has anything else they want to bring up today. All right, then enjoy the rest of your day. Hopefully. Thank you, YouTube. Thank you very much. See you next meeting. Bye.