 Yeah, I guess we get started. So welcome to our backstage maintain the track talk, the state of backstage in 2024 We can do some introductions first. My name is Ben. I am a senior engineer at Spotify and a core maintainer backstage Been working at Spotify for like Five years nearly and four of them on backstage and I win with Patrick. Oh, and it was Patrick go by rugby pongit I'm also senior engineer core maintainer and those other things. Yes Q. All right, so let's dive into an agenda. So we are today going to be looking at project updates Yeah, we're gonna dive into some updates around the project. What's going on? I'm gonna touch on some governance updates that we've been working on Project areas from things that sort of a project areas that's happening inside backstage I'm gonna stop around some of those and do some updates there too And then we're going to dive into some core framework things that we're working on So first off projects updates So backstage has been around in some shape or form within Spotify for the better part of like eight years Our developer portal journey, I guess you could say was first spreadsheets And then we kind of moved into this developer portal journey for a single pane of glass And then that's kind of turned into what is now called backstage So just over four years ago me Patrick Febben and some other People from Spotify got together in a hack week to see if we could open source our developer portal And on the 15th of March in 2020 We released backstage framework to the world and open source it which makes last week is fourth birthday. Whoo. Yeah A quick numbers update to so these numbers are taken from KubeCon in Chicago when we were all in deep dish pizza So it's like that last four months So we have grown it up to 10% contributions are up 15% so that's showing some good progress and good growth in projects overall All right next are some updates for what we've been doing for project governance There are no big changes this time around but just some additions to our process So what we've recently added our backstage enhancement proposals or BEPS This is a new process for suggesting iterating and approving the design of new features for backstage It's of course heavily inspired by Kubernetes enhancement proposals, but it's a bit more lightweight And up until this introduction of BEPS our main tools for iterating on design were GitHub issues and pull requests Both of those are fine when the scope is Fairly limited, but for larger proposals with more complicated designs than harder decision It gets a lot trickier in particular It's really hard to collaborate on a design in something like a GitHub issue where you might have Multiple different discussion threads going on at the same time and not an always up-to-date proposal to refer to So what we've added now is the web process to close this gap It's going to make it easier for both for all of you to contribute and drive design And we also think it will help us arrive at more well thought out the side Design with clear ownership and path to implementation So let's take a quick look at the happy path for a bet It starts as an idea for a new feature in of some form It can often be good to circulate this idea on discord in the community discord or in a sigmating just to gauge interest before starting a proposal Assuming you want to move it forward the next step is the initial creation of the bet at this stage This proposal can be very lightweight. The most important parts are motivation and goals for the design you open up a pull request to create the bet and then have it approved by the owning project area if It gets approved and merged There is that is a commitment by the owners of the area that they want the work to be done and have the capacity to facilitate the Contribution, however, it does not mean that the owners will build it And it is not yet the approval of the design itself this initial approval and commitment to facilitating the contribution is something That was really completely missing in our old process for managing contributions Next up is the iteration phase This is where the proposal is worked on until it is ready to be implemented throughout this process Anyone is really free to contribute and or open up pull requests to suggest changes to the proposal And it's up to the owners of the area to approve or reject the proposal and the updates And this is where you start digging into the nitty-gritty details and figure out the you know What the proposal is about how it's supposed to be implemented and rolled out it can often make sense as well to have an Experimental implementation at this point just to help to get things a bit more concrete Finally once the proposal is settled and has a good roll out plan and an owner of the implementation It is marked as implementable. This is a sign off by the project area owners that they are happy with the proposal and want to see it implemented The best then used as a guide when reviewing each pull request towards the implementation This lets us speed up reviews by having a clear target design while still being able to split things into smaller Chunks that are easier to review if there are new discoveries that require changes to the design The best should be updated to reflect those Although once the best is fully implemented, it's frozen and no longer updated It becomes a historical document and that is a snapshot of design at that time If there are further changes needed to the feature that would be a separate proposal and you bet All right, that's all I wanted to say about the bet process and now over to Ben for some project area updates Okay, so let's look at some Let's look at what's been going on in some of the project areas in backstage So there's a new project area for notifications, which is maintained by a mark from red hat with some major contributions from Haiky from op group this area owns the new notification system that is currently in development Where the goal is to be able to send notifications to users? We're also happy to say that we've Started the work setting up a new community plugins repository And this is led by the community plugins project area with Andre Beth Nick Philip Thomas and Vincenzo as maintainers So let's dive in to the community plugins project area and see what's going on now So some of you might remember I'll talk from Chicago and remember this slide. So this was an RFC That we submitted about pausing the acceptance of new plugins into the backstage monorepo and some options for what we can do instead So a developer portal is using backstage plugins play a pivotal role They use the backstage framework to provide different functionalities to your developer portal and giving developers the tools that they need to work efficiently So there's plugins for kubernetes catalog scaffolder tech docs, and then there's nearly like a hundred other repos in the backstage monorepo itself Sorry under the plugins in the backstage community book and back to the repo So the discussions in this RFC suggested and refined an idea for us to have a community plugins repository Underneath the backstage organization, which acts as the designated home for all these community contributor plugins for backstage And with red hat wanted to join forces to sell this repo. That's the path that we settled on So how do I back a little bit and talk about why we even need a community plugins repository? So let's recap for the discussions from the RFC. So we want to make it easy for Plug-in owners to contribute plugins as well as having other tooling around publishing and releasing Which puts plug-in owners in charge rather than it being coupled to the backstage core releases that happen weekly or monthly for mainland Another thing we also noticed is that plugins published in separate repositories sometimes can become stagnant and then unmaintained over time And this is all fine and kind of common in a healthy open source software ecosystem But we also noticed that there's also sometimes other users ready to contribute and want to help maintain these plugins So having all these in a common repository is a good way where we can get people to come together and share the maintenance for these plugins It's also going to help us as co-maintenors to focus on the core of the project without having this cognitive load All these community plugins and having to maintain them also It said there's going to be designated community plug-in repo maintainers as well as plug-in maintainers working at repo to help support triage reviews All that kind of stuff So what's the scope of the project area? What they're doing and where we are so obviously as I just mentioned the project area is going to be in charge of Setting up this repo getting all the tooling in place and then moving the plugins across into the community points repo We've already started moving some plugins across so expect a big bang of releases for and plugins be moved across shortly We're also going to be changing the scope of the MPM packages too So they're likely going to be using the backstage community scope instead, but of course this is going to be migration We want to make this super simple and kind of transparent So we're going to be updating a lot of the tooling that we've got to kind of do this for you And we expect that the quota sorry all of the core plugins the framework and potentially some of the accompanying modules We will expect will stay in the main monoripo for now So this is an exciting step store step forward and scale in the project and keep your things moving in the right direction It's a lot of talking cheers, okay open API, so this is a project area With a ramist, I think he's here. We'll be on somewhere. Yeah, there is and he's making some great progress here With integrating open API tooling into backstage So we've now got the ability to generate type routers from open API schemers So basically open API. I'm all for anybody that's used open API before you can basically generate a Type express router for you to implement in your backstage back and plug it We've also now got the ability to generate type clients through so from those same schemers You can generate types of clients or whatever client you like And this is also going to be very useful for creating and packaging clients for integration with those same backend plugins And it's in use today in the catalog client package. We use this And there's also some experimental optic support for a schema aware testing So Ramesh has got some open PRs right now, which enable things like fuzzy testing Automatic detection of like breaking API changes to there was a talk that he did yesterday at backstage con go watch it great They explained this a lot better than what I can Yes, so yeah, really excited to see what's going on here and rolling out some more schema first open API stuff Scaffolder updates. Yes So this is the project area. We also announced in November in Chicago With help from Bogdan from bull.com For people that don't know what the scaffolder is It's a collection of plugins and modules which power the create flows in backstage sometimes called software templates if you're familiar with that And we've been having regular sync meetings And going through the open issues in private But recently we moved that workout into the open in the form of a sync so that we can have regular meetings with the community Projectary maintainers go maintainers and going through all the things that's happening with the scaffolder Bogdan has also been driving some work around retries of steps So if for instance a step fails or the task dies It's going to pick it up and resume from a previously known good state closely related to his rollbacks So being able to undo things like yeah the steps like the creation of the git repositories undo that rollback and start from a clean slate This work is kind of under a lot of planning right now Don't really know where it's going to land but super excited to see what's next there And there's also some new scaffolder actions a test utils for scaffolder action Sorry Which will allow you to create a test harness for testing these actions a little bit easier Rather than having to mock out some of our internal types like the context and things like that And also some final and also finally some improved support for secrets in the scaffolder So the secret field extension is now going to make sure that the secrets that you collect from users in the form They're going to be properly protected and redacted in logs So if you're dealing with anything sensitive, please make sure to go check these stuff out And the last update from me is on the microsite and documentation The CNCF kind of funded a documentation review for the backstage IO microsite We've found some areas for improvement and how we can make these dots more user friendly Always good to get a fresh set of eyes on documentation in a project You spend every minute of your working career in just so you can kind of get lost amongst the weeds, I guess There's a collection of issues underneath this umbrella issue here There's come out the back of the review some big ones some smaller ones some rewrites some restructuring And contributions in open source is not always code, right? So if you want to please go if you want to go and help contribute and help us fix the docks and please reach out All right, let's round off the project area updates To with a look at what Marek and Hakey have been working on in the notifications project area So the current scope is rich in app user notifications But we're also preparing to support notifications rather through external challenge channels in the future as well The notifications area also owns the signals plug-in, which is an underlying piece of infrastructure That let's plug-ins push lightweight signals to subscribers in the front end which can be useful in some other places, too So I'm gonna jump over to a demo There we go So here we have a very lightweight backstage app No content basically. I'm gonna show you a notification. So keep your eyes on the notification sidebar at them They're to the left and I'm gonna send a request to the notifications back-end with a little payload And there we go. We got a badge indicating that we have a new notification coming in Hello, CubeCon. Can you mark this red? Move on Now I also mentioned signals. We have the dev tools back-end here, which just shows information about the Instance running backstage, which in this case is my laptop And you can see here that we have live updates of the memory usage and load Which is a signal that you subscribe to when you're on this page All right, that was a quick demo back to the presentation. Yes, sweet Now again, this is all work done by Hakey and Maret in the notifications project area Also, I'm gonna give a shout out to Patrick Jungerman from Borneo and his work on the event system Which the notifications rely on Okay, I want to switch gears and focus on the core framework changes that we have been working on Starting off Something that we're currently working on are improvements to the auth system and this is not about user authentication But rather how communication with a back-end is secured Our main goal for this work has been to make backstage secure by default to no longer have this requirement to deploy backstage Find something like a VPN. It's too common of a mistake to deploy backstage unprotected And it's something that we just wanted to remove as a problem Another benefit is of course that you remove this external dependency to make backstage easier to set up While at it, we also wanted to simplify and improve our service-to-service authentication Moving to a better distributed solution that doesn't require management of static secrets And finally we wanted to improve handling of our user tokens Avoiding the pattern of forwarding user tokens for service calls and instead using service credentials on behalf of users with a reduced scope Now I'm not going to dive into how we solved all of that in this talk We did the lightning talk on the topic yesterday Those I'd encourage you to go watch the recording of that if you didn't catch it if you want to dig into more details You can also have a look at bep number three which covers all of the API design of this new improvement now this is all something that we're still working on and Rolling out yesterday. We released one dot 24 which included all of the API changes we needed for this Overall improvement and then we're targeting 1.25 the next release in a month Where we shipped underlying improvements for the service and user auth Now most of these changes only apply to the new backend system, which means that if you haven't migrated yet This is another reason to do so Lastly we're following all of this up with a security audit sponsored by the CNCF just to double-check our work and for your peace of mind Okay, that was it for features and I'm going to switch over to talk about the framework itself the part of backstage that glues everything together First let me remind you of our overarching goal for the last year or so which is declarative integration We want to move away from using type script to integrate plugins allow you to set up and manage a backstage instance Without the need to write any code. We shift much of the responsibility out towards plugins and we make it more Clear how plugins are supposed to communicate with each other as well Now we're working on this for both the frontend and backend systems. Although we treat those as separate efforts Starting out with a new frontend system We did a full talk on this in the kubecon a na last year in Chicago And we'd encourage you to check out this talk if you want more details In that talk, we promised an alpha release at the end of last year, which we delivered in our 121 release With that release, you're able to build a fully working backstage app with the new system But it's still a early version with plenty of work left to do and at this stage You should not be using this in your in your own projects unless you want to be on the bleeding edge Now looking forward we have identified if it identified a few critical areas for improvement Which came out of feedback from our own usage internally at Spotify in particular. We want to make it easier to make Making create customizations both if you want to override existing extensions or create new extension types We're also going to make improvements or testing utilities and on top of that There's a long list of smaller improvements that we want to work through to polish the system Now to get to the end goal of this system being the default the next step is to get to the second phase of rollout This is where we add compatibility With a new system in our existing frontend system The goal is to make it easier to have plug-in seamlessly support both systems at the same time without a bunch of boilerplate Compatibility code now that requires confidence in our design of the new system, which is why it's important that we do these other changes first From what we see right now This isn't something that we're going to be able to start working on until the second half of 2024 Now over to the back end system Okay So first I want to celebrate a couple of milestones from the last year the new back-end system Marked as ready for production and we've already seen significant adoption We've seen many of you able to move to the new system early on with the help of our compatibility APIs even if all of the plugins weren't ready yet But there's also much less of a need for that now across all plugins and modules in the main backstage repository Have now have support for the new back-end system along with extensions for customization of those plugins With the 1.24 release that went out yesterday the new back-end system is now also the new default for backstage projects Now the work towards our newest back-end system started before the clarity of integration The declarative integration initiative although it was already heading in that direction But now we've properly properly added support for declarative integration in the new back-end system allowing you to integrate new back-end plugins Without changing code which Ben is not going to give you a demo Okay, so I'm going to prefix this with the note that it looks like I feel like can't type on all the demos that we do And it's because it's the Swedish keyboard So this is gonna go one of two ways it will be fine. I'm sure okay Let's have a look Oh, it's command now. Okay. That's what it is. And it's also practice keybindings which make it a thousand times worse No, it's fine. You're good. All right, let's head over to the demo So No, that's not it This is fun. Okay. Here we are and then where's the Chrome window for this? Here we go. Look at that was easy Fine, right. Okay, so Yes, back-end system don't so before we dive in a little bit to either new Declarative integration I want to kind of take a step back. So I'm actually going to go to the code window instead here. So For everybody that's using backstage already this might kind of look familiar from before so a lot of In this file, you see like a lot of setup code like there's a lot of boilerplate for creating your back-end right a lot of it is like creating the environment to create these plugins and then obviously registering the plugins themselves into the Back-end There's kind of a lot of boilerplate stuff here And this is kind of what we started this whole new back-end system initiative and declarative integration to kind of fix because this is a lot of like manually Maintaining this yourself is not great, right? So we headed in the direction of the new backing system which allows you to do the same code or like this is the same functionality as the previous Code example, but this is a lot smaller right now. So we're just adding in the plugins Themselves all of that boilerplate has been moved to the framework instead So now we set up all the dependencies for the environment and register these plugins for you as normal now There is another step for this which is what we've just been alluding to which is the core of integration Which is the basically ideally is to make this a lot smaller too and this is the stuff that we're heading So is this too small for everybody come people say don't make it bigger. I was good. It's too I'll just make it a little tiny bit bigger. I'm there we go. Yes risky So yeah, this is The new standard that we're looking at we have a feature discovery service which we Register in the back end and then that is in charge of discovering all of the new plugins and modules and adding them to the back end for you So arguably this is also very standard like there's nothing custom in here So we could also kind of provide this for you and if you wanted to provide your own you could do that But yeah, this is the head of reason we're heading in this direction is to make setup of back-end plugins a back-end much more simpler So, how does it all work? All right, let's go start So I'm gonna just pop back to this turn the window here. I'm gonna cancel this and I'm gonna run your own start Back seat plus. That's not a plus back end. There we go So you'll see how it when it starts the back end Here we go. You'll see how we have some new log lines, which is Detected these packages down here in the corner. I'll make this also a little bit bigger To so you see How we are detecting these plugins and modules and registering those into the back end Now I want to just take a quick jump back to this if we go to the catalogue here Oops, I'm just gonna refresh this I have a feeling yes So we have nothing in the catalogue right so I want to show you an example of how we install a module Which is going to help us populate the catalogue So I'm gonna pop back to the code window. I'm gonna pop over to the config so you can see how we've Got some config here, which is setting up a Module to help us populate the catalogue and in this case this is configured to point at github.com It's gonna crawl the backstage Organization and it's going to populate the catalogue with the users and groups from the backstage organ github So how do I settle the module? Well, I just need to install it. So I'm just gonna go to yarn add Nope backstage plug in catalogue back end module github Great got there eventually so this is gonna go and add the module It's gonna add it to the package Jason in the back end and then what we're gonna do is hopefully it's gonna restart the back end Yes, it has done and it's picked up our new module in here as you can see it detected Catalogue back in module github.org If I now go back to here and give this a refresh I am hoping that now We have users and groups in the catalogue great. So these are all the user groups pulled from Github and added into the catalogue and that's it. That's all I've got for now. I'll give you this back Sorry, it's all good. Oh, excellent. Cool Back end system demo and now I'm gonna talk about the roadmap for the new backing system So next up is the 1.0 release There's going to be a well We're gonna need to do a review of all of our API surfaces and the polish and refactor as needed in general Though we're not aiming for any new features This is work that's expected to start in roughly a month and take a couple of months to complete We will of course also aim to face out the usage and support of the existing system Although that's gonna be somewhat gradual Gradual it's mostly up to each plug-in to decide whether they support the old system or not And we're also going to see an increasing number of plugins that only support the new system Overall though, we're expecting support to be phased out after the 1.0 release and to have a fully 1.0 release of the backing system and have fully removed support for the old system by the end of the year Now just looking a little bit further into the future I want to highlight another goal that we have in sight which are dynamic features Even with declarative integration and all of the improvements that we've been talking about You still have to rebuild your backstage project when installing new features For example, if you're deploying with Docker every time you want to install a new feature You need to add the dependency rebuild image and redeploy We want to add support for dynamic features directly into the backstage framework and ecosystem itself The goal is to be able to install new features directly into your backstage deployment without rebuilding the project This will really improve the simplicity of managing a backstage instance and lower the barrier of entry But we also see it as a tool to help scale large Backstage deployments as well making it possible to deploy plugins separately from the main backstage deployment and integrate them at runtime Even for the front-end This enables individual teams to own the life cycle of their own plugins and remove the need for a huge monoripo Now this work is of course dependent on declarative integration and the new front-end and backend system But we're already seeing some exciting development in the area It's work that's already been going on for quite a while with lots of contributions in particular from Red Hat and I'd encourage you to watch yesterday's lightning talk by Lightning talk on the topic by Tom Kuhl from Red Hat. I also want if you want to follow along We're iterating on the design of a This system for front-end plugins in BEP number two And that's all we had for you today. Thank you We do have time for some questions if anyone does have any questions. I don't really know how we're gonna do it There's a microphone there. I can run around maybe let's do that. Oh dear Hello And to Patrick and Ben, Shana Patrick. So I'd love to ask are you, you mentioned front-end alpha, right, in your old map. And we are, you've been using backstage like three years for now And I'm really looking forward to some improvement in terms of front-end because usually our developers They complain mostly about how UX and UI Do not allow them to actually to understand sometimes what's happening And for us as an infra team, it's hard to actually Implement what we want because of the limitations you have So I would love to hear how you want to address it in your roadmap It's an area we're excited to work more on once we got the new backend system out because that's going to free up You know headspace and there's a lot going on in the project getting the backend system out and having that more stable Let's us really focus more on the front-end system both You know the overall framework and also building front-end plugins Or we should expect it next year then One of those two Sorry Well managed Hi, just very quickly about operating it at scale, we've got documentation, I don't know, some advice Maybe not quite that big but still quite a lot of engineers What's your sort of expectation for organizations that would do a deploy to that sort of scale Like have we got any good models or documentation about how we should have organized And size the teams to actually just run and manage back to each course That's an interesting question. I know that we have some blog posts and stuff around that how we are set up in turn It's Spotify. I'm guessing when you mean scale you mean like People power and like teams contributing to the backstage rather than the actual scale of like scaling on kubernetes for instance like It's less the scaling on the meta is more just like solving a problem We don't only do we have 15 000 engineers, we've got like tens of thousands of repositories and all of them are different All specials no play, so it's not just We can solve we can we go on board one and all the rest of the same So that's going to be quite some help, you know, I didn't develop this real reporting stuff It's like dev realm in a sort of weird way. Yeah All right, it's it's tricky backstage. There's a lot of different tools that you can use to solve different problems in an organization Some tools are better for solving problems that you have for your organization right now if it's very You know different across the organization You might want to focus on something like the catalog first getting kind of control and like tracking everything rather than something like the scaffolder Yeah, or the other way around So what stage is here's a backup question. We want to reverse card. Um, what stage are you adopted backstage yet? Or are you looking at is it like Okay, sure Yeah, I mean one thing that we recommend to a lot of adopters is like Don't try and solve everything at once like try and pick something Super isolated that you can go and fix solve that problem prove that it works and really are more gradually it might be maybe different You know for for adopter for organization, but yeah, I think it's tricky I think to get kind of set up at that scale as well. So that's very important start small start small Yes When you're managing plugins and things it's like Maybe it's just my dislike of mpm and JavaScript typescript ecosystem Like managing all the different plugins. Do you have personal recommendations of like Like you're doing with the community plugin. You think about in a separate repo We have a bunch of custom plugins That we'd be using to do our own customizations So you have your own recommendations of how to manage that stuff in different repos and building them and integrating them Other than obviously the magical lovely new backend system that means I don't know if we're in one of those So we're we're really hoping to see improvement areas dynamic features later on and being able to deploy from a separate repo into the main backstage deployment Yeah, without having to kind of go and integrate and redeploy I'm assuming as part of that picking things into containers is terrifying Yep, yep Cool Similar to other people in the room where we run a backstage team. We've been doing it for over a year And now we have tons of people within my company who want to contribute custom plugins So I'm seeing the dynamic declarative extension work and I'm like, oh this could simplify my code base I can have other teams have full control over their plugins and that way we don't have to manage it And you know, I can see a really nice relationship where people want to develop custom plugins A question I have is how does integration testing work in that type of environment? So if I'm loading in all of these Dynamic plugins that are putting my backstage, how do I make sure the quality is there as people are developing and pushing updates to their plugins? How do I manage that from a team point of view? To internal contributors outside of my backstage team What we're really aiming for is to isolate errors So that if someone's plugin breaks only their plugin breaks That's kind of a high priority for that and then it's up to them to make sure that their plugin is well integrated and works Some of these declarative extension points Encapsulated testing are ways that I can have guarantees into the dynamic plugins that I'm using Because I would say that debugging plugins is what my team spends the majority of our time doing for sure Thank you very much Thank you, and we have run out of time I believe If you have a question, is it one more? Last one, okay, last one Lost Yes It's quite a quick one Just a new backend system as well, like a couple of people asking questions about that Is it possible to inject custom dependencies into plugins with this way from the boilerplate? I saw it and it didn't look like you had anything like that Because I have like Specifically like a matrix that I like to inject into dependencies that the plugins require So we got modules for extending plugins themselves They have plugins export extension points that you can use to customize functionality that's specific to the plugin Then we got the services, which are the framework level Kind of features for all basically everything the routing logging All the different kind of baseplate things and you can override those And you can customize the functionality of those per plugin as well, if you want to Yes, sounds good Yeah Thanks. Thank you all. Thank you