 So, hello, everybody. Welcome to our talk, Simplifying Backstage and Breaking the Code Barrier. So, this is the maintainer track talk for Backstage. Let's get started. So, first, let's do some introductions. My name is Ben. I'm a senior engineer at Spotify and a maintainer of Backstage, and I have with me Patrick. Hello. My name is Patrick. Go by rugby on GitHub, also a senior engineer at Spotify and maintainer of Backstage. Cool. So, let's dive into a bit of an agenda. First off, we're going to start with some project updates, and then we're going to head into something we're calling the Frontend Framework Evolution, which is what we've been working on for, yeah, the last six, eight months. So, some numbers first. So, these numbers are respective from Amsterdam, KubeCon in Europe this year. We have now 2.3 thousand adopters, which is up 90 percent, 1.2 thousand contributors, and 24 thousand stars. I also want to call out some new project areas that we've added to the project since Amsterdam also earlier this year. We've got three new project areas with some dedicated people taking ownership of these parts of the code. I should talk a little bit more about the difference between incubating and I guess you can call them graduated project areas. Incubating is basically going to be partly maintained by the maintainer team, too. So, we're also going to, like, help kick things off and be a part of that project area until we feel like we're a comfortable place from both sides to hand over a little bit more of the maintainer cost. Things like setting up sync meetings and going through issues and, yeah, signing people to review pull requests, et cetera. So, there's an open API project area, which Aramis is taking over, is in the front row down here, give everybody a wave. This project area is looking about how we can improve the tooling that we have around open API for backstage, especially at a back-end plug-in ecosystem. So, this allows us to have things like typed clients when communicating with back-end plug-ins and also the providers with, like, request, response, validation, and express. There's a scaffold of projects area. So, for anybody that's seen me and Patrick speak before, you'll know that the scaffold has got kind of a very special place in my heart. So, this project area is really exciting for me and to see where we can take it. It's a heavily used feature backstage and it's gone a lot in complexity since we first made it, like, three years ago. So, it's great to get Paul Cowan from Front-End Rescue and Bogdan Netshi Parenko from boll.com on board who have been contributing since the very early days of the scaffolder. And, yeah, it's great to get them on board to help take it forward and maintain it. And lastly, there's a permissions framework and all the packages related to that have been taken ownership by the imaginary Goat Squad at Spotify. Don't ask about the names, there's some very strange names for other squads at Spotify. But, yes, they were the initial inventors of the RFC and the implementation. So, it's great to have them on board too. So, let's talk about plug-ins a little bit. So, I guess backstage loves plug-ins, right? The point that backstage goes from being a framework to a platform is plug-ins. And if that's adding in open source plug-ins or even creating your own plug-ins in your own organization, it's what gives backstage the power of being that single pane of glass for your company, connecting all of the different pieces into one place. So, in fact, the main monorepo backstage backstage has over 200 plug-in packages, as you can see by the gif that's really long here. And some of these plug-ins we provide out of the box, like the catalog, the scaffolder, and the tech docs. But most of them have been actually community-contributed since we started the repo three years ago. And we're so grateful for this. Like, personally, I don't think we would be where we are today, although you're here on this stage with an incubating project if it wasn't for all these plug-ins and these contributors. And they're vital for the backstage ecosystem, right? And because they're free, because they're open source, they're also free, right? Which is great. But just because they're free doesn't mean that they don't have a cost. So, when we were adding these plug-ins originally, early on it was great to be able to see, like, all of these different use cases and make sweeping, breaking changes across the project and see how that impacted different uses of these APIs. But I feel like, or we feel now, that we're kind of past the point of this critical mass for this to be valuable anymore. So, the other thing now is that backstage core is also a lot more stable. Like, we're not making so many breaking changes, especially to the core framework. Not all plug-ins, when they're contributed, have clear owners after that initial commit. So, contributors come and go, but the package still stays there, and then that maintenance burden falls on us as called maintainers to keep it going. And we as maintainers, we have a lot of stuff on our plate right now. You'll see later on what all this new stuff we've been working on. And backstage is going at a fast rate, as I've just shown you previously on the number slide. So, there's quite a large amount of our time maintaining these packages, approving code changes, shipping bug fixes, bumping packages, et cetera. For plug-ins that we don't really use or test that often, it's kind of hard for us to work out if they're going to work or not, or if the changes are good. So, what can we do about this? So, we think as maintainers that we own too much in the repo, so we still want to shrink that basically scope that we own. So, we got our heads together and came up with a plan for how we can trim down the scope, and our first step was to put together an RFC, potentially stopping new plug-ins from being added to the main monitor repo. So, instead, we would provide guidance and tools to make it easier for people to host their own plug-ins in their own repo and their own organization outside of the backstage org. So, of course, we don't want to stop growing the amount of plug-ins that we've got, but we just want to free up some time for us. So, we got some great feedback on this RFC. In particular, we heard that people love this idea of having a central repository for community plug-ins, and that these are it provides to be able to contribute plug-ins with all of our tooling. So, we're going to be exploring this idea a little bit and to see how we can set up this governance so that we're not super involved in that repo. So, if you're eager in being involved in this or being a community plug-ins maintainer, please reach out to us, either on Discord or come grab us here or wherever, and let's talk. I also want to mention one thing as well on similar sort of a topic to that. There's been some steps to moving out some of the plug-ins that initially lived in the main repo before. So, these plug-ins were also contributed by maybe people that didn't work at these vendors before, and it's awesome to see and it's great that we've got such a great open source community that can contribute these and then then being picked up by the vendors to maintain. Seeing these vendors take ownership of these plug-ins is great for backstage, it really shows the investment from them, as well as giving the new home outside of the backstage monorepo, they're now being maintained by the right people that know a lot more about these plug-ins, which is great. So, maybe if you work at a company who has a plug-in in the core monorepo and you want to help move it out, please create an issue and let's get started. Cool. Over to Patrick. All right, so that was a couple of project updates. Now it's time to look at what we've been working on for the backstage front-end framework. So, since the beginning of the backstage project, you've always needed to integrate plug-ins using TypeScript code. Typically, you found one or a few snippets of code in a plug-in readme. Tells you what to do, import a few things, render them somewhere and whatnot, some form of integration. Now, it's usually fairly simple, but already here, there are a few points of friction, right? Quite often, someone that wants to set up a backstage instance might not know TypeScript, they might not know React, they might not be even that familiar with coding at all. But even with that knowledge, it can get pretty complicated as your project grows. One snippet of code at a time and eventually you're here and you're sifting through your entity pages trying to figure out how to add a card for your Java services. So, what we've been working on is a way to integrate plug-ins in a backstage application that doesn't require this code. What we want to do is remove all of these small snippets and instead just keep the, but still keep the flexibility that allows you to customize your apps to your own needs. We call this overall solution declarative integration. Now, something that's worth pointing out here is that this is for both the front-end and the back-end systems, but with the changes on the way to the new back-end system, there's much less work to be done there. So, what we're going to focus on today is completely the front-end part of this solution. Now, if you've been around, you know that we presented declarative integration as part of our roadmap at KubeCon EU earlier this year. We've been working on the actual implementation of this since around the end of summer. So, what we're showing you today is work in progress and it is subject to change. All right. So, when we sat down to design this new system, there were a couple of goals we had in mind. So, first and foremost, we wanted to remove the need to write TypeScript code to integrate plugins. But we didn't just want to shift it all over to YAML. We wanted to make sure that plugins can provide defaults so that the only configuration you need in your application is for customizing your setup. We also wanted to clean up the composition patterns, making it easier to build plugins that extend and compose with each other. And finally, we wanted to build towards the possibility of dynamic plugins as a first-class citizen in the backstage framework, allowing you to install a new plugin without rebuilding that. Now, to be clear, dynamic installation is not part of what we are building right now, but we had it in mind as a future step as we were designing these things. Okay. So, that's our background. Now, we'll look at the design of this new system. So, this is a very high-level overview of the new framework. At the root, there is an app instance, which is just an empty shell. It knows how to wire things together and gives you base to build upon. To populate it with content, you have to install plugins. Now, plugins are also still just lightweight containers that provide extensions. It's in the extensions where all of the functionality actually lies in the backstage application. It's when you take all of these extensions and wire them together that you actually get an app that you can render and use. If you're familiar with the current system, you might know that nothing has really changed here. It's actually the case that the app and plugins more or less look the same from the outside. The big difference in this new system are the extensions and how the extensions interact with each other and come together to build that. So, that's what we're going to focus on now. Okay. So, let's return to this snippet of code that we used to install a plugin and see how this has changed in the new system. So, in the new system, each extension provides their own attachment point. And the attachment point is the location in the app that an extension is integrated at. So, in this case, we're installing the GraphQL page as a top-level route in the app. But in the new system, the extension is able to provide this information itself, so we no longer need that code. Now, extensions are also able to provide a default integration configuration. In this case, it's the route path that we want the page to be rendered at, so we can get rid of that, too. We have also moved away from React as a first-class building block when you're integrating plugins. It was nice to be able to, you know, introduce extra React elements in an app, but it was also really confusing the way we used React in some cases. So, extensions are just plain values that you import directly into an app. And finally, we have added the option to automatically detect plugin packages and import them, which means that we can get rid of this explicit import as well. Sweet. That's all code gone. Nice. How do we actually build anything out of that? So, to understand this, this extension structure, when you bring an entire app together, we'll start by looking at the structure of an existing backstage app. So, what we can see here, it's a thin slice of an app. There is a tree of React elements that determine the structure of this application. Some of these elements are extensions provided by plugins, some are provided by the framework itself, and some are just plain React components. And now, as a side note, it's this mixing of different kind of React elements that we want to avoid and have a cleaner separation between the backstage framework and React. Okay. But what would this look like in the new system, then? So, if there's no code, there's going to be a lot of YAML. So, to the right here, you can see the same structure represented in the new system. We no longer have an explicit tree structure, instead the extensions are defined as a flat list where each extension declares its own attachment point. It's not until runtime that this is all resolved into an extension structure, a tree that actually builds up that. Now, however, the good thing about this YAML is that you don't need to write it. The entire structure is provided to us as defaults by plugins and the framework itself to build it into an actual app. So, what you typically have is something that's going to be closer to this. If all you want are the defaults in a backstage app, it can more or less just be defined by the dependencies in your package JSON. Now, of course, we want to be able to customize our apps, too. And we're going to show you that in just a second. But first, I want to talk about something else. So, one of the main strengths in our current system, the one on the left, is how well it visualizes the structure of the app. That was one of the main goals as we were designing that system. Now, of course, in the new system that no longer makes sense, we wanted to get rid of the code. But it's still very important that you can get the sense for what the app structure looks like as you're making changes and debugging. So, let me show you that. The tools we have built for that by jumping into a quick demo, and we'll also explore the app structure in a app using the new system. There we go. Cool. So, what we have running here, and I'm going to remember the key points, there we go. So, what we have running here is a backstage app using the new system. It's a small one, just a couple of plugins installed. We have an entity page here with a couple of cards. But, as I said, it's all in the new system. And we have this visualizer down here that lets us see the structure of this entire application. So, each of the blue square boxes here is an extension installed in this application. Each extension is linked to an input, the gray round node. And the inputs, in turn, are linked to parent extension. And it's this relationship between with an input and a parent extension that forms the attachment point of each extension in the app. You can see before I form up this tree, every single extension is attached to a parent all the way up to the root of the app, where we have the root app extension. Now, you can see here that we have, for example, a couple of extra, or a couple of extensions here that are themes. So, extensions are not only React components, they can also provide other kind of functionality in an app. Also, jump to this more detailed view. It's the same list of extensions, but in a slightly different format. We can see here, for example, the catalog page extension. I can navigate to it. So, the catalog page, it's all the content you see here to the right, everything except for the sidebar. Now, these color dots here on the catalog page extension, they represent outputs of this extension. So, among other things, the catalog page outputs a React element that renders the entire page. This is shared through the attachment point to the parent extension. And this goes on all the way up to the root of the app. All of the extensions compose what inputs they have all the way up to the root app extension that outputs a single React element that we can use to render our entire app. All right. That's enough for this demo. And back to the slides. And, Ben. Okay. So, let's shift gears a little bit and talk about how we actually use this new system. So, we'll mostly skip over how to set up the app itself and building plugins because that's going to remain fairly similar to the existing system. Instead, we're going to focus on what it looks like to actually create extensions and configure them in the app. So, let's return to the previous slim example of the app. So, so far, we were using the defaults which are provided, right? But let's say that we want to change the path of the settings page from settings to be preferences instead. So, in the old system, you can, of course, just directly update it in the JSX, the scene on the left. And with declarative integration, the path is instead changed by passing configuration to the settings page extension. So, we do this by referencing the extension by its ID and then passing this extension specific config to update the path. So, this next example is a little bit trickier. What we'd like to do is disable the extension in a certain environment. And in the old system, there's no real great way of doing this. You end up coming up with your own switch or a feature flag to be able to do it. And for simplicity, we're just going to comment it out. With declarative integration, however, we can simply disable the extension using the configuration that you can see on the right. So, just by passing false in the building. Now, these are just two small examples of how you can customize your app using declarative integration. Extensions can, of course, declare their own configuration schema so you can make any form of static configuration possible. And you're also able to change the attachment point of extensions and, of course, provide an entirely new implementation of an extension with your own. So, let's look at how you create extensions. So, almost all of the time, you're going to be using these predefined extension creators like create page extension that you can see here. So, these predefined extension creators come with a useful set of defaults and an implementation. All we need to do is give it an ID, a path, and a loader and a way to load the actual page component itself. So, we've created our extension, but now how do we install it in the app? So, as mentioned before, we use plugins as containers, plugins as containers for extensions to install them in the app. So, the recommended pattern is to stick to one plugin per package, and by making it the plugin, the default export for that package is going to be automatically picked up and installed into the app. So, there's also a lower level API under the hood, which is plainly called create extension. So, this is what you're going to use if you want to make your own extension creators or if you simply want more detail control over an extension itself. So, what you can see here is the definition for our call layout extension. So, the call layout extension is basically like the shell of the backstage app itself. So, it's got the sidebar and the content for the actual plugin all wrapped in a sidebar page component. You can see that right down the bottom where it says factory. That's why it returns this React element. So, here you can see when using create extension directly that we've got a lot more control over what we actually want to do in the extension. So, we have to declare an ID, an attachment point, some inputs, which I've said are going to be the sidebar on the app content, and then our outputs, which is the React element. So, we can also define a configuration schema, but for this example, it's not necessary. You should also be able to now see kind of in your head how you can wire these inputs and outputs of different extensions to build this tree that Patrick was just talking about. Now, and obviously, in these examples, we're taking inputs of React elements and then composing them together to build another React element output. That was a mouthful. But you can do this with any form of data type. It doesn't have to be React element. And to reiterate, this is a lower level API. You're mostly going to be using these predefined extension creators. And whilst we're not going to be using create extension directly today, I want you to just bear in mind this call-out extension, as it's going to come in quite useful later in the demo. I'm about to show you now. Okay. Let's just switch over to the demo. This. Okay. So, I'm just going to go over to the catalog, to the example website. So, hopefully, this is going to seem quite familiar to people. What we have on the right is the app config, which comes apart of the backstage app. And on the left is the catalog entity page. So, you can see how we've got a list of extensions on the right. And these are extensions that have just been added to the app. So, what we can see here, we've got some entity cards here. Now, these are for extensions that order is important. They're disabled by default so that you can enable them in the extension array in the order that you want to see them. So, for instance, we've got the about card here. I can just go ahead and move this down, save it, and hopefully it updates in the right order on the left. So, let's look at configuring extensions. So, let's look at disabling first, actually. We can just go ahead and disable this. So, I'm just going to pass in false to the about card. I say we don't want that anymore. It's going to remove it there. We have a to-do plugin here, which has also been added to the entity page. This to-do title actually comes from the extension itself. It's default config. But I can override that as we showed earlier. So, let's say, let me just pop ahead to this plugin itself. Let's say I want to change the title of the tab content. So, I can go ahead and do config. Oops. And then title. And then let's say I want to call this roadmap instead, right? Sure. It goes ahead and updates. Great. So, this is all nice. What about adding new extensions? So, I'm just going to pop over to our settings. You can see the themes here. So, I'm going to go and add a new extension. So, we're going to add our new theme package that we built. This is called theme. So, by adding this to the workspace and to add it to the package JSON, what that means is that the DI system is going to collect any new dependency additions or removals, which I'll show you shortly, and then automatically bootstrap them into the app. So, you can see that we've got a new theme here, which is the aperture theme. So, I can go ahead and click this. Oops. Sorry. There we go. And it's automatically enabled in the app. So, what about replacing extensions entirely? So, I mentioned the call layout. So, the call layout is kind of what you can see here. You've got the sidebar and the actual content. So, I've got a new extension here, which is called top nav. So, what this does, what's this installed? What this does is basically override the implementation for the sidebar with a new navigation component, and then override the call layout extension with basically changing the order of how it renders. So, now we can see we've added a top navigation and the actual content of backstages rendered underneath. So, it's a super powerful system to be able to change extensions and provide you an implementation and do what you like. I'm just going to quickly show you how removing extensions works, too. So, I'm just going to remove the top nav. Hold on. I need to crouch down a little bit here. I can't write it properly. Remove top nav. Yes. Great. Okay. So, let's look at creating our own extensions. So, I'm just going to pop back here. Oops. So, this is what our index TSX looks like. So, for you already, it might look a little bit like this, which we're going to come on to shortly, which is kind of a lot of React and JSX. But for now in this new system, it looks a little bit similar to this. So, you can see that I have an example plugin already, and this has been passed into these features right here and being registered inside the app. But because this is not a separate module, it's not automatically detected. We have to register it manually. So, what I'm going to do is I'm going to use the create page extension to create as a new root or a new page inside backstage. They're just some simple content. So, let's go ahead and do const example page equals create page extension. And this is the extension creator I was just talking about. We need to provide an ID for this. So, I'm just going to call this welcome page. And then we need to provide a default path. So, this is the root that it's going to mount on, which we're just going to call it welcome. And then we need to provide a loader, which is going to tell us how to go and get the React component that's going to be the page component. So, I'm just going to cheat a little bit here and just do promise.resolve. This with, like, a welcome. So, thank you, co-pilot. That'll do. And I'll give this a save. And then I need to add this to the extensions. So, I'm going to add this to here. And then, hopefully, if I go and go to the welcome page. Oops. Welcome. We should see. Welcome to backstage. Yay. Great. And this is all cube gone. Cool. Nice. So, that's kind of the simple demo I had for extensions. I'm going to pass you back to Patrick now. Oops. Let's talk about compatibility. So, it's very important for us that there is a smooth transition to this new system. In particular, we want to minimize or preferably avoid any breaking changes to plug in code. The code that actually implements the functionality of your plugins. Now, the new system is built out of two new packages mimicking our existing structure. They're, say, a front-end API and a front-end plugin API package. While these packages won't be complete drop-in replacements, a lot of the common APIs like use API, use route ref, they're going to remain the same. So, a migration of a plugin or migration of most plugin code is just a matter of running code mods or search and replace. Now, there's been a couple of changes under the hood, though. So, plugins built using the old core APIs for them to be compatible with new apps. There's a compatibility wrapper that you need to use for now. The good news is that once the new system is stable enough, we'll be able to add forward compatibility in the old system that lets you use plugin code in the new system. And hopefully, this makes it as easy as possible to have one plugin that is able to sort port both systems at the same time, which is also going to help out with app migrations. And on the topic of app migrations, they're a little bit different. You know, app is more or less all-wiring code from the framework. What we found that was what works best here is a top-down approach where you start by switching out the route of the app first. So, for this to work, we, of course, need to be able to use the old react-based app structure in the new app, which is why we are going to provide a compatibility wrapper for exactly that. So, after switching out the foundation of the app, you can then slowly move over to the new system, one plugin at a time, until you have migrated the entire application. So, let me show you what that looks like. And I'm going to switch keyboard layout for this one. Hey, sweet. There we go. Okay. Cool. So, I'm going to jump to that app Ben showed you before. So, this is an app using our current system. You can see it here to the left as well. It has about the same plugins installed as the other demo app. But, of course, you can see that we have this react-based structure that determines what the app looks like. So, what we're going to do is migrate this up to the new system. And as I said, we do top-down. We start by switching out the app instance itself. So, I'm going to change the import of create app here to instead be from our new system. I'm also going to add in a compatibility API helper that is going to come in handy. So, this helper allows us to take a structure from our existing system and convert it into plugins and extensions that work in a new app. So, down here at the root of our existing react structure, I wrap everything up using our convert legacy app. I then pause that on as features to our new app instance. When I save that, this app is now migrated and using the new system, but with the old react extension structure. And I can show you that by heading over to the visualizer here. So, we still have the old kind of hard-coded sidebar here. So, we haven't switched the sidebar to the new system. But we can see here that we have a couple of extensions, right? And this tree is a lot smaller because this conversion creates a much flatter structure. It just migrates the pages. It's not aware of additional extensions that might exist inside these pages. Okay. So, we're running the new system. We still have the old structure. What we want to do now is migrate one plugin at a time and move them over to the new system. So, to do that, we want to remove the definition from the old system and we want to install the plugin from the new system. Now, as you know, plugins in the new system, they are detected automatically. And, in fact, all of the plugins are already installed in this application. All of the plugins using the new system are already installed in this application. The only reason we don't have duplicate installations is because these explicitly declared features down here have higher priority. But that means that all that I need to do is go ahead and delete the definition from the old structure and that plugin will have been migrated to the new system. So, you can see we now have a bigger tree of extensions because that entire plugin now runs in the new system. Now, of course, for your own apps, you might have internal plugins and so it might not just be a matter of deleting code. You would need to migrate that plugin and once it's ready to migrate it, you install it in the new app and you gradually migrate like that. But, in this case, it's all open source plugins that already support both systems. So, I can keep deleting plugins one at a time until what we have left is just, well, nothing really. We can get rid of this, too. This is just empty routes at this point. We can clean up our imports. And this app is now completely migrated to the new system. And it has a lot less code. There it is. And we have slides again. Cool. That was the demo slide. And now, okay, so I want to show you the timeline. I know you're going to be curious about this. Our hope is to be able to ship an alpha release at the end of this year, but no promises. We're looking at still an early release and you should not be migrating your production applications even at this point. But there will be a complete system for you to try out and we are looking forward to your feedback. Thank you.