 Welcome to this talk describing Spotify's journey from spreadsheets to backstage. We'll start by walking through the struggles we had over the years and how we gradually evolved the way we track software at Spotify. And once we progress through the dark days, we'll arrive at present day with an introduction to backstage. We'll talk about how it's more than just an inventory of your services and give you a demo of what it looks like. Lastly, we'll share our learnings and help you get started with backstage. But before we start, here's a brief introduction to us. My name is Johan and I'll be co-presenting this session together with Patrick. We're both engineers at Spotify and have spent many years building infrastructure platforms. We currently work as open source maintainers of the backstage project. And let's begin our journey in the beginning of 2010 and let's call it a Stone Age. Spotify was still fairly small at this time with an operations team of just six people. These were the days where servers had names and feelings, the only thing that matters were servers. They ran big monoliths, tasked with solving all the world's problems. And we had yet to figure out how to shrink them down into microscopic scale. Provisioning of new hardware was a completely manual process that was coordinated through spreadsheets. We did find an opportunity to sprinkle some automation on top so we built a software or a system called ServerDB. When purchasing hardware, we received spreadsheets from our vendors which contained basic information such as MAC addresses and serial numbers. We then imported that data into ServerDB which in turn gave it human names and a purpose. These were then re-exported into yet another spreadsheet that got sent over to remote hands in the data center which did racking and installation. It's worth pointing out that there were no fancy user interfaces at this time and all changes were done using a CLI. And as years passed, we ventured into the century where microservices were invented. Our engineering team had grown into the size of a small village and we've even convinced some of our platform engineers to learn HTML. The Stone Age monoliths were starting to break down into microservices. But with this shift, our engineers starting having trouble to keep track of all the pebbles. And since ServerDB only knew about hardware, we had no place to keep track of all the new services. So we built ServerDB which in true microservice fashion was completely separate from ServerDB. And here you can see ServerDB in action with its beautiful Bootstrap UI. It allowed engineers to browse services as well as figure out where they were deployed. Of course it was written in Python and in CoffeeScript which was the hotness of the time. And so we entered the Iron Age. As time passed, Spotify had grown into the size of a small town and our engineers had started looking upwards towards the clouds. This transition away from physical hardware opened up an opportunity for another level of automation. All of a sudden, something that had previously taken weeks or months could now be done with a single click of a button. Our platform organization also started inventing a whole new set of tools for our engineers to enjoy. They were however scattered across a multitude of different sites and many of them were hard to find without the tribal knowledge of where they were. We also wanted to know more about the service than simply where it was deployed. Things like, is it running in production? Who owns this service and who's on call for it at the moment? So, as a solution to all of these projects, we built System C. Staying true to fashion at the time, it was written in Angular rather than CoffeeScript. Spotify engineers now had this place where they could browse through all the services but when you logged in you were presented with a list of all of the services that you owned. And you also had the possibility to create new services. Each service in the catalog also had its own dedicated space. There was an overview page with general information like metadata, things like links to external sites and contact and paging information. For each service there were also additional tools like build history, capacity management, deployment status and traffic routing. This made System C more than just an inventory of all of our services. It became a developer portal where you could interact with the service through its complete life cycle. There was however a big flaw with System C. It was developed by a single team with a huge backlog. The system was just not built for extensibility, making it hard for others to contribute. We again started seeing fragmentation as developer tools started being built outside of System C, especially in other domains like data engineering and machine learning that really hadn't been brought in. But we didn't give up. We doubled down on consolidating all of our engineering tools in one place with a focus on reducing fragmentation and speeding up onboarding of our new engineers. To alleviate our current struggles, System C was reworked to make it easier to scale development across multiple teams. It was migrated from Angular to React and given a plugin framework that allowed for more isolated feature development. The catalog model was also expanded to accommodate other categories of entities like data pipelines and websites. As a tribute to our roots in music, we also gave the project a new name, Backstage. And so we entered the golden age. We saw a soar of new plugins entering Backstage. The introduction of TechDocs made it easy to write and find documentation. Tech Insights brought tracking of fleet-wide migration efforts and cost insights started suggesting ways that you could improve your cloud spend. All of these plugins and many more were developed without much interaction of the Backstage core team. One of the metrics we used to measure productivity at Spotify is onboarding time of our new engineers. More specifically, the time it takes for an engineer to send their 10th pull request. This metric had been increasing steadily as our organization grew. Developer tooling had become more fragmented and it got harder to find information. But in the years after introducing Backstage we were able to cut our onboarding time in half. Which was a great result. As mentioned, one of the changes in Backstage was that we no longer handled just services. This spiked the growth in tooling in other domains like data engineering. What we're looking at here is an overview page of a dataset. You can see that we bring delivery status and access control to the foreground. While for a service we might have highlighted things like the deployment status. There's also a whole different set of tools available through the top bar. And thanks to the plugin system, this page and all of the tooling could be iterated on in isolation by our data platform teams. A benefit of using one interface for all of this tooling is that it makes it much easier for engineers to venture into other domains to broaden their knowledge. For example, a backend engineer can now easily manage data while building a recommendation engine. Staying in the same familiar interface that they use for their services. Spotify has always been open about its engineering culture. And we were showing Backstage to different companies in the industry. While doing that, we got a lot of positive feedback with many people asking if we could open source it. So we did. And that was in the beginning of 2020. And almost two years into the project, we're now part of the CNCF and have roughly 50 public adopters and know many more who are building their developer platforms using Backstage. We're especially happy with the reception, not just from companies that publicly stated they use Backstage, but also from our close to 500 contributors that helped build out the project to what it is today. So what is Backstage? Well, Backstage is best described as a framework for building developer portals, along with an ecosystem of plugins. It provides a set of common APIs, a design system and a pluggable architecture that helps you scale development. The way you adopt Backstage is perhaps different from other CNCF projects. It's not an off-the-shelf Docker container that you deploy straight to your Kubernetes cluster. Instead, we supply tooling to help you create, manage your own project. It's an application framework that you extend with plugins, both ones built by the community as well as internal ones. Our own Backstage installations is built using the open source project, just like everybody else's. You make Backstage your own with its own visual theme and name. And we've heard quite a few creative names out in the community, such as American Airlines naming theirs runway. So let's imagine that I'm an engineer interested in a particular service called the Greeter. If I head to Backstage and look it up, I'm greeted with this overview page. Here I can see a collection of different cards giving me general information about the service, showing things like latest GitHub workflow runs, links to documentation and on-call information from page duty. Each of these cards is provided by its own plugin and you can access more at the top bar. Basically, this page takes information and tooling scattered across your bookmark bar and makes it available in one view. The plugins that you see on this page are selected based on the type of component we're looking at. In this case a service and you might have different set of toolings for things like websites or libraries. We call these frontend components and the backing metadata store the software catalog and it's one of the core features of Backstage. You will often have a lot more pages and plugins in a Backstage installation, but for the rest of this talk we're going to stay focused on the catalog. This was a quick look at the frontend. Now let's have a look at the data which is back in this page. At the core of Backstage, as with all CNCF projects, it's the technology that we all know and love, YAML. What you see here is a catalog info file for a service, actually the service we saw on the previous slide. Typically these files live with your source code and are named cataloginfo.yaml. And you might recognize this format as it's heavily inspired by the Kubernetes resource descriptions and you will see many common concepts between the two. We call each instance of these things an entity. A component is a kind of entity defined by the kind field in the YAML. So far we've talked a lot about components, but Backstage aims to model the bigger picture. So components can be part of a system, depend on resources and of course implement APIs. Each of these are different kinds of entities within the catalog and they have their own catalog frontend. It's important to mention that this is not a model that is set in stone for you as an adopter of Backstage. It can be reworked or extended to fit your organization, just like custom resource definitions in Kubernetes. In our own installation for example, we model things like native applications and machine learning models. So going back to the YAML, let's look at some other things that we share with Kubernetes, like the metadata and spec sections. We use the spec sections to model things that are specific to each kind, while the format of the metadata section is shared across all entities. A bunch of these fields are standardized in our model, like name and links, but all information related to plugins are stored under annotations. That's because they provide an open namespace that can evolve over time. So let's look at how this data is used to build out the frontend. The highlighted sections in this example drive different parts of the UI. The top links section produces a small link card on the overview page. And the page-to-duty annotations causes the page-to-duty card to be shown. The page-to-duty plugin then uses the value of the annotation in order to associate an entity with a service in page-to-duty. This pattern is something we frequently use where we keep a lightweight reference between an entity and an external system. Plugins then use that reference to fetch more data or take actions. In general, we strive to keep integrations lightweight, storing only what's necessary in the catalog. And something we haven't talked about yet is how engineers add entities to the catalog. And one way is to use manual registration of a YAML file in the UI. The locations get stored in the catalog and are then used to ingest the metadata. This method of registration allows us to give direct feedback to the user if something goes wrong, such as a bad YAML or a duplicate name. Although manual registration is perhaps not the most common ways of getting data into the catalog, it's mostly used to register existing software. But if you're already using Backstage, you will be creating new projects through Software Template, which is another core feature. They help you automate the creation of new projects along with taking care of additional tasks such as registration. At Spotify, we use software templates to save engineering time and drive standardization. They're a great way to encode common patterns while also helping engineers. You run a template by filling out a form that it generates, which in turn executes a series of actions. These actions can range from creating a new project in your version control system to manage more specialized tasks like creating a monitoring bundle in Grafana. It's a bit similar to GitHub actions if you ever use that. And we won't dive further into software templates in this talk, as they could really have a talk of their own. Now let's look at the machinery that takes care of all of these registered locations and make them available for consumption. At the core of the catalog is a set of pluggable processors, and in a typical setup, the catalog will cycle through these locations to read YAMLs straight from repositories. It's important to note that the catalog itself is not the source of truth here, it serves as a collector and cache of information. However, processors can do much more than just reading YAML. They can, for example, handle validation or call out to external systems to decorate or even generate new entities. The catalog does give you a default set of processors, but writing your own ones is a key point of customization when adopting backstage. Finally, after the entities have gone through this round of processing, they are made available through an API that powers the front-end. We recognize that each organization have their own particular way of structuring software, and we intentionally did not attempt to make one solution to fit everyone. Because of that, we provide a couple of different methods that can help populate your catalog with data. The one we already covered is manual registration. This method is built into the catalog, and it has its own API for management of these locations. However, it might be that you simply want to ingest all catalog info YAML files that you can find in your Git repositories. For this, we can use automatic discovery. You can, for example, point your catalog at a GitHub organization and have it ingest all of the catalog info YAML files that it can find. This is great because it skips the manual step, and it makes your catalog a bit more resilient as you're not relying on the database to store their locations. It does, however, give you a little bit less control over when things are added and removed from the catalog, and it's also difficult to provide the same level of user feedback as with the manual flow. Another convenient method is through configuration. When deploying the catalog, you can provide it with a list of locations for it to include. This method is great if you're the one operating a backstage instance. And you can, for example, use this to lock down who is allowed to add entities of a certain kind through rules in the catalog. A typical use case for this is to statically configure all of your software templates and then forbid users from registering any more of those. The last method we'll call custom processors. This is actually how all of these methods are implemented under the hood, but you can, of course, also build your own as the API is open. These can leverage patterns that are well established in your organization, and they might even be able to allow you to skip YAML altogether. You can, for example, rely on parsing Java POM files or package JSONs in order to generate your entities from scratch. Or you might have an existing inventory system that you want to bring in. An example of this is the Microsoft Graph Processor, which was contributed by our awesome community. It uses the Microsoft Graph API to ingest organizational data and then makes that available as users and group entities in your catalog. Let's look at the more concrete example of automation using a catalog processor. Tectox is another core feature of Backstage, and it takes markdown documentation that lives with your code and makes it searchable and readable within Backstage. To enable Tectox for your component, you need to add a specific annotation to your catalog info file, which in turn then makes the documentation tab show up. Now, within an organization, we could agree that we always keep our documentation in the docs folder, and if that folder exists, it must always contain markdown documentation. If that's the case, we can use a processor that checks if the folder is present and in that case automatically adds the annotation. This can save us from keeping this annotation in YAML, but it does mean that we all need to adhere to this pattern. This example also highlights how when you read entities back from the catalog, they will contain more information than your source YAML files. This is similar to how Kubernetes works, where when you're reading an object back, it has more information than the original manifest that you applied. Now, let's have a look at the demo of Backstage. So, I have my own local Backstage instance. We have chosen to use the catalog index page as our home page, and this is where we can browse all of the components in our catalog. Right now, I'm looking at all of the components that I own, but I can also switch the filter over to see all of the entities in the organization. We can look at some of the other top-level tools, like the TechRadar, which is a great way to align what technology you use. There's also software templates, which we talked about a bit. We can see here that we have a couple of different templates that we can use to create new components. Now, going back to our catalog page, we're heading into the Greeter service, which is the one we talked about earlier. We can see the overview page here with an about card, a glimpse into the readme, a collection of external links, and information about the languages used in this component. We can scroll down to see our recent workflow runs, results of vulnerability scans, and pager-duty information, who's currently on call, and if the situation calls for it, we can even trigger an incident. We also have these tools along the top bar, so we can head over to the CICD tab to get a more in-depth view of our builds. We can also look at the technical documentation for this component. We can also see what APIs this component provides and consumes. So let's look at the Greete API that we are implementing. We see the same kind of overview page, but with slightly different cards. This is because we're looking at an API entity. We also have a different tab up here, definition. Here, we can straight in backstage see our open API specification for this API. But let's head back to our Greeter component. Now, we talked about a lot of open-source plugins that you can install in backstage. So let's try installing one of those. We'll have a look at the to-do plugin, which is able to scan your source code for to-do comments and then make them browsable within backstage. So I have prepared a bit here to not have to wait for downloads and do any kind of live coding, but installing this plugin is split into two steps. There's a front-end component and a back-end component. So looking at the front-end, we install the plugin really just by taking content from it and giving it the space in the UI. So in this case, we're creating a to-do tab on our entity page and showing it there. Now, the installation of the back-end plugin is a little bit more complex. This is all well-documented in the README, and to be honest, it's mostly copy-paste. So heading over to our to-do branch, where we have the plugin installed. I head back to backstage. We'll wait for it to reload. And now we should have the to-do tab. So let's head over into the to-do tab. And there we go. Now we have all of the to-dos in our project. And we can see here that I commented that we should not be hard coding our greetings. I can also click this link to head over to Github to see the documentation, see the comment in our source code. Alright, let's head back to the presentation and wrap this up. We want to leave you with a couple of strategies that can help you be successful in adopting backstage. The first one is to provide incentives. I, as a selfish engineer, should save time by adding my service to the catalog. Use carrots rather than sticks. As an example, at Spotify, a newly created service in backstage gets monitoring, logging, CICD, cost insights, and much more out of the box. It's important that being part of the catalog does not become a chore. It should be an obvious choice for all users, regardless of how much of the common tooling each and every particular service relies on. Engineers should benefit from having all of the tooling becoming available in one place rather than having to bounce around the bookmark bar to accomplish a task. Speaking of bringing together all development tooling, the next strategy is to start small. You might be tempted to build out a bunch of new fancy plugins for your existing internal tools, but keep it simple. Often what you want to start with is just a link out to your existing tool. Then over time, as you build out your backstage instance, you can start bringing information and actions from these tools into backstage and eventually completely repelase the tools. Also, be sure to check out all our open source plugins that have been built by the community. They can add support for a lot of commonly used software and vendors. And finally, prioritize. Backstage as a framework can do a lot, but it's important to focus your efforts early on. Where to start really depends on where your organization is suffering from the most and where the low-hanging fruit is. At Spotify, we had a mountain of microservices being created every week without knowing who owned them or their status. We were, however, quite aligned on what languages and frameworks to use to build new services. And because of that, it was quite simple for us to introduce software templates. Once we had them, they simplified the lives of our engineer and they also helped us track all of this new software. And that's it from us. You can learn more and get started by visiting backstage.io. If you're interested in adopting backstage, there is also more information available at backstage.spotify.com. We'd also like to give a big shout-out to our awesome community. They contributed to all parts of the project and helped build out backstage to what it is today. And it's been growing more than we could ever imagine. Also, be sure to check out our Discord server where all the maintainers and community members hang out. You can find links to all of that on the website. And thank you so much for listening and enjoy the rest of your day.