 Hi, everyone. I'm going to go ahead and get started here. My name's James Russo. I'm going to be talking about how Brex leverages backstage for FinTech engineering growth. A little bit about me. I've been working at Brex for the past three years. I originally started doing product development, but helped start up a team that focuses on internal developer productivity and experience. And that's where I've been spending the last two years as a tech lead, helping to improve our back end developer experience. For anyone not familiar with Brex itself, Brex is a financial technology startup founded back in 2017. We mostly focus on corporate credit cards and expense management for software. But we have been a pretty fast growing company. And there's been a lot of work done in the last couple of years to help with that growth. So that's what I'll be talking about today. In general, the agenda is going to be some background on Brex a little bit more, as well as why we went and decided we need to adopt an internal developer portal. Our adoption path for that internal developer portal, any learnings that we've gotten from it over the past couple of years, and then maybe some time for Q&A, hopefully, at the end. So in general, as I said, Brex is a pretty young company. But we've grown very, very quickly. When I first joined in 2020, we were around 300 or so total employees, and less than 100 of those were in engineering, product, and design. Now three years later, we're about 1,100 people with 500 of those in engineering, product, and design. So we've had about 5x growth in just engineering, product, and design. And with that rapid growth and addition of teams, we've added a number of new workloads internally, as well as a number of new tools. And managing this growth has been a big issue that we've been working on over the past couple of years. So in order to try and solve for this rapid growth, we started a new team at the start of 2021 that I was a founding member of, where our mission was to provide a centralized platform that simplifies and caters to the daily workflows of developers at Brex. This was before we even knew what Backstage was, or that we wanted to build an internal developer portal. This was the mission that we decided on, kind of led us down this path. We had all of this different tooling internally. As you can see here, this is just a sample of them. There's lots more today. And it was becoming increasingly harder and harder for all of our developers to maintain the context necessary to use this tooling. And we wanted to simplify it as much as possible and lead developers down the right path when it came to solving their problems. So our vision here was to build a unified interface where our different platform teams in our infrastructure and platform organization could build tooling into a single file, a single place. This could be anything from managing our preview environments, service management in general, resource management in AWS, and the infrastructure parts, serviceability, secrets management, et cetera. We wanted product developers to have this single place to go to to find access to any of our tooling as well as lead them along their developer journey. So with this in mind, we kind of started looking at different developer portals and building our own internal developer portal. For anyone not familiar, Backstage is an internal developer portal that is an open source software project maintained by Spotify. They used it for years internally and then open sourced it in 2020. We started adopting it in 2021. It's a framework and set of libraries for building your own internal developer portal. There's also a series of SaaS offerings for internal developer portals. But we quickly latched on to Backstage because it was solving for the exact need that we initially wanted, which was a software catalog. That's a core functionality, which we'll talk a little bit more here in a second, as well as it was really, really extensible and easy to build your own internal workflows and functionality into it, which was a big thing, not in a lot of the SaaS offerings where you kind of are stuck with whatever they kind of give you tooling-wise. But Backstage is completely managed internally by yourself and has a lot of opportunities for you to hook your own functionality into to extend it for any personal needs. So after a quick POC with Backstage, we were able to get up and running within a couple of weeks. We quickly saw the value in it and we were super excited about the extensibility part of it as I've already mentioned a few times here. So at the beginning of 2021, we kind of started our Backstage journey. And I'm gonna kind of walk you through how we went about adopting it, what our adoption path looks like, and where we are today. So Backstage offers a series of core plugins. This is functionality developed and maintained by Spotify internally and open sourced. The four main ones that we've kind of adopted are the catalog, which allows you to catalog and group your software as well as add additional metadata around it so that your software engineers and your organization can easily find information like who owns what service internally, how to reach that team, et cetera. They also offer a scaffolder, which is a way to get from zero to one quickly and in the right way, basically a way to generate a new software project with all of the right things built in, any infrastructure pieces and things like that. They have an API docs plugin, which allows you to understand the interfaces between your different workloads and how they interact and how you can interact with them. And they also have a Kubernetes plugin that we've adopted that allows you to troubleshoot your software running in your different environments. This is just the four main ones that we've adopted. They also have a few other ones that they maintain like TechDocs, which is a knowledge base that allows you to use documentation as code alongside your code and service it in Backstage. And there's a few others as well. But these are the four main ones we've used. This graphic that I have here on the right is the system entity model, which will be important for the next slide around the catalog. So I'm gonna quickly touch on it here. Basically Backstage comes with its own model for representing software entities and the relationships between them. At a high level, there's components, which are JRPC or HTTP servers, Cron jobs, websites, libraries, things like that. These can depend on resources, which are things like infrastructure components like S3 buckets, databases, et cetera. And then these components can provide and consume APIs. These can be JRPC, HTTP APIs and the documentation associated with them. All of this is grouped into a system, which is a group of related entities that work towards a common goal or functionality. And then you can group the different systems into domains, which can be products or different lines of functionality that you offer to your customers. That's gonna be important for this next slide here. So the first thing that we did in our adoption journey was utilizing the catalog and populating the catalog. A question I often get when talking to people who are looking to adopt Backstage is like, how do you start with the catalog? How do you populate it? How do you get the data into it? And generally what I say and how we started was to focus on a specific type of entity first. So don't try and get all of the information in the world into the catalog off the bat. It's important to focus on a specific type of entity. In our case, we focused on JRPC services as our first entity that we wanted to catalog, as this was the most abundant resource in our software world and also the one that we utilized the most and wanted to build the most tooling on top of. So that's the first step. I would also then say work iteratively from there. So slowly build in more and more information after we got our services into Backstage. We added stuff like cron jobs, asynchronous event consumers, teams and users, APIs, and things like that. But each of those were different steps along the journey. Our initial approach to this was to utilize Backstage's built-in entity ingestion. So here on the right, I'm going to quickly go through kind of how the basic Backstage entity ingestion works. So they offer this catalog plugin which comes in with a bunch of functionality that knows how to read information from certain locations. You can figure this in a YAML file saying I have resources located in this location I want you to read for me. The basic one that they kind of use internally at Spotify and that we've adopted for most of our software is a YAML-based configuration file. So they have a catalog info.yaml file and it looks very similar to what you described as like a Kubernetes resource. It's very similar to that. You place these files alongside your code in Git repos and then you tell Backstage where to find these different files and it periodically reads from GitHub, stores that information into the database, and then allows you to display it in your software catalog so that people can go to different places and see the metadata related to it. So that was our initial process. We basically went and automated generation of a bunch of these initial catalog info YAML files by looking at the runtime environments where we deploy our software. In our case, Kubernetes wrote a script that generated a bunch of these YAML files, placed them into the repository in the right places, and then opened up PRs for teams to quickly adopt this. There are other approaches where you can try and get the teams to do this themselves, but we found it very difficult to get adoption, especially when teams are focused on building other things. So in order to facilitate and quickly adopt it, we tried to automate this and do it for folks as much as possible. After utilizing the catalog info.yaml approach, we've also gone now and created custom entity providers for internal APIs to source that information into the catalog. So Backstage allows you to extend it by writing these things called entity providers, which basically are ways to ingest data into the catalog. You can ingest it from static configuration files. You can ingest it from internal or external APIs. In our case, a couple of use cases that we've done internally are ingesting data from an internal service that stores information on all of our employees and the teams that they're members of, as well as additional metadata around it. The service is read from periodically and then puts that into our catalog. So we have all of our engineers information on the GitHub profiles, their Slack profiles, and stuff like that displayed in Backstage. We've also used it to read directly from the AWS APIs and ingest resources into the catalog, like RDS instances. So after we adopted the catalog, the next step that we went around is adopting and utilizing the scaffolder. So we wanted to automate software and resource creation. Previously, this was a very, very hands-on and manual process. There was a bunch of different CLIs for different languages that could generate code. These weren't owned by core teams, often out of date. You also had to do a bunch of manual steps to generate the resource infrastructure, like Docker, repos, databases, et cetera. So once again, we tried to work iteratively and build on top of the open source version of Backstage. So Backstage comes with a bunch of what they call custom actions in their scaffolder, which allow you to get up and running very quickly to take a template or a scaffold template, inject a bunch of values into it, and open up a PR, or merge it directly into a repository. We started there by just making code scaffolding for the best practice of what we thought a initial service looked like. From there, we iteratively added onto it and built more automation around infrastructure pieces, so making sure that we automate creation of the Docker repo for the RDS instance for you, any other configuration via Terraform modules, et cetera, so that when you open up a PR from the scaffolder, you can instantly merge in your code and see it deployed into your environments, and everything works off the bat. I will say that at a certain point, we couldn't automate everything and make it synchronous in the scaffolder. It takes about 30 seconds, maybe less, to scaffold all of this out via the scaffolder, but some more complicated things like the scaffolding out databases, making sure that the users for the database were set up, any additional infrastructure that relied on that were a little bit slower, so we integrated with a workflow engine internally managed by another infrastructure team that allows us to implement more complicated workflows, like creating the database and all of its dependencies, and we kicked these off from the scaffolder so that every time a new piece of software gets created, all of the additional resources will eventually get created. It might just take a little bit more time as it asynchronously spins up the database and its dependencies. A great thing about this is it allowed us to make sure that we have best practices by default or golden pathways is a big thing that Spotify really talks about in a lot of their literature, basically making sure that all of our security best practices or coding best practices are displayed by default, and we have good documentation around the scaffolder so that developers know exactly how they should write software or how we can best support them in the way that they write software. Scaffolder itself allows us to also get contributions from other teams, so the core backstage maintaining team at our company aren't the only ones contributing new scaffolding templates. We have a bunch of other teams internally that create these scaffolding templates and are able to easily allow developers to automate generation and creation of additional resources like additional AWS services like ElastiCache and S3 buckets can be created through backstage now so that we're not the only ones contributing and it's really a platform that we have a lot of different folks building on top of, which is great. After we adopted these first two initial core plugins, a lot of the functionality that we've built since is more so internal plugins custom to our infrastructure and our platform teams, so we wanted to make it really, really easy to build internal plugins and custom plugins. Something to know is that backstage is a TypeScript React front end and Node.js back end and in general, at Brex, we utilize gRPC and Protobuf as our messaging format between services, so this isn't something you can easily call from a web front end per se and instead of making all of our teams develop a Node.js HTTP router that would translate JSON into Protobuf and call the back end gRPC services, we decided to automate this for them and create a reverse proxy utilizing another open source project called the gRPC gateway library. This is developed by Google. Basically, this library allows you to translate Protobuf into a RESTful JSON API and then does the communication to the gRPC service for you and vice versa. So we set up this internal reverse proxy gRPC gateway that allows backstage front end plugins to call it via JSON and HTTP and then it translates it and calls the underlying back end service. This way, we reduce the amount of code that our developers need to write internally to build these custom plugins. They can just build their back end service as they would for any other project at Brex, hook it up to this reverse proxy and then focus on building the front end components via React to call that service. Once we did that, we've gotten a bunch of different custom plugins. Here's just a few examples. The main one and probably the one that we see used the most is one around deploying release information and managing your deploys and releases. So internally, we use a GitOps workflow for deploying our software and we allow people to easily with one click roll back to previous releases via backstage for a given service or workload as well as roll forward and just manage the deploys and see previous deploys in general. As I mentioned earlier, we've integrated with the workflow engine internally so that you can create more complicated workflows and kick them off from backstage. We have a notification timeline in backstage which allows you to see different events associated with your software. So this is things like when your software was deployed, when an alert went off and dated dog or paid your duty all in the context of one timeline. And then lastly, another well-used plugin is around our asynchronous event consumers utilizing Kafka. So when an event enters a dead letter queue because it can't be processed by the consumer, that information is displayed in backstage alongside the consumer and you can republish or delete these events and manage them through backstage. These are just a few different plugins that we've developed internally but like I said, this is where we see most of our development nowadays are the internal plugins. So there's a bunch of other use cases we've also implemented. Here's just an example of kind of our internal backstage. Even this has kind of evolved since I took this picture but basically here you can see for a given piece of software in our catalog, this backstage demo app, you can see its deploy and release information here on the left and this is how we manage it. You have a list of all the different release candidates that have been created. You can click back to previous one and roll back to it. You can see when it was committed, what different PRs were associated with it that made up that release, et cetera. And then on the right, here's this event timeline where you can kind of see the different Kubernetes events related to your service in one place alongside the release information. So one very important thing around in adopting an internal developer portal is around evangelization and education. In general, organic adoption does happen from early adopters and champions but you won't be able to get full adoption just by building stuff. You need to evangelize and educate users. So a bunch of different ways that we've kind of tried to evangelize and educate is around engineering-wide presentations or any large format presentations. I've given a talk like this multiple times internally to two different groups. New hire training, so we have when a new engineer joins, we have onboarding and 101 training that talks about how to utilize backstage in your developer journey. Team specific training sessions, we've reached out to very specific teams especially when we see them have issues that we think backstage can solve. So something around incident management and stuff like that. We've set up specific training with teams to make sure that they're aware of backstage, all the new functionality that we built into it and are able to use it. We try and find partner teams to try out new features. So whenever we're building something, make sure that we're listening to customers working directly with them and solving their problems. Lastly, we use a lot of customer research. So we are constantly trying to talk to customers, synthesize that information and figure out what pain points are the most important for us to solve via backstage. So now that we've kind of gone through our adoption path and talk a little bit about the learnings we've had on this adoption path. So in general, the Scaffolder was a massive improvement for us, but it didn't drive weekly or daily usage in backstage, which we were hoping for. Our main goal is to have developers going to backstage every single day, starting their day with it and using it throughout the day to solve their daily issues. But the Scaffolder is more of something you do once for a project and don't come back to backstage to do that. Again, you're not creating new software all the time. But we did get a lot of success and a lot of people were very excited about the Scaffolder, so we wanted to try and continue that. The next thing here is the two-sided marketplace. So there's not just the users of backstage that you need to think about. There's also the other platform developers that you need to think about. And you are constantly at odds trying to build new functionality for your users as well as making sure that other platform teams think about backstage and build their tooling into backstage. So you actually have that unified interface that everyone knows to go to, and there's not more tooling sprawl that they have to deal with. Metrics are incredibly important. I'm going to touch on this in the next slide as well. But in general, we've taken a very metrics-driven approach to this as you would for any product and any application that you have on the web. Usage-driven, usage is generally driven by actions more than information. So this kind of relates to the Scaffolder, but we initially thought that the catalog was going to be this change for everyone, and then everyone's going to come daily to the catalog, find information, know how to reach out to folks. But in general, that's not what happens. In general, the plugins and the functionality that we built that actually allow you to manage and take actions are the ones we see the highest usage in, such as the Play plugin that allows you to roll back and manage your software releases. Initially, we use a mono repo internally. Backstage uses multiple repos. So initially, backstage, when we were adopting it, did not have very good support for mono repo functionality. We had to build a bunch of custom stuff, and in order to do that early on, we had to fork the entire project or the plugins that we wanted to use, build our functionality into them, and then make it possible for mono repos to be ULAS. But over the following couple of years, there's been a lot of work by the open-source community to make sure mono repos are very well supported. So as part of that, we moved away from our fork. In general, our fork at Backstage made our lives very difficult. Backstage is updated almost constantly. They have a monthly release cadence where a bunch of new functionality that is often very useful. So we were falling behind on these updates and wanted to get away from that and moved to the golden pathway development way of utilizing Backstage, which essentially is to write basic Node.js and React apps that pull in NPM packages. And it's been a much, much easier process and allowed us to take advantage of all of the ongoing updates. And in general, Backstage has a very, very good community. They have a Discord where the Spotify maintainers are very active, very helpful, have learned a lot from myself, as well as monthly community sessions and adopter sessions where you can attend and ask your questions, presents to other folks on Backstage. So it's been a very, very good community for learning and adopting Backstage. So one thing I wanted to touch on are metrics in general. The Brex product is a web application as well as a mobile app. We use segment in those applications to capture client-side metrics. So similarly for Backstage, we use segment for capturing client-side metrics, such as page views, users, things like that, as well as triggering custom actions. Some of the metrics that we've kind of looked at over the past couple of years are weekly active users. So are we getting more active users as we build new functionality? Most active users and teams, who's coming to Backstage the most? What teams are utilizing it the most? Which plugins are being used the most? So we make sure to break information down by plug-in usage so we know which ones are the most valuable versus which are the ones that maybe we should sunset or revisit. We also have scaffolding metrics, so what templates are being used, which, how long does it take to scaffold out templates, things like that. And something more recently that we've kind of looked at is power usage. So which users are coming to Backstage multiple times a week, as well as utilizing multiple different plugins so we can talk to these users and see the value that they're getting out of it and make sure that we're replicating that for other users. In general, we've been very metric-driven, making sure we set up dashboards for all of our new functionality and plug-in launches so that we can quantify how successful they are or not. Lastly, conclusion. I just want to echo the maintenance to solve for a very specific problem when you're adopting Backstage. A lot of folks get really excited by all the different functionality and problems that it can solve, and they try to solve too many things at once. But the maintainers always try and push everyone towards solving for a specific problem, and I want to echo that as well. Starting with one problem, make sure you're successful in solving it and then moving on to the next via Backstage. Evangelization education needs to be a priority. Without it, you're not going to see the adoption that you hoped for. Upstream contributions are super helpful as well as super important for you to contribute back to the community. As I said, Backstage has a very vibrant, open-source community and makes the releases an important part of the process. It's also important that you try to contribute back to it as much as possible. Balance platform work and future development, as I've already mentioned, that two-sided marketplace of users and platform developers. And then lastly, make sure you monitor and track adoption and impact via metrics. That is everything in my presentation, but if there's any questions on Backstage or adoption, I'm happy to take the time to try and answer those right now. Yep? How do you design new plugins? Yeah, so I think it's the same as you would for any kind of product experience. You make sure that you're doing your customer research, getting actual data that backs up the findings that you want to do, go through the development lifecycle process, working directly with your customers and getting feedback iteratively throughout the process as well as working on it iteratively. So start simple and then build to a more complex experience, making sure you're solving that problem. If there's no more questions, any other questions? If there's no more questions, thank you all for coming to my talk. Appreciate it. See you around. Thank you.