 Thank you. Everybody, my name is Ryan Emerly from Comcast, Senior Principal Software Engineer. And today I wanna talk to you about how Comcast is democratizing developer experience using Backstage. Before I get started, I wanna give a little bit of background about how we ended up using Backstage in the first place. So for some context, when I first started at Comcast about seven years ago, it was a little bit overwhelming. Comcast is a pretty big company. Lots of really cool internal products and services and APIs and platforms. But when I first started, it was like really hard to find anything. So I've kind of come up with this analogy of the dark warehouse, so bear with me. Just imagine this giant warehouse where you've got shelves floor to ceiling in hundreds of yards in every direction. And on those shelves are all the really cool products and services and APIs that you can use to build the next cool product. The only problem is in this warehouse, it's pitch black. And all that you have to light your way is a single flickering candle. So you're in the warehouse with this candle trying to see what's on these shelves. But all you can see is kind of what's right in front of you. So how do you find the things that you're looking for? How do you know what's available? How do you know what all the cool things are? So effectively, we need to know the people that put the things on the shelf in the first place. Hopefully they know where it is. And these people tend to be the subject matter experts. So here's kind of how it goes, or at least my experience. You have a project, there's something that you're trying to accomplish. And you know that there's some, let's say you need the billing service. You know it exists, you need to find the person that maybe knows where it is or how to interact with it. So you manage somehow to find this subject matter expert and they're able to guide you through the dark warehouse to that component. Great, now we know one path. We know how to get to this one thing. But we need other things in order to make our cool product. So we ask another SME, and then another, and another. And as you see, we start to illuminate some of this warehouse. But, and if this was to scale, there'd be a lot more black here. Of course I didn't have time to make it to scale. There's still a lot of undiscovered area here. But what's wrong with it, right? I've got my answer. I've talked to my SMEs, I've got the things that I need, I can move forward, what's the big deal? Well, so subject matter experts, they could be a team, it could be a person, it doesn't really matter. What if they're not available? What if it's a person and they're on vacation, or it's a team and they're busy? And I've got a deadline, I've gotta build something and I need these subject matter experts in order to figure out where this thing is in this dark warehouse. So it's not ideal, right? It's gonna cause delays, I'm gonna be blocked. But let's say everything works out, subject matter expert comes to me, it's like here, Ryan, take my hand, I shall guide you through the dark warehouse. They do, great, now I know. I know, one person, I know there's been a knowledge transfer from one person or one team to another. Not super efficient. What's more is now that I know, I'm gonna share it with my team, maybe and kind of propagate some of this knowledge, but I'm not the subject matter expert, they are. So now their system changes, my knowledge is stale, and the whole process has to start over again. And this is where we needed to come up with some kind of solution to start centralizing knowledge and that's where Backstage comes in. So Backstage gave us this magnificent foundation to start from. We had other internal efforts to try to build something similar, and we originally looked at Backstage before at its 1.0 release, and it wasn't quite mature enough for what we were trying to do, but we decided to look at it again, became a CNCF project, and it fit our needs. It was really a magnificent starting point for us. And this started to offload our subject matter experts. They had to do less hand-holding through that dark warehouse. And effectively, we began, as we began to centralize all of this information, we created basically a hub for our developers. So we thought long and hard about what to call this hub for developers. We came up with Dev Hub. So the reason I mentioned what we call it internally is because I'm gonna call it that, and I just wanna disambiguate vanilla Backstage from Comcast deployment and implementation of Backstage. So we have Backstage, but there's still some room for improvement as far as what we're trying to do. We're still not using our subject matter experts super efficiently. We still don't know what's available. Well, now we have a software catalog, we have tech docs and stuff. We do kind of know more of what's available, there's still this kind of support aspect that we are still relying on our SMEs for. What's more is that they're also not just producing documentation, they're producing valuable data that might wanna see in our development hub. There are also processes that these teams might have. Let's say that you're on an observability team and it's a platform and you want people to come and use it. The only thing is in order for them to come be onboarded, there's some provisioning that needs to happen. So there's an onboarding process. You need to answer some questions and typically what will happen is it would go into some ticket queue, someone would work it, eventually you'd come around to getting the thing that you needed. So what we're looking for to kind of get around some of this stuff or to improve the developer experience is first class integration with some of these other systems and services that we don't own as the Dev Hub core team. The Dev Hub core team isn't big enough to go out and implement all of these things from all of these different teams. There are thousands of teams. So what do we do? Well, this is where I don't hit the laser pointer and instead move the slider forward. It's where we democratize the developer experience. And in short, when I say democratizing the development experience, I think it's easy to think about making the developer experience everyone's responsibility. If you think about it, you build a tool as a software engineer. You necessarily want people to use that. You don't wanna build something that goes unused. That's kind of like a fate worse than death for an engineer to build something that no one ever uses. So how do we democratize the developer experience? Well, as I mentioned, Backstage is a fantastic platform and foundation. And we decided that we were gonna leverage the two main extensibility points. Plugins, which we've all heard a lot about today. They're fantastic and Backstage is basically just a system composed of plugins. But I think what kind of goes unnoticed, at least from what I've seen, are templates. It seems like the focus of templates is on creating a repository based on some code templates that then render out some final version that you can then go ahead and get started. It's a quick start. But we see it more as a generic thing doer. It takes some user input and can execute any number of actions. And those actions can be customized to be whatever you need them to be. So we think about that onboarding scenario. Let's say that there's some API that we can call that can handle the provisioning. Well, create a software template, ask whatever questions you wanna ask, and then create a custom action that calls an API. Now, users can go to that software template, click a couple of buttons and they've just, now they have self-service and that team didn't need to build their own self-service tool. They didn't need to build a slack bot that could answer a couple of questions and then provision something for you or send up their own service. So I think that software templates are a little bit overlooked and we're trying to use them as an additional extensibility point. So our initial approach to this was, hey, we're using Backstage and Backstage has tons of documentation. Go check it out, our extensibility partners and SMEs. Go read all of this documentation, try not to get overwhelmed and good luck. The only problem was they got overwhelmed pretty quickly so we realized that we needed to do something different there. We need to put some sort of guardrails in place and some process in place so that we can funnel people and get them to focus on what actually needs to be done and not spending tons of time trying to figure out how to even get started. So we formalized this process by introducing the Dev Hub extensibility program. Dev Hub extensibility program, when we set out with some goals, basically we wanted to encourage contributions. We wanted to make it easy and enticing for these subject matter experts or extensibility partners to create things. We don't want to make it difficult. We don't want to add more work for them. They already have plenty of work to do. We also want to gain some control over the intake process. We don't want Dev Hub to end up like an ass car car with all the sponsor stickers all over it. We want it to be a cohesive, meaningful product where everything looks like a single unified product, not a bunch of MySpace webpages, if anybody remembers MySpace. We also want to reduce duplication. We don't want multiple teams trying to build the same thing because there was a lack of communication and now we have to say sorry, guys, we already have something similar by this other team. And lastly, we want to reduce friction. We don't want the process of making the developer experience better to be a terrible developer experience. So we started out by introducing it a process, just a very simple process where we do some intake. We've got an intake form. We get to understand what someone is trying to implement. That will go through an approval process where we just decide, does this make sense to do now? Or are we, as the Dev Hub core team, building some functionality in that will make whatever you're trying to do easier later and we can say maybe you just hold off on that. And then if it gets approved and it hits into the development phase, well then, our extentability partners just they can start development. This is more focused on plugins than templates. Templates are a little bit easier, obviously. So, but the only problem is we need folks to be building it in Dev Hub. The version of Dev Hub that's going to be in production. We need them to see the plugin within the context of Dev Hub because building a plugin in isolation can lead you to believe that it'll work fine and then you go to integrate it in tire screech. Oh, that didn't work. So the only problem is that it's a little bit difficult to test against the production version of Dev Hub because it's the production version of Dev Hub. And what's important is that Dev Hub, as I mentioned, it's kind of a superset of backstage. Backstage is at the core. It's got its Postgres database. But we're talking to other external systems. We've got a really powerful federated GraphQL product and we use internally to bring together all kinds of disconnected data and connect them. And we have to talk to GitHub and we have to talk to other services. So it's not something where we can bring people in and say, just have at it here all of our secrets. Just don't mess anything up. We can't share those secrets. So what do we do? We need people to test in Dev Hub but they can't test in real Dev Hub. We could always have them deployed to our development environment, but that's minutes of time that would have to go for every iteration that you're making. We want people to iterate quickly on this and, again, reduce friction. So what we did was we introduced Dev Hub Lite. And I would share the logo that we created internally for this, but for copyright reasons I can't. The extensibility program, or I'm sorry, Dev Hub Lite, as you can see now all those little keys are missing. Basically is exactly, it's Dev Hub. It's the same code repository, the only thing that's different is the configuration. And then that configuration, instead of pointing to production Postgres database, it's pointing to a Docker container that has a sanitized version of production data in it. Some of the external API calls that we have from our upstream services, those are mocked out for the more consequential ones. And for our federated GraphQL, we've got a proxy in place that will automatically authenticate. And get how we don't really need because we've already ingested everything into the database. So now we have version of Dev Hub that folks can use that is very close, 99.9% the same as what is in production. So that would give our subject matter experts or extensibility partners confidence that when they go to integrate with Dev Hub and then we go to deploy to production that everything is gonna work. But this Dev Hub Lite was a huge success for us and it enabled a lot of folks to get much more familiar with how everything worked together and let them use common tooling. They could use VS Code or IntelliJ and put interactive breakpoints in instead of again working in a silo or in isolation. But not without its challenges. So we added some processes to help funnel people in and we added some tooling to help reduce friction. So now we have another set of challenges that we need to consider. We're now dealing as the Dev Hub Core Team, we're dealing with code that's coming from disparate sources from different teams of varying skill levels. They might not know how backstage works. They might not know about best practices for TypeScript and performance and might not know the difference between a back end plugin and a front end plugin and might accidentally include credentials in a front end plugin. We also need to ensure that the UI and the user experience is consistent. As I mentioned, we don't want this to be like the NASCAR car with all of the stickers on it. And we also need to consider support. Who's gonna support this? Not the Dev Hub Core Team. We don't know how these plugins are working or what they're talking to. Maybe they're talking to some upstream API and what if that thing goes down? We have no control over that. It's not something that we maintain or control. So we need to make sure that we are routing people to the appropriate sources for support if and when they need it. So to address the first one, best practices, what we did was we stood up a scanning service. And this is just a service that we wrote that can do some static analysis on the code that from the extension, our extensibility partners repository. It's a simple, it's a little CLI that can run. It'll send the code off to be scanned and it'll come back with errors warnings or it will let them know that everything's okay. Everything's okay. Can actually also package it up into an NPM package and help deploy it to an NPM registry. Also, we have a corresponding CI tool that does the same thing. So this allows us to insert ourselves into the development pipeline. It gives us at least a place where we can do some checks. It's not perfect, it's ever evolving, but at least we have now a little bit of control and as things come up and when we note different implementations, we're like, well, when we didn't like that, let's make sure that doesn't happen again. Now we have a place to put that onto the consistent UI. Again, this is a big one, we want it to feel like a cohesive product. So we created a Dev Hub UI library, which is basically just a wrapper around material UI, but all the components have been modified in such a way that we can enforce what we want in as far as the UI is concerned. We wanna make sure that the people aren't using make styles, for instance, they're not doing inline styles. This has the added benefit this UI library has and added benefit of being like Lego bricks where you only have a small bag of tools to use. You're not overwhelmed by, well, here you can do absolutely anything. Instead, it's here's your set of tools, now you can build a Dev Hub plugin, one that's gonna look and feel like it's part of Dev Hub. This provides a nice abstraction and allows us, again, a little bit more control there and enforces best practices as much as possible. What's cool about it too is that we can now use that scanner that I talked about and ensure that anyone that's creating a front end plugin is using our library, the UI library. If they're using Material UI directly or some other UI library or we detect inline styles, we can flag it. Now, it's not necessarily a, nope, you can't do that. It might be a, well, if you wanna do that, let us now make a PR to the UI library and we can add it there. It's not like, never, you can't ever change anything. And speaking of changing things, if we wanna change the UI, all we have to do is update this UI library and now all of the plugins that were built using it will benefit from those same updates. Now, to support, this is a big one. I think it's one that might get overlooked by some teams, some organizations. We've now started to democratize the developer experience. We're having our subject matter experts or extensibility partners that are coming in and they're building all these really cool things. But again, we don't have control over it and that's what we're trying to exert some level of control with the scanning service and whatnot, but now it's in production, now it's out there. What happens if something goes wrong? Well, all of our front end plugins are in a plugin boundary. And basically what that will do is it will provide some additional metadata on the front end that says, hey, here's some plugin support. Just so you know, this is a plugin. Here's who's responsible for it, how you contact them. Here's their Slack channel and so on. That will pop up in a modal dialogue and that will also be an opportunity to provide feedback for that particular plugin. So we can start to gather some information for our four-hour extensibility partners. Like, hey, we're seeing, this is the feedback that you're getting. And then they can take action and improve their plugin accordingly. We also similarly have an error boundary. If an error occurs, we'll say, sorry, an error has occurred, please contact this team. Please don't contact us because we're not gonna be able to help you. And then if things go catastrophically wrong, we have feature flags, we can turn things off temporarily if we need to. Again, we have all of these things coming in from different sources, which we want to enable, we want to encourage, but we need to have some level of control over our product. So I'm gonna bring this all together. I'm gonna kind of summarize where we're at and see how this democratization can lead to a better developer experience. So we have our subject matter experts. We want to leverage them. We want to avoid our knowledge silos. We want to avoid that one-to-one communication. We want, let our SMEs build these really cool things and share their knowledge with everyone, not just one person at a time. We want to reduce friction. We want to invite our subject matter experts in. We want them to want to contribute. It should be an easy process. It should be a fun process. Let's make the developer experience awesome and amazing. We should want to do it. It shouldn't be a chore like, oh my God, no, I've gotta do dev hub stuff, it sucks. And so we need to respect our subject matter experts time, right? Let's let them get on and do some cool innovation stuff and let the developer experience kind of be just part of the process. We want to do all of this too while being thoughtful about the product. We want to have thoughtful contributions that make sense in the overall developer experience. We want to, again, to have a cohesive product and we want to do all of this by guiding folks with some unobtrusive guardrails. Basically, we don't want to waste the potential of our subject matter experts. We want to empower them. So if we think back to the dark warehouse, that's the first part of this picture on the right here. That's all the people at the top with the question mark. Well, that's all the people that had to deal with that dark warehouse. How do I find things? Where are they at? Okay, now you've led me through. Now I know where this thing is. How do I find the next thing? So on, it's friction. It's time spent where we're not building business value or we're not innovating or waiting. Not ideal. We started the democratization. Now our developers are in the center. They can see everything. They can access everything. They can self-serve. They can onboard. They can do everything that they need to do. They can see signals from various different systems. So you've got an observability system. We don't want to duplicate the Grafana in Dev Hub, but wouldn't it be nice if at least there was a status indicator of okay, warning, critical? And if you can click on that and then go in and see the dashboard for more detail. But if our developers are in Dev Hub, they can see that right away and act on that signal. So basically what we've done, gone from that dark warehouse to simply turning on the lights. That's what I have. Hopefully it's been useful and helpful to see how we've come to democratize developer experience and the journey that we're on so far. And it's one that we're continuing on and then continually evolving. So again, my name is Ryan Emmerley. If you guys look at the QR code, if you want to provide feedback. And all of my contact information on the bottom, if you want to get ahold of me. That's what I have. Thank you. Thank you so much, Ryan. Any questions for Ryan? One, two. Thank you, Ryan. I work for Adobe, our backstage, we call it Dev Home. Oh, nice. So I think we should talk. Yes, very nice. About Dev Hub Lite, is that an environment that gets built up for contributing teams, trying to contribute to Dev Hub? Like is it something that is short lived and per pull request, let's say, or? It's actually a local environment. So you're just pulling down the repository and it's just the configuration that dictates where to point things. So we use a make file to have all of our actions. So basically when a user comes in, they want to run Dev Hub Lite, they can just do a make run Lite. What that does, that will start up the Docker compose set of containers for that sanitized production database, some of the mock things and the proxy. It gets all that started up. And then it's just on their local workstation. That way you can iterate really quickly and not have to worry about deployment. All right, and my other question, are you, the repository where Dev Hub lives, like the code base, is that a monorepo? So contributing teams contribute directly to that repository or are they managing their plugins in like a multi repo kind of setting? That's a good question, yeah. So we have our developers, extensibility partners, whatever, they build their plugins separately in their own repositories because they own them. And then they build their, the NPN packages and publish them through a registry and then we pull them in as part of the integration process. Thanks. So not to sound like an interview question, but I'm over here. Okay, thank you. But once you decided to actually start expanding into asking for more contributions to the experience, what would you say your most technical and your most non-technical challenges were? The technical challenges are ongoing. It's difficult to have, you know, we're a small team and ultimately we want to review all of the code that's coming in. That's kind of what is unique about this is we've got these teams creating these plugins which is code that's ending up in our product. But we're kind of a little bit blind there. So that's why we've had to put in some of these processes and these technical, like you know, the scanning tools and whatnot. But so that's still kind of an ongoing thing. How do we ensure quality? How do we control what's coming in? Non-technically, I mean, just getting people to understand and see the value of Dev Hub. Again, we have, these folks are very busy. So telling them that this might be beneficial to you to add this functionality into Dev Hub is kind of a, not a hard sell, but it's one where you're gonna get a little bit of pushback at first, where they're just like, I don't really wanna do any extra work. Thank you very much. So yeah, so that's where we're at. But I think the biggest one is still the technical one because I'm kind of a control freak and it bothers me that I don't know what these repositories have in them, but that's me. Hi, I'm curious how many contributions you've actually had since like launching the program? Sure, yeah, the program is relatively new. We've had half dozen so far, so half dozen teams integrating their functionality with ours and it's been very successful. I mean, we have our CI CD team, we have a plugin which we're gonna be contributing back to the backstage community because one of our CI platforms is Concourse, which there is not currently a plugin for. So we have got a plugin for that and yeah, I've lost my train of thought, but hopefully I answered the question. Great talk, thank you. Over here on your right. Hello? Hi. I was curious about your choice to use backstage as like a general action runner, as well as just an initial setup. We kind of considered that and were nervous about trying to shove too many things all into one tool. Did you have any reservations? Are there any gotchas or is it something you isn't unalloyed good? Yeah, that's a good question. So the thing-do-or-action-runner part, I think it comes down to how you organize those. So we have, I forget what backstage calls it, but we call it the marketplace where you can see all of the templates there. And with Dev Hub Lite, we also added some functionality where you can point the template discovery thing to a local file system. So that also makes it easier so that people can very quickly iterate over their templates. There's also the template editor that's built into Backstage, which is super nice, but it can only do things that allow dry runs. So having it running locally really helps there. And trying to control the complexity of that so that there's not so many things in the marketplace that you become overwhelmed. I think that's gonna come down to how we organize that. And that's gonna come down to, we're gonna start to implement feedback and starring system where things that are more popular start to rise to the top to help control some of that overwhelmingness of things that come in. We're also thinking about, well, we're also going to be building a interactive template builder where you can just do it all from the UI. You don't need to write YAML so you can drag and drop UI components, even the custom UI components, drag and drop the custom actions that are available to compose those things. Because we wanna enable these, not subject matter experts to be able to come in and not have to learn all of the intricacies of template YAML to be able to do things. And I think that's really gonna open it up so everyone can come in and contribute. Even folks that have just been are used to Microsoft Forms should be able to come in and create even more powerful forms that can actually do something, not just collect data. Thank you. Thanks. Do we have time for another? I didn't think so. I'm sorry. Yeah, that's a wrap. That's a wrap. Thanks. Again, if you need me, I'll be around. You can talk to me. And also my contact information, you can...