 Mae'r dra也可以ch ar ein chylyfan bearla, le mae'i fyfans breakfast, a'r bodan baeth arbennig am unrhyw Majesty 2023. Felly i eitr dweud ond yn cramldi. Mwdech chi. That's perfect. The audience is very, very happy to be with you. So, thank you to the audience for being with us in this new project. If you're new, I appreciate you being here and for the support. I'm going to go in and back, thank you. I'm just going to go in and go back in and do this in a different way. I want you to see what the audience thinks. So, thank you. Thank you for being here. Thank you. Thank you. Thank you. Thank you so much. And thank you so much to you for being here. Thank you so much, and thank you too for being here. hachwych, yna y gallwn i gyd yn ymgyrch o'r bwysig o'r bwysig o'r gweithio'r bwysig, a'r bwysig i'n rhoi'r fath ar y bwysig yw'r gweithio'r bwysig arall, boeddwn ni'n meddwl i'r rhywbeth o'r fath o'r companyau arall, nad yna'n mynd i gael ymlaen o'r bwysig. A'n gweithio, os ymlaen o'r bwysig, mae'n gwybod yn fwy o'r bwysig, oherwydd mae'n mynd i'n gwybod. I dwych, roedd y cyfrifiadau cyrraedd cyfrifiadau yn gweithio'r project yma CNCF, yn eu hwyl o ddodd yn cyfrifiadau, ac yn cyfrifiadau cyfrifiadau. Yn dwylo'r cyfrifiadau, mae'r grwp hefyd yn gweithio'r grwp hwnnw, yn y dyn nhw, yn y dyn nhw, ac mae'r argynwys cyfrifiadau yn y project, ac mae'r edrych yn cyfrifiadau. Mae'r cyfrifiadau cyfrifiadau ar 100 o'n cyfrifiadau yng Nghymru yn y dyn nhw'n cyfrifiadau cyfrifiadau. As we've just seen, the project has grown a lot, and this has also led to some challenges, in particular how we scale development and involve the community in the project. As part of tackling this, we took a look at evolving our project governance at the end of last year. Our original governance structure was a single maintainer team as many projects start out. Over time we've had more teams at Spotify that came and helped build out different parts of the project. Of course, a huge number of contributors from all around the world, anything from students doing a course on open source to whole teams in large organisations. Some of these contributors have been around for a long time, they know the project very well, and they want to help out more. With our existing governance structure, it was really tricky to help empower those of you that want to do that. This is another big reason. We went back, took a look at our governance model and how we could update it. What does the new model look like? At a high level, what we've done is introduced the concept of project areas with isolated ownership of different parts of the project. We've also added a contributor ladder, which had more roles between contributor and core maintainer. This is all based on the contributor ladder template from the CNCF Tag Contributor Strategy. Shout out to them as well. They've provided some very valuable feedback as we were going through this process. First, let's take a look at the contributor ladder. This is less of a ladder we think everybody should be climbing and more of an order of progression for someone looking to get more involved in the project. We start out at contributor, which is a wonderful world of no responsibilities or requirements. All you need to do is adhere to our code of conduct. This section of the governance is more about just highlighting the different ways in which you can contribute to the project and get involved. Once you've been contributing for a certain amount of time, you can become an organization member. This is a lightweight way to get more involved in the project and become a member of our GitHub organization. It's a way of showing more interest and also requires you to stay an active contributor. Moving on to plug-in maintainer. We have the first role that has direct ownership of code in the project. If you contribute a plug-in that we think is a good fit for the backstage project itself, then you can take on maintenance of that plug-in, assuming that you're already an org member. This is also fairly lightweight, but it does require you to commit to maintaining this plug-in that you contributed. Now, mentioned project areas. Naturally, we do have project area maintainers as well. This is a role where you make a larger commitment to the project, where it's expected that you spend a certain amount of time working in that project area. This role might look a little bit scary, but it's intending to support a wide range of experience and for maintainers in one area to work together. Also, as we onboard new project area maintainers, the core maintainers were here to help out and get going. Last, we have the core maintainers. It's the new name for what we previously just referred to as maintainers. As a core maintainer, you're expected to spend the majority of your working time towards the project. You're expected to facilitate communication and solutions across the project areas, and we're also the default owners for anything that doesn't fall into a particular project area. Let's look at project areas. We didn't aim to split up everything into project areas from the beginning, but instead continuously worked to find new areas and split things out wherever possible and where it made sense. Initially, we have the catalog discoverability tech box areas. What I'm really excited about is that we now officially have maintainers from other organisations than just Spotify. More specifically, we've got Andrew and Tom from Red Hat joining Vincenzo as maintainers in the Helmcharts area, and Jamie from VMware has joined Matt in the Kubernetes project area. We'll keep expanding this list of project areas, of course. Next up, we're working on separating out the permission system and that part of the framework, as well as software templates, which Ben is going to talk about soon. If you have something that you think should be a project area and that you want to drive, come talk to us. Also, if you've been contributing to Backstage and want to become an organisation member, you can head over to our community repo and open up a request. Now over to Ben for software templates. Thank you. I'm going to go through some software template stuff now. Before I get to what's new in software templates, I just want to give a quick recap as to the why of software templates. Backstage has been all about solving three main things from the start, which is to create, manage and explore. Being able to create new things inside your organisation, the managed part, so being able to see everything that you own or are responsible for, and the explore part of being able to see everything around you inside your organisation, like from other teams' perspective and what they've got running. Software templates is all about the number one in that list, which is to be able to create new components inside your organisation. I guess why. Every company is different. Every organisation is different. Look around you here at KubeCon and the explore whole, the amount of companies that there is that's solving for debugging, login, tracing, all these different technologies. Every other organisation has got a different combination of these. What we want to do is make that more simple for people and engineers to get set up and get running. Because onboarding in any company is hard, and having to have this knowledge of getting set up and being able to create a new project is a challenging task for any developer, like new start or not. What we're trying to do here is give engineers the ability to create new software components with all of your best practices built in and remove a lot of the manual steps and knowledge needed to be able to get something running. This is what one example of software templates might look like. It is plain, right? Yes. A template author can ask for bits of information that's needed for them to create a new software component inside their organisation. So in this particular template, we've got some accompanying boilerplate or like blueprint templating files, which are going to become the foundation for the new repository that we're going to create. The information that we collect here from the user is going to be templated into these files, and then our new repo is going to be pushed and ready to use. And it's these blueprint files that you can include yourself or your build tooling, your logging or whatever is needed to create components inside your organisation. So this example here is just one that we ship out of the box with Backstage, but we've got a few more like heavyweight examples, things like Create React App and Next.js skeletons and things like that that you can go and check out, but this is just one example and I'll come onto some more use cases later. So how does this all work and what do they look like under the hood? So I mentioned that creating repositories is just one use case and this is kind of what I want to focus on here. So this is what you can see that the Node and Next.js skeleton looks like. So we've got a template YAMO file and a folder called Skeleton which contains all of the boilerplate files that I was just talking about and that's the image on the right-hand side that you can see. And you'll see there's like package.json files and basically a structure of what's going to be a very simple, very empty Next.js service. So you can either choose to house all of your software templates in one big monorepo or you can break them out into smaller repositories with different ownership or a hybrid, which is kind of what we do at Spotify. So let's take a little peek into the template YAMO and this is where we'll come onto the two main concepts of software templates which are the parameter schema and the step definitions. So this is the parameter schema and this might look familiar for anyone that's used JSON schema before and that's because it's exactly that. It is JSON schema, just YAMO. And as you can see from the sample YAMO here and the image on the right, we take this JSON schema definition and then produce a form which is going to render an input field for the user to satisfy a bit of that JSON schema. So if you imagine you want a bool in, you're going to get a checkbox or if you want a string you're going to get a text area. Shout out to the guys that maintain this, which is the RJSF core team, React JSON schema form. And obviously there's a lot more opportunity for more advanced things here but I'm just going to cover some of the simple use cases. The other thing to notice is that this is all React, right? So you'll see the UI field part where it says entity name picker. So that's just a custom field extension which we'll get onto shortly but this opens up the ability for integrators to render any custom React component that they choose. So instead of providing like a simple text box you might want to provide a React component that can call out to an API to get your Kubernetes clusters or something, then the developer can choose that cluster to deploy into it for example. And then moving on to the step definitions. So this is the main meat of the software template. So this is basically what's going to happen when the user corrects create and this is the pipeline of steps it's going to go through. So steps are basically an array of actions that are going to take place in sequence. And here you can see we've got three of them. We've got the templating step, the publish step and the register step. So you can think that these are just basically typescript functions behind things and they're just going to call them with the input that you get in the animal. These are also super configurable too, like you can provide your own actions that can do anything you like inside your organisation. They're just typescript. So just to recap a little bit here what you can do with software templates. So you can write your own blueprints for templating to provide any, like to create any form of repository, whether that be declarative infra, the data pipelines or just something simple. You can create custom field extensions which are going to enrich the experience for your engineers when filling out the form. And of course you can create your own actions to do any additional logic that you might want to do when running these workflows. So that's enough of the recap. Let's talk about what's new. So I should mention that scaffolder plug-in, which is what powers this whole software template story. You might have heard references to scaffolder next, scaffolder alpha and stuff in the open source report and it's all the same thing. It's just a different name. What I'm going to show you is available for you to test and try out today. I don't much appreciate if you're going to test it out and try it because we're going to ship it soon and we don't want to make sure we break anything. And a lot of the work here that we've done came out of an RFC that I actually posted around basically improving the test coverage and the way that we can test the scaffolder itself because there's more interest grew for the software template, sort of the amount of pull request and contributions that came in and it was getting harder and harder to ship things and the new features that people wanted. So we spent some time evolving the UI features and also refactoring and writing some more tests to give us more confidence moving forward. It also allowed us to ship some new features which we're pretty excited about. We bumped the underlying library that I just mentioned that does a lot of the drawing of the form for us so we can do a little bit more complex things with the new JSON schema draft like if then else and things like that so hiding forms depending on input from another field depending on another input from another field for example. Async validation for custom field extensions and embedded workflows which I'm about to show a demo of. I think it's now the demo. There's a chance that this might not work but we're going to give it a go. I just need to join to my personal hotspot because the wifi here is a little sketchy. Let's go to here. Let me just start this and let's bring up this. What I'm going to show you first is the Async validation for custom field extensions. I just want to bring this back here. Can everybody see this? By the way I didn't need to make it a bit bigger. Shout out the back. Is it okay? Bigger? Okay. One second. Now my screen gets even smaller one second. This is going to be fun. This is fine. Now can you see this right? Hold on. I'll say what I'll do. I will make this full screen. Is this better? I'm just going to do it one screen at a time so it's not going to see side by side but it's fine. This is what a custom field extension might look like. It's just a React component and we're rendering a little input label here and the actual input field. This is just movie components. I can just try and do div like hello in here. We already have some templates loaded into backstage for us to try out so I'm just going to go to the create part here. I'm going to come back to the tech.spoilerplate thing in a second but I'm just going to click on this example template here so we can see we have our little hello thing. It's rendering the extension. That's good. We have our input field here and right now the version that's currently released to the public, we can only do synchronous validation here so let's think like regex and things like that but what happens if we've got some async service that we want to call to go and make sure that this entity that we're trying to create here doesn't exist already and we've got the catalogue for that so what we're going to do here is add some validation to the field extension so basically call the catalogue to make sure that the value that we're providing here doesn't exist already as an entity. I'm just going to go back and head here so we can see here that this is the definition of a field extension in backstage and we have this validation function now right now this does nothing there's a few arguments that get passed into this validation function one of them is the actual value that we have and the other one is going to be the error context or the validation context so I'm just going to call this validation oh my god I've fat fingers so this is going to be fun right so we have our validation function I like the co-pilot that's trying to help me out as well here this is going to be fine but we're going to just go ahead and basically make a call to the catalogue to make sure that this thing doesn't exist already so there's another argument here which is going to be the API holder so this is going to allow us to get access to any of the APIs in the API context in backstage and one of those is the catalogue API so I'm going to go catalogue API equals APIholder dot yet catalogue API and that's not how we use that but let's do this even nearly right co-pilot is nearly right okay I'm going to import this this is going to be that that's going to be great and then what I'm going to do is I'm going to go let's do const entities equals a catalog API dot get entities and then we have our filter and this is going to be a can component so we're going to check for all the component kinds and we're going to see if the value of the metadata on name is the value and then if the entities is more hey it's getting better now it's getting better and then we do a validation dot add error like this entity already exists so this is going to be fine this also needs to be an async function okay I'm hoping that this is going to work items there we go thank you typescript alright so now we're going to go ahead and try this so I just want to pop back to the catalogue for a second here so this is like some of the demo and demo entities that we've got so we have example website is one entity here so in theory if I go back to this and I type ben it goes next that's good if I type example website I should get an error let's try no well that's fine it does work please promise this is fine right we can do some debugging together and this should work right what I'm going to do is I'm going to not trust it I'm going to do a reload and I'm going to try it again example website yay there we go right that's good alright so what I'm going to do now is I'm actually going to go and create a entity using this template so let's go let's call this ben cube contest and let's just give it a simple description yeah we don't need this but anyway and then I'm going to just put this in my test repo which is in Blam demo and we're going to call it cube gone one two three four something that I've not created before so it breaks and then we go here and we go click create so this is now going to run the workflow for us it's going to go and fetch these template files go and create a repo and go push it for us and then go and register it in the catalogue so we can see at the end we've now got these two outputs and one's the entry in the catalogue so I'm just going to go I can just show the demo of the repo really quickly that it exists it does real it's a real demo but then I want to go back to the catalogue so I want to talk around the second use case of the scaffolder I want to give a big shout to Paul from Frontside here because he's driven quite a lot of this work for embedded workflows and it is cool to show on demo so I want to pop over to the docs part here so just a quick show of hands has anyone used backstage ever seen a screen like this before yeah quite a few so how we basically how we intertwine plugins and connect plugins to entities by annotations right so this is saying that when I want to use the tech docs plugin that I don't have this annotation in my entity now it's a little unfortunate right because we know what is expecting it's just that the plugin doesn't have the ability to do this but there's something that's like really good at editing files and creating repos and pushing repos and pull requests and it's a scaffolder right so why don't we try and use a scaffolder to basically do some of this work for us so what I want to do now is I'm just going to pop over to the entity page which is here and uncomment some code I don't have to write anything this time which is nice so I'm just going to uncomment this so what this is going to do is it's going to use the embedded workflows that we've been building in the software templates and I need to select all this and that one now there we go what this is going to do now is it's going to basically check to see if that entity does not have that annotation and if it doesn't it's going to provide us with a form that's pre-populated and this looks very similar to the scaffolder and that's because it's basically an embedded workflow and it's going to pre-populate it with the information that's needed and then what this is going to do is it's going to run a workflow on top of the repository and that consists of going to grab the repository grab all the contents of the repository run a transform on the top of that to go and add that annotation to the catalog catalog info YAML and then it's going to raise a pull request back so I'm just going to go through this we're going to run this in place we're just going to create this workflow it's going to go through this you can see it all running now it's going to go and fetch the entity go and add the technical annotations to the YAML that's on that repository that we just created so I'm going to go over here to the pull request, we can see it I am not going to go about waiting to build I'm just going to merge it, it'll be good I'm actually going to probably show you what it's created first before I merge it but anyway, here we go so we can see that it's added the annotation for us and it's also added some sample documentation for us to get set up with tech docs so now if I go back to this if I give this a refresh now tech docs is set up automatically for us now so this is just kind of a specific use case for tech docs but you can see how it can be applied for anything we can go and add we can go and transform people's repositories and go and add code for people so they don't have to go and do this manual management we can just delegate this stuff in terms of workflows and by that it's just generated some documentation for that entity and that is embedded workflows I am going to hand back to Patrick now actually no, I have another slide one second actually no, I don't get to the pic there we go, sorry one second where's the slides gone where are you sorry one second, I'll just open the slides again on this one, yeah they are thank you thank you, hold on I just need to click this button how do I try it out so there's a link here go ahead, there's some documentation that's also set up here for you to get set up and running with the new scaffolder please go and try it out, we want some feedback on it and before we ship it because it's getting kind of soon cool, and I'm going to hand you over to Patrick now to talk about framework updates shout out to the demo gods alright so recently our focus has been on sorry I know where I am I'm going to talk a little bit about how we have been evolving the backstage core framework so recently our focus has been on evolving the backstage backend system at the end of last year we did a talk at backstagecon where we presented our plans for a new system we mentioned our goal was to make it available at the beginning of this year and we ended up shipping the alpha in February in our 1.11 release now for a deeper dive into this new system you can go and check that talk out or if you prefer reading we now have quite extensive documentation for the new system I will give a summary of that talk our original backend system was grown out organically with the rest of the project unlike the frontend we didn't really have that much of a framework it was more of a collection of patterns for how to build plugins the way you install a plugin was up for each plugin to define themselves although there was a common set of services for plugins to use to keep things a bit consistent now this isn't very scalable it leads to a setup that is hard to maintain for both adopters and plugin authors so to address this we set out to evolve the backend system with a couple of high level goals we wanted to simplify the way you install and integrate plugins into your backend we also wanted to make it much simpler to evolve and extend both plugins and the framework itself without breaking consumers and we wanted to do this while keeping the system recognisable both if you're familiar with the backstage frontend framework but also relying on the existing patterns instead of inventing from scratch now let's look at a concrete example of how the existing system can be less optimal in this example we want to install a new backend plugin so the way you install a backend plugin backstage this means you need to copy paste a bunch of code that's not what we're looking at in this example what we're looking at here instead is if a plugin that you're installing needs a common service that you don't already have integrated into your backend so to make this service available first you need to add a dependency on the package which is what we see here then you need to add a service interface to your plugin environment for everybody to recognise the service then you need to wire up the implementation of the plugins to use so this is manual dependency injection going on here right and then lastly you have to forward the service from this environment to your plugin that you installed so it can use the service now this is a mess and it doesn't even stop here because of this complexity many plugins added fallbacks in case the service wasn't available so this both makes backends harder to maintain but it also makes the plugins harder to maintain too so this is really one of the things we wanted to improve as part of this new system so what does this example look like in the new system well for one we have automatic dependency injection your backend code is no longer responsible for doing all of that wiring instead that happens automatically under the hood we also introduced something we call default service factories which lets you have plugins that provide the default implementations for a service so in the end all the changes that were required before are now handled by the plugin and the framework itself and none of the code that I just showed is necessary in the new system so here's the new architecture quick high level overview there is a backend instance which wires everything together in the backend you install plugins like the catalogue or the scaffolder that Ben just talked about they communicate with each other they do that through network calls so it's microservice architecture essentially and this of course allows for a distributed deployment of plugins and backends so to extend the functionality of plugins you can also install modules so for example you can install a custom set of actions for the scaffolder modules run in the same context as the plugins but they communicate through an extension point which is an indirection to exist to make sure there's a loose coupling between plugins and modules making them both easier to evolve over time and in the middle we have services they exist to make it easier to build plugins and modules so you don't have to implement all functionality from scratch this is things like logging, database access task scheduling so on both plugins and modules can use these services and the backend instance can inject custom service implementations if needed so here we have the code for a minimal backend setup in this example we just have a simple backend instance where we install the scaffolder plugin along with two modules that provide additional custom actions for the scaffolder so it's just one plugin but you can imagine the middle section of this example here repeated for each plugin that you want to install I omitted the imports at the top in this example to keep it brief now if you know what the typical backstage backend system looks like you know that this is a lot less code to manage compared to what we have today and we know because we counted down from nine separate files of 348 lines of code the equivalent is now just 26 lines of code this gives you an idea of how much simpler it will be to maintain a backstage installation once we move over to this new system and we released the alpha earlier this year and you can expect a stable release in well fairly soon so what are we doing next now 26 lines of code is nice but it's not as nice as zero so this is what we've started working on now it's something that we call declarative integration this is a feature that has been requested many times more or less since the beginning of the project the core of the idea is to be able to install plugins without the need to modify code declarative integration is now at the top of our roadmap what's the scope of the work we don't know the full scope yet but we know where we are starting what we're working towards is to make it possible to install plugins without writing typescript like I said and we're doing this for both the front end and the back end now this is still very early work there's a lot left to do and we're keeping the initial scope very limited and there are already plenty of tricky problems to solve either way we're very excited for this new direction of the project and what it's going to unlock for us in the future and that's it hopefully we have time for questions thank you does anybody have a question do we have a microphone or do we have to run with microphones we can repeat this yeah so what's the question the question was tech docs is there any plans to change that so there is I've seen some issues around people wanting to add more support for just markdown and make docs especially I'm not really sure where we're at with that issue I know it's been explored but I don't think I would say that part of our governance updates is that the tech docs area is not something that we're leading or very involved with to check with them and see what the plans are