 Thank you for joining us today. My name is Summon, and this is Jamil. Jamil has been an engineer on the Bosch team for more than two years now. And I've worked across a variety of Cloud Foundry projects, as well as a few core components of Bosch. This is us. So we want to talk to you today about an interesting problem that exists in the Bosch universe. Imagine the following. You have multiple massive deployments, properties, variables, and links being shared all over the place, properties being shared through links, variables being generated left and right, certificates being presented, and flying everywhere. And in the midst of all of this, an error occurs. And you are the one person that they have decided is responsible for fixing this problem. It's all on you. The question is, how are you going to do this? So we are here today to help you solve this problem. Ideally, by the end of this presentation, through the visualization of Bosch links and other Bosch components, you will be able to solve this problem. But before we get started, let's talk a little bit about the history of links. Links were originally created to solve the problem of sharing data across components in the Bosch manifest. They were also created because static IPs had to be assigned to all the components in the manifest. So for example, if you consider the case where you have an app client, a server, and a database, the client needs to know the address of the server, the server needs to know the address and port of the database, and the database needs to expose the username and password to the server. Originally, all these components were defined in the manifest, and there was a lot of duplicate information. And static IPs were assigned to every single component. Tools like Spiff and Spruce were created to help reduce that so that they would automatically generate that data and insert it into the templates in the manifest. But they created dependencies and large deployments were still tricky and hard to manage. So these are some of the problems that links were hoping to solve. But what is a link? And quite simply, a link is a connection between two components, a provider and a consumer. A provider and consumer both define the type of link that they are consuming. And Bosch, at runtime, resolves the connection between the two, creating a link. So an example to understand this is when you try to understand the connection when a database is providing a link to a server. So a database is providing a link connection of name and type con, and it's exposing the property port. And the server is consuming it. At runtime, in the job spec, Bosch will resolve that property for you during deployment. They also solve the problem of sharing addresses between links, because when the link is provided, again, in this example, the database is providing the con link to the server, the job spec resolves it, because the link itself contains the provider instance address. So the job spec is able to resolve it. So now that we have a basic understanding of why links were created and what they are, we want to get a little bit into the latest features that we have developed around links, the first of which is the links API. So if you have a deployment that looks kind of like this, you have a provider that's being provided. The links API provides a way to get all the entities that are being provided in a deployment. So this is the response you get, and this will be able to define, like, hey, there is this link. In this case, it's defined in the release spec with name and type peers and zookeeper peers. It lives in the zookeeper job in the servers instance group. It's aliased to custom peers, and it's shared across multiple deployments. So on the left, you have the actual definition of a link provider. Like, this is how it exists. The link consumers endpoint lists all the entities in a deployment that are able to consume a link. And the entities which can consume a link are a job, a variable, and a custom external consumer, which we will get into in the next few slides. So again, querying the link's consumer endpoint is able to give you the definition of the link in the release spec for this example. And it defines, hey, I live in the Nginx job in the app server instance group. We have also created the ability for you to create a link. So now when you create a link, you are going to be creating a link as a third party entity. So you specify the provider that you specify your provider and you set your type as external. So when Bosch resolves this link, it will find the provider with type external and then be able to resolve it. So it's very important that you set your type as external for this example. The response of this request gives you back the generated links ID, which you can use to query and get the link address. This becomes super useful in the case of service brokers when they're trying to resolve a link's networking information to generate the Bosch DNS address for the service instances that they're creating. So super handy for that use case. We've also created for you custom providers. And a custom provider is only really different from any other provider in the sense that it only exists in the deployment manifest and not in the release spec. A custom provider can still be aliased. It can still expose properties. It can still be shared. The only difference is that it only exists purely in your deployment manifest. The purpose or the reason for this existence comes in handy in the next slide when we talk about variables. Because now we've added the ability for variables of type certificate to consume links of type address. So in this example, we have the app address job, providing a custom link of server app address of type address. This will be consumed in the alternative name down below in the variable. So when this variable is generated, the sands of that certificate will actually contain the Bosch DNS address of that job. It's really cool. It's also a very important problem. So taking a step back for a moment, we now understand what links are, why they came to be, and a little bit about the latest features that we've developed. But we still have that same problem. We still have that massive deployment. We still have properties, links, and variables flying all over the place. But does it still have to be a headache? I'm going to let Jamil do a demo of the links visualization tool and give you more information. Thank you, Simon. Hello, everyone. My name is Jamil. As Simon have mentioned, I've been in the Bosch team since almost two years and a half right now. You probably, if you're interacting on the Bosch channel, you'll probably see me as Gargamel avatar from time to time. So that would be me. Yep. So hopefully today, we're going to demo this tool. And can I just get maybe a quick survey? Who uses links here on a daily basis? Just can you raise your hand? Cool. OK. Who have some times trouble understanding whenever you have a big deployment? Let's say CF deployment or any other deployment. Who's consuming what? Who's providing what? Cool. OK. Hopefully this will help. So let's go do the demo. Before we start, let's actually take a look at maybe a real problem. Let's take a look at the CF deployment manifest. Just that's the open source one probably most of you are familiar with. And surprisingly, or maybe not surprisingly, it is 2,133 lines of fiat. It has so many jobs. It has so many links. It has so many things, so many variables. It has 17 instance groups, 90 variables between certificates and passwords that gets generated, 29 releases. And just good luck figuring things out sometimes. It gets a bit challenging if there's no, if you don't have previous understanding of the internal components before you even look at it. So this tool was created like a few years ago. Just the one we look at it. It's a static analyzer of the Bosch manifest. It's just a website, just some JavaScript. It will try to parse it and then display a few components for the Bosch manifest. But still, it's going to tell you a few things on it. Let's say the variables releases instance groups, but it doesn't really tell you, let's say you have this job and it's consuming this thing from another deployment or from this deployment, which currently actually there's no way you can tell that in Bosch. It's just like basically you're running blind. So the tool we're going to experiment with today is it started as something called the Bosch extension CLI. It initially started by the previous Bosch PM, Dimitri Kalanem. So he created that tool. It's just a Go tool created so that it can help him debug different Bosch environments and parse some logs. So what we did is we took it a step forward and we added a web component to that tool. So we forked it and we added a web component of that tool. What does it really do is we made it very simple for any kind of dependency just to work for this Bosch extension CLI. If you're using Bosch, then you probably use the Bosch CLI and you have, let's say, all the environment variables ready in your shell. You just run that tool in the same shell and it will automatically shell out to the Bosch CLI and it will use those exported variables. So it would be very simple if you're using the Bosch CLI just to add to use this tool. So for example, I'm going to just start now. I'm going to stop it and let's run it. I'm going to just build it and run it and then the command is visualized for this Bosch extension. What it will do is that it will create a local HTTP server and everything is contained in that binary and we're going to just serve that some content locally. And I want to just mention one thing. Everything that you see is on your local network. It's not going outside of your machine. So that would be helpful later on whenever we're dealing with variables from a security perspective. So let's go ahead and then click it. So initially when we created this, we added as if at the beginning a quick way to visualize some components of Bosch and then links came over and then we added it. So at the beginning, let's say it will have, let's say how many deployments you have, how many running tasks, how many releases, just basic stuff. So if we look here, it will just have all the list of deployments. If you click on it, just usually this is what the Bosch CLI will show you and then you can quickly see all the releases. Let's say used by this deployment, one CF or let's say by cloud cache. So just a quick thing, just let's say if you're new to the environment, I want to just quickly debug something, what kind of deployments without typing too many Bosch commands. So till now it's not that different from the actual Bosch CLI. So let's take it to the next step. This one is the Bosch releases. And if you use Bosch, you know that whenever you list the releases, it will tell you it's being used but it will not tell you by which deployment. You need to do some gymnastics to find that out. So this one actually behind the scene, it's doing some aggregation. It will list all the versions of the Bosch releases you're using. And let's say we have binary build back. It's used by CF deployment. And Bosch DNS is used by all those deployments. Let's say it's used by nine deployments. This way whenever you have a big environment and then you want to quickly just glance over all version of releases and then you want to see which deployments are using those. And just quickly find out. And this is mostly to help the operator. And we added a couple of more before we get to the links which is the meat of the stock. I'm gonna just go through them quickly. So here we click on the task. We are displaying all the tasks of the director. Just whatever you get from the Bosch tasks you meant and some helpful information from it. And then here you can see there's two links for every task. There's an events and there's an ID. So if we click on ID, it will go fetch the actual debug logs for you and then it will try to colorize them. So if you have a medium to small size deployment, if there's any error, you can just go and then quickly see it without just with some color at least so that you can distinguish what's really happening. And this is also happening in the background. You don't really need to type so many Bosch commands here. So for example, in this deployment, this was successful. I can just, if something went wrong, I can see the debug logs here. Now, if I go back to the tasks, I can also frame the tasks if probably if you're a Bosch operator, you have used the Bosch events, which could be very helpful in tracing exactly what happened if something went bad. So here from each task, if you click on it, it will take you directly to another page which will filter all the tasks in the Bosch director by that ID. So here we can see that it's creating deployment CF or updating sorry, deployment CF. And then here you can just start to see the actual linear time of linear events what's really happening. And even myself, when I was on Bosch for two years, sometimes it's not easy for me to know like what really happens at this point, what's Bosch doing? By just following this, it will quickly tell you like to put in your head some kind of a flow of what really Bosch director does. And it has some interesting features where if an event has a context, if it's an error or not we're gonna see the error, you can click on it and then it's gonna show you the actual context for this event. So here we are updating the deployment from here and then this is sorry, this is the after state and this is the before state. This could be helpful actually whenever let's say let me clear the filters and whenever there's a problem in your deployment. So let me go here and then I wanna and I wanna also mention that Bosch events will give you only 200 by default. So you need to do before ID to get the last 200 it can get messy. With this one, you can just copy this one and then put it before ID here and then it will directly filter the previous IDs. So you don't really need to type that 17 dash dash dash dash something in the last CLI. And here for example, quickly let's say we know like line 50 task 59, there's an error in this event. If you click on it, you can just directly get it. This is whenever you're dealing with so many deployments in the Bosch director it's quickly easy just to figure things out without pulling your hair out, hopefully. So this is the Bosch events task releases deployment. So this is just the quick things we added to this tool. And sorry, one more thing is the configs and now probably maybe if you're aware or not Bosch supports different kinds of configs before we had the runtime config and cloud config. Now we have a resurrection config. There's also CPI configs and potentially it could be expanded to anything. So with the addition of all those configs, let's say I wanna see which version I'm using, what's the content of that? We added this quick tool which will aggregate a few things together. It will list all the configs. In a real world scenario, you will have so many of them but in this one it's only two. And let's say I wanna see the cloud, if you click on it, it's gonna go fetch directly the cloud config and then it's gonna display it for you so to quickly visualize it. This can also hopefully be helpful whenever an operator dealing with multiple configs. Now let's go to the actual, to the big part of this presentation about the links. So you have the C of deployment, you have, I don't know, X number of deployments and you have shared links between those deployments. So maybe sometimes let's say if you are updating a provider and by some chance whenever you're updating a consumer it's gonna start to update something that you would never even imagine before. How did you know that before? Probably not. Maybe as an operator sometimes you don't really know like is this deployment consuming from that or what's really the topology of the deployment. So with the help of links API, with some aggregation, what we did is we separated the concerns of links into two, the providers and consumers. We said choose a deployment and we're gonna tell you all the providers that this deployment has. For example here, let's go to the providers per deployment. I'm gonna choose a deployment. This is the C of deployment. It will give me all the actual providers of this deployment that all the entities, it could be a job, it could be another thing that can provide something. Additionally, it's also doing some aggregation. It's telling you how many times this provider is being consumed. So for example here Nats, the link Nats of type Nats which is shared, it's coming from a job and that job belongs to the instance group Nats and it's called Nats. It's consumed eight times. And to know who is really consuming that, we just open this one. It's behind the scenes also doing so many aggregations. It will give you the actual provider definition and the database, which could be very useful. Also it will tell you the list of all the consumers that are consuming this. For example, we know Nats is consumed eight times by Nats itself, by the router job in instance group UAA and deployment CF and the list goes on. This before links API, before this, there's no way actually you can know that unless you go to the database and then do some queries. Even with that actually it was not if links were not the first class citizen of the database. It was put in JSON. So it was not, I don't think it was even positive. So this one for the providers. And one thing for example you can see here, you can start to see some patterns is let's say you can see that some of the providers they are consumed zero times, which is expected, some of them. And you can quickly let's say I'll be interested in the huge deployment and when I see like the providers that are consumed the most amount of time. So here we have like Doppler. If I open it, I can see the Doppler link from the Doppler job in instance group Doppler. It's being consumed by all of these. And this one will tell you now it will only see one deployment. If you have multiple deployments consuming this provider it will show you here. So if you are just having so many dependencies and then you wanna make sure whenever you're doing some update on the deployment, I wanna don't wanna trigger something or I wanna expect or I wanna really know what I'm expecting. This hopefully will help you just predict that by whenever if I'm dating a provider maybe I wanna also update the consumer deployment as well so that I will not face any interesting facts later. So this is the BOSH deployment, this is the BOSH share. So this is the list of providers for a specific deployment. Now we flip it around and then we're gonna see let's say the consumers per deployment. I wanna see I have a deployment. I'm interested in seeing where all the consumers in this deployment where they are coming from. Let me choose the CF deployment one more time. And here it will give you some statistics. So we have 78 total number of consumers in this deployment. We have 72 fulfilled consumers because the consumer could be optional. So it could be nothing actually really it's not consuming anything. And we have 70, most of them, all of them actually are consumer of type job. We're gonna see the variable later and then we're gonna talk about it. And then the consumer here, you choose any consumer. Let's say if I'm interested in the UAA, let's say in this consumer, the database consumer coming from the UAA job, if I click on the plus sign to expand it, I see the consumer, I see the link, and I see the provider. I know exactly this job where it's consuming things from. And the reason this is helpful is because this kind of link could be provided by, I don't know, maybe 20 providers. And at deploy time, you don't really know which one. Unless you really coded that link, but if you're an operator, even if you coded that link, are you sure it's that one? Behind the scene, like anything can happen. So this is from a link consumer perspective. I'm gonna choose actually now another deployment and I'll talk about the variables. So now with the ability of, before let's say with CrudHub, you can define the variable section in the manifest but then it's gonna go generate the variables for you. But probably you most aware of is that you need to put the common name and the sense, subject alternative names to the Bosch DNS aliases, like something CF internal or something. So you need to hard code it. So what we did is we had, now we said a variable can consume a link from a job. Let's say it's the UAA job, it's a UAA job. This variable, whenever it's coming, that's link of type address from that job, whenever it's just generated before we send it to CrudHub, we are automatically adding the Bosch DNS record for the instance group that this job belongs to and injecting it in the common name or subject alternative names so that you don't really need to deal with it. And it will automatically be sent to CrudHub, signed, created, signed and sent back to you. Here, imagine let's say you have 190 variables and it was like, okay, like I'm consuming something from a job as a variable but what this thing ended up doing? I don't know, it generated the variable. Did it really have the correct sense or not? It's a big question. Here in this deployment, we can see that we have 12 owner, we have 12 consumer of type variables, all of them. And then we can see they are fulfilled by which a job exactly, the same thing as a consumer. So the consumer link and the provider. And one thing, let's say if I wanna investigate this specific variable, what is it? I can just take the name of it. Let's say app server two and then I wanna go to the variables tab which we didn't talk about. I'll click on this deployment. It will list all the variables used by this deployment using the Bosch Vars command. Additionally, it will shell out to CrudHub and it will fetch that variable for you. So for example, if I'm interested in this one, it will open it and it will automatically fetch that. Now this one might seem a bit scary but everything is local. We're shelling out to CrudHub the same way that all the environment variables are already in the shell. And then you can quickly take the certificate and decode it. One last thing I wanna show is the deployment dependency. If I wanna quickly glance, I have the CF deployment. Who are the other deployments consuming links from the CF deployment? I can choose it and I can just quickly see, let's say here I have the provider deployment which is the CF. I can quickly see all the deployments that have links consumed from the CF deployment. This way, if I'm updating CF, maybe you wanna update the other ones. It depends of what you're really updating. This is, I think this is the last thing I wanna show here. And to go back, this is hopefully, it was introduced to help to bridge all this gap off. I have those many links, many providers, many consumers, and now it's getting more confusing with the variables. It's getting more confusing with the custom provider definitions. And I imagine you're facing a manifest with around 3,000 line of code. Okay, just tell me what's really happening. This is a prototype that we started doing. There's one limitation. There's a couple of limitations with the API that Bosch is exposing. We are doing many queries behind the scenes. So our next step, if this one is found to be helpful, is to submit requests to Bosch director, PR, sorry, to the Bosch director to help us visualize more entities much quicker and understand the topology of links. And I would always think about it with one person actually on Slack messaging, like, hey, I have 500 deployments in my director, and he's a bit scared. He was a bit scared, like, what is happening with the links between those deployments? It's just basically like click and just wait. Hopefully nothing breaks. Hopefully this will at least start to make things easier. And basically, yeah, basically that's it. Thank you. Questions? You said that this was, that you all had forked Bosch XT CLI. Is this still on a fork, or has that been merged back into the main line of Bosch XT CLI? It's still forked, but eventually it will go back. I would be curious, at least, to know what the name of the fork is. Maybe I just missed that, but was that in one of the slides? It should be at the end of the slides. There's a link for the actual GitHub, yeah. Awesome. Any questions? Quick question I have is, I was expecting some graphical representation. I'm guessing that you have something like this, maybe you were thinking of. A bunch of packages, but it's right. Yeah, only thing with that is that whenever we started to do the presentation for the graph in D3, it got very messy very quickly, because there's so many links, there's around 100 links, any deployment, let's say, or even more, that's the basis of deployment. And it will look a bunch of lines altogether. So at the beginning, it's not that helpful actually, as it seems. Do you have any ways you should use? Do you have it? Not currently, no. It's only the, basically, we only left it at this point. I see. Yeah. Any other questions? Oh, Tom, again? All right. That's a different question from the first question. How well does this work when you're, I assume you all are currently authored as admin locally? How does it work, like if I'm a user, like I'm on a team and I've got some team scope configs kicking around and stuff like that, how well does this to work if I don't have full admin privileges to the director? It will show you whatever the bus CLI can show you. So all the deployments that the bus CLI show you, all the configs that will show you. So everything that technically this tool is only shelling out to the CLI and whatever permissions that the client that you are logging in with, it will have, it will show you. So if there was weird behavior, it's really more a question of like, that would be weird behavior in the CLI because this thing just uses the CLI to fetch all of that information. Or maybe some of the business logic whenever we're aggregating all those endpoints together. Okay, cool. Thank you. Thank you.