 Okay, so you are in the view, so you should be on that, and you can go, you can see it there. Yeah, I see it on the screen here. So basically, if you are on the left edge right now, so basically if you stay near this red box on the screen, that's the perfect place. Okay, I'll be here then. You can go here, maybe at the middle at most. Good. I'll stay here. Good morning everyone. Welcome to the first talk of the final day. Before the talk starts, I would like to remind you that today we will have a final session, or win-win show, session that Raddeck will be handling. I think it will be in this big room, right? Yes. So please come there, stay until the end of the conference. There are still a lot of things to enjoy. With this, I'm passing over to Ricardo on Automating Kuiper Nets with event-driven Ansible. Please welcome. Good morning. Thank you. Good morning everyone. Welcome to this talk. So first, that introduction, Ricardo Kruse. Everybody just can call me, you can call me Ricky, everybody calls me Ricky. I'm a Principal Software Engineer at Red Hat. I've been at Red Hat for six years and a half. I'm a member of the project decision team at Ansible. We're responsible for building, delivering the various components of the Ansible Automation platform. And we're also involved into some of our offerings in cloud and software and whatnot. And we're also responsible for the installers and the operators for Ansible. Prior to that, I produced company, HP. I was working heavily in OpusTac. I was part of the infrateam, both upstream and downstream. Then I joined Red Hat for the engineering, did networking, CSD, cloud stuff. Then moved on to OpenShift Engineering and doing networking. Then moved on to your office to do some telco stuff and back to my leadership at Ansible. And I'm very happy about it. So I've shoveled it around quite a bit. So first, a super brief introduction to Kubernetes. I assume that you know what it is, but just mandatory slide. So what's Kubernetes? It's an open source platform that orchestrates container as applications. So you have your container as applications and you declaratively put a manifest. I want to run this application with this many replicas and want to expose this board. And Kubernetes does its magic and make sure that they get one and all things. Developed by Google, now a CNCF project. So we have a lot of contributors, Red Hat included. We have a lot of people working on it, especially because we have the OpenShift product, which is Kubernetes distribution. There's a wide ecosystem of tools and projects around it. If you go through the CNCF landscape, you will see there's a massive picture of observability, monitoring, provisioning, all sorts of different use cases that are around Kubernetes. So it's a big thing. You can either run it on a prem or host it in clouds. You have Kubernetes in AWS, Azure, Google, Lineout. We have Red Hat authorship in all those clouds. You can also run it on prem. At its core, Kubernetes, it allows you to self-heal your applications. So make sure that it will be always running. If, for whatever reason, it dies, it will bring it back up. It allows you to do scaling, horizontal scaling. And it also allows you to do running updates. So if you update your application, it will do a downtown update. And it can also do a rollback. So when you adopt Kubernetes, they're typically what they call another architect. But it's like a known term or pattern. It's like they call it in date phases. So typically in Kubernetes Day Zero, you gather the requirements for the workloads that you're going to run in Kubernetes. How are they going to run? Where is it going to be your data? Is it going to be on prem? Can it be on clouds? Will you be having hybrid cloud Kubernetes like on prem and on cloud? And on cloud, how are you going to connect that to your environment? So in this phase, you do all that. We're going to be gathering the design and the architecture. Once you have that, then you move on to the next day, which is installing the cluster. So you install the cluster. So you probably have some months of playbooks, or terraform, or cloud formation, or I don't know. Arm templates, whatever you're running and installing your cluster. You install it, and then you configure it. You put the initial users. You probably hook it up to your infrastructure's code repositories with your CSD. And that's the initial bootstrapping of the environment, so to speak. And then in day two, that's where you deploy the workloads. And then you do the ongoing maintenance of the whole applications and the underlying infrastructure. So this is where you take care of observability. So you monitor your applications. You monitor your infrastructure. Is it sane? Is it like, maybe, is it for women? Well, not that kind of thing. Then you take care about the scaling. Maybe your workloads grow over time, and you need to scale the nodes. So that's something that you have to do in day two. So how are you going to do that? Security. There are new users coming in, users coming out, maybe contractors. How you're going to handle certificate management, secret rotation, that kind of thing. That's what you do. Storage, same thing. So probably you will run out of space at some point. If you're on cloud, maybe it's taken care of for you by the underlying cloud provider, but maybe you're on-prem. So you need to, how can you expand the storage so you can still run your business applications? Networking. Imagine that you have a site, an initial site for your cluster, and then you run out of IPs. How can you expand that? Or how maybe you need to interconnect to some other internal site? That's another thing you do in the area. Update management. Both from the application front of you and the cluster. Inevitably Kubernetes is like a fast-paced project. So new versions come out, and you may want to keep up and update them. Disaster recovery. Backup and restore of its CDs, volumes, databases of the applications. Do you have a strategy for that? How can you backup? How can you restore in case of a total disaster? How can you bring up a mirror cluster environment? Or maybe you have a pilot-like environment there. And when there's a recovery, you can quickly bring it up. That's the kind of thing that you do. So in Kubernetes Day 2, you have tools and practices for doing that. Typically for within workloads of Kubernetes, you use Kubernetes operators. So they're like Kubernetes native applications that contain the operational knowledge to maintain the lifecycle, manage the lifecycle of applications. So typically, if you have a password SQL that you use in your environment, you will have a password SQL operator that will do the installation, but will also take care of the backups, restore, upgrade, that kind of thing. You probably want to use GitOps to manage and deploy application infrastructure. So keeping Git declaratively the configuration for both your underlying infrastructure and your applications, and have that system to automatically apply the application. So no human intervention, hopefully. And when you've done, or the things that you've done, fall into the two previous packets, you probably have like runbooks that define day 2 processes. So best case scenario, you probably have like Ansible or Bathscripts to automate that. Or case scenario, you will probably have a DDoC or some wiki space somewhere. Worst case scenario, you hit an issue and you're just trying to figure out what to do and fix it. So now I'm going to talk about Ansible. I assume that you probably know Ansible, but I don't like to presume things. It will be quick. So Ansible is an IT automation tool. It allows you to automate tasks in your infrastructure. So open source, that's what we do with Red Hat. It came by an acquisition. It's very easy to learn and use. It uses YAML. The learning curve is really, really easy. You can get going, you know, fairly quick and automate things. It's agentless, meaning that for the things that you want to automate, be servers, network devices, firewalls, I don't know, cloud endpoints, you don't have to install anything on the target. You run your automation from a bastion control machine and you perform the automation against those targets. Highly sensible with plugins. Everything is a plugin in Ansible. So it's a very, you know, extend the extensible system and you can swap in, swap out, you know, behavior by changing plugins. Use cases, configuration management. So create users, install packages, you know, system services, firewall rules, that kind of thing and servers. First of all provisioning. So we have modules for provisioning pretty much everything, like for AWS, Google, GKE, Azure, VMware, Office Stack. There's also stuff for parameter provisioning. Application development. You can use Ansible to deploy containerized applications or legacy RPM applications or DevEvent. Orchestrations, this is my favorite. Since Ansible is so easy to learn and it has such a wide integration with pretty much, you know, every system you can find in an IT environment, you can use it as a glue, as a universal automation language to orchestrate all that. I would count, as we said, if you have public cloud stuff and on-prem, you can just use it, you know, to interconnect them. Networking, this is a big one. So Ansible has a wide variety of modules for managing networking devices. As a matter of fact, we have, like, a content team just for that. So the Cisco, the Arista, the Juniper, you can manage that. Same for security. We have modules for Palo Alto, Checkpoint, CM systems, IBS, that kind of thing. We're also getting a lot into Edge. There have been, like, quite a bit of talks this week about Edge automation, so we are getting into that area a lot. So how it really works. So you have a control machine, which is a bastion, which is where you install Ansible. That's where you run Ansible from. Then you have your target nodes, which is your servers that you want to automate, or maybe your network devices, or your security appliances, or your cloud endpoints. Then in the inventory, it's where you keep your authentication and connection details on how to connect to those target nodes and then the playbook, which is the artifact in which you define the things that you want to automate. Typical and inventory static file. Very easy to read. Here we have a databases group with that IP and a web servers group with two IPs, meaning that if I run a playbook against the databases group, it will run the task against that IP, run automation against web servers, it will run the task against those two IPs, for example. This is a playbook example. Here we have the host, so we're targeting the web servers group from the previous inventory file. We're telling, hey, we want to use like root to get into those machines. And the task, which contains the automation itself. So it's going to run sequentially this task. So in this case, as you can see, as for example, it will install the HTTPD package with Yom to the latest version. Then it will sympathize with Jinja the config file. So Ansible, so that's like, you know, from a generic point of view. So then, you know, if you drill down, you have like other components or other abstractions. So the modules contain the code to perform a particular task. So there will be a module for typically a Python for containing the distance logic for this task, for this task, for each one of the tasks. So we have like a module for doing the thing. There are mostly reasons in Python, but any language can be used, providing that a module, a module really just needs to like read JSON and output JSON. I mean, how it does it, you know, internally it's not really, you know, important. I've seen, you know, modules done in Bash, even in Golang, so they're easy to develop and ship. So you don't have like to have a module in your system libraries for Ansible or, you know, push up ER and hopefully, you know, it gets merged and then, you know, it's in a release. You can develop a module, put it into a library folder alongside your playbook and Ansible will auto-detect it and use it. So which is nice because it allows you to get going and also test modules. So Ansible has built-in modules. So whenever you see Ansible.builtin.whatever, so those are built-in modules. So they're part of the Ansible core package. So it's something that you get, you know, bundled and built-in with that RPM package. There are also Ansible content team modules. So we have content team with Ansible for networking, cloud, also security thing, and also people doing edge stuff right now. We also have partners doing modules. So some other like vendors from all kinds, they also have like their modules and they develop and maintain them. And you also have community modules. You know, we have a great library community and we get a lot of contributions there. So once you start, you know, doing a lot of your playbooks, you will start realizing that, you know, you can maybe refactor and reuse that automation for all their environments. And it's certainly tying that to a specific host of your inventory. So that's when you use roles. So roles allow you to encapsulate Ansible content such as fast, change of templates, static violence variables to automate that particular thing. So you typically would have a role for, I don't know, NGNX, for Apache, for PostgreSQL, and that's self-contained and can be reused in whatever environment that you have. Once you have a role created, the way you use it, then it's you included or imported in a playbook as a task. So you would do include role, my role, and it would include it and run the task within your play. So that's the typical structure of a role. Here we have a common role which contains Task folder, a main YAML, some danger templates, static files, some bars that are specific to the role and some defaults in case you don't have, you don't specify the bars on common line or in the playbook. There are some other things, but these are like the most common ones. You also have like handlers and some other things. You can also, you used to be able to ship plugins, but now, you know, with collections which we'll see in a moment, that's not anymore, you know, a thing for roles. So plugins, as we said, and the Modifier Code Ansible capabilities, there is like a large variety of plugins that you have connection plugins. So typically, when you use Linux and you run on your automation against Linux, it's going to use the SSH connection plugin, but maybe you have Windows machines. So Windows is not usually, SSH is not something that you're usually using Windows, so there's a Winner-Rend connection plugin for networking devices. There are like NetComp connection plugins. It really allows you to extend and, you know, to adapt to different kind of devices and things that you may use in Ansible. There is it to develop by using Python. These ones, you have to use Python. I don't think you can use other language. Okay, so now you have playbooks. You have plugins. You have roles. So how can you package that up? So back in the day when I joined, so we had Ansible Ansible. And we shipped all the modules, all the plugins and everything within one repo. Then that became, you know, how to maintain and how to contribute because, you know, we only had, you know, a certain bandwidth to review and get things going. So there was a decision to decouple. So I have Ansible Core, as I talked about earlier, to just contain the core of Ansible, the binary and the built-in modules. And then the rest have it in a separate artifact, which is called a collection. So we have a collection of playbook roles, modules and plugins. So they're typically like for domain automation. So you will have a collection for AWS, collection for Azure, I don't know, for Rista, you know. And those collections will contain all Ansible things for automating that particular domain. Distributed by Ansible Galaxy Private Hub. You can just go to Ansible Galaxy, it's like the content store for Ansible. You can just go to browse for content and download it and install it in your machine. You can also have like a private Galaxy, which is called Private Hub, to install that. So for Kubernetes, it's no exception. So there's a collection for Kubernetes, which is called Kubernetes Core Collection. It contains modules and plugins for automating Kubernetes and Upshift since Upshift is Kubernetes. You install it via Galaxy. Ansible Galaxy, you can install Kubernetes Core, you install it in your machine, then you can use that content. So this is a subset of modules that are included in the collection. So you will have modules for Helm. Helm is a package manager for Kubernetes. It's like an RPM kind of thing. It uses something that you call a chart, which is like a definition for an application. So you can just, instead of using Helm, you can just use this module. It will use it under the covers. Then we have modules for managing Kubernetes objects. See, this is probably the most used one, the HATES module, because it allows you to either slap into the module as parameter the definition for an object or you can also specify Kubernetes manifest that you may have, meaning you don't have to redo your Kubernetes manifest in the Ansible DSM. You can just reuse your Kubernetes manifest and just use it. You can exact in two parts, get information about Kubernetes objects, lock parts. You also have imperative modules like rollback and scale, not only declarative. Some other specialized like service modules, which are probably not super useful because you can do the same thing with the case module. This is an example. So here we're using a Kubernetes manifest that is within testing, the problem in YAML. This is a Kubernetes manifest and we're telling, hey, apply this manifest. I will just do it. That's why I said that it's good because you don't have to use the Ansible DSL. You can just reuse your Kubernetes manifest. You don't have to change anything in your environment. Here, another example, using an imperative module. So here we're saying, hey, the elastic deployment, which is under namespace, my project, I want it to be three replicasas and it will do that to get all five to have that replica set. Okay. So what's EDA? So it's a new Ansible project that allows you to execute automation based off events. So what is? Typically in Ansible, okay, you have a playbook and you want to perform something. I don't know, maybe you want to do a backup and there's a human running a playbook or something happens and something opens you a ticket. Hey, I need to do a restore. Then you run the restore playbook. So what EDA allows you to automate that trigger of that answer automation. You can think of it as the if this, then that of Ansible. So basically you, with EDA, what you do is you observe events from your environment. Then when those events trigger, you evaluate those events against some rules, which you define. And then, you know, if there's a match, then you run some action. So EDA uses an abstraction or a concept that is called a rulebook. You probably, hey, it's like a playbook. Yeah. So it's a rulebook. It's the playbooks for EDA. It gives you a hint what they're all about. So rulebooks contain a set of uniquely named rule sets. Then within rule sets, you have a host section, which, you know, you define what you want to target for your automation. The sources you want to listen. So what kind of the events you want to react to and the rules to evaluate those events. Then the rules themselves contain conditions and actions. So, and then if the events match the conditions, if there's a match, there's an evaluation that is performed by the Joules engine. It's a Jala-based engine for rules. Then the rule action is executed. You know, rather than me trying to explain things abstract, let's see a rulebook. So here we have a rulebook called Hello Events. It contains a host section. In this case, it's a local host. And it has a sources section, which is where you put your events sources plugins. So here it's Ansible EDA range, which I'll go through it, you know, in a moment. Then you have the rules, which have this one rule called say hello with an addition and an action. So in this case, the events source plugin Ansible EDA range is kind of a test event source plugin, which basically does a Python range. So it will go from zero to five and it will create an event for each value. So there will be an event for zero, for one, for two, for three. Then as a rule, we want to catch when the event.i, the key i within the event, is one. And when there's a match, we will then run this plugin. It's probably not super, you know, fancy, but probably in this next thing, you will probably see, you know, what we can accomplish with EDA. So in this case, we have another rulebook. Check my web app. We're targeting all the machines on our inventory. And we're using the URL check event source plugin. What it does, it will pull for the URLs that you may have as a parameter continuously. You can put that delay or timeout value for when you want to do that check. And then we turn back the status code, 200, 400, 500, whatever. So here in the rule, it's called help and we have the condition, hey, trigger it when the status code is different to 200. And then run a playbook. We just call restart app. So basically, this is a rulebook for self-healing, remediating a web app. So this is the kind of thing that we can do with EDA. So in the rule sections, as you can see, we can have multiple rules, not just one. It's like a list. Within each rule, you can have multiple conditions. You can have checking various things within the event, so different fields, compared against different values. Each rule can have multiple actions. So when there's a match for a given rule, you can have a list of actions and they will be executed sequentially. Although I think they put a patch recently which you can have, like, run actions in parallel. I need to get back to it and check. So the rule conditions is like a, you know, a Python Boolean expression. It allows you to put, like, Boolean operators or we may take operation and yes, so you can do things, does my event contain this key or is this key within my event equals to this value greater than, less than that kind of thing you have on or not, you know, the usual that you would expect for that kind of expression. It can contain multiple conditions. You can have, like, list of conditions and you can also use, like, any all operators to say, okay, if there's any of the condition matching the event, then execute the action or whether you want to have, like, all of them to match in order to get down to the action. So the final part of the actions are, like, the final part of the event we have closed. So we have event. We have the evaluation of the rule for the event and then if there's a match, that's the action. So that's, like, the final part. If you define the execution part when the event is matched by a rule, so when there's a match, then you run the action. You have, like, a built-in set of actions bundled with BDA, event source plugins. Like, separate from the EDA core. Actions are, like, bundled within the binary that EDA uses. There are no plugins right now for actions, but I think they're open to also open to bundle actions apart of separate content. So probably will be implemented down the line. These are, like, the actions that you get built in from EDA. You can run a playbook, as we saw in the examples. We can run a module. This is, like, the same when you do ansible all-mping. Imagine that you want to do, like, a quick execution with Ansible. You don't want to create a playbook for just one task, and you want to specify the module and the parameters in a one-liner. That's where you can use with one module. Run job template. So maybe you have a WX automation controller somewhere containing your inventories, your job templates, and all your automation. You can hit that controller, AWX, and run automation from it. You can create facts just in, like, an Ansible post-events. So maybe you want to have, like, a flow of different events, like in a flow chart. So you hit an event, and then you want to jump into another event. So that's something that you can do with post-events. You can also retract paths. You know, from the rulebook execution. Pull it out. Print event, which is just for debugging. And shutdown, which I think is for restarting rulebook if something goes wrong. Or you want to restart it for whatever reason. So we saw event source plugins. They contain the business logic for the sources. So for the URL check, there will be a Python thing for doing that. For the EA range, there will be a Python thing for doing that. For more interesting things, which we'll see in the next slide. They are Python files. They use the AsyncIO library. So they are asynchronous. The plugins can be either calling for external sources. Or they can also be listening in a port. So an external source can send events to the plugin, and then, you know, it reacts. Once, you know, it catches the event, it puts this into a queue, that the EDA runtime gives it to the plugin, and then, you know, it can evaluate it with the trues engine, and then, depending if there's a match or not, run the action. So how does it really look from internally a plugin? So this is like, you know, the meat of the URL check event source plugin. I'm not going to say a guy or guy. I mean, I've got it in Python, but it's not hard to read. It's an infinite loop, in which it gets a session with AIH TTP of Client, and then it tries to, for each one of the URLs that you pass as a parameter, it will try to hit it. And it's going to get the status, and then it's going to put it into the queue as a deck. So then it can get evaluated with the trues engine. So what are the events for plugins? Oh, yeah. So right now, we have like the one, official one that we, the EDA team maintains and develops, which is the Antigo EDA collection. They have like a bunch of different plugins. These are like the most, it's just a subset. So you have like an event source plugin for Kafka, so you can actually get events from the Kafka queue. Also from a layer manager, which is a companion service for Prometheus. Prometheus is getting metrics, then a layer manager alerting when the metrics go over or below a threshold. And then you can get, you can trigger automation from a layer manager. CloudTrail, the, one of the services for AWS for security that, you know, keeps a log of the API calls for a given account. SQS is like a messaging system. It's also very, very handy Azure service bus, same thing. Webhook, this is a good one because it's a generic Webhook. Event source plugin, this is like a listen, event source plugin. So it's going to listen in a port, and you're going to have external systems send Webhooks to the port, and then react to those events. So along the Antigo EDA collection, so we're getting partners developing their own, Antigo EDA collection. Those are some things that have been announced. So CrowdStrike, a security vendor, they're developing their own for the product Cisco, for the NXOS devices lines, and the Thousand Ice, which I think is a monitoring product line. Dynatrace, an APM. IBM, Instana, another APM, Trubonomic, I think it's a cost management system. Palo Alto, you know, security vendor, F5 networking security vendors, Zabx monitoring solutions, they're developing their own EDA plugins. So then you can also, you will be able to integrate, if you're using it in your environments, to integrate with that. Besides the eventual plugin, you have event filters, which allow you to filter data from events. So at times the event from the plugin can be really big, and there may be information that you're not interested. So they allow you to remove or transform the events. You can change the event filters. Imagine a Linux pipe operator, so you can change them. They define after the source definition, they're like a source object. Built in, you have a JSON filter, event filter, so it allows you to include or exclude keys from a JSON, convert dashes to underscores to maybe, because maybe you need like a snack case, insert host to meta, if you want to search some host as part of your role activation to the event, normal as keys. This is also something that you can develop, like event search plugins, and bundle into your EDA content. Let's think about just EDA content, how it gets distributed. So EDA content is going to leverage Ansible collection framework and tooling, so meaning you're going to have Ansible EDA rolebox and event source plugins and event filters using the collections framework. So you'll be able to just use Ansible Galaxy Collection install, do, and you put your collection and will install it. You can also publish it with this thing, and it will be able to publish into Galaxy, just like an Ansible collection. So as we said, the Ansible EDA is the first EDA collection available. It's the one that, you know, bootstrap the project. Let's go dig deeper. So what are the components? So the main component right now for the EDA is called Ansible Rolebox. It is the CLI and worker component for event-driven Ansible. You can either run it in a standalone mode, so you can run Ansible Rolebox. You provide an inventory, your Rolebox file, and it will run in foreground. Or you can also run it in working mode, provide an ID and a WebSocket endpoint so it can connect to a WebSocket and to get, you know, what Rolebox should I be using. This is what's going to be used largely for the EDA controller server offering. So the EDA team is working on the same way with Ansible. We have the automation control, which is a central console containing your automation principle. There's going to be an EDA server containing your Rolebox. You can log in, log out. You will get, you know, logging about your events and it's a Web application, so to speak. So the working mode is what gets used so we'll go connect into that system. So how we can... So we've gone through Kubernetes, Ansible, Ansible, Kubernetes content, EDA. How can we mesh EDA with Kubernetes for the way things... So Kubernetes clusters can emit events for changes. So the EDA server is constantly, you know, every single component in the Kubernetes system is hitting the EDA server. It's the one that is responsible for performing change in your cluster. So when you create a CR, deployment and conflict management space, you know, you get to create things, you can update the product operations, right? Now the cool thing about Kubernetes is that it allows you to... what they call it, a watch. So a watch is like an API call you can put in a given resource version. So you can basically say, hey, I want to put a watch for conflict map objects under this namespace. And then the Kubernetes API is going to stream me back any kind of conflict map changes in that namespace. Whether there are no ones, conflict maps are updated or they're deleted, I'm going to get those events. So given that kind of functionality, that's what we need that for. We need to plug into that capability from Kubernetes for getting watches. So get events from the API and then so we can react to them. So there's an event source plugin for Kubernetes. It's called Saber1041.dda. It's a collection that was written by Andrew Block. It's another fellow bread header. I think he's around. I haven't met him. If you're interested, as a matter of fact, he also published a blog post about the idea collection, how it gets used. So how it works, it uses Kubernetes libraries, the event source plugin. So with this event source plugin, you define which resource you want to watch. Maybe ABA version B1, conflict map, namespace full. And then it connects to the Kubernetes cluster and puts a watch against that. And then the plugin gets events from that watch. So that's one way of getting events from Kubernetes. Or you can use the generic webhook, which is it's already bundled within Ansible.dda. The webhook plugin is a generic event source plugin that receives webhook events. So it's an event source plugin and you can get external systems sending webhooks to that port. And then, you know, the plugin will get the events in that way. So as you can imagine, we're going to need to have something because Kubernetes kind of just do it that itself. We need to have something to do that watch mechanism for whatever resources we're interested in and send those events as webhooks to this endpoint. So this is what we can use Robusta.devCubeWatch is a tool that was developed by Bitnami. It's a CLI tool which basically you create a config file and you say, hey, I want to get the watches for deployments, config maps, nodes, whatever. And then for those events send webhooks to this endpoint. So what we can do, we can have a faster sending webhooks to our event source plugin running webhook on their port. So now we're going to do a demo. I don't know if is this going to be a hard one because I don't have a mirroring. I'll try my best. So Can you increase the Yes. Is it good? That's better. This is my laptop. I have a cube config. I have a cluster and I can just run to get the commands. So here I have CubeWatch with some with a config file which is CubeWatch.demo So you can see it's a demo which you specify the webhook section. So for the events, I have a webhook against my cluster where I want to send those webhooks. So I'm sending to my machine on port 5000 which I'm going to have my event source plugin listening for events. And then for resources, I can put the kind of resources that I want to put on watch. So in this case we want watchers for deployments, config maps, it's, you know, on the program and it puts a watch for those resources against my cluster so whenever there's something happening for those resources in my cluster, it will pick it up and send to a webhook. So if we, as an example so if I create that namespace RickeyTest Okay? And if I go back to the CubeWatch it got, oh, processing that to namespace RickeyTest and it got that event and it tried to send to 5000 but there's nothing, you know, listening. So what we want, that's because I don't have the event source plugin as a robot with the event source plugin for webhook on that port listening yet. So let's do some some cool demo. Let's see, let's imagine that we have users in our clusters doing deployments whatever and there's a certain image and tag version for that there's a CVE or for whatever reason we don't want any users to deploy it in our cluster. So what we want, we want to catch whether there's any kind of deployment using that into in this example the port namespace we want to catch that if it's using the bad image we want to scale it back to zero so it doesn't run in our cluster we want to notify to some slack that I have and post a message hey, someone tried to do this don't do it, okay? So if I go here I'm going to show you so this is the robot so using the first plug in, listening on 45,000 and we have one rule which is name and scale down by the image deployment what are the conditions? So get event that the kind is a deployment and whether you know the reason for the event is created so basically we're going to get Qtl create deployment now the thing is that the event itself doesn't give us information about the image being used so we will need to do some introspection based on the name of the deployment and that's something that we will do as part of the playbook so when the condition is met we're going to run this playbook scale down by the image deployment and notify and we pass some bars my slack token the deployment name which comes from the event so we can do that introspection within the playbook and my Kubernetes API token so I can do this scale down if we look the playbook that corresponds to this worldbook so here we have some bar it's called bad image which is NGNX 1.14.2 so that's the bad image that we want to detect so if there's a deployment using that image we want to scale it down, notify so what the playbook does by leveraging the Kubernetes modules this is why Ansible is so powerful because it can integrate with everything and it's really a great great glue for your automation we're going to introspect the deployment and we're going to register into the deployment variable then I have a block so a block is you can group tasks for a given condition or a variable expression so when the deployment that we get earlier if we drill down into the JSON because that's a JSON so this gives us the deployment object so we'll introspect it so we'll get into the part template of the deployment we will check for the container it's using the bad image and if it's the same then we run the task within the block and the task within the block is using the case scale module so set the replica to 0 and then notify to Slack that someone is using the bad image so I have to scale down the deployment please do not use it if I run the Rolebook which is this one so now I have Rolebook listen in 4 or 5,000 I have to watch so get in events from Fubinitis and for whatever events it's configured to it will send to 4 or 5,000 which is where we have EDA listen enough it's manufactured ok it creates an nginx deployment we replica and it uses the bad image nginx 114.2 so we want to catch that so if I apply as you can see Rolebook got triggered and it's running a playbook so if I switch back quickly and I do an also get the deployment oh it was quick enough if we hit the deployment we replica but we catch that event and we scale it back down to 0 so we were able to remediate the thing in our crashlet that we care about and we don't want to happen and if we go back to here hopefully I can see the so this is my Slack yeah so it's 10.16 right so let me so we got the Slack notification on our Slack about what happened and we scale it down so I hope that you can see you know how we can leverage all this because we can build that by using all the integration that we have with Ansible we can react to events from our environment whether it's at the in cluster level or at the underlying infrastructure where our cluster runs and we can do, we can run the automation for it at the end of the talk that's what I have any questions? is that clear? yes as a matter of fact I know yeah so someone asked in the audience if there's a way to integrate light speed which is an AI assistant for writing playbooks in VS Code and assist you with some automation to integrate with that for sure because light speed has an API so you could create an event source plugin or some automation and plug it as a matter of fact I know that there's some people within my team that they're looking into integrations of lightspeed.eda because you know if you can get suggestions about how to remediate imagine you get an error on vocal automation for it if you could hit light speed API and get suggestions for a playbook to remediate that that becomes extremely powerful so that's a great question oh there is as a matter of fact so my team is not my team so my team we've been developing an operator for an EDA controller offering so the same way you have a WXL automation controller for Ansible you know it's just the single tool for running previous but you have a WXL automation controller as a central console for running your automation there's going to be a product EDA server or controller which is the same thing for EDA so you will have a central console which you can have users logging in and logging out you will be able to upload your rule books see your activations your event data that kind of thing I want to also say rather than a young project we would love to get contributions and people involved in the project especially as you can see the Ansible EDA yes it could chase event source plugins but you know the sky is the limit ideally we would like to have it would be nice to have event source plugins for integrating with whatever you have like the environment whatever cloud whatever sass you use whatever so if you want to contribute by all means I mean the EDA team will be happy to look at your PRs and you know contribute with them as a matter of fact we've already got some contributions there was another talk the other day Cube Alex I think is the guy he contributed an NQTT plugin which is really nice and you know if you want to be more would be happy to hear from you we don't have any questions from Matrix so that concludes this session thank you very much thank you one minute before we start I would like to remind you that today is the last day and there is a final session I think it's 5 or so that actually will be a win-win session so please come to play a game and win something in that game that will be in the big room next to us next building with that please welcome Nikita which will be talking about Red Hat CoreS on the mainframe not exactly ok please welcome hello everyone my name is Nikita I work at IBM and today I would like to present you IBM secure execution for CoreS and OpenShift so before we start let me ask you two questions please raise your hands if you are familiar with CoreS and Ignition and second please raise your hands if you don't fully trust cloud providers yeah so I have to explain about what CoreS and Ignition is so CoreS is a small operating system it's efficient secure and designed to run containers in cloud environment and Ignition is a configuration tool for CoreS it runs in it run the first time and takes care about system configuration so you have your config you give the config to Ignition and after installation or provision you have your own CoreS instance so it's too used for configuration ok what secure execution is maybe it's some kind of top secret place with one PC inside no secure execution is hardware based technology developed by IBM and it's available since Z15 or Linux 3 generation system it's technology which fully isolates virtual machine memory and CPU state from hypervisor and other VMs but why do we care about security and what kind of security we use on daily base mostly everyone the most simple example is a phone so a lot of people use touch ID or face ID so even if phone is stolen bad guy cannot unlock it and see our messages, photos or something like that for similar reason secure execution exists it enables the notion of secure VM or confidential VM so confidential VM is VM where you cannot see what's in runtime in memory of VM you don't have any access to it or it's blocked secure execution not only enables this notion it also helps you to deploy your VM securely so you don't trust your cloud provider but you have your VM so you can send it as a file and try it secure execution guarantees that your VM is unmodified or if so it couldn't be executed also it makes it possible to specify on which exact frames your VM can run so now let me explain a bit how it works in general in any Linux distribution so Susie, Redhead, Ubuntu so during manufacturing each mainframe becomes a pair of host keys so you have three mainframes and private host keys stored somewhere in hardware firmware doesn't have access to it it's only available by hardware public key is published on IBM website so if you have serial number mainframe you can fetch corresponding key then we have a special tool called nnprodimage which creates a unified kernel image we call it sdboot it's here in blue color so it's a file encrypted with one of several host keys which contains kernel so once created this file can be only decrypted on corresponding machine or machines and run only there any other Z cannot run it but is our VM now secure? actually not at all we still have to encrypt the disk itself with wax encryption and there is a headache of system administration because every kernel update we have to run nnprodimage again create new sdboot and using the IPL tool which is IBM's bootloader you have to install this file onto disk we don't use grab we use the IPL so there is a special tool which creates a mapping on a disk and you start directly on a disk so all I mentioned is good for security but it's not ready for cloud because administration, wax encryption a lot of hand work it's not acceptable so at IBM we set three main goals for open shift and chorus first goal is security must work from the beginning there should be no gap between you run your VM and to enable secure execution second user experience shouldn't be different or shouldn't differ that much because if you have to read ten pages of manual how to enable it it's not good it's also people are used to something they don't want to see huge changes and third goal was to protect ignition config I'll get to it so why ignition config is important chorus image itself contains operating system but without any user nothing so in ignition config you usually specify your user ssh keys, password, system dunits actually a lot of stuff you can create and it's your secret we want to protect it so in order to reach those goals we modified building system of chorus it's s390x only on the left side you can see how regular you call image for s390 looks like of chorus we have two partitions labeled boot and boot correspondingly so ssh boot contains what you usually have under boot, kernel, BLS entries and root root is just root for secure execution we added something more first of all vd1 replica picture is a special small partition which contains just sdboot or unified kernel image and this image is encrypted green partitions are exactly the same no difference but during the build we enable dm-verity option who knows what dm-verity is ok so dm-verity actually is a protection which calculates the hashes of partition it splits it into blocks calculates hash of each partition stores it on the file for another partition then it repeats it calculates hashes of this partition and continues until it has just one hash it's called root hash and we append both root hashes for boot and root to our sdboot image so command line contains both hashes it's like a merkle tree then we also want to protect ignition as I said ignition shouldn't be readable by anyone and we also want to prevent notification the easiest way to do it is just to encrypt ignition so during build of chorus image we also generate a pair of gpg keys private key we append to our encrypted sdboot image so after neutron affairs we have gpg private key and then command line public key you can fetch on redhead openshift website so each build of chorus contains its own unique gpg key two different builds have different keys but we still have an issue how we boot our sdboot image because each mainframe has unique key and if we don't have all the keys for all mainframes we cannot run for that purpose we have a special key called universal it's a special key which is only accessible by IBM and a bit by redhead so during a build we use universal key and sign our image and encrypt it using this key this key you cannot find you cannot fetch even I don't have access to it so only built environment uses that key to sign sdboot with universal key so you can boot it on any mainframe sorry too fast but we still lack locks protection we still have no encryption for our partitions so when user has its ignition ignition supports feature called merging so we enforce locks encryption we wrote a small ignition config which stored in nitro mfs mn system boots first it checks verity hashes if they don't match installation fails if it doesn't work again installation fails if everything is ok or condition submit we merge user ignition config with our small which creates two locks partition we completely erase the file system of via and create boot effects and root effects with locks encryption and integrity option on all the keys we append to sdboot image so image is encrypted locks keys are there keys are random so the same build of chorus image if you run it several times on different mainframes prison image has different locks keys so on subsequent boot system automatically decrypts both partitions checks integrity and if everything is ok you have your chorus image we secure execution and that's it I managed that we didn't want to change user experience but unfortunately for security we have to pay a small price user must provide his own host key he has to fetch the keys for his mainframes or mainframes he is going to use in the cloud specify them within ignition config with predefined path and that's it we take care about re-encryption, about system updates so every time system updates the kernel we create new sdboot image we append again locks keys, crypto config and that's it, you have security from the beginning in the cloud and this is the first solution which is available for public usage before I end my presentation I would like to thank people who helped me a lot during development it was Red Hat team and IBM team so without them it won't be possible to make this feature available so with OpenShift 4.13 secure execution is available so if you have access to IBM cloud or you want to try you can use secure execution and make sure that your config and your IBM is protected against hypervisor or other VMs links are appended because presentation is not that technical but if you're interested in technical details on IBM website there is a big article with a lot of technical details about how secure execution is designed how sdboot image is generated because it's not only two host keys we also generate intermediate keys symmetric keys, all the stuff and if you're interested about how secure execution works specifically for CoreS on github nmd file where all the design is written with all explanation about why we use that or why we don't use something so thank you for watching and if you have any questions please ask and I'll try to answer them looks like no questions thank you for watching, thank you for your time Thank you Thank you for your time Thank you for your time Thank you for the session for the call from this year and we'll almost at your time show you can go to everyone I can start now So hi everybody, thanks for coming I'm going to be talking about the state of continuous integration in the upstream Linux kernel it's a little bit one-sided from the side of CI and testers and it's targeted in a big part to people like me but I would also want to have kernel developers say what they think about it how they think it reflects the reality or doesn't Do we have any kernel developers here? Oh hi, great, thanks for coming OK, so I'm Nikolai Kondrashov I work at the ECI project at Red Hat I'm doing continuous integration for the Linux kernel for internal and upstream as well I'm also working with the Linux Foundation's kernel CI project and I'm developing the KCI DB project within there I do electronics as a and embedded as a hobby So we will go briefly through what's the landscape like for kernel and then I'm going to be defining continuous integration and measuring it you know, pro-legman metrics I think could describe it for the purpose of this talk and then we go over what we have and what we cannot have and what is hard and what we can do with it or at least what I think we can do with it So as probably many of you know there's a whole bunch of different systems that test the Linux kernel sample each of them sends their own reporting mails and results in their own way, in their own time during the development process and most of them have some sort of dashboard and they're all different as well So kernel CI is the Linux Foundation project that aims to be the kernel CI for the Linux kernel but so in the big part they are a CI system themselves so they check out kernel build and test them they have hardware and they use hardware provided by third parties connected to one central system so they have the dashboard and their own reports but also within the project there is the case IDB project which aims to get the results from whereas other CI systems and put them in a single database and generate reports notifications and have a dashboard for that and extract some value from aggregating this data together The structure of the case IDB is very simple you just send a JSON containing your test results, build results etc and put them in the database show it on the dashboard and then monitor what's coming in to generate subscription notifications about various results So we get around 300,000 test reports per day and looks like 20,000 builds and around 100 revisions per day in case IDB right now so the dashboards look like this it's a Grafana prototype and we have reports which aggregate the results if you look real closely you can see that this report will generate from results from four different CI systems So let's define just an abstract way to continuous integration so it's quite simple you simply test every change done to the code and all as close as you can and then give feedback on the result we'll stop on that So out of this you can come up with some metrics so the coverage of course which tells you how much functionality you are testing what's the percentage then latency how fast you get the results after the testing after the change was made or presented and then how reliable results are, how many false positives false negatives it generates and how easy it is to interpret the results that the CI system produces like how hard it is to figure out what's actually broken and for these metrics of course the ideal CI would test everything, provide instant feedback it will be always spot on and it will just tell you what's exactly broken not requiring you to go in there and date the logs and try to reproduce those problems and the worst CI would of course test nothing or test something else and it will take forever and it never produces the correct result and the result it produces basically it's impossible to comprehend so the worst CI is much worse than no CI So with upstream kernel CI nobody seems to know since there is no single system just a bunch of separate systems and they don't really verify coverage that nobody seems to know what the coverage is like so everybody focus on particular functionality but nobody seems really measures code line or function coverage of course that's not very useful I mean like it doesn't really tell you how much functionality is tested exactly but it is something that could measure could look something like this from the people who saw these reports before so this is from a single run at Red Hat CKI for run 64 and just typical set of tests and this includes just some directors most important ones but not all of them so this percentage is just some percentage so nobody seems to know exactly what's going on there typically it takes from several hours to multiple weeks to get results for your change or for whatever you merchant in a tree a lot of CI systems still do manual reviews of any complicated results because there are still false negatives and accessibility of our eyes based on the CI system some systems provide very clear and very useful reports like SysBot or Zero Day others more bare bones that's different but the main thing is that they are all different so there are of course limitations and first of all we can only get as much coverage as we have hardware because the kernel is an abstraction layer for hardware so we need all the hardware to exactly test everything but we cannot get all the hardware so that's a hard limit for CI as well as latency because to go faster you need more hardware to run on so the reliability limit is obviously the hardware reliability as well but also how well the kernel is working the kernel is buggy and tests fail all the time like in interesting ways that the creators of those tests don't participate the reliability of the CI system suffers but that's what CI is supposed to fix accessibility is again hard limit is hardware availability because often you get a failure on some hardware that you cannot access yourself, you cannot reproduce it so it's hard to figure out what's going on so you have to do the backtraces ask the owner of the hardware what's going on so it's a difficult process of course the kernel complexity is a limit for accessibility so tests can only be as simple as the kernel is ok so the coverage I think is mostly good there are many people who want to write tests and who write tests but because of the other metrics such as reliability, latency it's difficult to get that coverage out there and to deliver it to developers and it's kind of sometimes depressing how hard it is so that's a challenge so problem with latency is that of course not much pre-merci is done and when you do it you have to be extra careful because there is no authentication for patches that people send to mail it's like anybody can send patches and you would think 10 times before running that on real hardware in case it blows up for example so that's a problem you can run tests in nbms but that's a limited effect and then of course there are slow human reviews that require reviews for the actual test make it longer if the test completed right after your left work and then you come into the morning and then you start going through your mails and checking those results and it takes a while so it could delay the results and developers quite a lot so the main problem with reliability I think is the fact that tests constantly distinct from the kernel state and that's also a problem because the latency is too high so by the time test result arrives that somebody decode the debug is already in the kernel and it's getting stuck there until we fix it and then we get more failures with other systems and that's a big problem as well as well another problem is that you have multiple kernel branches multiple trees and bugs jump between them as maintainers pull the changes in and it's like hitting the flies with the flies water so and the accessibility the only problem I think is the fact that we have all different reports as developers have to try to interpret them all in their own ways and it takes time so it's sort of a network of problems that affect each other and for example low reliability and accessibility lead to reduced trust towards results from kernel developers so they are reluctant to look at them or might say I don't send them to me they are unreliable or hard to comprehend as a result gating is impossible if they are unreliable and as well the feedback is less than it could be towards decoding towards test themselves improvement is as slow so people just don't get their feedback don't run enough tests and it's not improving and again the reliability and accessibility lead to high latency because of human reviews etc. and high latency as well is another reason why there is no gating and the reason why bugs stay in public code that everybody uses longer and result in more time wasted so there is again less time for improvement you have to review those results and try to understand what's going on and this leads to even more latency so I try to summarize what I think how I think this whole system works in this little graph but we will take a closer look at this later but if I had to pick one thing it would be latency that's enough room I get to drink a little ok so what can we do with this well at least what I think we can do with this or rather what we can't do first thing that we must remember is that the kernel community is not a single team and moreover it's not a single company where you can just say like now we enable gating with whatever we have and fix tests and that way we start the whole process and we tighten the feedback loop and it doesn't happen because nobody holds anybody much the companies that run the tests and the developers kind of have to work with trust and you have to first fix the test and gain that developer trust before they will start listening and looking at those tests I think that's one of the main things in this situation so what I think we can do is for coverage obviously we need more hardware, we need to attract more companies to run tests and to provide the results and if you have your own CI system you can you don't want people to control your hardware you can send your results to KCIDB and you can also help work on the report and to make it more efficient if you would rather contribute hardware and not do your own tests you can have them connected to KCI these links lead to instructions how to do that and in any case you can write to our medallist and ask questions for latency I think the most important thing to do would be to pre-merge testing because again it reduces number of public bugs and has compound effect on many things on reliability as well so if you'd like to do patch work or not patch work but patches then you could use patch work and that's what some CI systems used before pick patches using the patch work and then test it safely as much as you can probably in VMs but while researching this I noticed that there's about 50 depositors in the maintainers who are already using GitHub or GitLab instance and that could be a way to by for example connecting the GitLab or GitLab CI to your CI system like kernel CI is thinking how to do that right now with this new kernel CI API that they're working on so the idea is that you for example publish a action in GitHub actions that takes your patch submits it to kernel CI and then gives you a check mark after it's done or a red cross and why this is better than patch work is because you get to authenticate the users who submit the merge request you can say okay these users get to the real hardware and the rest of them needs to be verified before they get access to the kernel CI and then you actually get your tests running on real hardware and that could be this effect could be a selling point for other maintainers to get them at least to have a repo on GitHub and GitLab for testing to get access to real hardware in an easy way and perhaps merge do the testing of merge requests and pull requests so since reliability is a problem like you should not try to do all the tests at once it's enough to just start with one test into pre-merge and talk to the maintainer decide which ones we need to do and help start the feedback loop well I have an idea to use KCIDB subscription for that and the KCIDB subscription system that allows you to pick for example which branch you want reports on compiler architecture like particular CI system anything so if you're interested in those results send a message to the mail or just talk to me so then the thing is the latency is that manual reviews are hard and slow so it's great if you can automate that and that's what various CI systems have been doing and that's what we are trying to build in KCIDB for example GraphicsCI does this and you can see the system for various parameters you know patterns and parameters too much and test results to detect them it's a known issue and it's already a bug open so that you don't identify your maintainers about this problem or developers and you know keep the CI working same thing is done in CKI as well as I've had and CISBOT is doing a similar thing they additionally are able to automatically extract identifying information from kernel crashes and then group the crashes into a single group under a single bug automatically so that works very well for them and with this they are able to produce a lot and a lot of crashes in reproduced bugs etc automatically without human review so this will be probably controversial so what I think is could be done is that the best way to approach it apart from fixing the test all the time etc is to make sure that the tests are in sync with the code and what we can do is you know move at least the most popular and most impactful tests into the kernel repository proper like LTP for example and integrating the kernel documentation make them official so that will join of course K unit of tests but with you know wider coverage the problem of course is that once you integrated with a single branch then it starts like if you integrated into master then it starts testing master and all the branches will stop working so you would need to backport whatever you integrated with those branches and fix them but afterwards the complexity of the tests decreases and they become more reliable because they don't have to account for different kernel versions and you can focus on testing this one and you can keep them in sync but that of course only works if we actually run those tests and run them fast so in the CI system it's important to prioritize execution of entry tests they're executed earlier and the results produce faster so that you can shorten the loop and keep them working well accessibility is not so bad we have decent enough uniform reports it could save people time if they always receive the same report but there could be different opinions of course and often it's the XTACD comics of about the virus standards that comes up so it could be just another one more report as a result but doesn't mean we shouldn't try so come to me and tell me what you would like to see there just try this or help the development of the report system so here's the influence graph again with the improvements I think should go in there and that's how they could affect the whole system in particular developer trust and the ultimate targets the stronger code feedback and test feedback which lead to better quality that's all thanks everybody any comments questions ideas there's somebody behind there so the question was when patches are being sent for maintainers to look at are there any requirements to run tests on those is that right so in some subsystems and some maintainers they do have requirements and they define the test that you need to run but largely I don't think there's anything at large and that's what Veronica here wanted to work on yes yes and I think that's what Talis is going to do talking to maintainers and trying to determine which tests should be called canonical for their subsystem and to put that into for example maintainers file to have it documented and to have for example check patch output the recommendation or requirement for the testing when you generate the patches yes so you have an idea how many of that are possible to run machines of course there should percentage I don't have a number my feeling is that it's a fairly high percentage especially things like LTP could run in a virtual machine I can only say I think that our test database contains maybe 40% tests that can run in virtual machines anybody knows better it's just my feeling I'm just poking I haven't calculated that but we do have tests which explicitly marked as can run in virtual machine and that's one way to improve this and to reduce latency and to arrive at the primary CI anything better than nothing anything that will start to look faster back get us closer to gating anything goes yes how would CTI work with respect to proprietary hardware if you're working with a vendor that has hardware that's open or publicly available could you keep that private and long with the CTI yeah we have private tests and oh yes the question the question is how does CTI deal with tests in private hardware how do we make sure the tests are not feasible outside or the NDAs are not broken yeah we keep tests in a separate repository to test for themselves and I think I think that most of the tests are still in our KPAD database but I'm not sure exactly how it is like the very secure or the very secret tests added there but we do try to keep them away from the public anyone else is there kind of open source project or technology that you use in order to run a specific test let's say I change the network but I don't have to run the whole CI so what's happening now so at CTI there is a system for picking tests based on patches we worked with it for a long time so we can share that it's all open source but we can share the experience and the approach with anyone who's interested since we test on real hardware a lot we have to be careful with that obviously not only to reduce latency but also to reduce because there's a lot of patches but yeah we have a system where we can mesh when these files changed and run the tests things like that it's quite sophisticated and works quite well yes? Is there some way in which they can trigger the test for the things they are about like they collect the range for the next so the question was the question was is there a way do we work with maintainers do we provide a way for them to trigger tests for a particular change that they want to do on real hardware I'm not sure about CKI but exactly do they have special branches but CKI picks up the monitor's maintainer trees when they push something and trigger tests and so do clients of other CI systems plus there are agreements between maintainers and CI systems that when they ask can we monitor this specific creeper where I will be pushing it will to check my things and get a report on the range that they ask to test and then get the results after a while yes? Was the question is it possible for an individual contributor to request testing of a patch not as such as far as I know so some CI systems they do pre-merge testing of patches on a mail list like for some subsystems I think that Intel's graphics CI does that and some subsystems at least one subsystem has a GitLab repo and they have CI set up there so when you open an MR maybe you will get some test run I don't know what kind of access control they have there but there is no universal agreement how to do that and there is no single CI system that does this universally well if it's yeah for maintainers from outside that's the problem of course you just try your best to understand what the problem is to review it and send an email out manually that's what our main QE person does from time to time when they notice basically human review and explanations because we cannot normally give access to hardware that's inside that had to outside people for inside developers of course it's much easier we can just get to the hardware and we provide all the help to tell them how to exactly run that test and reproduce it that's the problem that's the accessibility problem I was talking about yeah of course of course of course and like sysbot is very good at that they basically can generate a reproducer for you like a C program automatically generated that you can run and expose a problem in a syscode yeah yeah VMs in that way are great but of course CI systems they do like to control their setup they have their own provisioning and everything and to arrive at exactly the same setup that the test was running in is not always easy to communicate to maintainer like for example CTI like we have our provisioning and we have our test setup but thankfully that's not the problem most of the time it was still like it is a bit of a problem to explain automatically especially what you need to do exactly to produce that I'm not sure I answered the question running something similar but for Windows driver the problem of KVM should actually manage it to some kind of distribution for that we have to provision the VM but that's how you can yeah of course when you can do VMs that's much easier but still kernel consists of a lot of drivers and those do need to test it and that is a problem okay because a lot of the back-up that's the need around real hardware and just cutting the hardware would it be better to write test as a virtual machine test to make sure so the question was the question was could it be a good idea maybe it's a good idea to at least run the test itself in a stable virtual machine environment and pass through the hardware to the VM so that at least the test setup and framework and everything could be automated yeah that kind of makes sense I'm not sure that anybody has tried to do that yes of course like two kinds of devices again like there has to be support in the VM but it's an interesting idea especially for some kind of devices anyone else comments? it's maybe more of a comment I do have hardware I wish I could contribute to that so that the maintainers would actually run test on that and catch the bug before they actually reach the main entry but from what I'm understanding even if I go there now and try to enroll the hardware in some way it's not going to happen because people are not actually running any sort of pre-merge test to that no no not pre-merge test they still catch the stuff that maintainers merge like for a particular sub-system and you might might get you might get your test before it reaches Linux or another maintainer so that's too it will depend on someone actually looking at this dashboard result and not merging the patches because they decide not to because that doesn't mean that's a difficult problem so communication there is difficult but I think it could still be worth it to talk to kernel CI and expose your lava lab there to talk about it so you can talk to us on the chat or on the mail just ask what they think about it and give you an extensive answer and ideas alright ok, thanks everyone that's it so is it already streaming? no we will start streaming on the algorithm but we will start recording to top hat bytes so there is a difference between streaming and recording yeah, yeah, we are recording from source and then streaming is whatever happens on the network stopwatch just turn it on as soon as it closed the door I hope it happens let's begin hello, welcome to our talk about kernel development learning pipeline program that Joel and I have started at Red Hat, KDLP for short I'm Julia Denham my name is Joel Savitz and today we will give you an introduction to our program so, first a little agenda we will begin by describing what this is, what KDLP is why we would want to do such a thing then we will give a brief overview of the program which includes an introduction to Linux kernel development course some various things related to internships and mentoring and this new kernel devs program group that's part of some initiatives that we have to get people involved in kernel and grow the kernel groups at Red Hat then we will talk about some of the program growth some of the exciting things that have been happening internal and external connections connection with the Linux foundation and their mentorship program then we will talk about some of the success stats a little bit of the numbers and finally wrap up with some resources so you guys can get involved if you like so, first of all just to make it abundantly clear what is KDLP the main idea is that we are trying to build a comprehensive pipeline for Linux kernel talent and it's difficult to get people into the kernel and so we want to build a pipeline so, okay, first a little bit of history where did this come from so I went to UMass Lowell and about halfway through I did a co-op with Red Hat just completely by chance I ran into this manager who was hiring at an event he wasn't really supposed to be recruiting at and I ended up in this co-op and if it weren't for that I never would have encountered Linux kernel I never would have known where to look to get started I wouldn't have even really known about it so anyways, as sort of a side thing I became involved in an initiative to improve Fedora usability, stability and performance on the Raspberry Pi platform and then along the way so I kind of focused more on the low level stuff and there are some gaps between what I had learned in the co-op and then what people generally were familiar with from their computer science education so long story short we ended up developing this into a bit of a study group to learn about kernel drivers for the Raspberry Pi and then that was a part of a directed study like credit bearing directed study type of course sort of independent project that eventually became more of a class like type of thing with a small group we ended up making some slides through the help of the Red Hat Research Department we ended up bringing in some interns to help work on the project and the Raspberry Pi stuff the course became more developed we got more interns eventually we pitched the program to the Red Hat Enterprise Linux the Red Hat Enterprise Linux the RELS Shark Tank program and we got the sponsorship of Mike McGrath big thanks to him for sending us here and from there we continued to develop the program until it became a full on credit bearing course or a regular course academic course that we created it's you know all this stuff is open source and we will continue to talk about that later on but for now I'm trying to go over to Julia to talk a little bit about why we're doing this so why did we start this program so we noticed that many Linux kernel engineers are getting close to the retirement age I started at Red Hat three years ago full time and I've already seen a couple retired and I've noticed that it's really hard to find qualified candidates for kernel engineering and most colleges nowadays don't even teach C programming which is hard to do kernel without C programming so we oh and kernel onboarding also takes a lot longer than the average onboarding time so our solution to this was to create the qualified candidates for kernel engineering so there are three main pillars to the program the course the internship and then full time work so students will learn concepts in the course they will practice them with weekly assignments that are similar to things that they would be doing full time at a company depending on how the class goes they could get an internship at Red Hat they could be turning on kernel using all the skills they learned in class helping with the program and then depending on how that goes they could be full time kernel engineer at Red Hat and Joel is going to go into the specifics of each of the pillars thank you Julia so first of all I'll talk a little bit about the introduction to Linux kernel development course so the general idea with this is a meta overview of the kernel because as you know it's a gigantic project there's not a single developer who knows how to rewrite the entire thing no one is a master of any of the subsystems so what's the subsystem do we really focus on and the answer is pretty much none of them we go into a couple but the general idea is to teach people how to teach themselves how they can get involved in the community how they can properly interact with the community using the community standards and so the prerequisites for this course are just a general background knowledge of C an advanced knowledge of C mainly and basic Linux experience I mean generally programming experience as well is extremely helpful but the overall idea is to get sort of a tour of the different parts of the kernel some of the different tools that you can use to interact with the kernel how to navigate the code things like CScope, Elixir tracing, BPF specifically the BPF trace implementation front end and some basic F trace concepts which are things that once you're pointed to they're very useful but you wouldn't even know really where to look unless you spend a lot of time running around in circles or you got lucky enough to find a mentor or just find yourself somehow at Red Hat in a particular department so everything we do is open source of course that's open source for open course and we develop a credit bearing undergraduate and graduate university level course that we teach at University of Massachusetts Lowell in Boston or in the Boston area and that we've been running in its current form for two semesters now we just finished the last one back in May and recently we've been piloting a mentorship program on the Linux Foundation's LFX platform thanks to partnership with Shua Khan who we'll talk about a little bit later but the main idea and our main philosophy with this course is that we want to teach difficult material from first principles so then the idea is with internships generally the idea is to bring people in who have some idea to test them out in an environment where you're not making a huge commitment to them and they're not necessarily siding onto the whole thing they're trying something out so you can opportunity to vet them and do a three to six months or a longer interview and see if they are A, qualified to work in this particular area and B, that they actually like it that they actually enjoy it something they want to spend their time doing so with that we thought what if we went back a step further we're working already at the university level kind of at a semi training wheels type of internship and we're already thinking like do they like the kernel, do they like are they competent at working on the kernel do they enjoy working on it and from there we can bring them in and internships already I mean I know a manager who is saying he doesn't even like doing three month internships in the kernel because by the time they have the absolute basics it's time for them to go I mean even six months is not very long I stand to mind eight months I still barely knew what I was doing at the end of it so the idea is if we can get people set up with some of the tools, the ecosystem some of the email patches, I didn't mention that earlier but that's a very important component of our course getting extremely familiar with get and email patches because if you can do that that's a big part of the hurdle into getting involved in the community but if you can do those things if you have this competency then when you get to your internship you can actually hit the ground running and accomplish a lot more in three to six months than you otherwise would and so by the time that we actually recruit them the full time, ideally they can hopefully shorten that two year onboarding window that Julia mentioned to a lot shorter and then saving everyone involved time and money which is great so at the same time we want to support the key initiatives within Red Hat and target people towards areas that are growth areas and train new talent while focusing on areas that people otherwise wouldn't really be exposed to and now I'll hand it over to Julia to talk about some of the initiatives that we're doing within Red Hat to bring people involved and bring new people and interns together to learn more about the kernel so one aspect of the program is this group called the new kernel devs group it's a way to kind of connect all the different pillars of the program it's an internal group at Red Hat where kernel engineers of any level can come so we have a chat so you can always send messages in chat or on the email list or you can come to our reoccurring monthly meeting and ask questions give presentations like if you're new and you want to practice giving a presentation it's the perfect space to do it or if you're nervous about asking a question you can come and ask in this group so program growth throughout starting this program we have partnered with different organizations within Red Hat including Red Hat Academy they have sponsored us to go to a few conferences they've helped us out learn how to work with colleges and we've also met with their curriculum team to potentially get our curriculum up on their platform but that will take a lot of time but they were very interested in it which is awesome we've also worked with the intern program in Waterford, Ireland with Julia Blay we ran a workshop a kernel engineering workshop once every two weeks for four months for a group in Ireland it was virtual and one of the interns in that workshop actually just accepted a full-time offer which we're super excited about we've also been working with a few other intern programs at Red Hat and we're hoping that we'll partner maybe with Forno or Israel in the future and then our newest program integration is with the Linux Foundation that Joel will talk about right now thank you Julia so to build on that point about the Linux Foundation so last year at LPC we were talking to different people about the program I ended up talking to Greg Goe Hardman telling him about what we're doing he was saying oh we already have something like that a kernel bug fixing thing he said you should talk to Shua Khan so he was introduced to her and we started talking about her mentorship platform comparing some of what we're doing and what she's doing and we found that by working together and by offering our course on their platform we could provide some complimentary content and strengthen the value of the service that we're providing while also adding something to the Linux Foundation she was very helpful in working with us on this and so we piloted having a Linux Foundation mentorship group cohort in our course this past semester and we learned a lot from that but it was a very good pilot program and at the same time we got UMass Lowell students involved with the Linux Foundation's mentorship platform and got them kind of a nice get their names on there for being involved in Linux it's a nice thing that they can have they can show hiring managers it also actually unlocks some Linux Foundation programs that are only open to graduated mentees so that's very cool it also allowed us to have probably the most diverse group of mentees students, different terms that we've ever had or that we could have possibly had if we were only working in Boston we had people from all continents not Antarctica but we had people in a few African countries I think Nigeria and I don't know, there's another one some people in Central and Eastern Europe I think we had three or four in India who were interested, a couple in New York in Texas, there's one guy in Peru really all over the world which was very cool so next I'll talk a little bit about some of the numbers some of the the technicals here so this three actually it was two until like a week ago we've had three full time hires we had a program in various swarms and through our efforts we were able to get them interested in kernel teach them a little bit about kernel and then place them within various kernel groups so we have some relatively new talent there which is exciting there were two people who did co-ops with us that really liked Red Hat but it was a difficult time for Red Hat hiring and they were both credited to Amazon and Microsoft because they were good at what they were doing one of them he liked working at Red Hat so much and he liked the environment he decided the Amazon juice really wasn't worth the squeeze and he wants to come back he really liked the Red Hat environment and yeah so he actually left the Amazon recently fun fact but overall we've had seven interns and co-ops who have been vetted and trained by our program and then come through and do internships with our group in various forms some of them less connected, working with other teams mostly with us, working on the program and overall it's approximately 30 students in mentees completed introduction to Linux kernel development during the past academic year which we're happy to announce is the most diverse group ever by gender and location so a little bit of program information so who runs this it's Joe and I we have Charlie who started out as a student, then he was an intern and now he's full-time and he is our course content lead and then we have Dennis who has interned with us well he took the class then he interned for a year he unfortunately has to go back to college but we're hoping he will come back to Red Hat when he graduates so one thing is this program is not part of our full-time job we work on the side and we work closely with our executive sponsor who we'd like to thank which is Mike McGrath and we'd also like to thank Heidi Dempsey who's been helping us out a lot with dealing with the interns and here are a few resources to leave everyone with we have a mailing list that we send a quarterly update newsletter to so if you just want to be informed about the program that is something that you can join low traffic or you if you have a specific question you can reach out to Joel or I and then we have a website and that has all the course information and like we said it's all open source so if you're just interested and looking through some kernel slides that is the perfect place to do it so any questions how do you feel do you react about shifting things around to make the kernel distribution work closer to what people usually see after they let the you know after they start working computing today now they like start to make your first decisions using like GitLab, Git Hub QCI, maybe GitHub, CTI or something like that and then you need them to the actual kind of workflow like ah, not only do you know how it kind of works and you'll send your first patch through to GitHub try to send a patch through the mailing list and all that mess and so how many people would react to that so our philosophy with the Git email patch stuff that we do is that well it's very tricky to learn right it's very finicky and so the first few assignments are we provide a couple of chances where it's the real challenge is just doing the email patches like you send some email patches, you get it going and the idea is it's a stripped down version of Git where you really understand the fundamentals of what Git is because there's a lot of extra abstraction that's added on GitLab, that's added on GitHub, those are very useful tools but they kind of build on this abstraction that Git was originally created by Linus Torvalds to work with his other project, the Linux kernel and so by making people do it many many times they get really good at it and they get a really solid understanding of firm foundation about what Git itself is that would allow them to go to GitHub, to go to GitLab and then in their minds have a more clear separation of the abstraction of GitLab and you know, pull requests, like what is a pull request well on the mailing list it's someone sends an email requesting a maintainer to pull from their branch right, it was a little confusing I remember when I started out to say pull request, like how does that relate but I think by learning, you know, the original workflow and really hammer it again so they really have it well trained in their mind, they get a good understanding of Git, they get a solid foundation of working with the kernel and then they're able to better understand the other abstractions and I think work with them even better. Introduction to the sub maintainer or maintainer or that sort of thing where it's an explicit, you connect someone with the sub maintainer in the part of the kernel or do you describe how to prevent yourself, how to interact how to submit your patch how to how does that part work? Well we probably could get more sub maintainers involved that's an interesting idea but what we have them do is they're submitting email patches against a base repository that's like our assignments repository there is a point where they do generate patches right now but they generate patches against the kernel and they include that patch as a part of another patch which is a little confusing for them and then they send it to mailing lists where other students as a part of their grade they review each other's patches and they get feedback from each other. It would be interesting to integrate more of the sub maintainer kind of workflow but I think that's an interesting point that we've integrated yet but it's a good idea Yeah it's Yeah We need the question We partnered with Edu Lipta Challenge I think is what it's called is the question and the answer is it's been shut down they're not letting other people in apparently it's a bunch of bascripts that people don't want to maintain anymore and I mean we probably you know I'll talk to Greg at some point if I see him maybe I'll mention it but I think I think they want to move on from that at this point unfortunately because it was sort of yeah Oh really Yeah Yeah so yeah the response is that maybe there's a new model going on Yeah I mean we've discussed this with Shua Khan a little bit and she was saying like she you know it's very they're very protective also about the code you know because they use it to you know the source because they use it to evaluate people and they really don't want this up getting out there you know some of the evaluation stuff you know you have people all around the world trying to try to break this stuff and you know show that they can I mean unfortunately you know kind of falsify that they've actually completed some of these challenges but you know it's interesting I mean definitely it's definitely an interesting model funny story someone actually mentioned they came up to me after I did this talk at Boston right and they're like hey you know do you remember Edu Lipta Challenge when was that from you know like what and it was like oh like back in like 2015-14 like well yeah I was in high school like I didn't know what was what was going on with Linux at the time so it's a bit before my time but it would be interesting to try and work with Greg if he's interested in that but I don't know from what we heard from Shua it's no longer operative There's no other questions thank you for attending our presentation That was the best version Should I start? Hi everyone so this talk is called Secure Python Development Tips, Tricks and Tools I'm Dominic de La Rere I'm a software engineer at Red Hat I thought I had Red Hat under software engineer but I think that disappeared somewhere I work mostly I work mostly on internal tools and and internal processes within Red Hat so what I'm going to present to you is not a project it's not really even anything new exactly what it is is a summary of things that you can that you can get started with if you want to bring some secure development practices into your Python projects this is just a summary of what I've discovered as part of a team working on trying to make our coding more secure and particularly in an environment that uses a lot of Python and uses a lot of CICD so I'd like to lay a few ground rules the first is that I'm not the security police I can't tell you what to do I'm not your mom so anything I say here is advice from my experience it's not the law security can only be improved not perfected you've probably heard it before if you've read anything about security but it bears repeating that anything here is not going to make your coding perfectly secure it's only going to maybe help you get to that next step of a more secure environment also I'm going to mention a number of tools here you'll see logos for different projects that you can use they're not really endorsements these are only the tools that happen to be familiar to me and they're here as examples so I'm not recommending one thing over another necessarily I'm going to focus on a few different areas for secure python development so first I want to focus on secure dependency management basically how to know that you're pulling in the right python packages and formalize that process secrets management because like any software project a python project can have a dependency on credentials api keys etc and python has particular ways that you can manage those safely static application security testing or SAST again this is not unique to python but python has a unique way of interacting with SAST some basic network security reminders around python how to enforce best practices in python coding so that you have a common starting point for secure development in your python project so here's the python package index I think I don't want to speak for everyone but I like the python package index I like being able to pip install things it's very nice but there are some security challenges around that model of installing software and adding dependencies to your project so the challenges we have are isolating python packages from the system-wide python making sure that the library versions match across different environments this is a challenge for everything but obviously one python environment could have different library versions from another so that's another challenge vetting package releases before trusting them in your application just as you would in any other language that has its own package management system but we have particular tools for that in python and verifying download integrity just the basic making sure that you are downloading in a verified package mitigating against and in the middle of text etc so in terms of isolating packages from the system-wide python the very basics that you likely have heard already are like please don't use sudo pip install on the on your main system so instead you want to be using a virtual and you've probably heard that they've been around for a while but it bears repeating but it also helps to use tox to provide dependencies tox is a great system for running all your unit tests in one place and in a predictable way but it can also provide kind of a front end to various other things that you might be doing with your python application or python scripts it provides a very reproducible way to install dependencies right when they're needed into a temporary environment and tear them down when you're done so that's good for lint checks and unit tests but again lots of other stuff you can run with tox you may also want to consider something more comprehensive than tox there are whole python packaging systems that will integrate that kind of dependency management so those include hatch and poetry those have both been recommended to me and we definitely want to make sure that library versions are matching across different environments tox for example takes requirements.txt often a requirements.txt doesn't have to have versions in it you could just be writing the names of packages and that could result in inconsistencies one thing to help with making sure that you have the same version everywhere across different development environments is pip tools pip tools is a project that has basically two main utilities in it pip compile and pip sync pip compile converts a requirements.in file to requirements.txt file so in a requirements.in file you can be a little bit vague you can say I want requests but you don't have to say what version of requests that you want if you're okay with it having a strategy of pulling in say the latest available version or the the current default and a really nifty thing about that pip compile is that you can generate hashes flag to it and it will check some sh8256 so a nice secure hash that you can use to validate those requirements when you're pulling them in so later when you run pip and install all of those it's actually checking hashes as it downloads the packages the annotate flag is also very useful if you've got this in a shared project and you want to see where a dependency came from because you install requests and then it pulls in something else there are certainly packages with much deeper dependency chains so annotate will show you how you end it up with that dependency in your project in the first place pip sync is the other command that's in the pip tools projects and that will update your virtual lm to help you keep those packages fresh so this is a smaller solution than one of those full packaging solutions that I mentioned earlier like Hatch or Poetry but it helps you manage the dependencies in a similar way of course as with any language with a package manager you want to be vetting the package releases that you are using them before you trust them it's really simple but again just asking the basic questions what other projects are using this package who maintains it do they respond to bug reports how active is this project and you can be checking change logs to see what's coming into the project what new features are in there to help you with checking those dependencies that you have there are dependency scanners so the example I chose for this presentation is called safety safety will scan your project for dependencies that have known vulnerabilities and tell you how to remediate it so if that means that there's an upgrade without that vulnerability it can recommend that or even if there's like a small downgrade that reverts that vulnerability it can recommend that as well you can integrate it into an ACV pipeline the thing is this is this is an open source project but it's on a model where for non-commercial use you can use their free database it's very good but for any kind of commercial use their license asks that you or requires rather that you buy an API key from the developer behind safety which is a PI up verifying download integrity is pretty easy once you've got a tool like those package managers I mentioned earlier or or sorry PIP tools so there are specific suggestions that the PIP team has for verifying download integrity the first is that you use PIP instead of running a setup.py setup.py is arbitrary code that you're invoking I mean you could read it first obviously but it's considered safer to use PIP these days because it does more than just running the setup.py there are certain checks involved you can pass a require hashes flag to PIP and that will verify the downloads using those hashes that you added in with the PIP tools and you can use the only binary flag that's recommended so if you run the only binary flag it will pull in binary packages instead of pulling in the source package which means that it will simply place the files where they need to be instead of running setup.py etc so this is kind of like an extreme case this is a lot more work but if you have an environment where you really need to place a barrier between yourself and PyPI there's the option of caching PyPI you can have you can have your own mirror of PyPI you can vet everything that comes into it all the updates and you can choose what you approve and what you don't approve this is just one example of an enterprise solution that one company has proposed they call it nexus repository I just provided that as an example but I should also mention you may have seen the table for a project called pulp lying around so that's another example of an open source project that's out there and we are trying to provide various ways of mirroring different types of repositories not just PyPI but other sorts of repositories as well now like anything a python application might need passwords, keys certificates, tokens these are all secrets the things you keep secret the obvious thing that you probably have drilled into your head don't hard code secrets into your Python source code but also don't store them in secrets sorry, don't store your secrets in source code management there are private source code repos obviously but really no source code repo was ever meant to hold secrets they are meant to share things not to keep things secret so if you really have to store secrets in SEM because I'm not a security police and I can't tell you not to do it, then at least use GitCrypt so that all the secrets are PGP encrypted GitCrypt is a transparent process that you, it's a plugin for Git so once you integrate it with GNU PG it will use your private public key pair to just transparently encrypt any secrets that you're adding into the project it's a little more complicated than that on a project with multiple developers you have probably multiple keys from your team but it will handle the math essentially of using all of those keys to encrypt the secrets so that any one of your developers can then go back in and decrypt them but that will probably make road vacation harder if you ever have to do that so rooting out plain text secrets in source code management also has some tools so some examples are projects called detect secrets and git leaks you run these either locally or in a CICD pipeline and they scan your project and say hey this looks like an API key or a password often it's scanning against lists of known credential formats or it might do a little bit of entropy detection but either way the idea is to point out these might be secrets and you can either mark that as a false positive or you can take it out of your source code and obviously rip out that secret you don't want it live anymore so anytime it's possible of using a source code management for your secrets you want to use like an actual secrets management system if you're sharing that with a whole developer team hash of corp vault is the big commercial example out there I mentioned it here because it happens to have good programmatic access to secrets and the way you can do that in Python is with HVAC the Python client for hash of corp vault that's a community supported client so it's not an official part of the supported hash of corp vault system but it's good and it's out there but moving on from secrets we have security linting there's a good chance you're already using your code for style but a security linter does something similar where it builds an abstract syntax tree from your code style linter will usually throw up an error if it notices that your code isn't syntactically valid and then move on to style checks security linting instead of going into style checks we'll go into security checks that will look for common security issues a good option for security linting is bandit that's open source again run it locally run it in CICD but it just looks for common security issues in your Python code including things like for example you've run a shell command with a sub processor whatever like standard library and you've got in Python for running shell commands and you've just thrown a variable in there this will throw out an error and you have to decide if it really is a security vulnerability in the context of your project so for example you can mark this as a false positive if what you're developing is a tool that developers run on the command line in their own environment and they're allowed to give input that is going to go right into a shell command they're just doing it on their own system that's not really compromising security or for example you are already thoroughly sanitizing variables that you're putting into your shell command you can say well it's not a vulnerability here because we've thoroughly sanitize these variables but maybe it is a vulnerability maybe you're taking user input from a web application and putting it into a shell command and then you'll want to fix that instead so static application security testing can kind of pull all of these things together there was a great talk earlier in the conference about effective sast so I won't repeat all of that here but the examples that were given there used mainly semgrep which is nice it's an open source sast tool the example I decided to provide here is sonar cube so this is kind of an open core type thing that you can use for sast there is a community supported open source version of it and then there's an enterprise version that they sell it does do all those dependency checks that I talked about that you can do with something like safety but it also performs the security linting and it provides you with what are called code smells so if you're not too familiar with sast a code smell is essentially something that is maybe just not best practice not necessarily a security vulnerability but something that makes your code maybe more prone to errors in the long run so for example the first time I started using sonar cube I got some code smells about strings that I was repeating string little rolls that repeated a few times in the same piece of code and it said hey this could be a constant which I hadn't thought of and I just went and corrected that easy to do it will also pull in your unit test coverage metric so if you're running pie test maybe you're running pie test through talks or you're running some other system or you're running all of your unit tests it can take the output metrics and parse them and a wonderful thing about it is if you're integrating this into CICD then it's going to be able to focus on just the code that's changed in a merge request or a pull request and tell you what percentage of that new code is covered or how many code are in that new code so for gating that's very good because you don't have to filter out a bunch of false positives off the bat you can just focus on improving the code that you're checking in now so you can start adding better formatted more secure code now and gradually improve the security and formatting of your overall code base and you can set up I said you can set up gating but anyway that's basically you've got a percentage I need this percentage of coverage or I need it needs to be under a certain threshold of code smells in the new code etc but this is again just one example I like it for this presentation because it does integrate with like high test coverage and things like that it will tell you how those metrics apply to your code, your pull request so basic network security reminders again nothing new please use TLS so anytime you're interacting with APIs you really want to be using HTTPS rather than HTTP as your prefix you want to be you want to be using that certification and this is like a super niche issue but as just kind of an example of the way this can go wrong say you're in an organization that has like a self-designed certificate for things on an eternal network requests has this weird quirk where it doesn't use your system CA bundle to verify TLS so you might have a custom certificate authority installed to check these things and requests just won't use it so the temptation in this example is to do something like just turning off SSL or whatever for certain certain operations but instead you can just use an environment variable to point to the your CA bundle so this is an example where unfortunately the secure thing to do isn't the easiest thing to do it's kind of a weird edge case but I like this example because it points to an area where you would ideally like to make the secure thing to do just the easiest thing to do the path of least resistance anyway we also want to check for precise correct URLs we live in an age of typo squatting you've got two letters switched up and then you end up at the wrong URL that is controlled by some attacker you want to specify the fully qualified domain name if you're doing container image polls this is not directly related to Python but a lot of us are going to be working with containers connected to our Python projects and there was an old way of doing things where people didn't specify like docker.io slash name of container and that's not good because nowadays there could be multiple registries the order that registries are queried in could be different and that is a similar problem to typo squatting and you want to rely on standard number cryptography and authentication libraries you don't want to be making your own solutions because there are tested solutions and if that's not the focus of your project then there's really no sense in creating something that's less tested now I want to talk about enforcing best practices on a team project you want to set conditions for code quality and some of these things might not seem security related the issue is that code that is following certain best practices and that is consistent across a team is going to be code that's easier to review and good reviews of code lead to more secure code in the long run so you want to be running unit tests on all merge requests you want to have unit tests to begin with I think I bring that up again later but it's always good to repeat you want to create code coverage or SAS code quality rating thresholds that's the gating that I was talking about earlier so there's a number of ways to do that SAS makes it pretty easy you want to be using code formatters maybe not everyone likes code formatters but they can help if you want that consistent code style across a whole team for again easier code reviews so if you're going to do something like that I recommend black they call it an opinionated python formatter you may have seen it around not everyone likes it because it has very specific ideas of how your code should look but it also takes all the work out of formatting linters like flake 8 are also good so maybe you're okay with having a looser format but you want to avoid anything that the python peps tell you not to do for example flake 8 is good for that and this is really what I said at the beginning of this section but code style can be something of a security issue just because it's easier to be on the same page when you have a consistent style but back to like these super basic things that you probably heard a million times already use python 3 if you've got legacy python 2 code I know what it's like to have legacy code lying around there's ways to make it a little bit easier there's libraries like 6 that you may have used years ago and maybe you want to pick them up again they're good for bridging that gap as you're moving from python 2 to python 3 obviously the reason for that is just that python 3 gets security updates and python 2 doesn't get updates anymore you want to have unit tests definitely so many things that you could catch with them that you might not otherwise notice and it can be incremental you can have like you can go from no code tests to like I don't know 10% coverage and that's better than putting it off until you can reach 80% so if you want to start with just requiring a certain amount of code coverage for new code that's coming in that's often a great approach to build up coverage over time and you want it to be meaningful coverage you want your test to actually make some kind of sense but if your test is just running a function then at least you're catching something catastrophically wrong that prevents your function from running so some coverage is better than no coverage sorry go ahead requiring coverage for new code I have an example of a way to do that technically or is it like manual engaging on a modular request yeah so the SAST tools that I talked about can apply code coverage metrics to just what's included in a pull request or measure request so it's not too manual it's like actually fairly automatic you can even I most experienced with and if you're integrating those kinds of tools into GitLab CI you can actually say throw up a badge on something if it doesn't reach that code coverage threshold sorry on just the new code so like if the new code is beyond 80% covered then it gets a green badge if it's like under then it gets a red badge and the code reviewer says hey it's not ready yet so that can be super helpful it doesn't have to be manual it can be easier if you have the right automation in place thank you that was a good question using established libraries for cryptography is super important I think I already said it but like again don't roll your own crypto as they say but also don't make your own SQL requests these are maybe obvious things but like maybe you're coding fast and you start to get into these habits of reinventing the wheel so don't there's great libraries for handling SQL if you're doing SQL and HTTP requests and stuff you probably want to have like a good framework you probably want to have whether it's something as big as Django or something lighter like you need Flask or something but you probably don't want to go into this territory where people have already done the work of doing these things securely and just kind of like forge ahead instead you've got community supported projects to lean on so I'm going to end now if anyone has any questions I'd love to hear them from the the food at the end of the beginning with Django 5.6 questions so can you imagine those things getting over the ages for example working libraries so how do you reconcile those two things do you have some pretty early bombings of libraries I think that's mainly what Pipsync is for you can run that periodically it can help you upgrade things or it's been a while since I tickered with Pipsync but like pip compile itself also you can run it again you can it will regenerate things and there's different settings you can run so you could basically wipe out your requirements.txt and re-run pip compile on the requirements in but you don't have to do it that drastically I don't remember all the options but you can do just portions of your dependencies that you pull in the newer versions I think you can target oh I want to upgrade this dependency and run pip compile that way and just say upgrade that one the rest of them are fine so there's some customizability there yes please poetry so as we're at this particular conference are you aware of the way to make poetry involve just the project you're in like the one you downloaded and what to work with for using in a spec file I have to be totally honest I have very much no experience with these comprehensive systems like poetry or hatch they were recommended to me but personally I've been using the approach of using all these other individual tools to focus on the individual problems instead of the all encompassing thing I think I'm sorry I don't have the answer but if you give a talk like that I will attend it any other questions yeah I'm not super sure about the provenance of the security linting within sonar cube for example I think there might be some in house stuff going on there but if you're not running something like sonar cube that's already going to do security linting for you you can run bandit as a standalone thing any other questions thank you all for coming especially those of you who are here live in the room in the middle of a Sunday thank you and we're almost on time thanks for joining me for this last session of this year's defcon oh well penultimate session I should say because there is a win-win-win session today with amazing prices I hear and amazing quizzes so thank you for bearing with me I'll try not to make this too long you know you've had an amazing conference and all your heads are filled with these amazing ideas and the new information that you got but the thing that I want to talk to you about sort of ties into the overall theme of this conference and that is that nobody knows that your service is open source so this doesn't work let's use that what is the talk all about first of all let's start with the challenge of services that are open source then let's continue to understanding what does it mean to be an open service what are the basic requirements for any given service to be considered open source and then let's actually decompose that a little and look into an actual tangible list of things that you can check and that you can check against to verify that your service is open then let's go into one example and see how open image builder is and finally let's figure out how you can let everyone know that your service is open source which sort of you know concludes the talk with a demo and then finally a quick poll for a logo for all of this so the challenge of services go watch the keynote this is something that we've talked about at the keynote of this conference I can say a few words about it but I don't want to talk about it at length because it would really mean repeating the keynote so open source is facing a challenge services and managed service offerings do not necessarily have the same implications that other open source services have for our way of consuming the software you know we're not really copying it onto our drives but we are a way of inspecting the code because you know if the code is not here how can we make sure that we can actually look inside and also for changing it and then for contributing it back so our entire flow basically is questionable all of a sudden and the question is how can we fix this how can we improve things and that is a very general and big question that I can't answer here and right now but there are already services to open source and it's just not really discoverable about how we can use our sort of the workflow that we know from contributing to open source and from inspecting the code how can we apply that to services and that's something that I want to look at today with you so first of all let's start with the first bit what is an open source service Steph already mentioned this at the keynote we have two very basic requirements that we have to define all the code and assets have to be open and the public should be able to contribute I mean these basic requirements apply to all open source software I would say like generally that's what it means you're able to look at the code and to look at everything that makes up the software and that you're able to contribute and the same applies to services but as I said in a slightly different way because you don't necessarily download a copy of the service locally and run it so you're not able to necessarily inspect it or to contribute back so let's decompose that a little bit let's look into what could this mean like what could these two requirements mean because just looking at the two of them makes you think wow this is a bit incomplete or at least it's very general you know like sure the source code could be somewhere and it could be accessible it's possible that you don't really know about it or that the contributor workflow isn't even described because nobody ever thought that anybody outside the team that is actually building the service is running the service would ever have any interest in contributing so how open is a service the first two requirements sound really familiar you want to have open assets and you want to have an open contribution workflow but at the same time this implies many things around those two requirements that also need to be met so I will go through all of these in more detail now but just roughly you know if you have a contribution workflow that heavily relies on documentation being available you need to be able to follow instructions you need to learn about the service to be able to contribute back you need to be able to submit tickets or read existing tickets in order to be able to understand you know which issues have been discovered for what has been figured out before you need to have a way to communicate maybe with the authors and maintainers of a service so open communication is definitely key and then finally and that's a bit of a stretch goal of course open operations so open SREing of a service so making that as open as possible actually Michael also had a talk today on incident management he came up with this open source service scorecard system that I based these six points on I think they make a lot of sense they may not be complete but I think they give you a very good starting point for understanding how open a service actually is so let's look into what open assets means because you know it sounds very obvious in a way but what does that all entail and which parts of the of a services code are necessary you will discover the differences between traditional software and services already so the first one is obvious the source code is open and the second one unit tests are open that's also sort of something that we're familiar with you know like we have unit tests in our software in general we might even have performance tests and functional tests but these become more and more important when software is long-living and running as a service then you need to have these types of tests necessary of course is also that all the dependencies are open source because otherwise how would you be able to understand how systems tie into each other and how many dependencies make up the end product you know the final service that you're consuming and then the sixth point is you know where it really differentiates you want to have the metadata on how the deployment works why it says metadata and not just deployment the deployment is open you will have secrets in your deployment instructions you know you might have a yaml file lying around somewhere that's called deployment yaml and it describes how your service is being run I don't know on OpenShift or on any type of cluster and you of course don't want to expose your secrets but you want to expose the template you want to give people an idea of you know how is the service actually being run really nothing bad about that I would say there is very little risk involved with telling people how the service stood up then the contribution workflow the first point is very obvious probably that external contributors can follow the same workflow as team members it sounds very obvious and it's something you know from traditionally working on open source software we used to you know we download the source as I said and you know we fiddle with the files and maybe it's very nice and included some editor config or other sort of instructions that your IDE is set up in the correct way that your contributions will go in smoothly but overall sort of traditional software is meant for local execution while services are not so you need to have a way of actually following the same workflow as a team member that means that could mean developing against the staging instance that is publicly available it could also mean locally bringing up the stack in some way be it through Docker or actually running a deployment locally if that's feasible of the service then of course the second one is also very obvious that this workflow needs to be documented it sort of ties in with the documentation requirements I'm not repeating it there but this workflow has to be very accessible and readily available the third one is sort of again opening up the workflow to everyone contributors that regularly contribute to a service should be able to run CI why is this I mean why do we have to mention this whenever we submit a pull request CI runs and that's it but running CI on services can be expensive it depends on which kind of testing you're running sometimes you're bringing up an entire cluster trying things out maybe there are other services that depend on your service it could be a very expensive operation so you might not want to expose this fully to everyone at all times that's why it says regular contributors so that you get the mental model of people who regularly contribute to your project get a way of doing this be it through a team member approving it or having a separate group and role based access control where regular contributors can get in and then their PRs will automatically trigger CI and then the fourth one is really more of a more of a social thing I would say I mean you can codify it you can make a rule in your team or you can really make it very clear but external contributions shouldn't be stale they shouldn't go unheard and this is something that the more contributions you get the more challenging this becomes in any type of open source project but external contributions are basically the future lifeline of any project if you don't review them at some point this flow of contributions and this interest might dry up and the project might die out and the same is true for services so this is very important in terms of having an open contribution workflow issue tracking and planning that's very straightforward I guess having an open issue tracker and exposing all the issues to the public and having an open roadmap I guess the level of detail that you include there or sort of what that roadmap entails that may differ but overall giving the public a way of understanding which direction your project is heading or which direction your service is heading which things you intend to implement next documentation is also fairly straightforward you want to have user documentation in this case the developer documentation needs to include things like operational instructions so it needs to be clear what the people that actually built the service expect around the maintenance and the running of that service well communication sure you need a public mailing list of sorts or a channel where people can reach you it could be matrix or IFC or anything else and ideally there would also be public meetings this is a little bit optional requirement I would say because public meetings really only make sense when you do have an audience of regular contributors that want to engage with you so I mean offering it and setting it up initially might be nice but if you discover there's no interest there's probably no use in maintaining public meetings when nobody joins ever but I think this is very good to keep in mind that the entrance barrier sometimes can be very high and very difficult for people to get into a new project and then finally sort of the stretch goal having an open status page having open logging monitoring and alerting that of course implies that you have a way of effectively concealing secrets in logging so you have to be very intentional in your logging you need to really know what you log and don't just print everything to a console and then just open it up to the public that might be actually exposing a service to attacks or vulnerabilities but I think it's sort of if you do it this way if you open up your logging then you very intentionally set up your logging so I think it's just a very good mindset and very good practice and open incident management that is really what also Michael talked about I think two hours ago or something in his talk open incident management is a very good practice because there is no shame in incidents that really happen everywhere there are some really good examples I think also from companies like GitLab really having very open incident management and yeah I've never really heard anybody laugh about their incidents it's something that of course you're not proud of but the important part is that you learn from it and yeah that you also give others an opportunity to learn from it okay so let's go into an example this was a gap analysis that we conducted in our team nine months ago so we looked at these criteria that I listed for you and we asked ourselves how open is our project what I need to add and I don't want to tell you what image builder is because Andre very skillfully did that about 40 minutes ago so the important bit to know is our project is a hybrid project in that it is offered as a version that is packaged in Fedora and in Centro Stream and in Well and it's also run as a managed service so let's put it this way we had probably an advantage to services that only run as a managed solution parts of our openness I think come from from this aspect you know you could always run the service on your laptop or on your local deployment so therefore you were able to download the assets and inspect the code anyway and all of our code was on GitHub at all times so the first requirement was easy for us to satisfy the contribution workflow was maybe a little trickier but we luckily had an existing Docker stack that would bring up the entire managed service like you could do both we have a basically a way of setting up the development environment for the managed service but also for the local deployment the next one was already a little more critical so our team is working at Redhead and of course the issues that get created on GitHub are public and of course the issues created on Buxilla are to some extent public but we have an internal Jira instance that we use for tracking and planning and where you know where we create sort of the tasks and the issues that make up for our plans and for our roadmap going forward and that was still internal at that time also documentation there was a very noticeable gap I mean our documentation was there you know it was all sort of upstream and also available to rail customers but definitely not about the service so basically there was some developer documentation that was really geared at people running the thing on premise and running it locally on their computer but never like we never thought that it would make any sense to include service-related documentation because well our team is building the service we're maintaining it everybody else is just consuming it so all we really need is user documentation but if you really want to enable anybody else to understand the service to be able to contribute to it you need to open up that information as well in terms of communication again I guess we were mostly thinking at the time you know it's our team that's working on it we don't really get a lot of external contributions you know why would we have much set up there and we basically had an IRC channel and we said if anybody wants to reach out you know they can but internally we were using Slack in the team and then I remember I think one year after I joined I hadn't looked at the IRC channel in a while and you know I looked and then every time the day goes by in the IRC at least there's a line injected and then I scrolled up and somebody asked the message and I said there's really weird and then I noticed that yeah that's really not that's not really the image that we want to convey to any type of external contributor that we don't really care about that so yeah that was very helpful to realize that actually our ISC channel was orphaned and that nobody would really be encouraged of interacting with us if we don't react and then finally open-sighted reliability engineering at that point in time we had none of this we had no open status page no open logging, alerting or incident management we hadn't really thought of this at all I mean maybe another piece of information at that time I think when we ran the gap analysis our service was I think in production for something like five months it was publicly available so it was still very early and that's I think why it was also very valuable and great that we were able to tackle these things very early on so how does this look now like basically from a couple of days ago I updated this slide and I looked at all of these things again but first of all there's no more reds anymore there are still things that we're not yet doing but in aggregate in every section we have at least one green check mark and yeah green and red for me makes orange in this case so if you look at issue tracking and planning this was a bit of a process it meant informing the team informing the business, anybody who interacted with our issue tracker it is public now there's of course still a possibility to write hidden comments as there is in Buxilla or probably in any issue tracker because sometimes you want to put information there that is somehow relates maybe to secrets or to customer information that you don't want to expose but generally speaking all of our tickets are open in terms of documentation we worked really a lot on our service-related documentation to make it easy to understand for everyone looking at it how is the service deployed what's the architecture which components are involved where are they deployed how is everything working in terms of communication we started by creating a public mailing list because we figured that if you have to actively look into an IRC channel to get a message if that's not really part of your routine then messages may get lost but everybody somehow looks at email and if the email pops up in a mailing list there is a big chance that somebody will see it and will respond and we also registered a matrix channel and several team members were very active and wanted this so I think it will not be often soon so one of the remaining gaps in this area is that we still won't have any regular public meetings but as I said for me this is sort of more a thing that is well ideally we would have public meetings but then we'd also have to have an audience and we'd have to have regular contributors we of course have that because within redhead we're collaborating with other teams we have meetings with those other teams but we don't really have an external audience yet and I would hope that this would become a thing so that is the hope and open site reliability engineering we have an open status page now so you can look where the image builder is working and running and we still don't have open logging and I think that we will work on open incident management because I think this would be very valuable and it's very achievable I am at this point less sure about open logging and open alerting maybe alerting but logging I'm still very uncertain about I think that would take a lot of effort from our team right so this is the example I mean as I said our team is working at redhead and we're not the only ones going through this process so this is really an effort that goes across many teams in redhead and this is why I'm particularly proud of the next slide which shows that on console.redhead.com and in the inside services all of these teams have gone through this gap analysis so these are both screenshots basically of the analysis that they've done you can see there's a lot of greens there's also a lot of reds but they're working on it and this is something that I am really proud of you know that really speaking people have really understood that message so when I approached them they were very happy and very ready and actually Steph was key in that effort and so yeah I hope that we can get to similar results that we got to with image builder with all of these other services right we haven't really spoken about the actual problem that you know image builder is an open star service we consider it one but you probably didn't know that this room maybe you knew about image builder from under this talk or from Achillea's talk but you probably didn't know that it was open source or how open source it was so we've defined four requirements together with Steph like what how can you make sure that your service that anybody recognizes that your service is open so first of all there is a visibility requirement if you know you should see something that lets you know that the service is open secondly there is sort of a criteria of understanding people should be able from the reusable thing they should be able to understand what is the service architecture what are we looking at what are the components that make up the service the third one is actually more tricky than it sounds it's introspection somebody should be able to inspect the code but not any code if you want to create a skitsha or your experimental branch they should be able to go to the exact version of the code that is currently running in production because otherwise really it's a little pointless you might have fixed things in your main branch already but maybe for some reason it hasn't really been deployed yet and it would be really misleading it could also mean that if you run into a problem with an API and you want to look into the source code maybe there is no typing or nothing that will prevent a certain error from occurring if you don't really have the code that is running in production you might not really get to the answer you're looking for and then finally also contributions somebody looking at that at that reusable pattern should find a way how to contribute right that's it, doesn't sound very difficult overall I think and we've tried to come up with something that I will show you now and if you go to this URL you will also be able to experience it so this is image builder in reddit's console.reddit.com and at this point in time and we will be talking about this in the logo poll this looks very integrated so here you can see a little branch icon that sits up here and if you're a software developer you've probably seen this before and you might think what does that mean and if you click it it tells you this service is open source and you can explore all the repositories to view and contribute to the source code following this link I've conveniently opened the link already in the next tab that's what would happen anyway but just to make sure that we don't get any network hiccups let's just jump to that page you can see immediately what is the architecture of the service you see all of the components that make up image builder at this point you see the front end you see the back end that both of those are in the console.name space you see that image builder composer is actually in a different cluster somewhere completely different and then you also see that there's a work of lead and there are some databases that are running attached to both services if you want to go into one of these when you want to click into one of these components to explore the source code you can of course click and you will see that it does take you to the exact hash that is currently running in production that works for all of the components of course as you can see this satisfies the requirement of introspection of being able to introspect the code that is currently running and the criteria of understanding you can see which bits and pieces make up the service and maybe from your error or from your interaction error you will already know oh this was a front end error whatever I got some weird pop up that shouldn't be or the button was three pixels to the right and I really hate that let's fix that upstream that will be my reaction probably then of course you will be wondering how can I fix things upstream and then there is this section that tells you how to contribute so we have a developer guide we have actually a development stack that you can set up both for the front end and for the back end the back end of course includes the entirety of the stack in this case we have a way how you can reach out to us sort of the matrix channel the mailing lists but also the issues and pull requests the links are still missing here I should add those in but they are also publicly accessible and then finally you will see the assessment that I showed you as part of this presentation also here so you can see which things we are actually doing and which things we are currently not doing I will probably try to improve this a bit over time you know and add more hyperlinks that you see that the source code is open and unit testing so that you get a bit of a hint to each of these requirements how we satisfy them but at this point in time I really just wanted to show how open the service is and make sure that everybody who finds a gap or thinks that we don't deserve a green light where we currently have one raises it with us that's one of the reasons why we want to make this transparent and we want to expose this right so this is how it looks this is basically the thing so a badge that you can slap on your service makes people understand this service is open source but that also takes you to a page like this that then explains you know how does the service work which version of the service is running how can I fix issues in it right and now as I said this is just a button but we did try to make this as easy for you to use as possible it's really just three lines of code it's a reusable thing it's always to create something reusable also to make it consistent you want to have the same experience at all times it's very easy to write the java script thing that does a pop up and stuff and then every service the wording is different or maybe the icon is different the placement is different you really want to have a consistent experience that's why we started this within our organization to use the frontend components library and just add this there the open source batch and as you can see it only takes one argument and that's the URL to the page that you want to refer people to we don't want people to change the wording we don't want people to you know fiddle with the icon or anything like this it should always be uniform because that is how we satisfy this first criteria of visibility it should be recognizable you should immediately know what to expect and how to get it and then finally we're at the interactive part of this session we're currently thinking of what kind of branding do we want for this you've seen that I currently have added a branch icon because that was available in pattern flies icon library so that was easy but we were really thinking or really into minds a little bit do we want to create custom branding something that stands out something like the OSI logo that after a while you get repeated in many many places you will understand what it means and it will mean something very specifically and that would be opens our service and that would be option two custom branding or should we go for something that we currently have maybe add some color coding to it just to make sure to make it stand out to not make it sort of look too integrated into the service that's very technical it means something to everybody who's a developer so maybe it already reaches the audience but maybe not so I think we're a little bit into minds here and I would be really interested after you've all heard my talk which option you would actually be interested to see or which you think would really solve that problem of having an icon seeing it and knowing I've seen this before if I click here this service is open and I can actually inspect the code so maybe just with a very quick hands up I know you had very little time to think about this now but maybe just give me a gut shot two of you would go for the option one basically the known icons the forky me the generic icons okay that is 15 and who would go for option two custom branding okay are you holding both of your hands up okay so that was 15 to 9 that sounds like there's a slight preference for the forky me direction we haven't really decided anything at this point I'm very happy to receive more opinions on this to be honest but generally speaking I want to create something very soon that can be reused and that can be shared out through all of redhead services and through all of console.initially ideally later on also you know for anybody it's not it's maybe something that starts at reddit but it shouldn't be something that is restricted to reddit through its services and with that I'm at the end of my talk thank you for coming and for listening to me and now it's question time if you have any go ahead the problem with option two in my mind is that how to establish it that's something that people recognize of course until you do that the risk is that nobody will understand what it means that option one is preferable on the other hand if you can't establish option two then it's of course better that the term something was specific but until you get there the risk that nobody will understand it so how do you get to establish it for something new? exactly thank you that is the summary of the conundrum that I was facing where I thought I'd ask all of you for your opinions that's exactly the point how do you want to get the thing off the ground how do you want to get people to click it and to actually look for services becomes more mainstream that's a better time to try to move to something more specific for them people know the basic concepts the thing that I'm afraid of a little bit is there are other projects that do open source services already there is like GitLab for instance GitHub also to some extent like some of its code bases open even at lesson chairs their code base not in a very open way like they don't have a repository that you can commit to submit changes to but you know sort of this concept it's not really it's not new or it's like the main and that's also the thing I don't want to necessarily establish a Reddit brand I would rather have something that works for Reddit but also for the community something that can be very easily repeated without anybody thinking I'm slapping a redhead logo on my service why would I do that so I think the the barrier has to be low and it has to be accessible immediately and you have to understand it when you see it and that's I think the tricky part with custom branding as you say you need to first establish it a little bit and have several services on board which is why I want to have the reusable pattern because then of course we can swap it out at some point Martin how should I build the code okay so we have some practical experience with actual contributors from outside the team that tried to read the documentation and set up the circle yes that was a that is a very good point so of course yeah exactly so this part this part actually we should probably make that explicit here at least one contribution that proves that this is all working so if you don't like how can you prove that external contributors can follow the same workflow you need to have a contributor that has done it so yes we have had this in our case it was somebody from another team in redhead but we've also had many contributions from the public maybe not necessarily always to the entire service stack and as I've showed you architectural diagram there are lower level components anyway that you know it's very different to contribute patches to those you know also the development setup is very different for the lower level components you can run them basically on their own they don't necessarily need to bring up the entire stack that runs in console world but of course if you want to fix something in one of our APIs you have to be able to do that otherwise it doesn't work and yeah so we have examples for this I know this is something that affected us very much like until a couple months ago even inside our team you won't have different development environments and there will be like several co-students coming in and trying to follow these and some completely stock and this is where we said now that maybe they are not as good as before I think this is why this is overall you know even if you know the question can always be what's the business value of this whole thing why should your employer or why should anybody go for this you know why should anybody support this and it does make you much better as a team like these are the workflows that we used to the traditional open source workflows and currently if they don't work so for instance in Reddit at least I can speak for that environment a little bit we've broken this paradigm with services so teams I'm contributing to each other's services because it's not obvious it's not the same thing anymore you don't just go simply to the one repository that has the same name as the package and that is in a place where you would expect it and there are all of these links sometimes you have to click through several repositories until you find the thing first you go maybe to Fedora's package and then you realize ah that's the upstream and stuff but there is a way you can sort of retrace the steps and that's not necessarily the case with services it's sort of much more uh, intransparent and then, yeah, even across teams or even when you onboard somebody new into your team these are just best practices initially when somebody new onwards onto your team they are like an external contributor they maybe have no rights because they don't want them to do all of the things already so I think it's very general and it's just many of these things are general best practices they're not revolutionary and not new in that sense but I think overall they make for a very good assessment of how you're running your project any more questions? because we are out of time so thank you for your attention and yeah, thank you for that