 Hey everyone, welcome to my talk. Welcome to the last day of Kupcon. I'm super excited to be here and sitting on the other side of the podium is really curious for me. So thank you all for having me here today. So who am I? My name is Jonathan Pulsifer. I'm an infrastructure security engineer at Shopify. Certified Kubernetes Administrator number 89, if that means anything. You can find me on GitHub and Twitter at John Pulsifer. Previously, I was a team lead at the Canadian Forces Network Operations Center. That's where I got some of my security operations experience and donned my tinfoil hat. A network defense instructor at the Canadian Forces School of Communications and Electronics, and I did some work for SANS, some construction, and so some packet analysis and all that stuff. So I am a certified packet ninja. So thank you again for having me on stage today. My first Kupcon was back earlier this year in Berlin where I was sitting on the floor watching Jesse Frazel and Alex Moore talk about securing GKE clusters, or securing the Kubernetes clusters. So I'm really glad that the venue has a little bit larger room than they did in Berlin. So that's fantastic. Fun fact, for those who didn't know, the Kubernetes store, the CNCF store actually runs on Kubernetes at Shopify. So that's really interesting. I went and talked to the folks who were running the booth down there, and let them know that they should be letting you know that if you purchase anything, you're actually purchasing something through Kubernetes. It was one of the actual first doors that we moved to our cloud platform at Shopify. So before we start talking about security controls and how we do this, first we need to gather some more context about how we run services at Shopify. So we have what's called a service tier model, where as an application sort of moves through the software development lifecycle, we start off at tier four where we have a lot fewer requirements. It encourages that rapid prototyping really gets agility in there. And as we move through business importance, we impose things like service level objectives and the traditional software development lifecycle and controls, and impose things like page rotations, CI, alerting, redundancy, incident response, these sorts of things. And so when we were moving to Kubernetes and setting up all of our clusters, and I was thinking, why don't we do the same thing with security? So I've coined what I've called the security tiers. So first and foremost, when you create an application on our cloud platform on tier four, we impose things like strict R back, crities attestations. We'll talk a little bit more about what those are later. But as we move through the tiers, I have this vision, if I were to have one, of restricting the runtime of containers with secomp, app armor, the network policies, strict security context, dropping all the container capabilities and these sorts of things. So this is the vision that I have for doing this on my team. So we're going to explore a little bit of how we do this. So how many namespaces do we have? How many clusters do we have? What does it look like? So Kubernetes namespaces by tier. We have about 60 namespaces in tier one across a number of clusters. Tier two, we're about 70. Total, we have 450 namespaces operationally for services running at Shopify on GKE right now. So that's pretty outstanding. How does a team of seven cloud security individuals maintain sanity and sleep at night with this many namespaces running amok? And it's important to know that as we move to Kubernetes, we're not there yet. We're still in, I guess, an exploratory phase. We've moved most of our internal applications over, but it should be noted that we still have two data centers which are very much hot and we're exploring some Kubernetes there, but we're actually going to be focusing on GKE for the portion of this talk. So I'd like to introduce our cloud platform. Our cloud platform is running on GKE. So again, we have about 600 projects in Google Cloud Platform today, maintained by 15 folders, 700 Google groups, and 20 GKE clusters. So this isn't like the running one service per cluster idea. We have a number of clusters in a number of production projects which serve a lot of traffic. So that's some numbers for you. With that many clusters and this many resources that we have to control, does security work when you have to rely on people to do things correctly? Obvious answer here is a big no. So it's important that as we scale our operations, as we scale our clusters, as we scale all this, we need to introduce this automation into our platform so we can actually maintain agility with our developers, with a thousand active developers at Shopify, and how do we give them that Heroku sort of feel? How do we let them push code and how do we then secure those running containers? So this is a general overview of our cloud platform architecture and what we're going to do is we're going to step through a number of components here and we're going to introduce not the operational patterns, the operational automation because it's not the focus of this talk, but it's going to be the security automation here. So at each sort of stage in this process, we have our traditional, I guess everybody has a builder, right, CINCD is not new, there's been a lot of talks on this, but there hasn't been a lot of talks on introducing security controls into those pipelines. So we're going to step through those and see what they mean. So we're going to start off with services DB. So services DB is this sort of front end for services at Shopify where we have about 300, 400 services that run, how do we visualize or how do we make sure that everybody has their playbooks set up, how do we make sure that their logs are there, how can we visualize this? And this is a tool called services DB. I'm not going to focus much on the actual front-ending of operations again, but with services DB, because it has sort of the application context, this is the first place where an application sort of begins its life cycling. You have a repo and you sign up for services DB and then you can do some things like automatic patching. So on the right-hand side here, you see we have this robot who's creating a pull request on my repository that says, hey John, your Shopify cloud gem is at a date, so you should probably get that patched. We're a big Ruby shop, Shopify. So most of the automation we have built in for our dependencies revolve around Ruby. So this is an example of one of those where we actually provide our developers who are building Ruby on Rails applications with a gem, the Shopify cloud gem, which builds in some more operational automation, as well as things like providing good patterns around, say, storage, for example, is a good example of that. So with this, we actually, when we have a repository and we create service runtime, create cloud runtime, it's actually going to do a lot of magic in the backend and this is ground control. And ground control is really curious, because we use this to assign sort of identity to our namespaces. So we wait for, and we've heard of the container identity working group and Spiffy and we're getting to a point where we can then attach identity to a pod or a container. We can't do that today. So how did we do this at Shopify is where we actually give identity to the namespace. So when you create your cloud runtime with services DB, what's going to happen is we're going to create you a GCP service account that's going to be bound to your sort of namespace environment pair and we're going to mount that secret for you inside your Kubernetes namespace. At the same time, we're going to provision you an eJSON key pair. So what's eJSON? So go check it out. It's actually open sourced. It's called encrypted JSON. It allows us to use asymmetric encryption techniques to keep our secrets inside of Git repository so that we can actually have those secrets encrypted in there and when we deploy to our Kubernetes namespace, it actually decrypts that runtime. So we don't need to worry about these secrets being baked into containers, that bad anti-pattern that sort of has come up over the years. And we can avoid this by using encrypted JSON. So if we have an encrypted file full of secrets in there, who cares, right? But with this, we actually get the auditing for version control and all the secrets. So if somebody goes in, checks in a secret, we can see who did that when. So that's really unique. And we also annotate the namespace with your GitHub and who owns this, where does this repo come from and give a little bit more context to that. So now that we've taken a repository and we've sort of built its Kubernetes groundwork with ground control, how do we get your container built? So this is the fun piece. At Shopify, we do about 6,000 builds average on weekday. This is not like container builds going into production. This is everything from CI. This is our overall builder stat. So we build about 6,000 containers per weekday. Life-cycling is really hard. So we have about 450,000 images in GCR today. And that has given us and the folks doing the GCR vulnerability scanning, a little bit of load testing help, because as they're building these products to help us out, the amount of containers that are actually shoving in there has actually given us a hard time. So there's a bit of a latency problem there, but we're going to chat about that a little bit later. So our builder is called PIPA. And our builders run on build kite you saw previously. So PIPA in Portuguese actually means kite. So that's where that name came from. And it's a Docker in Docker, Golang creature that does a lot of discovery. So what I mean by that is, if you have a repository as a developer, should you understand explicitly how to build a Docker container? I mean, arguably no. So what can we do here to help get a secure container into production? So with PIPA, what it actually does is it discovers using the same Heroku build pack model, what code lives inside of your repository and will provision you the appropriate build packs needed for your application to run. So on the right-hand side here, you see the sort of build kite screenshot there and a couple different stages of our pipeline. Through a pipeline setup, we trigger the validation build, then we build your container. At the very bottom, you see this Grapeus and Creedy stuff. We're going to chat about that next. But with our builder, it's important to know that when it comes to building a container and actually building that container is when you have the most context as to what's going to be run inside of that container. You see the packages being installed. You see the versions of build packs. You have the most context. So this is where we can actually do a lot of our initial discovery and auditing and actually sort of check out what's going on. So we can actually do some audits like, does this image run as root? Do you see any vulnerable packages and container attestations? So what are container attestations? What does this mean? I want to talk about Grapeus and Creedy's. If you all have seen the sessions that have been going on, there's been a couple on Grapeus. And Grapeus and Creedy's has really been a large focus for the cloud security team for the last little while. So Grapeus is a software component metadata API where it allows us to store details or metadata about a given software package, right? You can have a dev, an RPM, these sorts of things. But isn't a container just an image that we could also have metadata for? Absolutely, right? So using the namespace and image and the digest as the key, we can actually produce what's called the note at build. So we can sort of start this chain of provenances who created this container, when was it built, what other, is it deployed in production right now? And these other pieces that we have context for at build time. So we produce these notes at build. And if you actually use Google Container Builder, you get all this stuff for free on the inside, which is sort of neat. So as a personal customer, I didn't know this, but building my own containers and my own personal projects, I said, okay, well, what is this stuff? What does this look like? You can see what's called an occurrence or a note of a package vulnerability. So with Grapeus, it's just metadata, right? So we can actually start pushing things like vulnerabilities there. So you see this is a TCP dump 490 package that's vulnerable and it's fixed in version 492. So if we have this information about this TCP dump package, a container build, we know that we're installing a version of TCP dump in a container. On the other side, then we can say, hey, look, this container, this specific container contains this package with this vulnerability, right? It's that container introspection. And that's where Grapeus is really powerful for keeping that container metadata. So if you want to learn more about that, check out Google Cloud's blog or Shopify's engineering blog, I wrote a little post about how we use that. Metadata is one thing, but Creedies, or as you'll hear if you hit the Grapeus repository, binary authorization, is really the cool piece. So it's using this metadata that's stored in Grapeus to create and enforce policies at runtime. So this binary authorization means that we're going to step back here and show you at the very bottom you see this Grapeus and Creedies stuff. What we're doing in there is actually taking the digest of our container, we're signing it digitally, and then we're pushing that back into the Creedies sort of binary authorization API, and what we're going to do here is then make sure that we can deploy no image other than the images which have been built by us, which is really unique, because this sort of takes away the idea of an adversary gaining access to your cluster and downloading their own container and running their own tools, right? That could be described as bad, and we don't like bad. So we like to enforce these with Kubernetes. There's actually a demo at the BOF talk for Grapeus of this actually working with an emission controller, which is awesome. So now that we have our repository checked in, our Kubernetes primitive started, we have our container built, and how do we actually deploy this to production? So Kubernetes deploy is really cool. It's a gem that provides us a lot of automation around deployment to Kubernetes. It's actually open source, so go check it out. It's essentially a plugin for our build tool called Shipit, which is also open source, so you can go check those things out and see how we deploy to Kubernetes. But the main features that are really exciting for me and for our developers is that we have the clear, actionable pass-fail results. Green is good, red is bad, it's like the pre-deployment of certain resources, right? Like you have to deploy a service before a workload so that you can have the appropriate environment variables, these sorts of things. Now we do a little bit of that, and the decryption of the eJSON is what's happening here. So what we can do with Kubernetes deploy since we built all these tools, they all have the context, is we can actually put this other structure inside of eJSON to say, create me a Kubernetes secret from this blob named this. And this is really cool because somebody can keep these secrets inside of version control and then deploy them to encrypt them at runtime so we don't need to worry about any of that plain text secrets and stuff. There's a few other checks in there as well, so like we can't overwrite a Kube system, for example, or delete Kube public, you know, these sorts of things, some checks in there to help us out. But there's an example of me deploying a little Go app. So you deploy it in 4.2 seconds, it's good, it's green success, and we're all happy. So this is an example of some of the automation that we have here around getting secrets into the appropriate namespaces that are inside of our containers. Oh, it's black, works for me. Let's try it again. Cannot reproduce, that's right. Oh yes, my fault, always. Thank you so much. So stepping back to Kubernetes deploy. Kubernetes deploy, TLDR, gives us clear actionable passfail results. Also decrypts us our secrets from EJSON into Kubernetes and really helps us with that developer experience. So if we have robots deploying for us, this has actually prevented us from having to give our developers Kube CTL access, which is really unique. So with a thousand active developers, then wanting control over their infrastructure is a really big deal. And you know, given that with Heroku, you can get a little bit more intimate with your application. We built tools like Kubernetes deploy, and another one which I'm not going to show, but I'll talk about is Kube Shell, which will actually sort of preclude us from having to give Kube CTL access to our developers, which is really awesome. With Kube Shell, what we can do is actually, with TTY.js and a little bit of OAuth, we can actually give them a terminal into their own namespaces with access provisioned via config map. So we don't actually want to explicitly give all of our users that admin access and deal with our back-end that way, so with Kube Shell, we can actually provision a little bit differently. So this is where the phone automation begins. We're going to talk about our cloud buddies. So cloud buddies internally marketed as friendly Kubernetes controllers, keeping the cloud fluffy. Is our brand for our Kubernetes controllers that we built. There's a lot of them, about 15 of them run per cluster. It depends, given the type of services that run there. But this is where we can introduce our service automation, our security automation. So we have things like RedisBuddy, for example, MemcacheDBuddy, that would provision you an appropriate resource under whatever circumstance that we've decided to build. But the ones that we've decided to build for security or implement security controls in, that I want to talk about are AccountabilityBuddy, BucketBuddy, NetpolBuddy, ArbeckBuddy, and SecretBuddy. So it's pretty fun. Here's an example of what AccountabilityBuddy is designed to do. So remember when we created our Kubernetes namespace and assigned identity to it with the GCP service account? It's how do we give access to a given Google API to that service account without somebody pinging me and having me to go and manually click some buttons there? That's not very scalable as we deploy a lot of applications to our cluster. So if they just drop a config map in the repo and say, hey, I want to give this service account, now say container viewer, well, we can do this. And we do have some checks in AccountabilityBuddy where you can't give yourself sort of God mode on the entire project, right? Like this is something that we've accounted for and we don't want to do. But we do have some APIs which we will very happily allow you to check into diversion control and allow us to version that access for you. So this is how the seven of us sort of maintain sanity moving forward with this. With storage, again, every developer would like access to their storage. And since we use G Suite for our actual business products, we can actually leverage that identity on Google Cloud Platform. And then with Bucket Buddy in this example, there's an example of a custom resource where we are giving storage admin to a robot and to a group and giving object creator to you at Shopify.com. So again, with all of this inversion control, this allows our security team to actually then audit who requested these permissions, when, and we have that trail, right? Of course, there's that, oh, anybody can change your identity and get, and that's all good. This is a way that helps us early on build that auditing into our platform, which is really unique. So with this, then we can assign appropriate permissions. Auditing clusters are hard, especially when we have 20 of them. So how do we ensure that all of our workloads are running securely? When I started down this road, I said, well, the tools didn't really exist at the time. I said, how do I effectively audit this cluster to make sure that my workloads are running securely? And this is where Cube Audit came in. We actually have a couple of dedicated developers on this now. We open sourced it very recently. And this is just a little Golang application that allows us to perform audits against arbitrary security controls on Kubernetes. So we can audit for things like Automated Services Account token. We can make sure that the appropriate image and tags are running. You can see an example in the top right there. We have a screen cap of the terminal we do. We have some human readable output, but we also have put in JSON so we can automate this. We can audit for network policies to make sure that you have appropriate default denies policies in there. And every control under security context we can also audit for. So if somebody comes to me and says, hey, John, this image has been determined to be vulnerable. What can we do? Well, we can just quickly run a little Cube Audit check to make sure that we're not running that image anywhere. This is probably going to go away with Grafayus once this becomes a little bit more... Once Grafayus becomes a little bit more fleshed out and productionized, but this is a really easy way for us to do this today and get some answers very quickly. There's this other continuous security monitoring piece that we're missing. So with 600 projects, scaling is really hard. So we have developed a couple tools inside to help us out with this. So there's a tool called Nosey Bastard which does scheduled scanning. It's like our traditional sort of VA or ZMAP, NMAP Nessa scans. It does some discovery of cloud resources on GCP, on AWS, on Heroku, and other operating environments, which is pretty cool. And we also decided, as we moved towards a really strict RBAC control clusters, we needed some sort of visualization there to help us out and look at this, to see which service accounts are mapped to which roles and these sorts of things. And we built that into Nosey Bastard so we can actually go and clickity-click and see which RBAC roles are assigned to which service account, which is really awesome. For setting security, if you all haven't checked that out and are using GCP, please do. It does really, really good comprehensive GCP inventory and there's been a lot of horsepower behind it over the last little while. And we used this for the discovery of other resources and also the enforcement of IAM policies, which is awesome. So we can make sure that if you or a developer at Shopify creates a new project, they can't add any external identities to that project for an example. So we're using GKE and the concept of the MIG or the Managed Instance Group that these nodes come up and go down a lot and moving from a traditional sort of config-managed Chef Ansible sort of environment where we had to ship around SSH keys and these sorts of things, that's not really something that we want to do. So the Google Account Statement that's actually running on the container-optimized OS will actually provision you access via SSH by taking the SSH keys out of the project metadata and then you can take the authorized keys so you can SSH into it. And if you haven't done a G Cloud Compute SSH, ticka-ticka-ticka, and bang, you're in the server, right? But where are these keys held? How can we manage them? Do we have to? That's sort of arguable, right? So our Google Accounts today are actually backed by multi-factor authentication, right? So if you want to talk to Google Cloud Platform API, you got to sign in. So we already have some controls around identity there. So accessing a cluster via SSH or a GKE node via SSH, this is all backed by our G Suite identities today, which is, again, backed by MFA. So do we have to ship these identities around? These SSH keys, if you don't have one provisioned, if you don't specify one, it will create an ephemeral key for you. So what if we just delete these all the time? And then you have to provision them again and again. It doesn't take too long. How often are we actually SSH-ing into a GKE node? Not very often. So given that, we're going to build a tool called SSH Janitor to go in and discover and delete these stale project-wide SSH keys. So every time that we have to log in, we got to do that little dance. Maybe it takes like 30 seconds, something like this. But it's really not that big of a deal if we can actually reduce that threat by having these stale SSH keys around, right? We have to provision them all the time on the fly, which is really cool. So what's missing? People at Google.com, please don't hate me. We were missing, at the time that I wrote this talk, a lot of things. But Kubernetes moves so fast. And given that, like, GKE is open-source Kubernetes, plus a couple bells and whistles, we haven't... We've actually scratched a lot of these off. So for the longest time, we actually didn't have access to the API server logs because Google hosts the master for us, right? So given that, how do you get these logs? Kube CTL proxy, curl the Kube API server log endpoint and pipe it out to a file and do some other things. You know, that wasn't available to us. But as of GKE 173, it's actually in cloud audit logging now, which is phenomenal. Network policies are finally here with Calico, which is great. As of GKE 176, egress network policies work as of 184, which is really good for us. We're still missing pod security policies. I've been poking for a long time. I can't set my cluster-wide pod security policy. Now, why is that important? Why is that important enough to grab a slide? Because we have what, 300 services maybe deployed there now? So if in Kubernetes 1.8, we have the new flag in the security context called allowed privilege escalation, right? So how do I ensure that allowed privilege escalation is set to false in all my containers and all my workload manifests? Well, that's a couple hundred PRs. I would really just like to do that once. So that's really where that comes from. So that's really where that escalation has been really, really fun for us. Not. So on GKE, we have a couple of different authorizers. We have the webhooker, the IM authorizer, and we have the RBAC authorizer. So for those who don't know, when you make a request against the API server, we have this long list of authorizers, and if any one of them succeeds, the request will succeed. So if I don't have an appropriate RBAC role but I've provisioned myself access through IAM, my request will succeed. I can get my pods, I can list my nodes, these sorts of things, but I'm going to see RBAC failures keep showing up in the API server log. We page on RBAC failures in the cloud security team, and this has been quite the journey for us in this. So we're really missing that IAM and RBAC synchronization. That would be super awesome if that could exist. The GLBC configuration authorizer for the identity aware proxy, I know stuff's in the work there, but I would really like to enable the identity aware proxy because we are a big users of the Google Cloud load balancer. So the GLBC, if I could configure some IP there, that'd be sweet. And container identity, I cross it out. It's not here yet, but we've seen all the work going on with the container identity working groups, and I have faith that it will show up at some point in the future. So that's all I have for you today. Ready for any questions? Thank you. Cross cluster communication in different projects. That's a little hairy. So I mentioned we have four service tiers, right? Tier one, two, three, four. There's actually a couple other ones that I didn't really mention. One is tier shared, and tier shared is that sort of providing access like Splunk and zookeepers and these sorts of things at Kafka, and how do we connect to that? We don't actually connect any other way than we would normally. So we'll still front it with an appropriate load balancer, and we'll still do TLS in the front and clients are often these sorts of things. So we're still using the same traditional login methods that we would have used before. If this was running in the DC, we'd use the same method. Maybe an extra sort of TCP tunnel on there just for funsies, but that's... We haven't really played with that yet, so we're really looking forward to the what's called IP alias clusters, right, where we can actually make the pods and service first-class citizens on Andromeda. That's really cool. We haven't done it yet, but we're looking forward to exploring that when it comes. Yeah, Kubernetes Deploy. So you're guessing the Kubernetes Deploy has access to the project? Yeah, we protect those keys the same way that we protect any other key, and that would be... At Shopify, we have what's called the default to open culture, so typically our engineers have access to all the things, and given that, they're still held in encrypted JSON and deployed the same way that we would deploy another app. So it's just another key, a secret to secret to secret. We do have auditing around this, so if we would determine that it could have been compromised, and with IAM, it's just super easy. We're going to spit out another JSON, kind of good to go, right? And we do have auditing through GCP there. Does that answer your question? No, there's not like a monthly rotation. It's more whenever we feel like it would need if something has gone awry than we would do that, but if there's no indication that it keeps them compromised, there's rotating it to the provider's value, I think. It's not like... So the question was if we have feature parity of sort of cloud trail there, it's not the same, but our models are very much different now. Moving in GKE, all of our cluster operations are logged appropriately through cloud audit logging, right? And we're satisfied with this, but in cluster sort of operations, app level operations, that's all Kubernetes now, so it's sort of like build your own auditing, I guess if that makes sense. We use Splunk for our logging pipeline, so we're going to hit the fluency aggregators, kind of Kafka Splunk, that would just be our ideal pattern, so nothing really changes there for us. Like when we venture into multi-cloud, I don't really see any huge differences. So the question was we still have a number of applications running in our data centers, and what were the pains sort of moving to GKE? Operationally, I can't really speak to this, but security-wise, it's all about the snowflakes, right? So not every app is the same. A Python application is built differently than a Ruby on Rails application, so we've tried to stick to that build-packs model and tried to introduce as many controls as we can there, but some applications are different, so we've actually had to walk teams through appropriately building a Docker container, or just keeping on top of these things, because not every app is the same. It's more of like an operational question, I guess, sort of migration there, but as an infrastructure security engineer, traditionally everything below the app in a stack, now it's like everything below the app in a Docker container, right? Does that make sense? Is that any question? Cool. So the question is, how is the GCP sort of identity that we provisioned for this namespace actually consumed by services? So when we at the beginning in the services DB, when we click create cloud runtime, and that's when all that stuff is provisioned, we actually mount that secret in Kubernetes and then provide that sort of as a volume mount, you know, sort of a secret volume mount inside the container, Google application credentials, you know, the typical place in the file is where we would mount the path to that, and we provide this in the workload manifest sort of for everybody. We have like a number of blessed stacks, right? So, you know, for the snowflakes, it doesn't really work, but if you're deploying like a Rails application, and everything's going to work magically for you, and that's what we get. We haven't done any really like YAML templating, like any templating like I've mentioned before, so that's a little bit hairy, some manifests are quite long, but this is where we can mount those for our developers, so they don't have to worry about these things. No, so this is the difference between the Kubernetes service accounts and the GCP service accounts. They're both like completely separate, so the GCP service account is used to talk to the GCP APIs, and we mount that distinctly as a secret. Yeah, cool, that's awesome. For, yeah, so the question is like, when we do our automated patching, do we actually, is it fully automated, or does a developer have to actually click a button? And sometimes they do. Some, you know, some, for example, like a gem bump or something like this, and then with automatically patching, yeah, they're going to go through and be merged automatically, but that's a risk, right? If we're serving production out there, it says, oh, here's your patch, boom, boom, push it through, like, and it breaks. I don't want to be responsible for that, you know, so not really yet. I mean, we're still pretty young in our Kubernetes journey, so I would assume next year, asking the same question, it'll be different, but right now, no, there's still some manual steps required, so if we're active right now, we can't do it fast enough to put it in the CI and make it worthwhile for us. So our, is Accountability open-sourced and we have a plan for it. A lot of the buddies that we actually have are really sort of contextually bound to Shopify and our patterns, so it doesn't really make much sense for us to open-source those right now. We thought about it. You may see some buddies show up in the future, but right now, they're sort of all monoreboats, so we can componentize those right now. So you want it, right? Yeah, that's awesome. Okay, that's great to hear. That's great feedback. I love that sort of feedback and if I can push that, I will. Yeah, so the question was going to explain a little bit more about why we don't give key pedal access to the devs and it's more about management, I guess at this time, so rewind back a year, and it was myself and a couple others doing the cloud security stuff, so how do we do effective RBAC for a thousand devs at a time, sort of automatically, and that was sort of like a design decision that we had to make, and did we want to expose or teach Kubernetes to developers? Maybe next year, but to keep us moving fast and to keep actually services being created and not having to pause and learn Kubernetes has actually been a really big win for us. So over the last couple of days, we had our company Hack Days, which is two days where we just build the things that are cool, and I think we have 50 or 60 different applications now deployed to that cluster, and it's pretty phenomenal that they can do that without having to worry about the Kubernetes patterns, right? They can just click through it and have their storage and have the Redis and do all the things. We do actually spend a lot of time supporting deployments. We have what's called a cloud help rotation, so we go through and some of us will swap out and actually provide sort of front end support to the devs who are onboarding onto the cloud platform, and us on the security team actually do that as well, so we can actually course correct if necessary if we're seeing these sort of anti-patterns showing up there, which has been really helpful for us. But yeah, we do provide sort of cloud platform front end support. How are we handling the image scans? It's all GCR vulnerability scanner is the main tool that we use for that, and right now we can pull out if containers are vulnerable or not via the gcloud command line utility or hit the API or in the GUI, but again, it's all reactive right now because of the latency. It's just, again, when we have that 450K images in GCR, like we're really hammering the API and we're giving them a hard time as much as they are us, so it's reactive for sure. Yeah, the question is do we see a lot of duplicate images or the same sort of services, and absolutely we do. A lot of applications aren't too dissimilar, so we do actually provide blessed base images to start this off, and that's sort of where we make those decisions, but yeah, there's a lot of duplication. We thought about injecting stuff into the image after deploy, but again, we're following that Heroku model, so Heroku-ish in the build packs really just it works, and it works well, so I'm going to stick with it, I think. Anything else? Yeah, so how are we handling alerting? Alerting on what? I guess. The thing that I mentioned was we page on our back failures, so we're actually looking for returns on roundtripper for our back failures, and we're digesting the application level logs that are anything that contacts Kubernetes API, and just alerting standardly through page-to-duty as we would. Regular logging pipeline sort of stuff. Cool, well thank you so much everyone.