 Kansas City is capable of. So let's try that one more time. Good morning. Exactly. So hi there. My name is David Blankettle, and I work for a company called AppSara. And one of the things that I tend to hate the most in life is when you go to a talk or someone says I'm going to give you a talk and it's supposed to be a technical talk and it turns into this marketing talk, right? That is like full of sales. I cannot stand that. So given that, I want to say sort of a proviso, which is that I will probably want to say some things that sound a little marketing-y, not very much. And so here's what I propose to do. I have here these three wooden blocks, red, blue, and yellow. What I propose is every time I want to say something that is, you know, could even be construed as being sort of like a marketing sales-y thing, I take away one of these blocks. And then if I've run out of them, I cannot say a single marketing thing anymore. So this way you know that my talk is not going to be all that. Can we agree that this is a fine and dandy system? So let's do it. So I'm going to take a little one at the start to tell you what the company I work for does. Okay? I think that's reasonable. Yes? Okay. So I work for a company called AppSara. They're in San Francisco. They make this software that allows you that DevOps love because it allows you to sort of take whatever work that you want from a shell script to a container to an operating system, toss it to the public clouds or run it on prem or move it around. And there's a policy engine in the middle of it that allows you to set up these policies. And we might talk a little bit about policies in this particular talk. You'd be very surprised that that will show up. So that's all I want to say. So let's move on. I think the best way to start my talk is to go to the end of my talk. And the beginning of my talk and the end of my talk is talking about production. Because in my opinion, production environments are all about trust. It is super important that you're, that your thing works. It is super important that you understand in production what your workloads contain. What are you actually running there? You know, that's kind of important. It's really important to know what can whatever it is you're running. I'm using workload as sort of the generic way to say things. What can I consume? What resources? What CPU? What memory? You know, what network bandwidth? That sort of stuff. You might have some constraints on where your workload runs. If you're dealing with EU data protection laws, they can't leave Sweden. Or if you have this machine over here that has lots of memory you want it to run over there. Or maybe it is you have this server that you have that you run all your PCI stuff on. So you want to make sure that you can determine where it runs. And finally, and perhaps one of the more crucial things is you need to know what your thing can talk to. It really is important to say, okay, this thing can talk to that and that's it. Or this thing can talk to the internet or the internet can talk to it. But it's really a bummer in production if you find out that suddenly the thing you're running is somehow talking to large swaths of the internet and you didn't expect it. Right? So that's really a bummer. So you have to kind of pay attention to this sort of stuff. And I think everybody here knows that the bigger the deploy you have the harder it is to know all these things and to have to maintain this trust. And if you're doing multiple clouds, which some of you here, is anybody in this room doing at least one plus clouds? Yeah, like so. So at least like a good 30%. It gets really, really hard. So what I want to do is I want to come up and tell you something I think that will help with this sort of stuff. But the problem that I have is that the thing I want to talk about typically gets associated with like a sleepy time thing, right? So I want to warn you that I'm going to say something and you're going to go like, oh, no, I'm just going to nap during this talk. Okay. But I want to let you know that I won't let that happen. I promise. So what I want to talk about, and I think part of the solution is this one little word here and for those of you in the back on your phones, maybe this helps. Okay. So this is what I want to talk about. Now, when I say policy, invariably people have this reaction because they start thinking about these awesome things like the CBB security policy and procedures handbook or maybe they talk about this one, which I highly recommend you wait till the movie comes out for, you know, like fiscal policy or maybe even your employee manual, which is just rocks. Now, I know I bet there have been people in this room who have had to sit in on meetings where you're deciding policy, right? Right. For your organization. Can you raise your hand if you have? I just want to know it will help. Yeah, look like a large percentage of you. And, you know, chances are the meetings look a little bit like this, right? Right. Or if you've really been doing it for a while, more likely that your meeting looks a lot like this, right? Right. And so I want to suggest that it doesn't have to be that way. I want to suggest that it is possible to have policy be more sexy. Okay. Then just what I'm saying. And those of you who are at the corner is come on in and there's plenty of space up here. So you know, you can come in. So here, let me prove to you can be sexy. Here is a diagram for policy that my marketing group made. I don't think that counts because, you know, you could make that too. But if you don't think that's sexy enough here, let's do this. Okay. So does this give you the sense of just how sexy policy can be? Because we're going to stick with the music for just a little while because I like it so much. Okay. So here's what I'm going to assert. Chances are that your environment looks a little bit like this, right? You have this long road going straight down that way. Or maybe, maybe just maybe your environment looks more like this, right? Couple of curves in terms of like when you're trying to get to production or possibly like this or more like that. Well, I'm willing to bet actually that your environment probably looks more like this, right? Now, one of the things that you may or may not have paid attention to when you saw these pictures, if you were watching is all the pictures have these things. What are these things? Guardrails. Exactly. So here's the point that I want to make to you is we're going to be talking about guardrails because if you can set up guardrails for your environment then it is possible to allow whoever is using your stuff to do what they want and you can sleep at night because they can't do anything that you didn't say they can do, right? So this is why I think this is important. And in fact, I'm going to ask you to go demand this sort of guardrails from all of your vendors, from all your open source projects, from all these sort of things. Now, just because I know what kind of crowd this is, who can tell me where this slide comes from? Yes. This is why I love people so much. My people, you know, it's so great. Yes, it is indeed from young Frankenstein. You're absolutely right. So what do I mean, what do I mean by like demand policy? The situations you have that you might care about is like when a developer wakes up at 2 a.m. goes, you know, I'm sure more memory will make it work better, you know, lots more, right? Or, yeah, the fact that you're chuckling is a bad sign. Or now we're a public docker image, what now? We're like, so like, I don't know, did you know that the official MySQL image, it probably still does, had shell shock in the middle of it. Doesn't that sound great? Go run out of production. Go, people, go. It's awesome. Or, you know, all of a sudden your container is talking to, I don't know, shouldn't be doing anything. Or the new open SSL exploit is called like, I don't know, Shave Fuzzy Bunny or something, right? Right? And we all have to deal with that. And are we vulnerable? I don't know. Or what version of Java are we supposed to be using in our environment? You want to be able to constrain that. Or here's even better one. Who has the database credentials? It's in a spreadsheet where, and what can those credentials do, right? So the question I think that is reasonable to ask is where is your policy in your environment now? In my experience, people have one of two different kinds of environments. First off, they often have a department or a person or a group of people whose job it is to handle sort of the compliance and the policy and stuff like that. And they're like the customs guard. Their job is to say no, right? At the right time in theory. So the problem is, is that doesn't scale or check. It means that every time you want to do something, you need to go talk to this person. If there isn't any of them and there are any of you, by default you're gonna have to queue. And that doesn't scale. So it's maximally frictional. Or, in my experience, people go with what I would call the Wild Wild West, right? Where anybody with a credit card can provision infrastructure and go get what they want. Okay? Now the problem with the Wild Wild West, for those of you who watch Westerns, is that invariably somebody takes a bullet. Like a straight bullet. Like, they, oh, darn, we forgot to shore up the mine and it fell in. Or, you know, somebody forgot to back up that database or whatever, right? And so that's not so cool. So the tricky thing is, is that what invariably happens is when you have the first kind, often people go off and try to do stuff on their own. If the first kind is so frictional, people go off and do the shadow IT thing on their own, right? Like you've seen it where they go off and provision their own stuff, right? And then congratulations, you have the best or the worst of both worlds. You win or lose, you know? So what would be better? So my opinion, what would be better? And this is by the way cosplay, so I can show you that one. It's not, you know, not required. I would assert that if you have something that was pervasive, you know, like everywhere in the things that you wanted to control that was explicit, so you could be really super careful about what it is you want to state in your policy sort of world. And this is the key, this is the kicker. It needs to be automatically enforced. If you have a situation where you have a lovely software that handles policy and every once in a while it goes, is this okay? How about this? Is this okay? Like it's not worth anything, right? It needs to, it needs to, you need to let the machines do what the machines do well. Let the people do what people do well. Like Todd Underwood, an SRE manager at Google says, you know, stop feeding blood to the machines. And I really truly believe that. So let's talk about what you would need if you were going to do this sort of automatically enforced thing. So first off is the easy one, right? You want to be able to say, I want to constrain things in terms of CPU, disk space, network usage, number of things I'm running, or maybe I want to know how memory all together, you know, stuff like that. It's important to be able to control workload to workload stuff. This thing here, my app server, talking my database, that's okay. These two things can talk and only they can talk. And I want to sort of give a little hint about that that I think is that will be useful to you no matter what you decide to do in your life. I think it's super important that when you're giving permission on these lines that it's really granular, that it's per port, not per container or VM, right? It is the case that you want to make sure that if this guy is talking to this guy, and this is the app, and this is the database server, that it can only talk on the database port, right? Because the reason is, is if someone pops your database server, and it happens, right, they can, if you have this lovely transit of trusting, go attack your app server. But if in fact you've only said only these things can only talk on that port, then your attack surface is much lower. And I think that's important. And per protocol as well. You want to be able to say, but you want to be, you want to be able to specify, right? So this automatic bi-directional trust, because this is what we often do, we often say, okay, I'm cool, this thing can talk to that thing. But what that often means is that thing can talk to this thing in the same way, and that's not necessary, and that's perhaps a less good security posture. Okay, so we talked a little bit before about ingress and egress. It's really important to make sure that you can control what comes to your thing and goes out from your thing. If you can't, that's bad. External connectivity and routing, if in fact you have something that's providing a web service and you want to move it around to another cloud, you better have a way to redirect when something comes in for that URL. Instead of it going to this thing, now it's going to this thing over in that cloud. You need to be able to control that. Ideally automatically. You know, when it comes to multi-cloud stuff. It's really super nice to be able to say, only allow this version of Java in production. Or anything that is currently running this version of Java in production can keep on running, but don't allow anyone else to run anything that is of that version. Because you want to be able to retire stuff. Right, so having that sort of stuff is cool. You want to be able to say things about what is allowed to deploy in terms of CI-CD systems. You want to be able to say, okay, only things that have passed a security check are allowed in production. Or only things that don't have personal information in it, PII in it. Or maybe you want to have something that says don't allow the letter E in any of my code. You know, whatever it is, whatever your business needs, you need to be able to control that when it comes to deployment. And here is one of the tricks that nobody pays attention to usually. If you're in a situation, and I bet you've had this situation, we have a developer and she is trying to debug her application. And that application talks to a database and she's like, you know, I can't get this thing to work. Can you please give me, can you please tell me what transactions are going on on the server? So your choice at that moment is one of three things. One, give that person direct SSH access to the server. Do we all like that? Yes? Is that good? Yeah, okay, good. I could, the sarcasm is dripping. Or you give this person a copy of the logs, but that's kind of static, right? And they want to run more stuff. So congratulations, they come to you n times, where n is potentially infinite, right? Or you set up something else. And I'm suggesting you want to have some other way to give them access, to allow them to say stream their logs, if you can, so they can see it. So you want to try to build that. If you don't have that now, I bet you want it. And clearly we're talking about policy, so you need to control who can edit it and when. Okay, now what I'm going to do is I'm going to take another one of these away. Can we agree that it's okay? Yes, everybody cool with that? I just want to say about one thing that our stuff does that I haven't seen before, and if you want to go implement it, go for it. I think it's a great idea. It's not, it's, some people, when they talk to me afterwards, they say, oh, that thing that your thing does, and this is only one of the cool things we do, but it's one of the ones that makes me happy, so I want to tell you about it. It is really cool if you can say, I've got an application and it's talking to the database. If you can have something that will stick, they don't like me to say man in the middle because that gives a bad connotation to it, but you all know what I mean when I say like, I guess the proper term would be inline component. So you have the application, you have this thing that we'll put there for you, and then you have the database. And your application talks to this thing, this thing talks to the database. Now what we can do with that sort of stuff is first off, the real credentials to the database can live in this thing, and it can have a federal credentials that are only good for the time of that discussion, which means that if your developer were to check the credentials into GitHub perhaps, or something like that, it doesn't matter because the credentials don't mean anything, they're only good for that particular connection. The other cool thing that is also really cool is we can do policy right there. So we can say things like, don't ever let my app drop a table. And what will happen is your app can send drop table, drop table, drop table as long as you like, and we'll just throw that on the floor. Or we can introduce artificial latency. Do you want to test like latency monkey? Do you want to test whether or not your application works well when your database gets bogged down? You can do stuff like that. And all sorts of cool stuff you can do if you get in the middle. So I think that's just a cool pattern. I suggest you check it out. Okay, so what did I leave out? I don't actually like this, I don't actually like this list. I tell you what, what I'm going to do, let's mirror this display right now, I don't like this list at all. Let's just give rid of these. What did I leave out? Let's actually go here. Oh, you guys thought that you were, that you were just going to sit here and listen. Oh, no, no, that's not the way it works. No, no, no, this is not like prayer. Okay, I want you to participate. What I want, I guess, sorry, that's going to get me in a weird place. Let's not talk about that. I want you to tell me what I left out. What do you think you need in the land of policy that you really want that I haven't discussed? And I'm cool with output silences applying the policies. So you want a better way to apply policies, okay? That's good. I like that. Number three, these guys are working. This is good. You need a way to communicate policy. That's really interesting. Okay? No, I totally mean it. That's an interesting approach. I mean an interesting thing that I think we should think of. Obviously, you can tell I can spell communicate. Let's pick that one. Oh, come on. Let's see. Okay. Well, I think that I asserted that I said I automated enforcement. So let's just assume that's in that, right? Because I had that picture of the RoboCop. I think that counts. What else? But I mean, I keep on going though. What else? One at a time, please. Updates. You want to figure out how to handle policy update? I agree. What else? There were two at the same time. Okay, tell you what, I'm going to call that doc. I'll tell you I'll use your term. Totally fine with that. References? Yes. Okay. Let's get one more. I'm sure there's more. Someone behind me. Someone's up there. Go ahead. I heard testing. I want to write that down, but tell me what you mean by usage so I can capture it. Oh, yeah, you want some sort of auditing, right? Auditing, stuff like that. Okay. So I think that's actually no, no, no. Cut it out. Cut it out. Stop mirroring. Oh, we did stop mirroring. Excellent. Okay. So five other things that I think you really want to have in any policy system that you use is better way to apply standards and references. Policy. Oh, that's horrible. I cannot leave that on the slide. Sorry. Why didn't you guys stop me? You were sitting right there. It's not. Yeah, that's, that's true. Just ask my spouse. Testing, right. Exactly. Testing and auditing. These are things that you have to absolutely have. I'm glad I could share that with you. Okay. So instead of being annoying and just not telling you where you can actually find this sort of stuff, let me give you some idea, but unfortunately, I have some bad news for you. The bad news is that you're not probably going to be get this level of granularity of policy from your cloud provider if you're going to go to them. So Amazon is usually ahead of the curve on a lot of things. They have the ability to set instance parameters and other stuff like that, ACLs like that, but it doesn't give you the granularity that we're talking about. Like you can say, I only want this kind of, this kind of EC2 instance, but you can't be more specific than that. Azure is getting better, but when I made this slide, they only had role-based access control that had three-set roles, and if you wanted another role, it was great. We have these three-set roles, and if you really want another one, you could get those three. So, same with GCP, which is now GCP, where they were just basically saying, hey, if team members have edit permissions, congratulations, they can do stuff, which is not exactly the level you want. And I don't know who here has the pleasure of working with SoftLayer. Anybody? Okay, so if you've not worked with SoftLayer, when I actually contacted them and said, hey, where can I find policy controls in your system, they said, well, it's actually, you know, whatever you can set in the GUI is going to be fine for you. And so I went to their GUI, and there were like 70, and I counted them, some odd checkboxes. Not all about policy, but just in general, and I thought, oh my lord, like, you know, it's so many checkboxes. So, yeah. But what about your platform? Okay? Anybody know what that is? So, the bad news is, they're getting slightly better, but in terms of, in the very, very latest release, but not really. There isn't really anything there when it comes to this sort of policy sort of stuff. For reasons we can talk about. This one, let's play, let's play, guess that, guess that thing. Which one? Stack or Shift? Come on, folks. Okay, thanks. Yeah, not yet. So, there is some work in this world called Congress. And what Congress is, it's meant to be this sort of generic policy engine. And the way Congress is supposed to work is that all of your components get feed their logs and other stuff to Congress. And Congress will then say, oops, somebody did something wrong. And then maybe it can run some sort of linearization script. Now, that to me isn't sounding like what you really want. You know, like, I mean, I guess that's the way sort of, you know, sort of police work to a certain extent, right? It's not the case that sometimes, sometimes they can prevent things from happening, but often they have to show up after the fact and say, I'm sorry, let me help you. So, I don't think that's the way you want to go. And I want to pull my last one here. Oops, sorry, not yet. I'm going to pull my last, not quite yet. This one? Sorry, we know, who is this? Yeah, so Kubernetes is actually a little further along than both of these in that it has things like quotas and stuff like that. And they're starting to add some role-based access stuff to it. It's getting better. Okay, here's where I pull my last one. Okay? It is the case that this is something we've been thinking about for like, you know, four and a half years at this point. And if you want to sort of play with policy and action, you should definitely check out the stuff that we do. And there's a free version that I'll point you out later if you care about. But the thing is, is that we have a platform whose whole idea is to stick policy in the middle. And what could you do if you have policy? And I want you to play with that at some point. So I think what I've just done is given you your own pitchfork and your own torch so they can go to all places. And I don't mean just to me, but go to everyone out there and say, where is your policy control? Where is this sort of stuff there? I want it, I want it, I want it, right? So now you have your own thing. And with that, I'm going to get, oh, I'll point to this. If you want to actually grab the free version of our thing and play with it, it's that you're all. But with that, I think I'm going to come to the end. Oh, and before I do, one thing I should mention is, actually, because I'm a DevOps person who had another block in my pocket, just in case. Anyway, thank you very much. And I used to do a lot of role-based access stuff for some in Netscape. And when I first looked to Amazon, what I saw was IAM. And when I looked at IAM, I said, well, there's my role-based role management in terms of defining a user and then driving that user back into the system at the code level to say that, okay, developers, this is the user, this is the guy that does this work. Right. I can shut them down at that level. So how does IAM, those things, fit in with some of the other things? So IAM is the closest thing, I think, to what you want in that day. They do have, they do have more obviously, the ability to do this. I think the thing is, is that you want to be able to tell sort of more complex stories because sometimes you want to control down at the user level, but sometimes you want to control down at the environment level. Sometimes you want to say, in production, this is okay. In development, this is okay. And those could be different things, but that's not per user, right? Or maybe it's per team or maybe it's this year or maybe it's this project. And so you want to have some sort of ability to be more expressive. Our stuff uses a policy language to be expressive because you really want to, you want to say more than just, this person can do X because it's very complex. It also gets really tricky if this person is on two different teams, right? And now, okay, great. You know, like, what does that mean? You know? So, up another mic, pointing at mic, yes. Other questions? I saw another hand go up. If you could do a smaller edition of my way, I'd appreciate it. No. So it sounds like you, there were maybe a couple of different categories of policies that you talked about. Yeah. So with the men in the middle, sort of like a messaging policy between microservices, obviously like admin type provisioning policies, what other kind of categories of policies would you consider to implement? So, okay. So I started off with that sort of notion that there are kind of four things you want to control. What can you run? You know, like for everyone, what can you, what can you run? What can it talk to? Where can it live? And how many resources can it consume? And so you basically take those four things into every one of your little realm, every one of the realms that you're talking about. Like as you said, exactly microservices are between applications and databases is one place. Provisioning you want to think about and using stuff. You also want to think about communication. You also want to think about when you move stuff around. Ideally it's, you need your policy system to deal well with moving your workloads around. Like if in fact, if you moved your stuff from AWS to GCP, because let's say, let's say Google is having a fire sale, right? And you just, you want to do some sort of cloud arbitrage. Everything has to be true over there too, right? So I think that there's, so I think that there are, there are other realms and I think it's kind of environment specific as to where you want to apply it. And that's why I like the idea of language because language is a very expressive and it can be used in, you know, we can use, we're using language now in this context. It would be a very different context if we were outside on the hallway and I was yelling at you because you didn't wash the dishes, right? But the same language is going to hold for both of those. And make sure you do the dishes. Yeah. So how do you do with that, Spain millionaires? I think you're saying, right? Is that what you're saying? Or something like that. Yeah. So there are two ways that, there are two ways that I know that you can do that in various places, including us and like Kubernetes, do have this. Thing number one is do what you're talking about, which is export your, export your logs after some sort of thing that consumes logs, right? And that could be, that could be, I mean like, it's not free in user time, but that could be like an elk stack or it could be like super spiffy, spunky, please give me the thing that has the, the jewel encrusted, you know, log management system. So that's one possibility. The other possibility is you can build into what you're doing something that allows you to essentially let somebody have the ability to tail the logs of whatever's going on, right? So in our thing, you just say APC job, job log, and it's going to show you everything that's coming out of standard, standard in and standard error, standard out and standard error, right? Because it's just built in to know that it's attached onto that and send what the contents of what's coming out into that container somewhere else. So either you built it in or you export your stuff, in my opinion. But what you don't do is you don't, is you don't copy your logs to somebody because there's, that's for a, for a number of reasons in addition to the like it's static, I mean it's fraught because like they might have some data they don't need. And what you don't do is you don't give them SSH access in, you know, as awesome as they are. In my opinion. Is that, is that good? Oh, thank you very much. Thank you very much.