 I was just speaking in San Francisco last week and they were less mellow than you guys. So let's try that one more time. Good morning. Good morning. Morning. Much better. So good morning. My name is David Blank, gentlemen. I am the technical evangelist for a company called AppSera. We're a company out in San Francisco that, in case you haven't heard of us, makes software that DevOps love because it lets you take whatever workload you want in the public clouds and privately move it around, have a policy thing and so on. But that's not what I'm going to talk about. What I do want to make sure to do is because the thing that I tend to hate the most is technical talks that are really marketing talks. I can't stand them. I don't know if anybody here has the same despise of them as much as I do. However, it is the case during my talk that I would like to say a couple of things that might possibly be construed as a marketing-like thing. And so here's what I'm going to propose I do. I have these three wooden blocks. I'm going to put it up here. What I am proposing to do is any time I say anything that sounds like a marketing-like thing, I will take one of the blocks away. And this is my total amount of things that I could possibly say to this market. But you know you're not going to get an entire marketing talk. Now I'm only going to do this if the audience agrees this is a fine compromise. Are you cool with this? Yeah. Yes? OK, good. Then let's go forward. So I think the thing we need to do is we need to start at the end. What I want to do is I want to start at sort of the idea of what's a production environment? What's crucial in the production environment? To my mind, production environments are kind of all about trust, right? Because you have to figure out, like, what are the things that I'm running in my production environment, you know, from a software perspective, and et cetera? You've got to kind of understand what can the things I'm running consume in terms of resources, CPU and disk and network bandwidth and that sort of stuff. You probably want to know where is my workload running? Is it running in this data center? Is it running in Sweden where it can't be because of data protection laws? Is it running on the machines that have a high memory? Is it running on PCI certified machines and such things like this and that sort of stuff? So you probably care about where it runs. And finally, and probably one of the things that comes up when we talk about security, like the last talk, is what can that thing talk to, right? Can it talk to other workloads? Can it talk to the big bad internet? Can the big bad internet see it often in production environments? This is considered bad, right? So, and I think that there's nobody in the room here who would disagree with the sense that the bigger the production environment, kind of the harder it is, right? If you have more dots, keep you making sure all the dots connected properly tends to be trickier. So here's what I want to do. And I want to also say, by the way, this is something else that we've all found out is that when you have multiple clouds, it tends to be super hard. So here's what I propose to do. I want to tell you my answer to dealing with this sort of stuff. But my answer unfortunately has sort of, let's see, there should be sound in the answer. I'm going to change the sound. Oh, there it is. Can you pop this out of it? Oh, I can. I'm not sure we can. I think that's on the answer. So the problem that I have is my answer to this sort of thing is a word that typically gets associated with sleepy time. And here it is. We're now like in the middle of the morning, your caffeine has started to warm up. So I just want to warn you about that. Yes, I'm sorry. See, you're already joining. So the thing that I want to talk about is this little word here. And if it's a little difficult for you in the back to read, it's this word. Okay, so this is what I want to talk about. Now the tricky thing with policy is when I say policy, most people think about these awesome books like the CDP, Security Policy, Procedure Handbook, which is riveting. I'm telling you, but we should work with a movie. Marketing? I'm not marketing for CDP. Too late. Handbook, you know, like the fiscal handbook or maybe your employee manager, right? So the thing is, is that typically when people get into the policy realm, they think that it's like this, right? If you've ever been in a meeting where your company is deciding policy, it either looks a little bit like this, which happens to be the U.N., or let me just do a quick show of hands. How many people in this room have been in a meeting about policy, per se? Okay, did that meeting look a lot like this? Roughly? But so what I want to do is I want to tell you that policy can be a lot sexier than this. And if you don't believe me, here's a slide which is actually from our graphic artist who's saying that this is what policy looks like. But I don't find this sufficiently sexy, so I'm going to do this. Okay, so I just want to be clear. Is this now sexier? Okay, great. So here's the thing. When the car reporter is smiling, that's a good sign. So here's what I want to give you. I don't call it a jazzy. So I think that probably your environment looks a little bit like this, right? Where you're trying to move down this road, you're going straight down this road, or maybe it looks a little bit more like this, or, I know this sounds like a car commercial at this point. Or it's a little bit more like this, more like that, or chances are your environment actually is more like this, right? And so one of the things that I want to point out about this and about these pictures that you may or may not have noticed is these things over here, these white and red things. What are they called? Guardrails, exactly. So here's the thing. My assertion is, is that if you have the proper policies in place, and we're going to talk about that, that you can put these guardrails up. And what happens is that the people that need to use your system can go as fast as they want, and you can sleep at night because they can't do things that I'm supposed to. Now, here's my goal. My goal is to make, is to sort of make you guys rise up. Rise up with your pitchforks and your torches and demand policy from your tools, from your redenders, from everybody outside who is sponsoring the conference. Go out there and demand it. And just because I know that you're an awesome peak audience, where does this fly from? Where's this from? Young Frankenstein. Somebody is from London. Young Frankenstein, yeah. See, this is why I love the DevOps crowd because they know stuff like this. Okay, so what sort of things am I talking about? I'm starting talking about these awesome scenarios where somebody comes to me and says, it's 2 AM, you know, the developer comes to you, and I'm sure more member will make whatever I'm working on work better. I mean, that's lots more memory, you know? Or is there malware in a public Docker image? What malware? Like for example, if you go pull down the MySQL official image from Docker Hub, it's got shell shock in it. Awesome, let's put that in production. Or the containers talking to who knows what, right? You know, nobody knows. You know, nobody knows. It shouldn't be doing anything at all. Or, hey, I can't believe how timely this is. If the newest OpenSL, XSL exploits some sort of combination of gruesome and fluffy, right? This is basically the new version of it. You know, dead, puppy-y, fluffy thing. Whatever it's called, I don't know what the latest one can be called. Or what version of Java are we using, really? That one, or who has the database credentials, and what can I do? Right, these are all these things. So the way to start, I think, thinking about this, is to look at your environment and figure out where is the policy now. And in my experience, in environments, policies are in one or two places. Either you have a department or a person or whatever whose job it is to make sure that you're in compliance and that the policies are being followed, right? And they look a lot like somebody at the border? Yes. Now, the tricky thing about this scenario is it means that everything you do is maximally frictional, because it means that you have to do a mother-may-eye every time you want to do something, because it's going to go through this person or group of people. And that group of people doesn't scale, because they're people. So that's kind of probably it. Now, the other place that you typically have is what I would call sort of the Wild Wild West, right? Where we're in a company and anybody with a credit card can get more resources. Now, if you have spent any time watching any Wild Wild West movies, you're very aware that invariably somebody catches a stray bullet in the middle of the movie, because, well, we didn't properly buttress the things in the mines, the mine fell down, or we forgot to back up that database, because we weren't following that procedure, that sort of stuff. Now, often what happens is you have this awesome combination of the two things, where you have, OK, it's too annoying to go through the compliance or whatever it is doing in the security or the stuff, so I'm going to go out and make my own little shadow IT thing. And guess what? The shadow IT thing goes a lot like the Wild Wild West. And so you and your environment have both things, and yay, you win. It's the worst of both worlds. So what would be better? So one thing that would be better would be to make sure that you have some sort of policy system that is pervasive, explicit, so you can say what you really mean and make sure it's taken care of. And we'll talk about that. And this is where the Robocop picture comes from. That's cosplay Robocop, so it's not copyright school. You want to make sure that it's automatically enforced, because any sort of policy system that requires a human to go, yep, this was fine, is not the one you want. You shouldn't let the machines do what the machines do well, but make sure that the people do what the people do well and don't confuse you as a person like Google, Todd Underwood, said, let's stop feeding blood to the machines. And I really, really believe that. So let's talk about what it means to be enforced, where you have to enforce it. Well, the easy one is what we talked about, right? We talked about resource limits. You want to make sure that you can say stuff, and this is where you see most often about is your only allowed to consume this amount of CPU or memory or just spaced out sort of stuff, right? That all makes sense. We all kind of understand that implicitly. It's kind of important to be able to specify what can talk to what. And here's your little tip on that. It's important that it's per port, not per container VM or container VM or whatever, because it's not uncommon for people to say, okay, cool, my app has to talk to my database servers. I'm just gonna put the connectivity to the two of them. And so congratulations now I have this awesome, transitive trust thing, this bi-directional trust where I say, oh, they can talk to each other. Which means that if, for example, the SQL, you know, the database server gets popped, it can have all the fun it wants attacking your app container or VM, right? Because you've set up this nice bi-directional trust fully there. My assertion is you need to be more granular, is that sort of thing. And per protocol as well, in theory. Okay, it's also important, as you can probably guess, to constrain whether from a workload you can talk out and whether things can see yet. And if in fact you're using containers, as many of us in this room are, and you have the thing where containers can come up and down where they want, you need to be able to say, oh, well the service that is using that container has this URL, you need to be able to make sure that when you move the container from here to there, the URL points, you know, gets pointed in some way to the right way. And that makes sense. It multi-cloud and stuff like that. Software components and version control, it would be really super swell to be able to say, from now on, we are only using Java 1.9, 7, 12, whatever it is now, right? It would be really swell to be able to say, anything that is running this current using 1.7, that's fine, but from now on we only allow 1.8. You need to be able to say something about what kind of things are being run out there. Similarly, you need to say things about deployment. It is only possible, and this goes towards the previous talk, you want to be able to have something in your deployment pipeline that says, do not deploy anything that has any known vulnerabilities. Do not deploy anything that doesn't pass their scanner. It's perfectly reasonable. You're gonna want to do stuff like that, so you need to control your deployment pipeline. And here's another tricky thing that I think that comes up, that people don't think about it until they are forced to think about it. So you have a situation where you have a developer and she's writing some code that goes and does some database access and having a hard time, it's not working. And so she comes to you and says, hey, can you show me the logs from my application? I really need to see them. And so you have a few choices. Choice number one is, give that person access to your database server. Here's the SSHDs go for it, right? Anybody in the room like that option? No? Other possibility is, okay, here's a copy of your logs, but that's kind of static, right? Because then you write it on a query, so that kind of stinks. So really you need to come up with a third option that allows you to be able to say, let me stream your logs so you can see your logs, right? That's kind of crucial to come up with some services in the long run. Otherwise you set up bad things. And because we're talking about policy, it's also crucial who can actually edit it. Okay, and I'm gonna say this one. This one is gonna allow me to take one of these away. Is that okay? Can I agree? Can I get the agreement? This is fine? Yes? We're gonna say one thing, this is not the major thing in that app series, but one thing that we do is kind of cool that I find fun because I like to talk about it. So one of the things that we do is we will, if you're having, say, an application talk to a database server, we will set up something where when you set up this connection, we'll stick something in the middle, an inline component. They don't like me saying man in the middle because that makes people hear. But let's just call that an inline something, or a man in the middle, but we all understand where your application talks to it, it talks to the real database. And the thing that you can do with that sort of scenario is a few things. One, you can do this ephemeral credential thing where my thing in the middle will hold the real credentials for the database and I will give these fake credentials that are only good for the connection between it and the thing in the middle for that length of that time to talk. And so that developer or that developer's source code repo or anything else can contain credentials, but they don't mean anything. And so it talks to that using these credentials and then it does the real thing and it sits in the middle. But that's not where it gets fun because if you can be in the middle of a conversation and I suggest that you think about this as an interesting paradigm, because usually you're shot away because it's usually in security vulnerability, you can do things like, hey, don't allow my application to drop any databases, right? If you understand the wire protocol for say MySQL and if you do, I'm so sorry. But let's assume you do understand the wire protocol. If you understand it, the application can say, hey, drop table, drop table, drop table, and you can just ignore it because you know what it's doing. All right, so there's lots of interesting stuff you can do, or you can introduce some artificial latency where you can avoid things like the Knightsbridge Capital story, so we're in the middle of New York. How many people here know the Knightsbridge Capital story? How many people don't know it? Let's try that one. Okay, so enough people here. So let me give you the story you can tell your children when they're being bad and you want them to go to sleep at night or whatever it is to scare them into and scare them out of the field. So Knightsbridge Capital was an awesome thing and you should go get the SEC report and if you don't, you can't find it, send me a email and I'll send it to you. Where they had, if I remember correctly, eight different servers that were doing high-frequency training and they managed to put the wrong piece of software in one of the eight. Then for about 45 minutes at very high speeds because that's what they did, it took, you know, the actual training took place except for this one here that was doing it wrong. And the one that was doing it wrong over here managed to take the valuation of their company down by 75% in 45 minutes. They had to do the bankruptcy thing and change their name and stuff like that, but anyway, so you don't need that thing. So you can just imagine that if you had something in the middle that was acting like a circuit breaker that that might stop stuff like that. Just an idea. Okay, so here's the question I want to ask you. All these things I asked about policy, what did I leave out? Come on, let's edit this slide together, okay? What did I leave out that you would want in a policy system? Oh, you thought I was just gonna let you sit and listen? Uh-uh. Say it. Say it. Excellent. Excellent. Next, please. What else do you need in a policy system to make it usable for you? Be useful. Say it. Be useful. Nice. I wonder what debugging in as well because for sure that's a hard problem. Two more. Documentation. Say it. Documentation. So you wanted to be semi-self-documenting? Sort of stuff like that, how do you do that? One more and I'll let you out the hook. Yes, sir. Impact analysis. Oh, I love that. I really love that. So in my opinion, there are five other things that you needed to make sure are in a policy system. Auditing off for mediation, troubleshooting you about getting documentation impact analysis. Just an idea. So instead of like leaving you like without a compass, not knowing where we're gonna go with this or where you can actually find this, let's talk about where you might be able to find policy systems like this. Now here's the sad news. You're not gonna find it necessarily from your cloud provider. AWS is a little further along as they often are in some ways in terms of being able to set policy around EC2 instance parameters and network typologies and stuff like that. Azure has actually done it better since I made this slide but prior to this time it had role-based access control with exactly three set roles. And if you didn't like those roles, you'd have those three roles. And if you didn't like those roles, you can still have those three roles, right? Google Compute Engine, this is obviously out of date, a little bit out of date, but still was this really awesome thing where basically it said, hey, if you have edit permission, congratulations, you can do stuff, which is not the granular you're looking for. And for both of you who have the pleasure of dealing with soft player, I don't know how many people do, when I asked them about their, where is policy in their system? They're like, well, the policies can be found in our web app GUI sort of thing. And so I went onto the GUI and counted the number checkboxes, there were about 70 of them. Not all of them were policy, but oh, you know, I'd say that's just bad. Okay, so how about in your platform? Anybody know what this is? Oh, you must know this logo. Somebody that's, I know this guy does. Yeah, right. So sorry, as far as I can tell, they're getting a little bit better along these lines, but at the moment, their policy, there isn't much policy to be there, but they're isolated and stuff that's getting better. This one here, what was this? Am I causing people to have traumas since this is what I showed these logos, right? Sorry, this is what? Loud open stack. App open stack. Okay, so not yet. So open stack has a project out there called Congress. And Congress' idea at the moment is you feed into Congress all of the events that go on in your open stack instance, and it will tell you whether something violated policy. And then maybe it'll do a remediation step for you, which doesn't sound like the greatest idea to me, but you know, maybe you like it. But anyway, it's not my favorite. This one, at least one person dressed up for my talk, I appreciate that. Yeah, so Kubernetes is the one that's a little further along because it's got things like the analytic constraint resource limits, where I just get quotas and stuff like that. And that's very cool. And I think it's getting a little bit better, stuff like that. And here we go, I'm pulling one more of these. Okay, one of the things that we spend a lot of time thinking about and happen to spend a lot of time is about policy. We have a fairly sophisticated policy agent and stuff like that. If you want to check this out, there's a URL then, I always check it out. Okay, so here I am coming towards the end of my talk. The key thing that I want to point out to you is now you have your own pitchfork, you have your own torch, and I would like you to please take these things and go demand policy systems in what you're using. And that's what I want to say. So that's the end of my talk. Oh, and the other thing I guess I probably should mention is, because I am kind of in the DevOps world, I did have an extra one of these just in case. So anyway, thank you very much. That's the end of my talk.