 Thanks for coming. This is deep diving into GRPC security case study on API security and we'll have some fun protobuf reflections. I'm Dana White. I've been a software engineer for a long time. I have a lot of animals. This is Austin who's going to be speaking later. He likes dogs. And he's been a software engineer for a while. So what we do at stock at Stack Hawk, we make a dynamic application security testing tool to ensure API security and it helps engineers find triage and fix vulnerabilities before they see production production. We basically enable engineers to shift security left and we are part of the team. Austin and I particularly are part of the team that make engineer the desk scanning tool. We also really, really like GRPC. So I'm going to give a quick overview of what DAS is. Austin's going to dive into it a little deeper when it's his turn to talk, but kind of you need a basic understanding for what we're going to go into next. Like I said, dynamic application security testing, that's what DAS stands for. It tries to mimic a hacker trying to hack your site. So like live hacking what they'd be able to access over HTTP. And so part of that is to explore that is you need to be able to get the full surface of your application to test. So some of the things you think about when someone's trying to attack your site, SQL injections pretty much everybody knows that one. So what we do is we do SQL injection, but over HTTP. You're familiar with the XKCD comic. This is little Bobby tables, but over HTTP. So this it discovers how you don't sanitize your database. So one of the things we need to do to discover the surface of your application is to it is to. Oh, I forgot what we're going to be talking about. Sorry. Okay, so there's three parts we're going to be talking about a developer tool to discover and iterate on authentication, which I was kind of saying that you needed that to discover the surface of your application. We're going to talk about how at StackHawk we run our internal microservices with CRPC. And then we're going to talk about discovering vulnerabilities in GRPC services. So the authentication cars. Authentication is always a problem. Authentication is always different and it's actually designed to be unautomatable. And there is always some legacy authentication where you need three cookies and a bearer token. And the guy who wrote it left 10 years ago. So to actually discover the authentication for your application, you need a quick way to test authentication. So this is what auth might look in our CI CD pipeline. That's just an example of our config file. And it's really simple. And that's pretty nice. But other times you might need web inputs. This is a little selenium script designed to click buttons so that you can off. So that's kind of hard to work on that by starting the scanner or doing a PR to CI CD to write for the scan to run. So if we need to iterate real quickly, we decided to come up with a tool, which is and we use GRPC for it because we love GRPC. So in this instance, it's a CLI command that spawns a local GRPC server. Think it's running on local hosts at 20,000 listening for any client to connect to it. In our instance, we're actually using our CLI to communicate with the GRPC server. So and it's watching all those config files that I just showed you for change and then streaming anything back so that you have instant feedback and you get it immediately. So how it works. This is you at your computer. That's your application. It's pretty fancy graphics. I know that's you authenticating and it doesn't work. Let's step back a little bit and through our GRPC server in there. And now it goes to our GRPC server and goes back to your application. And this time it works online and never actually works that fast. You're going over it a bunch of times and that's why we have this tool running. So this is some code basically of why we liked GRPC for this is streaming back real time changes. Kind of longing by line like this. It's Kotlin. We like Kotlin too. Some co-routine magic, some co-routine magic and a little keep running function. In this instance, it's iterating over a log file, but it can be anything sending messages back to our GRPC server. And this is sending it back to the client just doing the on next and pulling everything back. So you have real time feedback on what you're doing in your authentication. But we also use GRPC in our platform. We have a pretty typical setup. I guess if anything is typical, we have our Microsoft's micro surfaces hosted in Kubernetes. And then we have a RESTful gateway API to talk to our front-end over HDB1. But we have GRPC communication all in the back end. For some of examples, we handle our quota buffs by having it all in one repo. And we wrap all our request responses so that if like maybe this underlying config changes or we add something to it, we don't break the contract. But we're security, so we need to test our services. So we could test our API gateway. And that requires every other service that it depends on up and running in Kubernetes, which we do that. But you're only testing the pathways in the microservices that it uses. And like I said, authentication is a problem. What we'd like to do is test our GRPC services directly. And yeah, authentication is still a problem, but traditionally we haven't had a scanner that goes over HDB1. We need a scanner that goes over HDB2. So I'm going to turn it over to Austin and he's going to talk about the wonderful world of DAST. Thank you, Dana. So I am going to talk a little bit more of the scanner and how we improved our scanner to be able to scan GRPC applications. But before I do that, I want to talk a little bit more about what DAST is and give a little bit more context of the problem we had to approach. So as Dana said, DAST is a dynamic application security testing tool that essentially looks at your live running application and tries to include malicious data in its request. Look at the responses, see what might have stuck, see what a hacker could actually get away with or a bad actor out there might actually be able to get away with. And we're trying to do this all with the shift left perspective of doing this as part of the development pipeline. So none of these bugs ever get to production. It's a lot more expensive and time-consuming and scary to have these types of vulnerabilities in production. So we want to fix it as part of the development cycle. Something else that's worth noting is that there's particular code scanning tools out there such as GitHub CodeQL, SNEAK, and what they're doing is static code analysis where they're going line by line and actually looking at your code to see if there's any outdated packages, known vulnerabilities that you have straight in your repo. What we do different at StackHawk and with our DAST scanner tools, we're completely code agnostic of that. We don't ever want to look at your code. We don't want to see your repos. We just need your live running application and a way to talk to it. Once we can talk to it, we can start sending those malicious requests and seeing what sticks by the responses. And give you a lot of nice triage nodes, a lot of possible solutions, et cetera, information on that. So the fact that we're code agnostic, that means we're totally able to scan anything under the sun. May it be web apps, spas, REST applications, SOAP applications, GraphQL, but the problem was up until now none of these DAST scanners including ourselves could do these GRPC applications due to the complex and efficient nature of how you communicate with them. Until now, StackHawk now and Hawkscan can now scan GRPC applications and that is mainly what I'm here to talk about is how we approach that problem and figured out how we could have our scanner communicate with these GRPC applications. So as I mentioned, the GRPC applications definitely deserve the same DAST love that all the other API types. They're susceptible to vulnerabilities just like REST and GraphQL and spas. So some of the challenges we ran into, as Dana mentioned, the GRPC stuff all runs over HTTP2 protocol but our product is built on top of ZAP, it's an open source scanner itself and so ZAP already had some support for HTTP2 so we were really easily able to leverage that and start sending these requests over HTTP2. As we also know about the complexity, which also lends a lot to the efficiency of GRPC, without access to the proto buffs, it's hard to determine what the kind of schema of one of these applications looks like. An open API, we have like a swagger doc or an open API spec in GraphQL, we have schema files. We don't really have anything like that in a GRPC if you don't have access to the actual proto buffs themselves. So what there is though is what's called the file descriptor set that you can generate in GRPC and this can either be accessed via reflection like you go to the application slash GRPC or whatever your reflection endpoint. Some frameworks actually support generating this. For example, we use Kotlin so we're able as part of our build script we can just generate this file descriptor set and this really encompasses all the objects that live within the application itself. But one thing about this is the file descriptor set is still really just a byte file. So as a human I can't open that and determine what the schema of the application looks like but it is still a starting point that we can start determining things about the application. So it's not impossible and leveraging the GRPC libraries that are provided to us, particularly like I mentioned we're using the Kotlin ones, we can start going through and programmatically looking at these file descriptor sets and seeing what the application has, what types of messages, what types of services, what types of methods, etc. So really the key to all this in terms of the scanner and actually being able to communicate with these applications is the dynamic message class that GRPC provides for us. So the dynamic message class allows us to start building these messages at runtime and give them the certain information and method signatures that they're going to need to be able to properly communicate given an application. So really all we need to do is by parsing through this file descriptor set and getting certain types of messages, whatever it may be, in this case say an all string message that just has a few strings in this field, we can start iterating through that, setting the values to what we want and then when we instantiate this dynamic message object we give it the original type that we pulled from the file descriptor and by the time it creates this message and sends it over the wire to the actual application it's going to look and feel exactly like that original message type. It will no longer be this dynamic object into the context of the application once it receives it. So like I said, the byte code is really essentially the same as if we really created one of these things. So just a little bit of the example code here that we use in Kotlin. This is one that can just, the first thing we do is we create the dynamic message with the builder of the message type that we're processing currently and then we can just from that point iterate through each of the fields and start setting values. Obviously as a very simple example, if this is a complex object with nested fields, nested things like that you need to be a little smarter about it and that's what we did. So by using recursion and some old school programming magic we all learned back in school we're able to iterate through and recursively go through all these different dynamic messages that are provided by that original file descriptor set and through that we start generating messages with values either through custom variable injection or leveraging the Faker library to come up with realistic type of data based on email fields or phone number fields we can come up with realistic data around that kind of stuff. But something I do want to touch on is very important that we've implemented as custom variable injection when it comes to scanning. So what this means is that as the scanner is running and you have certain fields in your messages or endpoints, things like that and you can start designated certain values to those specific fields. So for example if you have test users in your CICD pipeline and you do it in pre-prod and you have test users there in certain conditions you can start saying when the scanner runs and this user ID comes up use this specific user ID for scanning because they might exercise different branches. So really what that does is allows you to really make sure you exercise all the different logic branches of your code and let the scanner explore every nook and cranny possible. But if you don't have any of those custom variables we do use Faker to come up with very smart, properly formatted values. So once we do that and we have these messages constructed and we can start sending to the GRPC application we're just off to the races at that point. All the serialization is happening behind the scenes so when it's actually communicating to and from the GRPC application it's in the correct bytecode and then when we present the findings on our side we'll serialize that back into human readable kind of JSON stuff. So it's easy to interpret the data without having to go through all the bytecode and under the hood stuff of GRPC. So what I can show off here is this is our StackHawk platform that unlocks me well in the past five minutes. There we go. So this is an example of one of our scans that we ran against our vulnerable GRPC application. Let's make that a little bit bigger. There we go. So what we have here in some of our findings so the services as we know we have services and GRPC services and those services come with methods as well. So we get a nice listing and again this is all pulled from the file descriptor set that we got off the reflection endpoint for this particular service. So with that file descriptor set we're able to find all the different services all the different methods and map them out pretty easy to read. In our findings we can say Dana mentioned a SQL injection earlier we purposely put a little SQL injection bug in our vulnerable application. So when we drill into that we get to go down here and see all the different services and methods that that SQL injection was present on. And you can see down here in the evidence our description put a little comma there is one of the values and we ended up throwing a Postgres SQL exception. So in the response we get to see right there in the GRPC formatted response that a Postgres SQL exception was thrown which we shouldn't know you have Postgres SQL just by sending a request to your application. So that's how we know you are susceptible to a SQL injection attack here. And then just like everything else we'll give you nice information on it steps for mediation some cheat sheets on different frameworks. And yeah the big thing here is when it comes back we get to see it in the nice formatted JSON format rather than the bytecode that was actually sent to the application so it becomes very human friendly in this sense. I think that might be about it. Yeah I think that's it guys if anybody has any questions we're happy to take some questions back here. Does this scan any of our ability other than SQL injection? Oh does it here I can actually show you while I still have this up if we go back over to the scan we can see the plug-in summary and essentially with this plug-in summary is a list of all the different vulnerabilities that it checked for. So obviously all the OWAS top 10s all the big ones heartbleed SQL injections the list goes on and on and since we're developed on top of the open source project zap is they add more plugins that are more vulnerabilities. We're also bringing those into our system as well so our library vulnerability scanning is always growing. Sure. Any other questions? You mentioned your course at the beginning to support and you mentioned custom variables. Do you support adding custom headers of metadata that make. Yeah there's a few different ways that we can do that but yeah we support that whether it's configured or whether you like we have an off scripting thing where you can just basically make whatever you want happen happen but we do have some ways you can plug it in but yeah. And as far as the custom variable injection goes that's also available on headers or path parameters body parameters. It's really anywhere where we can identify the field name and inject those certain values very sure. Thank you. So in terms of these cans or are you guys using an agent on the on the box or it's going to be a live scan while while the services are running. Yep that's the whole thing is since we're dynamic testing we need the service to be up and running so we could actually contact it and we don't care about the code what framework it's written in what API type it is. But yes to actually scan it it needs to be up and running and we got to be able to access it. We suggest you don't do this in production. Understood. And what kind of you know what are the resources for example when the scan is running in terms of resources how heavy is this scan. A lot of that depends on how big your application is itself. If you have a pretty efficient small way applications a lot of times our scans are a minute long maybe not even that long. But some of our larger legacy and people that are scanning entire gateways rather than individual microservice a lot of times we can see slightly longer scan times. But as far as resource goes it's it's very configurable and you can pass arguments to say how much memory you want the scanner to use etc so you can control that. Thanks. One more. I was unfamiliar with Stackhawk and Zaap before today. I was unfamiliar with Stackhawk and Zaap before today so the root question is why would I use you rather than Zaap and the second question is can I use you for free to get started. To answer the first question why would you use us rather than Zaap if you have ever used Zaap before I think that would answer your question alone. It's hard. What is it a Java swing front end made back started 15 or so years ago. It's very cumbersome. It's very hard to configure so what Stackhawk does is we build on top of it make the scanner itself better and make it just so much easier to use and a very intuitive platform to go along with it to triage findings and learn about these things. And I don't think I don't believe it supports gRPC natively. Zaap no I will maybe if you do some crazy jump through some crazy things. But the second question is the second answer is yes we do have a free trial account where you can scan a single application as much as you want for free. We don't charge by scan time. So you can take a single application go to the races on it pull out full support from Stackhawk and go to town. And to we. That's our VP of product Lauren say hi. Thanks for coming. Does it matter where the application is deployed that you're standing like. No as long as we can talk to it. We like to run the scanner as close to the application as possible just for latency reasons. But yeah as long as we can contact the application in one form or another. So say it's like an intro. Sure yeah we have a CLI that you can run we can we're also dockerized so it's easy to stick in any pipeline pipelines or anything like that. But yeah typically as developers and us working on it we're using the CLI quite frequently. But yeah totally totally easy to do that. Alright guys well we appreciate the time thanks for coming over to the little room to check us out.