 Всем привет! Доброе утро и добро пожаловать на EasterHack! Сегодня я рад, что я делаю со мной проект, который я построил, который создал для того, чтобы разобраться с лимитами и эффективными инструментами, когда приходит в тестирование веб-эфинитикации. Рейдер изменил, как я обращаюсь к тестированию HTTP, и сегодня я покажу вам, как. Здесь agenda for today's talk. Я начну с introduction, чтобы говорить о рейдерах, и проблема, которая была создана для этого. Потом я покажу методологию, объясняю основные концепции в рейдерах, как это работает и как это работает, и как это сделано из других инструментов. Потом я покажу вам демонстрии, которые я надеялся, чтобы попробовать. Я начну сACEA-сесеня Imagination, и я начну с комплик小時а. Теперь я буду на 해GOd. Я Данил, я для шарiska для more than 5-х лет. Корректор в letzten год и немного время я работаю в пообедительности, предметы «Модис». Мне находятся 5 больше годов, работающие с ССС-админ, вructive implementation, особенно с ССС-админом. Я начинал строить рейдер два года назад, и после того же года он стал частью ОВАСП. И я ainda работаю на этом проекте до сейчас. Что это рейдер? Когда я был бранстурмингом с друзьями, титул для этой презентации, я пришел с этим. ОВАСП Рейдер, новая фрейморка, для manipulating HTTP. HTTP processes of persisting sessions. Это выглядит очень сложно. Мы бранстурмингами, и я надеюсь, что в следующий раз у меня будет более короткий титул, чтобы описать это. Но для меня это не так. Рейдер был, по-моему, инсцептированным с целью в том, чтобы помогать сделать веб-эфинитикационное тестирование. Но это было очень долго, оно evolved по-настоящему, и может быть использован для всех HTTP processes of arbitrary complexity. Как это работает? Рейдер делает новую DSL, которая имеет domain-специфический язык в контрасте с GPL. Генеральные программные языки, как Path and Java, и так далее. И это использует DSL, чтобы описать client-server information exchange. Это также построено с модуларией в голове. Так что, если невозможно построить этот титул, то он будет поверить все свои нужды. Рейдер делает это в аккаунт. И users can write their own code to extenderator. So the core is written in Python, while the configuration files are written in a little known language called HighLang or High, which is a list dialect on top of Python. And I will describe my design decisions in a later slide. Now let's talk about the problem. When I was testing the authentication with the barbsuit, I ended up with something like you can see in the screenshot. Basically, I had a bunch of poorly organized repeater tabs. So my process was as follows. I had to go to the first step of the process, do some modifications to the request, send the request, go look into the response body, find some piece of information, copy it, go to the barbsuit decoder-encoder, modify it as necessary, then come back to the repeater, paste it into the right spot for the next step of the process, and so on. And I had to repeat this for every single step of the process, which can become frustrating. And it's also error prone if you copy one character less, for example. I needed a much more flexible solution with fewer clicks. Once you understand how the process works, you usually want to automate it, or at least parts of it. With barbsuit, this can be done with so-called barbsuit macros. And they work for simple cases. You want to extract some information from a previous request and put it into the next one, but you can only extract based by some static strings or regular expressions. This will fall short if you need something more complex. Let's say you need to send the one-time password, the generated one-time password from a secret, or you want to run some other arbitrary actions. In this case, barbsuit macros will not help that much. You might be able to achieve this by writing your own custom barbsuit extensions, but this is also, again, not a very flexible solution. Now let's look at how Zaproxy automates authentication. Here you can see, you need to configure the so-called context in order to enable authentication and you choose from some predefined authentication types. Then you need to set up the pattern to detect when the request is logged in and when it's logged out. Also, you may need to configure the actual users and the session management parameters. While this looks trivial in practice, it's not that trivial to set up. It's also very hard to debug when it's not working properly. And the most important of it all. This will only help you if you want to automate the authentication part in Zaproxy. It will not help you if you want to test the authentication process itself. It's worth mentioning that Zaproxy also provides us with some so-called authentication scripts, which, in theory, will be used to automate all kinds of authentication. However, they can run only inside those contexts, which I've shown in the previous slide. You cannot run those scripts standalone. So, that was a deal breaker for me, because I was particularly interested in the authentication and testing authentication itself. Zaproxy also comes with so-called Zest scripts, which is interesting feature. It's the closest to what I actually wanted to have when I was building Radar. However, it has some pretty big limitations. And I believe it's because of its configuration file being written in static JSON. The earlier Radar prototypes were also using JSON configuration, but very soon I found out that it does not allow me to have the flexibility that I wanted with Radar. And I'm going to explain those JSON issues in a later slide. The challenges with authentication are well known in the Zaproxy community. Here you can see a screenshot from the web page, which advises you to disable or simplify the authentication when testing to make your life easier. However, this is a disappointment, especially if you are particularly interested in testing the authentication process itself, like I was. In this case, the tools such as Zaproxy and BarbSuit will not help you that much. So, now let's talk about JSON. As I said, the earlier Radar prototypes were also using JSON for its configuration. Turns out, JSON files get large and complicated fast. Editing them manually is painful and it quickly becomes awkward when you have a lot of requests and responses. You also need some special syntax to reference previously defined items. You also need another syntax to process items, to encode, decode, to some string manipulations and so on. And also you cannot really easily reuse parts of the JSON that you've defined before. And all of those issues could be easily solved if we had real code instead of static JSON configuration file. Which brings us to the interesting part. Why Lisp? To most people, Lisp is an ancient language which does not have much real-world usage today. However, Lisp languages are ideal for creating your own custom DSLs, to make specific languages. And one of the main reasons why it's so useful for my particular task for Radar is because of the concept so-called homo-iconicity, which is also called code-is-data and data-is-code. A programming language is homo-iconic when the internal representation of the program can be manipulated as data using the language itself. So the program is treated like a regular data structure and this gives the users the power of metaprogramming. So basically, you write Lisp macros which look like regular functions, but they're not. They do not return a value. They return pieces of code which is why it's metaprogramming. You write code that generates or manipulates other code. And this is very useful to abstract away boilerplate code to automate some tasks. And that's really important for configuration files, because you don't want to repeat yourself. You want to make it reusable and to make it more easily maintainable. And before moving to the next slide, I wanted to tell you my story with Lisp. I began using Emacs more than 10 years ago. And I still use Vim nowadays for some tasks. But I fell in love with Emacs extensibility. How easy it is to change it. It does not feel like a text editor to me. Not even an IDE. It mostly feels like an operating system. A text-based operating system. Nowadays, I use most of my time inside Emacs. And it's not just for coding. There are many tasks and goals for reading my emails, reading news articles and so on. I even wrote this presentation using Emacs org-mode by exporting it with Lattice Beamer. So why am I telling you this? For many years, my main issue with Emacs was its configuration file, which is written in Emacs dialect for Lisp. And I wish that I could configure it using Python, which is my main language. But when I was battling the JSON issues I described in the previous slide, at some point it finally clicked. I finally started understanding why do we need Lisp and all those things. Basically, I had to write configuration files, which is data. But I also needed to contain code. I could have invented a new way to do that, but why bother if Lisp is here? I started playing with Lisp in Ryder and slowly I started understanding why do we need all those ugly parenthesis in Lisp. They're not for us humans, they're for the machine. We humans are supposed to ignore them and focus on the code and its indentation. And it all started slowly makes sense. Macro started to make sense and I couldn't wait out to try it all out in Ryder. By now it's clear, I hope why Lisp, but why Heilang. Well, the most important reason is because it compiles into Python code. You can still access all of your Python libraries, which is super important, especially in the security field, where so many tools and libraries are already written in Python. So Heilang combines the flexibility of Lisp with the power of Python. And it's also relatively easy to learn if you know Python. It's basically just a Python code written with Lisp syntax. Now since we've covered the basics, let's go into the details. In my experience, a lot of pen testers see the authentication as a black box. They assume this part is safe and they don't try to look at it that much. Which is a shame, because it causes many bugs to get overlooked. And with this talk and Ryder in particular, I aim to make it easier for people to understand and test complex HTTP processes. Like authentication, but not limited to that. So Now I'm going to show you some techniques on how to reverse engineer such complex processes, like the authentication. So here you can see a screenshot of a metamproxy, which is another proxy tool. And I captured all the traffic that's being generated while logging in on Reddit. And as you can see the 392 requests here. Most of those are not relevant. So we need to remove everything which is not. Those are mostly javascript file images, videos and so on. So after we remove all those unnecessary ones we are left with something that is much easier to digest. But it's still not all of those requests are relevant. We need to figure out how the authentication works, which are not relevant, which are not and so on. One way to do this would be to start from the beginning from the first request and trace back all of the inputs and outputs being used and generated until the end. However this is not always the most efficient way because you can get sidetracked. It's very easy to get some cookie or some header is relevant in the process while in reality it's not. And sometimes it's easier if you start from the end. So you pick one request which only works if the user is authenticated. It should fail otherwise. And go back from there. So here's how one such request looks like. And here you can see there are several blurred fields in the headers which could potentially be used to identify the user. Now we need to figure out which ones of those are important. How do we do that? We remove everything piece by piece until we arrive at the simplest request possible which still gives us back sophisticated result. And here's what I came up with. The first two headers host and user agent are required by the HTTP protocol. So those are not relevant here. The three ones highlighted in red are generated automatically by Python request library. So those can also be ignored. So this means we have only one piece of information which identifies us in this request. And this is the authorization bearer token. So for this particular endpoint we only need this one input to identify us. But how do we get it? Well, we search the traffic for this particular string and identify the first request which gives us back the response. And here's what I found. On the left side you can see the request, on the right side it's the response body. And this access token is hidden somewhere deep inside the response body. And this is the only output that we need from this particular information exchange. But what are the inputs? Here you can see again a few blur fields which can be used. And we do the same thing again. We remove everything piece by piece until we arrive at the simplest request possible which still returns us back this access token in the response. That's how you reverse engineer such processes. And there are more details to it. I'm not going to show you everything until the end for Reddit authentication. I'm just showing you some techniques. We need to focus on radar. So now let's talk about finite state machines. Finite state machines are a mathematical model of computation that allows you to analyze complex computer systems. So a system is stateful if it remembers the preceding events. For example, HTTP protocol is stateless. Each HTTP request is independent from the other ones. While the authentication process is stateful because you need to complete every single step of the process in the right order in order to get actually authenticated. This state is the information which you remember about the system. And the system can be in only one of its finite number of states at a given time. And the so-called melee finite state machines can be used best to model the authentication systems. And we can have multiple inputs, multiple outputs and the output values are determined both by the current state and its inputs. And here is how one such single state would look like with its inputs and outputs if we model the authentication process from finite states machine. So, in this case, we have three inputs. The username, password and SSR are of token. The server verifies them and returns us back some cookies. To model the entire authentication from the beginning to the end, we need to chain multiple such steps together. And here you can see a simplified diagram of how that would look like Each state having its inputs and outputs hidden because otherwise it would get too complicated to put it here. So, basically, we start in an unauthenticated state we open the login webpage we send the credentials, the server checks them if they are wrong, we go to the login failed state, if they are correct the multi-factor is checked if it's enabled or not if it's disabled we go straight to authenticated if it's enabled we go through one more step and the server checks the code and sends us back respectively either failed or passed That's basically it. That's how you model the authentication system using finite states machines. Now, how do we do this with radar? The so-called radar flows are basic building blocks in radar and they are used to describe information exchange during one single step of the process So, you have one single request one single response To create a flow, you just need to give it the URL and optionally it also may have some outputs and some operations The outputs are in the outputs you tell a radar what information to extract from the response While the operations either run some arbitrary actions after receiving the response or it links to other flows and it can also be conditional Depending, let's say we receive a HTTP status code 200 we go to flow A otherwise we go to flow B Now, here's how two such flows interact with each other in radar Basically, each flow needs to have one request and one response The request may optionally have some inputs While the response might optionally return some outputs and also each flow also optionally can have some operations which I said can either run arbitrary actions when you receive the response or tell radar which flow to go to next Another important part is the outputs from previous flows can be used as inputs for the next ones This is how radar keeps track of the state That's how you share information between the flows Basically, to create a flow you just need to give it a request So, look at the request The only required parameter is the URL Everything else is optional You specify the HTTP method You can also give it some cookies, headers some parameters to append to the URL or use HTTP body data either in URL encoded format in JSON, in multi-part and each of those can contain plugins I'm gonna explain plugins in the next slide but those are used to share data between flows So, plugins are small pieces of code they can be used as inputs and or outputs You can use them to extract data from somewhere let's say from the user from the terminal You can also extract some data from the response body or from JSON from the URL query You use plugins also to manipulate data encoded, decoded do some string manipulations Some of those can also be nested and the user can write their own plugins without touching radar core Now operations They run after the response is received as I said and they can execute arbitrary code but the most important thing why operations exist is to control the information flow to tell radar where to go to next and whether the SCP process completed successfully or it failed Operations can also run other operations conditionally as I said, it can check for the status code it can check for response body matching something and so on You can even run real highland code by using the list code thing and the users can write own operations without touching the core Now, let's talk about flow graphs It's a relatively recent feature introduced in radar and they use to chain multiple flows together until the end and or until success of error operations return Basically, they just a pointer to the first flow but optionally, it can also have the second flow we just run an additional test to see if the process completed successfully or not You can simulate complex system using this architecture and as you can see in the screenshot you can have independent flows like flow9 which only returns success or failure or you can have complex networks of flows some with a flow graph pointing to it and others without some will return success or failure others will not and from flow9 you can run either a flow graph or any combination of flows and flow graphs as you want and if you run some flow without having some inputs radar will prompt you for those inputs manually so you can run from a common line let's say we start from flow4 then go to flow9 flow2 until the end till success then run flow3 and so on any kind of combination of flows and flow graphs is allowed and this enables us to test very complex scenarios now I'm going to show you the demo so for this demo I prepared I automated some processes of ASP juice shop which is an intentionally vulnerable application for this first example I automated the registration and the login part and here you can see approximately how that looks like to actually register a user we need to send 3 different requests so first is to get the security questions the next one is to register a user and the last one is to submit the security answer so we created those 3 flows and have the register flow graph pointing to the first one additionally we also have the login flow which needs just one step and as you can see some of those flows return success and failure others do not so now you can see of ASP juice shop and I'm just going to show you that I don't have the user yet so now it will fail but let's take a look at the code first so here is the project files for this for juice shop the first one the main file it's mostly just creating some plugins so basically the first one is just a string we created the base URL for ASP juice shop then we have here some plugins which will extract the question from Jason here below we have some other Jason plugins those 3 variable plugins email, password and password will be the information we created about the user here below and we have 2 more plugins one is a cookie which is created from a previously defined half token plugin and it will be named token and the last plugin is a regular expression which will just match some html code and print it use it later it's for another attack so basically here I created the plugins registration so here we have in total 3 flows and the first one is we'll get the security question and I'm using with base URL here it's a macro just appends the base URL we defined to this one and it will extract the question it will print it and go to the next stage now for the next flow and the post request with the json body and all the credentials here inside it will extract the user ID and then it will check for regular expression in the response body to see if the email is already registered or not so if we try to register twice with the same email it will fail and it will print this otherwise it will print the whole body and go to the last flow so here we send the post request with the json body and it's very similar except it will check for HTTP status 201 which means the user was created otherwise it will return failure and here at the end we have the flow graph which points to the very first flow here now let's look at the login on this is simpler so basically we have just a post request with the email and password and it will output the half token then it will check if the status code was 401 which means the login failed otherwise it will print it yeah and return success now here is how integration looks like basically you can see just from the project here you can see the filenames so for each filename you can see all the flows and flow graphs are in red and that's configured in this file so by default we also get the default flow graph which is just to run the very first flow just for convenience purposes but now let's run it so I want to register then login and as you can see it got the security questions printing them, registered the user printed the response body submitted the security answer and created the user successfully then it run the flow login which printed the token for us the half token if we try to run it again because we cannot register the user with the same email twice now if we go back here then we already logged in and it works so that was it for the first demo now the second demo will actually exploit an SQL injection in JuiceRob when you login in JuiceRob the email input field is vulnerable to SQL injection and we are testing this with a radar there is just one single flow which points to itself and there is a flow graph pointing to it so now let's take a look at the code to see what exactly that looks like for SQL injection so that's it we have this login SQLI flow which will send a post request and with the JSON body but here you can see that instead of sending some string for the email it will prompt us the user for the payload this is how we are going to test it so it will also output the out token print the response body and check for status code if status code is 200 that means we are logged in successfully otherwise it will go to itself or it will run in a loop otherwise now let's see how this looks like SQLI test so if we just input some random value it will give us back invalid email or password which is expected but if we include a single code we will see that there is some error some SQL error and explain how all of this works so I'm just gonna use the solution for now and we can see the login process completely successfully, we got the token and we authenticated as the admin user now for the last example this one is a little bit more complicated because you need to chain multiple things together the ultimate goal here is to exploit stored cross-site scripting issue which can be found in the change username flow here basically when you change the username you can use some access attacks inside this field but you cannot exploit it because of the so called content security policy header configured here but this can also be bypassed with another vulnerability which is why we have the second CSP test thing and I'm gonna show you all of this into the details very soon but first to actually run those we need the auth token so we can get them either by using the regular login or the SQLite test both of them return the auth token so now let's look at the code here's for the accesses and so basically we have two flows here and one flow graph the flow graph points to the first flow so basically the first one will just query the user profile and print the part of the html code which we are interested in and the second one will submit a post request with a username changing the username and it will also use the prompt thing to ask us the terminal line which username we want to have and yeah and it will run in a loop so from the first request it will go unconditionally to the second one and from the second one it will go unconditionally to the first one it will just run two flows in a loop infinitely now for the content security policy one we have also two flows and one flow graph pointing to the first one but here is a little bit different so basically we request the profile page we print the content security policy header we check if the user is satisfied with the result if the answer is yes the process returns success otherwise it goes to the next flow to change picture URL and here it will just submit this thing with what we told it in the prompt right here and for a reason you will understand very soon we need to wait for the server to time out 30 секунд so to do this I just use the least code to import the time library from python and sleep for 30 seconds then I go to the back to the first flow so this will run also in a loop now let's look at how this actually looks in practice so the first step could be either login or sqli test which I've showed before also gets us back this token but since I don't wanna do the sqli injection attack again I'm just gonna use login, it's faster the second one is CSP test and cross-executing test and I guess that's gonna work let's see yeah so first it printed the existing content security policy header and this doesn't work for us because we need one more attribute here we need unsafe inline we need to add new attribute so I'm not satisfied with the result and oops, sorry I need to repeat that yeah so now it asks for the image URL which will upload which will change the user avatar second, I'm gonna do something so now we give the server an invalid URL but at the end we have appended the actual attributes that we want and the server will try to get this URL which is why we need to wait for it to time out and now in a few seconds it'll time out and you will see that despite the invalid URL we will still get the content security policy header updated here so it changed to what we wanted so now hopefully it'll work and I'm satisfied with the result now let's take a look at the cross-site scripting thing so basically right now you won't see anything interesting but if I just test username you will see my string here and we can try the regular alert and you will see that it gets filtered so we need to bypass this filter as well I'm just gonna use the solution since I don't have time to explain it all here I'm gonna print documented cookie and I guess that works and if we go to the profile page you should see the cookies in the alert box I hope why is it so slow yes so here it is yeah that's basically it that's all for the demo part hope you like it so what's next basically right now the documentation part of radar is incomplete there are still many changes which have been done but has not been actually reflected into the documentation after that I need to focus on the way users interact with the radar so basically right now the debugging features are still missing and it's also a very limited command line interface and I think one great addition would be REPL basically instead of changing the source code and running radar from the command line you could do all of this in real time another interesting thing that I've experimented with was using GPT-4 to actually generate high files from natural language basically it almost worked like 80% correct but there are some errors which does not allow it to run and if anyone knows how to train it and is willing to help me with large language models I will accept your help and I'm also interested in adding some new features so basically the fuzzing part was kind of working at the beginning but now with a lot of contextual changes it broke completely so I need properly functioning fuzzing mechanism here I also want to have a rather keep check of the sessions so that you can recall or reuse previously seen inputs and or outputs I also want to implement more macros to make it simpler to actually write the configuration files and last but not least I'm also interested in integrating radar with other tools which will benefit from having something like this now I admit radar has some big limitations it has a relatively steep learning curve it already assumes that people already know finance test machines and are comfortable using Lisp Python coding and with Lisp it's a thing that many people are just scared of those Lisp parenthesis everywhere so I hope you don't be a big issue but there's also very limited community support radar is still a recent tool that not so many people using it and it's also I'm only working and testing it on Linux but compatibility with other operating system I haven't tested it so now to end the presentation I want to say that radar is not just a toy anymore it can be used for complex real life systems I've already used it for a lot of stuff during my job for penetration testing but there's still a lot of work to do and room for improvement and if you are interested in helping or learning more about it feel free to look into those links and contact me and now I'm open for the questions no questions I can hear anything Hi, you said earlier that you're interested in not so much earlier you're interested in integrating with other tools you mean like integrating with ZEP and adding the radar as an optional authentication provider or would this not be an option? Can you repeat please? I didn't get the last part If it would be possible to write a plugin for ZEP to use radar as the authentication method that the authentication then ZEP does the scanning Yeah, we are already in talks with ZEP developers about this but we still haven't made any progress yet so it's still in planning Thank you, perfect That's it I guess Okay, thank you very much