 Yeah, so sorry about that beamers or protectors, I don't like them, they don't like me either. So a little heads up, this is going to be the only slide I'm going to show you today, so slide, because I think doing stuff like that in the terminal might be a little bit more interesting for you, but sadly something is getting cut off, so we have to improvise a little bit. But anyway, so today I will be able to talk about two of my favorite things right now, which are free BSD and Dtrace, but this talk has been cut down to 30 minutes, so while we're focusing a little bit more on the Dtrace part, so there will be a little bit less BSD than I anticipated, and also I adjusted everything a little bit to fit better into the resilience track, so hopefully you will enjoy that. So before we begin, who here is actually using Dtrace? Okay, more than I expected, but still not as many as I would like to see, so hopefully after this talk you will think, oh, this is a really awesome tool, I got to learn it because I totally love it, it changed the way I do a lot of stuff. So for those of you who do not know what Dtrace is, first let me fill you in on this stuff. So it's open source, it originated on Solaris and has been developed currently on Illumos, which is a fork from open Solaris, has been ported to free BSD, NetBSD, OSX, there's also a port for Linux called Dtrace for Linux, I think it's done by a person called Paul Fox, it's been ported to QNX, and the open BSD folks are currently doing some work to get the technology like Dtrace on their system, and I've heard there's a port for Windows, I don't know if this is actually true, but it is, it's kind of cool because then that means it's basically everywhere. So most of you will probably know static tools like Strace, we have a very similar tool on free BSD that is called Truss, and what Truss and Strace are doing is you can attach them to a process and look at the system call that this process is emitting. So in case something is going wrong, you can look inside of the program, which can be kind of useful when you're trying to find a problem. It's kind of handy but it's also pretty limited because first of all it really, really slows down the process that you're currently looking at. So if you want to debug a performance issue, you're pretty much out of luck there. And also it's kind of like narrowed down, you can just look at one process, which is also like bad thing because the system that we currently have, all these systems, they're very complex. We have a lot of layers, you have virtual file systems, you have virtual memory, you have network, you have databases, processes communicating with each other, and in case you're using a high-level programming language, you might also have a runtime system. So it's a little operating system on top of your operating system. So when something goes wrong in a system that has such large complexity, something happens that we call the blame game. And the blame game, it's never your fault, it's always someone else's. So what we want to be able to do is we want to look at the system as a whole so we can correlate all the data and come up with some meaningful answers when something is really going wrong in there. And also we don't want to switch out all the processes for debug processes to make that happen because as these things are, every problem happens in production. It never happens on the development box. So like switching out all the processes, that's totally out of the picture. So to do that in an arbitrary way, to instrument the system in an arbitrary way, we need a programming language. So we need to describe when that happens, please submit data so I can see what's going on. So this kind of implies a programming language. And D-Trace comes with such a programming language. It's a little bit reminiscent of AUK cross with C. It's pretty simple to learn. You can pick it up in 20 minutes and you can start turning out your first D-Trace scripts. So like AUK, if you know AUK, AUK can be used to analyze large bodies of text. D-Trace is pretty much the same but for system behavior, it's a little bit mind-boggling but probably I can show you what I mean by that. And also as a bonus, we don't want to slow down the system. So we want to be able to do things like performance test like that. So I've prepared this little demo here. So since we had some issues here, probably this is not, oh, I have to play around a little bit. So what I'm going to do is I'm going to look at a very, very naive way to, excuse me for a second, very naive way to authenticate a user. And there's a lot of stuff wrong with this code but what we're going to do is we're going to take a user string as input and then we're going to just compare it to a secret. So I know the secret in here is like in plain text, I know this is a problem but it's a little bit artificial but I just want to get my point across. So from an algorithmic perspective, this check function is correct. So we take a string, we take another string and we compare them. Everything is fine and easy. So if you look at the way string compare works and what it does, it's essentially taking these two strings and it's comparing every character bit by bit. So when it finds the first pair of characters that do not match up, it's going to stop. So we can conclude something about from that. So if it takes very short, if this function, this check function takes a very short amount of time, then what will happen is it will terminate earlier and if our password guess is better, it will take, well, it will take longer. And if we can measure that, we can basically extract information from that running algorithm. So I wrote a little driver program in Haskell that basically just iterates over an alphabet and just feeds this one letter into that program and I'm going to use detrays to get some timing information. So let me start the driver. So this is now just running in the background and so you cannot see what I'm typing there, but don't worry, these scripts will all be, I will push them on my GitHub. So detrays now produces this nice little distribution. So if you were able to see the entire alphabet, you would see that everything except D behaves differently. So if you squint a little, what you see there is detrays, the D letter takes a couple of nanoseconds longer. This is the precision that I'm measuring here, 10 to the minus nine seconds, really precise. And D takes longer than everything else. So it's a little bit cut off there, but trust me. I know I sound like Donald Trump and I'm saying that. So yeah, and from that, let's just enter a letter. And now the password and now the script clears everything and it's going to guess the next letter. So sadly this is cut off because you would see that this distribution radically changed. It looks completely different. And so we can play that game a little bit. So let's just roll with that. And every three seconds the script is going to recompute looking at the new distribution and you can probably see where this is going. So here you can see, okay. And now it just takes about like three seconds for me to guess the next letter. So and this is not a problem that is only something that happens when you do string compares. This can happen with basically everything. So especially in things like cryptographic stuff where you don't want to have some information leaked out. So this is what we call a timing side channel attack. So I could essentially use detrays to analyze the real binary. So I didn't change the binary. I didn't have some debug code there. This is like the actual binary that I would put into production. So what's important about that is to take the actual binary is some of these timing side channels might be introduced by a compiler optimization. And when you insert debug code into that code, then it might actually go away. So you want to look at the real code that you're putting into production. Let me show you the script that I came up with to write that. So there are three interesting things in this script. So and don't worry, this is the more complicated example. I just want to inspire your ideas because the things that you can do with detrays, pretty much the sky is the limit. You cannot come up with the weirdest ideas. And so this is a more complicated example. I'm going to show you simple ones to demonstrate how we got here. So there are three interesting things in this code. The first one is something that we call a probe. So a probe is a point of instrumentation in the system. So whenever a certain event happens in the system, this probe is going to fire. And in this case, the begin probe marks the moment when the script starts. So the second interesting thing is this clause. So this clause is basically what this probe is going to execute, what's going to be executed once that probe fires. So it's a little block of code. And this probe is a little bit more interesting because it tells us something about the structure of how such a probe looks like. Because every probe is uniquely identified by a four-tuple. So it's like four components that uniquely identify a probe. And the first part of this tuple is called the provider. And I'm going to talk about providers in a couple of seconds, what they are. The second one is called the module, the third one is called the function, and the last one is called the name. So these four pieces of data, like, they identify probe uniquely. So the third thing that is interesting here is sadly something that I don't have time to talk about today. This is called an aggregation. And this single line that you see here is essentially responsible for accumulating all this data to print out this distribution stuff, to generate this distribution. So this is built into detrace. You don't have to do that yourself. When you look at this script, it's like 42 lines of code. And I came up with the first prototype after five minutes. So it's not a lot of stuff to get something out of that. So it's very useful to have things. If you use detrace, you will use this a lot for performance debugging. So it's kind of neat that we have that. So yeah, let's talk a little bit about providers. And this will probably also be cut off. So I'm going to cheat a little bit here. Just going to double that. So let's talk about providers. So here are a couple of providers. Oh, that's handy. So I got 27 providers here. And the number of providers vary from operating system to operating system. But so these are the ones that I can see right now. There are other providers that can come into existence when you demand them. So I have these 27 providers. And we're going to look at the syscall provider and the FBT provider first. So every provider knows how to instrument a specific part of the system. So the syscall provider knows how to instrument the syscall table. That's not very surprising. So if you can look at the syscall provider, and here you can see essentially every system call entry and return that is that FreeBSD offers. So here you can see this four tuple, like the provider syscall FreeBSD is the module and so on. So these are all the system calls that I have in my system. And the other provider that I want to look at is the so-called FBT provider. And that is pretty astonishing. The FBT provider, FBT stands for function boundary tracer. And what it allows us to do, it allows us to trace every single function in the kernel. So I can look at the entire kernel at functions as they are being called. So to illustrate that, I wrote a little very simple detrace script. And this is probably, so you look at the upper half, please. So this is probably one of the first detrace scripts that you will come up with. It's a fairly simple example. So let's break it down. So I'm going to instrument the M-map system call. For those of you who do not know what the M-map system call is, what you can do with it is you can take a file and map that into the address space of your process. So very dumb-down version. So whenever we enter the M-map system call, we are going to set the variable follow to one. And what this self-ad means, this is essentially a threat local variable. And we're going to associate that variable with the threat that we're currently inspecting. Then I'm going to do something pretty, that sounds scary, but I'm going to instrument the entire kernel, every function entry and every function return. I'm going to instrument that and say, please emit the data when you do that. And this is what we call a predicate. So this is where the awkiness of the detrace programming comes in. So this is a predicate. And whenever that evaluates to true, then the probe is going to fire. So in this case, when we are in the threat that we're currently tracing, we're going to emit data. And this is just an empty clause. We just want to know, hey, we got here. So when we enter, exit the M-map system call and the predicate is set, we're going to set the variable follow to zero because every uninitialized variable in detrace is set to zero. So this pretty much amounts to deallocating that variable. And then we're going to exit cleanly. So let me run that. So it takes a couple of seconds and boom. So you saw a little pause here. That was when the detrace God reverses the driver, the kernel. So now you can see it every function call that happens inside the M-map system call. And this is a little bit hard on the eyes. So let me pass this flag here. And now you can have nice and nice to read indentation. So now you might say, I don't like that. You are injecting code into the kernel. That sounds dangerous. And yeah, but let me show you something that I find really interesting. So this is, I'm not going too much into depth here, but this is a byte code. So every detrace gets compiled to byte code. And this byte code gets sent to the kernel. And in the kernel, you have a virtual machine that interprets that byte code. So in case you write a script that for some reason might go rogue on your kernel, it allocates too much memory, takes too much time, this virtual machine can just say, OK, stop it. And just going to revert all the changes that happened to your kernel. And that's kind of handy. And it's not a new idea. So if you're using TCP dump, it's basically the same approach. They also have this kind of byte code. So that's just a little excursion here. This is called BPF, Berkeley Packet Filter. So it's not an entirely new idea. So everything I showed you until now was, hey, I can look when function calls are happening. That's not very much information. So we're going to increase the amount of information that we get out of the system with every example. So let me look at the actual kernel. So I had to restart my machine, so my setup is basically gone now. So let's look at this VMFault function. So this is the source code of the operating system that I'm running right now. This is FreeBSD Current 12. And the VMFault function, remember the VMMAP system call that I told you? So the VMMAP system call I told you can bring, like, map a file into your address space. And it doesn't necessarily have to load the entire file. So whenever we are touching a page in the system, like a memory page, and this machine is four kilobytes, it's no superpages here. So whenever it touches a piece of memory that you didn't bring into memory, we're generating something that's called a page fault. And then this function gets called. So here, let's look at the arguments. And I'm going to skip the 0th argument. I'm going to look at the first argument. So this is the address that provoked that page fault. This is the type. And these are the flags. And I'm going to show you something to make that a little bit more readable. So what about this one? So you can see it's a pointer. And this is a big structure. So we want to be able to look at that structure. And I just probably should do this here. So let's look at this VM fault script here. So this is don't pay too much attention to this code. This is basically just boilerplate to make stuff readable. And this is where the actual action is happening. So what I'm doing there is I'm instrumenting the VM fault function. And whenever we enter it, then we're going to use some information that Dtrace gives us for free. So this is exec name. This is the name of the currently running executable that provoked the page fault. This is the process ID. And here we have a bunch of argument variables. So these arc1, arc2, arc3, that are essentially just integers. So nothing too fancy there. But we want to be able to look at that struct. And here I'm going to use this arcs array. And this arcs array is kind of special because it has typing information about the arguments. So when you run that, so you're referencing that pointer there with the star. And let's just run that. And maybe not just start. So this is an internal data structure that we can now look at. So Dtrace enabled us to look at in-memory data structures as the system runs. And this is really, really, really powerful. In the Dtrace script, I could use all these fields. Like, I can manipulate this arcs array, this value in there, just like every other variable. I can pretty much work like I was in C. So I was doing that. There is something that's called CTF that's not captured the flag. This is the compact C tracing format. So you can see that. But there is a man page in FreeBSD. And there's a little segment in the kernel binary where all this typing information is stored. I don't know how that compares to modern Dwarf, but yeah, this is what Dtrace is working with. So now you might ask yourself, why on earth would I do that? Why on earth would I look at virtual memory? Because yeah, this stuff is safe, isn't it? There are no bugs in there, except when they are. Anyone remembers Dirty Cow? So this was a very nasty vulnerability in a Linux kernel. And that was a problem in the virtual memory management. So it allowed you to write to a file that you didn't own as a regular user. So you could essentially just write to a binary that had set UID set. Very unpleasant. But I'm not going to bash the Linux folks here. This is just, I just want to show you, these things are hard. And the first fix for this problem was in 2005. And then it came back in 2016. So now that's fixed. And then it came back with huge Dirty Cow in 2017. I mean, this was there for way over a decade. These things are hard to debug. And this is what I like about these systems. So not having tools like Dtrace to figure out what's going on inside of the system somehow, to me, amounts to security by obscurity. And I've heard that some people are developing exploits for systems that have Dtrace. They say, oh, I really like developing exploits on these systems, because the tooling is so great. Yeah, but to be honest, this is cool. Because an exploit is a proof of concept. And coming up with these exploits quickly is very usable. Because you know what's going on. You can show, hey, this is going wrong. I had situations where people were telling me, oh, this is not a problem with our program. This is this weird operating system that you're using, like Solaris, weird operating system. And yeah, and then I turned out some Dtrace scripts. And no, it's actually your problem. Oh, now I can see that on my Linux box. Magic. So everything I showed you until now was very much related to function calls. And we want to have a little bit more semantics here. Because you might want to write a script that inspects like protocols, stuff like TCP, UDP, stuff like that. So you don't want to know which function inside of the kernel is responsible for handling your TCP-IP stuff. So Dtrace comes with something that's called static providers. And I'm just going to show the upper pole here. So these are, so every static provider has a main page, which is kind of handy. Documentation, woo. And you can see there is an IO provider if you are interested in looking at disk IO, IP for looking at IPv4 and IPv6, TCP. This one is pretty cool. It's about scheduling behavior. So what does my schedule do? And if you look at that, you can see some interesting stuff like land priority, if you ever saw things like priority inversion, stuff like that. Now you can see that happen. I'm a nerd. I find this interesting for some reason, I don't know. And it's also pretty interesting to figure out what's going on, why is this getting scheduled all the time, so some interesting things going on there. So I'm running a little bit short on time here, but I just quickly want to show you something. This is all kernel stuff right now. Can we do that with user space? Of course. So there was one provider that didn't show up when I had my provider listing, but was in the Dtrace script where I did this timing attack stuff. And that's called the PID provider. And the PID provider generates probes on demand because a process might have a lot of probes, and you will shortly see why. And this is why I'm going to use a very small program, which is called true. And true just exits with exit code zero. So nothing too exciting going on here. And this dollar target gets substituted and we get the process ID there. And this is everything that happens when I'm executing this program. You can see this is a little bit more fine-grained than the FBT provider, because now we can trace every single instruction inside of that function, which is kind of handy. It's a scriptable debugger. So these numbers are the instructional offsets inside of that function. We can also look at, so this is everything in the true segment, can also look at libraries that got linked in. And there's a lot of stuff happening in libc force when you run true. So one last thing that I wanted to show you, because it consumes like a week of my life, I'm using a lot of Haskell. And the Mac OS people, they also have Dtrace. And they have GHC Dtrace support, so the Glasgow Haskell compiler. And Glorious, excuse me. And they have probes to analyze what's going on inside of the runtime system. So I thought, I want to have that IFD trace. Why doesn't it work on FreeBSD? So after a week of fighting with make files and linkers, that just works. If you check out the recent GHC repository and build it on FreeBSD, you get all the nice stuff I'm going to show you now. So this is a very boring program. It just starts 32 green threads and schedules them all over the place. And now I can do something like this. I can ring a telephone. Now that would be interesting. So you can also use wildcards and not S, the name of the probe. And this is what's going on inside of GCE, Glorious Collection, and all this stuff. Now I can look at this and write useful Dtrace scripts that also take my runtime system into account. So stuff like that exists for, I think, Python. I'm not entirely sure, because I don't use it. Node.js, same. Postgres, I use this, but not with Dtrace right now. And what I find interesting, Firefox. When you run JavaScript in your Firefox, it actually has a provider. So you can trace JavaScript running in your browser with Dtrace. So after everything I just showed you, there might be some stuff going on there. So yeah, this is basically everything I wanted to show you. And I think I'm going to wrap out, because otherwise we're not going to have a lot of time for questions. And maybe you have some. So yeah, thanks. Thank you very much, Michael. We are actually over time already, but we have two more minutes, because we started three minutes late. So if there are any really quick questions, possibly from the internet, there is one. The signal angel says, let's hear it. Yeah, hi. OK, so the question is, which changes are actually necessary to do in the kernel of an operating system to support Dtrace? That's a lot of work. So it's not something like you do in a weekend. This is so the person who started the work on FreeBSD has sadly passed away now. But I think it took a couple of years to have everything in place. So you have to have stuff like the CTF thing that I showed you, which is what OpenBSD is currently working on. And then you have all those magic gizmos, like kernel modules and stuff like that. So it takes a lot of time. But it's been ported to most operating systems that are available and in use right now. So yeah, hope this answers the question. Excellent. And there are no more questions here in the room. I will thank Raiko. And you can find him outside of the room. And also on Twitter on RAICHOO. If you have any more further questions.