 Bonjour, I hope everyone is awake. I've considered giving this talk in French just so that none of you would know what I was talking about, including the French people. But instead, I'm going to give it in English and people will still not know what I'm talking about. This is work done jointly with the other lovely people on the slide here, Jonathan Anderson, Brian Kidney, who gave a previous version of this talk at BSD Cannes, Aaron Thomas, and Robert Watson. And yes, today is not the 23rd click. So what am I gonna talk about over the next hour to try and keep you awake after having lunch? So I'd like to talk about a few things. Generally, when I give a detrace talk, I give a bit of history of detrace. Some, in this audience, more people might know this than others. I'd like to talk about the motivation for the work we're doing with detrace and how the group of us that you saw on the first slide are trying to use detrace, sort of in broad outline, because that really underpins some of the changes we're trying to make to detrace. I'll talk about some recent improvements that we've made to detrace both in our own private repos, which we will eventually upstream. Some changes we've gone and upstream, some changes have not. Some changes we think are more acceptable to the broader community and some of them may not be. I'd like to talk a little bit about how people actually use detrace, because I know this is not a room of only kernel developers, but the majority of kernel developers who use detrace sometimes don't understand how normal people, can you see the quotes from there? Normal people use detrace or less abnormal. And then I'll talk about, ooh, I said abnormal and the lights came on. Because I need to be better lit. And then I'll talk about some future improvements and the changes we're planning to make as we continue to grow and improve the detrace code. So this is my histoire de couture. This is my obligatory history. So detrace originally developed by a company called Sung. Some of you may have heard of it. Back in 2005 was sort of the first public release that you could get in Solaris, but it was developed over time before that. First ported by John Burrell to FreeBSD in 2008. It's also been ported to macOS, so you happen to be running a glowing apple. You also have some, you have detrace, although it may or may not be completely enabled in macOS. And since its initial ports, it has been maintained separately with some cross-patching between different versions. So I'll talk a bit about that towards the end of the discussion, but it's an important point to come back to because we have this open source code base that has been modified by different groups of people with some amount of coordination, but not with a ton of help. Okay, so what are the motivations we have, we being the research project I'm working on, also some of the people we've been working with in the community for using detrace. And why do we care about detrace as a technology? So, detrace is not the first tracing system ever built for an operating system. It is a quite comprehensive system, but I spend a lot of time working with people who are older even than me, and as far as I can tell, everything in computer science was done on the Atlas in 1953, so I'm sure someone will tell me that there was a tracing system on that. And one of the reasons that we're very interested in detrace is because it's an extant open source system for tracing that is relatively well written and relatively easy to understand, and we believe that using detrace, we can improve the state of tracing on large systems. So for those of you who have ever seen the title of my tutorial, the subtitle is No More Print Desks, right? So what I would like to do is live in a world past the 1970s, where I don't have to use print depth to debug everything, and I'd actually like tracing systems to be a pervasive, common feature of operating systems. Another motivation that we have in the research project is to expand tracing to distributed systems, and I'll talk about this a bit more in a couple of slides, but for those of you who've used the current detrace, detrace is usable on your single node. It's a very powerful system, but what if you could apply that to a distributed system? How powerful would that be? And there are, again, there are distributed tracing systems that exist, but we believe that detrace is a good base from which to build more distributed tracing systems. Now, some of us, Robert Watson and myself, have found that using tracing we're able to teach about complex systems. So for this audience, that means we teach about operating systems. So when I occasionally teach undergraduates, because people are silly enough to ask me to come teach undergraduates, it seems like a mistake on their part, or when Robert teaches graduate students, which seems like a good idea, instead of making them re-implement some ridiculously simple device driver for which they will learn nothing about the operating system, we take them on a tour de force of the entire operating system using the tracing system. So a good tracing system like detrace allows us to, in eight weeks for Robert, or in a week of full-time work for myself, take a class of 20 or 30 computer science students and show them how does TCP work? How does the file system work? How does memory allocation work? Let's show you what happens with blocking. Things that take much longer when you're making them write C code from scratch. Now, of course, we all know that forcing them to write C code from scratch builds character. That's what we're out for. But we believe that tracing is really an excellent way to teach people about large complex systems and to give them problem-solving techniques to understand those systems. That's another motivation. We want detrace to always be production-running. Sometimes it is, mostly it is. And we wanna do it in open source because, well, open source is good. Okay, so those are our motivations for continuing to push the detrace technology forward. This is one of my favorite films. This is a film called Reservoir Dogs, which I have no idea how to translate that into French, but you don't know, again, we just get to see a Reservoir Dogs with my terrible, terrible accident. So, another motivation. This is a crisis meeting. And I have been to these meetings happily without the guns. So, when people debug real-world problems in large data centers or large distributed applications, usually there are two or three interested parties. So, in this particular, this is called a Mexican standoff, by the way, in this particular Mexican standoff, it doesn't matter who is who, but you usually have someone from, let's say this guy's from the systems group, they're the systems administrator, they keep the systems up and running. This old guy here is a mafia boss in this, but actually is the network administration group, and they're like, well, they keep all the computers connected together, and then Harvey Kitell, who's my favorite character in the movie, he's the programming group. They're the people who actually take the programs and run them on the distributed cluster. And what happens is, usually Harvey over here sends in a bug report that says, your data center, your massive compute cluster of 1,000 nodes or 5,000 nodes is slow. And then these two guys here go, no, it's not. And then they start pointing guns at each other, and bad things happen. Now, you'll notice there's a character lying bleeding on the floor down here at the bottom right. This is the actual consumer of the data who just wants an answer, but he dies first. So, this kind of discussion, and this kind of problem solving, by the way, the problem does get solved at the end. It does not go well. This kind of problem solving is really amenable to something like a distributed tracing system. Well, not the gun part. But actually having currently, what we have in large distributed systems are uncorrelated data, right? So a switch generates some type of data which is uncorrelated to what you get out of a server which is uncorrelated to what you get out of a program. Wouldn't it be wonderful? Wouldn't it be a wonderful world? If you actually had a system where you could trace all of that and correlate all of that data together to try and form a more coherent picture of where the problem is, as opposed to standing around pointing guns at each other and accusing the other one of breaking the system. So this is actually, well, I won't say that this photo, but this idea, and having been through these kinds of discussions with people, very loud discussions, are one of the reasons that I got interested in this sort of tracing arena. Okay, so I keep saying we. I'll mention the project that we're working on at the moment. This is Cadets, which is the causal, adaptive, distributed, and efficient tracing system. The idea behind Cadets is that we can look at all of the events. We can have transparency into a distributed system to answer an important question. Now Cadets is focused, Cadets is a DARPA program, so it's focused on security. But the problem, the question we're trying to answer when something goes wrong in any system, and certainly something goes wrong in a distributed system, is also the answer to the question my mother used to yell at me when I was fighting my brother, which is who did what to whom and when? And that is the question that you wish to answer at any sort of large distributed system. Who was the actor that caused the thing to happen and when did it happen? Show me the ordering. So Cadets, because it's a distributed efficient tracing system, attempts to answer that and we're using de-trace, and that technology is one of the components here, we have a component called Loom, to try and give more transparency into what's going on on a single node and eventually what's going on in all of the nodes in a distributed system. So there are two components to this, I will talk a lot more about de-trace, I will not talk about so much, I will talk about how we apply it. Mostly I'll talk about de-trace, we'll talk a little bit about Loom, which is work that's being done within LLVM to generate more in-programming instrumentation. So let's get all of that on the screen. So Loom is specification driven program instrumentation, this will result in things like that binaries. Of course, everything's built on free DSD because it's just me. And then we're using de-trace as our tracing framework because in order to have this transparency we're gonna use tracing. So how do we use de-trace, how does that suit it? So we use de-trace on a single node, we also leverage it for distributed instrumentation, which means we can have a single client that goes out to a large set of nodes and tells them to all perform certain types of tracing and feed those trace records back to a central point. So this has some interesting side effects. So if you want total transparency into the system, you're not gonna use de-trace as it was perhaps originally envisioned. De-trace was envisioned as a way of debugging a problem at the time. I have a problem, I'm gonna go look into it. But if you're building a system that's trying to build up a set of records that gives you total transparency into everything going on in a system or a distributed system, then it's not turn on, turn off, it's always on. So in our project we call this always on tracing. There are some side effects to that that relate to doing security work, which I won't go into in depth, but it does mean that there are parts of de-trace that can be exploited by an attacker against us, we're worried about that. And it also means that this is, to some extent, a novel way of using de-trace. People have certainly built full telemetry systems on de-trace, the official textbooks did that. I know that people at Isilon have done things like that. The Frinais folks do that. But they're not looking at everything in the system all the time, they're looking at subsets and that is different than trying to trace all system calls or all of the interactions between programs in a distributed system. So that requires some equipment. Okay, so safety first. So one of the things that's important to understand about how we use de-trace and how we would like to improve de-trace and move de-trace forward, you have to go back to the de-trace design principles. So when de-trace is originally built, it was built so that it would have no overhead when not in use. And people seem to miss the last part. They're like, oh, it's supposed to be free. At some point you have to execute an instruction no longer free. But the idea is that when you ship a binary, you ship a single binary, you don't have a debug binary and a regular binary, you just have the binary. And then if you don't turn on tracing, then the overhead should be zero or very, very, very, very small. That's, by the way, a scientific measure of very, very, very, very small. Really important, don't panic the kernel. Because it turns out if you turn on your tracing in the kernel panics, we'll get really, really annoyed, especially if you enter them in production. Unless they're Twitter, then it's okay. So, and you have to protect the kernel at all costs. Well, this has some side effects on things you can and cannot do in de-trace, which we then have to think about as we apply de-trace to our workload or add new features to de-trace. So, in particular, for those of you who've ever programmed in D, D has no loops. Because if you could shove a loop in the kernel, then it might loop forever, because it turns out, Turing, period. So, no loops. There are no basic blocks. There are a bunch of things that, as programmers, we kind of expect that of a language that don't currently exist in de-trace. And for the most part, they don't exist because the designers at de-trace were trying to make sure that the tracing system they added to their kernel, to Solaris, couldn't either accidentally or maliciously be used to cause the kernel to crash. It turns out when the kernel crashes, so did everybody's programs and people get cranky. Safety first. So, D, the scripting language is like C, but safe. The problem is that we need some of that safety to possibly be off. Another bit of background about de-trace, and I figured I'd put this in here for those who actually work with it. So, de-trace was built on or about 2005. It was a simpler time with smaller memories and slower CPUs. This is your grandparents' computer. Not really, my grandparents were not born in 2005. But thank you to the Wayback Machine for keeping all of the marketing material of every company for the last 20 years because when I wanna talk about what a computer was like in 2005, I just go to the Wayback Machine. So this was, if you bought a Sunfire V890 server in 2005, you could have up to 64 gigs of memory. I think, I probably don't have that much in my phone, but I could, so not a lot of memory. And this was, Sun Hardware was nice, top of the line, high quality stuff. If you went and bought a Super Micro at the time, you could have up to 16 gigs of 400 SD RAM. I don't know why it's SD RAM. And not a lot of memory, slow cards, et cetera. 3.6 gigahertz, 800 megahertz front side bus. Now, you wouldn't even bother putting this in a data center. So that means, what are we doing then? So sometimes when we apply de-trace on a running system, using the original defaults, we run out of steam. Yes, this is Montparnasse. Yes, that train really did go at that station. So for those of you who use de-trace in anger, you've seen things like this. And please don't tell me de-trace is broken when that happens, as some of you have. De-trace is not broken. It's actually working as it's designed to work. Remember, going back a couple of slides, safety first. All right, if we're gonna tell the operating system kernel to start recording events about everything that's going on in the system, we are not going to block on all of those events. We're gonna have a ring buffer, it's a thing. We're gonna have a ring buffer or we're gonna have some sort of buffering system whereby when we get behind, we can be like, okay, I can't keep up. So throw away some records. De-trace is not broken. But it is a bit mis-tuned at the moment. So there are three tuning parameters, buff size, which currently defaults to four megs and was for reasons that I have never understood, severely limited on previous need, but we'll change that by a tunable. So if you're having a lot of drops, you can try increasing that. The switch rate, which says how often we're pulling buffers out of the system, which defaults to one hertz, which you can increase again if you have a bunch of those drops. And lastly, so again, De-trace built for safety, built in 2005, smaller memory time. There are, you know, there's two main memory sizing things. One is buff size. The other is the dynamic variable. So you might see this, 103 dynamic variable drops, which is different than the buffer drops. So you might wanna increase this as well. Now, okay, so now you might say, well, you know, it is 2017, why don't we just make all of those bigger? Well, it turns out that De-trace, especially De-trace on a free VSD, runs on a wide variety of platforms. In fact, it runs on some embedded platforms as well as servers. So we can't have a one size fits all. We actually do need to need to have one. Let's talk about some improvements that we've made to De-trace, we being the cadets project. So the first one we did was machine readable output. So De-trace was built by people who loved AUK. Now I know we all love AUK. I love AUK, it's okay. But the output that comes out of De-trace is not easily parsable by program, and generally it's not meant to be. It's meant to be parsed by a human who's looking at the console or maybe they've built a flame graph, which we will hear about later. But it's not really meant to be ingested into another program. It's meant to be sort of read over by. So one of the first things we added was live XO support. So now you can have your output in XML. You're welcome. Also, Jason. Because we wanna be able to build tools that consume the data from De-trace. We've added some new providers. There have been providers added by other people in the community. So one in particular is Audit, Mac and Mac Framework. OpenCrypto now has a provider. So does SCTP, it's a transport protocol, the XPG stuff. So as the code has been adopted and adopted by the community, we've added more providers in the system. And providers are just a way of collecting trace points together and expressing some form of functionality within De-trace. The other thing we started doing is performance analysis. Now, going back to a much earlier slide, De-trace is nearly free when it's turned off. And people will then immediately ask, well, how much does it cost when I turn on a single probe? And the answer is, we're not really sure yet, but we have someone doing a PhD about that. So we started to do quite a bit of performance analysis to see where we can do improvements so that when we're doing always-on tracing, and there's a ton of trace points enabled at all times, we actually don't have a ton of buffer drops or other performance overhead problems. So we're doing that analysis now. We've also started documenting quite a bit of the internal. So the Solaris folks wrote great code. You're gonna look at the include files. They have really excellent comments, which I really appreciate, but they are not written in a way that you could then take a specification and, for instance, do a clean room implementation or explain to someone simply how they would do extensions. So we're doing that. So the comments explain the what. They explain a little bit of the how, but none of the why. So machine readable output. This is a simple retrace one-liner and you say something like, oh, hey, tell me when is this called, when write fired and bang bang bang, you get a bunch of text output. If we tell it we want JSON output, we get this nice little object. That can actually be parsed by something that we've JSON, right? So you're not, well, people aren't normal, but normal people should not read this. It should go into a tool. So the JSON output gives us the ability to do things like give a timestamp to every event and separate the things that happen. Like this is a function and this is its name into their separate elements. And this is done automatically. So instead of having to write a script that converts something from text into JSON, we just spit it out and then we can feed this into tools that can do analysis for us. Remember part of our overall goal in the cadets project is to deliver a system that can be used by security analysts to look at what's going on in a distributed system. Security analysts are not going to read this. They're not going to read this. They're going to build tools out of it. So machine readable output. We've made a few improvements to the D language. So have the, I have other people, actually some of them have been imported by people in this room. So what is D? D is a C-like language that supports all the C operators. It was fairly structured like AUK. Has threatened clause local variables, subroutines to handle common tasks, but you can't write your own. So we started doing a series of improvements to make D do more of what we want. One of the things we had to add was a way to get an M-buffs. So if you're working in a distributed system, it's fine if you know about socket connections and you can see TCP and all that stuff. But sometimes you want to do packet infected on packets that are coming in or out of the system. And in 3BSD we have M-buffs which is a very complicated but very useful method of encapsulating network data. So this allows us to read chain M-buffs in D. It's really important for all of the BSD derived systems. So for those of you who are attempting to port D-trace to the other BSDs, not BSD, not BSD, you probably want something like this because it does not exist in the original Rulimos slash Solaris code. There we go. And that allows us to do this cute little trick. So on TCP input, I've got an M-buff. I happen to know that it's in this zeroth argument and it doesn't matter if that M-buff is locally stored or in a cluster. The script writer, which is the person we want to enable with this, just calls this one routine and gets a pointer back and then they get this, I call this nice output but then I'm a kernel programmer, so I'm a terrible person. This is the IP packet. You can tell by the magic four or five byte at the beginning. And so this gives people using free BSD and if the other BSDs adopt this, easy access to network buffers. We needed this. So what are some other things we'd like to see? Turns out programmers really like if statements and D has this ternary operator. I usually defy people to tell me what this is and someone can email me if they want to. I'm not gonna leave it at that one either, so it wouldn't make it difficult for you. Ternary operators are fine, but they're not great. So if statements improve reliability, remember we want analysts to use tools but we also have to have people to write analysis scripts in D because those G scripts are gonna be distributed to the entire cluster or whatever it is and we'd like people who write in modern languages to be able to write something. So the Solaris, actually I guess this is the Alumos group, did a syntactic sugar version of this. So if a want of an F, I'll make a dispel in. So this is an F block in a D program. This is the source code and the way this works, one more brace, one more brace. This is from Matt Arendt, who did the work of doing the syntactic sugar version. So this is what happens when you compile this. Yep, it goes up the bottom of the slide on purpose. So what happens is it actually just blows out the script to still have a much larger D script that sort of encapsulates your F. It's not a real oscillator. It is sufficient for simple scripting but it is not what we want long-term. So I mentioned other things we added. So in the cadets project we are using audit, the audit subsystem in 3DSD to audit the system as it runs and we've added an audit provider. So this is a subsystem for login security related events. It was originally done for the US government's common criteria security standards. It's been optional in 3DSD since 2004. This is work done by Robert Watson and others. So we know what a provider is, Dtrace Code that collects together a set of trace points. What does this one get us? This gives us access to the whole audit framework. So that means we can turn on audit and then we can use D to filter the data and get statistics, which we would have to write special programs to get from the audit records. Dtrace performance. So I talked about tuning earlier, but tuning only goes so far. There are components of Dtrace that are not as efficient as they could be. So for instance, the decompiler that takes the source code and turns it into, you know, D bytecode that gets executed in the kernel is not optimized. I wouldn't call it a pessimizing compiler because that would be unfair, but it is definitely not an optimizing compiler. So the strips themselves generate bytecode that is not as efficient as it possibly could be. So the idea is that, the idea behind Dtrace, of course, is that it shouldn't degrade performance, you'd like it to not degrade performance too badly while in use. Now, one of the nice things about working on a DARPA project, when we first started working on it, I asked, as one of the engineering types, as opposed to a research type, like I just write code. I don't know how the research thing works. It was like, well, how much penalty are you willing to pay to get transparency into your shrewd system? And they said 100% overhead would be all right. I'm like, wow, okay, give me your money. But it's still our goal that when you turn on Dtrace, it doesn't make the system very slow. So right now, when Dtrace gets behind it drops records. The kernel can kill the tracing under high loads. One of the other protections that Dtrace has against someone asking for too much data is that if Dtrace sees that it's getting behind, it'll just kill, the kernel side will just kill the user side and be like, no, no, no. Too slow, go away, and then the kernel will recover. So there are some, you know, what are the possible solutions to dealing with this if we want to try and get every record? So if you think about a security application of a logging system in a security application, you never want to drop a record, right? But if you never drop a record, then you will block all forward progress on waiting for something to happen, like writing the record to a test. So we can look at things like changing our monitoring cycle, we can figure the buffer sizes. We can improve the decompiler, which is one of the things we're doing now. And then we can look at the efficiency of the underlying decode, both in the kernel and in the user space. We can jit the byte codes, and we can leverage LLVM. This is one of the areas in which we're doing a lot more research. Jonathan Anderson is doing this with the loom work, which is related, but not completely the same as with Dtrace work. But we believe that we can use LLVM and apply some of the, as compilers will get, to apply it to some of the problems in Dtrace. So I keep mentioning loom, here's a quick aside. So loom is an instrumentation framework based on the LLVM tool chain. What it does is it puts instrumentation directly into the intermediate representation generated by LLVM. The instrumentation is defined in their own policy files. The instrumentation can be done at any time. The one with the IR is still available. So I mean we ship a binary and then add instrumentation later, as opposed to currently in a Dtrace context, we might have to add a statically defined trace point to get it something, or if you're in user space, you're simply gonna be tracing a particular instruction if you guess the right instruction. The idea with one of the things we wanna do with loom is actually make it talk to Dtrace in user space. So this required another change. So USDTs, we've got to work kind of like this. We've got a provider file, we get it included, we compile that, we compile a couple other bits and we get this binary that comes up. With loom, unlike the USDT, the fat binary that comes out of LLVM can be instrumented later in the field. So, it's important. USDT performance is similar to the kernel performance, so, you know, we do this in opera placement, we do our overhead. So there are some problems with this in terms of shipping the code, which is why we're doing the LLVM trace. That's that the Dtrace tool is modifying the final binary. It doesn't leave it aside to modify it in the field. It doesn't play well with me, I'm told. It relocates quite a bit of things. So, we're adding this loom-based user land tracing where we just take a simple C file, we use the compiler to generate the fat binary and then we ship the app and then we can actually have a provider.dm file that can be moved later in the field and loom will actually move them together to do tracing. So, we're currently working on this. We have a prototype system called DT-Probe, you can want to be able to call probes from a user space program. So we added an in-system call, that is not in three days to see that. But this is used by the loom system to do instrumentation. No change in the binary when there's no instrumentation and it could be completed. So, some testing, provider generation. Ah, damn, I got to that slide too soon. Okay, so that's kind of what we're doing with loom. If you want to know more about loom, I'll point you at the work that Brian can be and Jonathan Anderson will be. Now, okay, I've given away this slide. There can be only one. So, I pointed out the D-Trace was not the first and it wasn't even the only dynamic tracing system that had ever been built. It turns out some people on another operating system have built a tracing system based around BPF called EBPF. Using EBPF at the lowest level is like programming and assembly language, which for this crowd is like, yeah, that's fine. But for everybody else is like, why would I do that? But, you know, things improve over time. EBPF now has DCC, a C-like front end, and more interesting to me, actually, is the PLI work, the PLI work of the Python something. So one of the real downsides of D is that you still had to be relatively conversant with both the kernel and user space to write effective, big effective D-space. Write D1-liners is not so bad. What you would like in a trace language is something more like the languages people use for scripting, like Python, God, they didn't use Perl. Yeah, I'm gonna get that one in every time I do the talk. So you want something like Python or maybe you'd like Lula, but, you know, you want a higher level language that gives you something that's readable after 15 minutes. So, Dtrace now has competition from EBPF and all of these other things, BCC and PLY, which are built on top of, you know, using EBPF. According to someone who actually is in the audience, and we'll be talking later, EBPF now has feature priority with Dtrace. So we've got serious competition. There cannot be only one. So, I mentioned the history of Dtrace. How does Dtrace get developed? So this is the current source flow of how code moves between various systems. You'll notice there's a Windows port. I haven't run it, terrifying idea. But the main current users, and I guess I'll have to add OpenBSD to the slide eventually. The main current users of Dtrace, Mac OS, Wyrmus and 3DSD sort of share code amongst each other. The original import was here. We continue to import some stuff. Mac OS pulls things from us, and I think they also get stuff from Alumos. We send patches back. There is a Linux port, which, you know, whatever, don't ever accept it. But I'd like to make them excited just to do it. So the source sort of flows between various projects, but there's no one project that can really be the ultimate upstream. Most people are still looking to Alumos to be the ultimate upstream, but we'll see how long that lasts. So what are we gonna do next? I don't know, this is an animation. How did that work? So we're proposing to do OpenDtrace. I will not put episode four on the slide. A new hope. So the idea is to have a cross platform. It's already relatively portable. We're using the same RFD process that Joint and Sun used to do requests for changes. It's already in an organization, and we've been, this is where we've started to put all the stuff that relates only to Dtrace. So changes that still need to go into FreeBSD, still go into FreeBSD head. Over, hopefully, a short period of months, there will then be a source base in OpenDtrace, which will be feature parity, a lot of feature parity across all of the operating systems and will be importable by any operating system that wishes, which is gonna be a big and interesting job, and if people are interested in hacking on that, she can talk. So I mentioned earlier the specification. So, you know, I know room pillar programmers don't want to write documentation, but it turns out to be really useful. So what are we specifying? Diff, Doth, and CTF. These are sort of the three major components. The data structures, the things that make Dtrace go as a system. We're improving the testing framework. We're supporting an execution of traits like the chip. That's why we would write the specs, so you can actually figure out how to write the chance. Makes it easier to do future extensions. Right at the moment, someone who wants to make an extension go and looks in the source code, they think they know what's going on, and then they add something and maybe it works and maybe it has a terrible knock on effect. Also it allows for a clean room re-implementation. There's at least one attempt to re-implement the CTF tools, which was discussed yesterday, where due to, let's call it a license allergy, people wish to not use the CDDL code, they wanna have a clean room re-implementation or something like that. So if we write the specification correctly by reading the code and testing a bunch of stuff, not only do we have a way of communicating to people who wish to extend or build on Dtrace, how to do it, but if people wanna re-implement a new version from a clean room, they can do it. So what are some of the things we wanna do as we move into the open Dtrace feature? You know, I like to have basic blocks in the view language if we're gonna keep D at all. Founded loops, turns out programmers like loops, but it turns out they're also super dangerous, so, founded loops. It'd be nice to be able to include one D script in another and have them refer to each other's modules. No programming like it. Improved performance, not just because the project I'm working on needs it for always on tracing, but because it's good overall. If you wanna build a really impressive telemetry system for some sort of storage device or some sort of network device, you're gonna need higher performance out of Dtrace so that the tracing system itself isn't stealing all of the cycles from the actual application. Improved test suite, we always want more tests. More OS ports, so one of the interesting things about working on the Dtrace source code, it's incredibly, it's well-written, it's portable. If you mean portable to two architectures or other solaris. Now, it was ported to FreeBSD, very clever, ported back to us, very clever, but we'd like to see more OS ports, in particular a port to RTEMs, which is an embedded system that's being planned. And I think doing a port to Linux will be fun because it'll annoy the Linux people. More architecture support, so again, it was quite portable to the two types of systems that Sun sold at the time, which was Intel and Spark. Someday I'll pull Spark out of the FreeBSD tree and then I'll be happy. But we would like to, on the FreeBSD side, we've already ported it to PowerPC, ARM 32, ARM 64, and MIPS 64, and we wanna end risk five, work done by Rustlin. So we wanna make sure that that architecture support continues and that the system can support more architectures in the day up here. Fine-grained libraries. So it's quite possible that we will tear up libDtrace into components that make it easier to put it into programming languages. That way I can have my PLY-like Python that talks to Dtrace instead of having to write my scripts and do it. And we want it to be, the goal of fine-grained libraries is usable from other languages. So it'd be great if I could write Rust with, have Rust talk to Dtrace or Python or Lua. I mean, I think, or I hope it's a thing called Go. Python, Rust, go. That's weird, I'm missing a picture. Oh, there it is. Animation's in the wrong direction. So when we talk about multiple architecture and multiple systems and the kinds of things we're gonna have to do to keep Dtrace working, so in the sun world, I mean, if you go back and look at that, there's quite a trip down memory lane to look at the way back machines, what could I buy from the sun in 2005? Actually, I couldn't buy anything from the sun in 2005. Everything on my page was super expensive. But OpenDtrace, because it's gonna run on multiple architectures and multiple systems, is gonna run across the spectrum of machines and it currently runs on all three of these as far as we know. This is the Beaglebone back, Black, this is what Robert Watson teaches on, this is a PS4, this is the machine that you have on your desk in front of you. So, there you go. So OpenDtrace needs to be able to run on spectrum machines from small memory to large, running from low power processors to high power processors too, that, right? So our goal with distributed Dtrace is to be able to run the same D script that we run on all the small machines across however many nodes that are ready to count and I'll start with the computer. How do we see Dtrace, OpenDtrace being used? We expect there to be more kernel trace points because a lot of kernel programmers work on it. So things like WebSack, network home players should probably have some more Gion and Cam trace points. More drivers. There are 75 different sets of debug macros in the previous kernel and 10,000 calls to device print depth in our device drivers. I would like all the device print depth to die in a fire. So my goal is to get things like Dtrace to request that. More user space tools. So we talked about machine usable output. I'd like to see tools like Jeff Robertson's SkedGraph be modernized to manage Dtrace to pull locking information and scheduler information into the flame graphs or something that's more like a software oscilloscope. SkedGraph, lockgraphing, looking at performance of subsystems. Flame graphs are all the rage so we're gonna flame graph all the things. And then our goal is to have a development model where we're gonna have a single upstream similar to OpenJetAppS and OpenDSM where there's an OpenDtrace and then we can share everything amongst all of the operating systems that we're just consuming. So macros, almost all of them are used to do it. The next window is, et cetera. I'm not doing it in this way. All right, and questions. No questions? No. Okay, so thank you. Thank you.