 So, my name's Kamil Tarowski, I'm an NBSD developer. I started to use NBSD since version 6.1. It's quite recent to other people, although other developers. And I'm quite fresh developer in the project. I'm mostly focused on the kernel part, user land and package source. And my main point, my main focus is to use NBSD as a desktop. Recent times on laptop and as a workstation. At the current time I'm an LL4M committer. This involves also committing to other LL4M projects like Clank, LDB, Compiler Earth, Poly and others. I'm also a GNU, GDB and GNU be new to use committer. Also a focus for NBSD part. And I became recently NBSD maintainer in the Tuemu project. At the moment I focus on compatibility with NBSD as host. I have streamed all the local patches. We are passing all the tests in the test suite in Cuemu. And I'm helping other project that are using package source to update patches. So I've moved to LDB. So there's some link. I will focus on LDB part in my presentation. I will start with some elementary terminology to make clear. So we start with elementary terminology to make clear what am I talking about. And what's my work about. And I will talk about LDB porting details. And at the end I will note some progress about sanitizers. They are also part of the LL4M toolchain. And I will end my talk with the list of current tasks on roadmap. That is evolution. So I wanted to tell clearly that debugger is not just some kind of third party optional software in that case on a Unix operating system. However, it's integral part and traditional integral part of Unix toolchain. It's as important as a compiler linker, libraries like Clip-C, IP threat for threats. And more advanced toolchains can contain also additional software like 3D graphics modeling, audio software and similar. Here's a picture of an example debugger. It's DDD affronted to GDB. And I'm defining here what's the debugger in my mini, in my use case. It's a computer program that can trace other process. And it approaches about stopping the program on certain kinds of traps and investigate the tracet program. And there are also other kinds of debuggers like now that don't involve stopping the process like detrace or cutrace for kernel Cisco tracing and similar events from the kernel. There are also kinds of debuggers. However, I focus on the debugger that it's using pitrace API, that it's using the approach of stopping the process and giving ability to a programmer to investigate the current state. So yes, I'm describing a usual workflow of using a debugger to find and fix a bug. So we try to reproduce an issue, isolate the issue to one library or one module, one function. We are trying to detect the source of the issue. And we are attempting to remove the reason of the bug. And at the moment, at the end, we find that the problem is gone. So it looks very simple from a big picture. However, in case of large software, it's more complicated, but because we have context of functionality that may be slightly altered by our bug fix. I also note that I recall a research from Bell Labs about telecom software bugs. There was a research that scientists were looking for, trying to understand what's the most common source of introduction of new bugs in software. And the results said that fixing old bugs is the most common source of new bugs. So it's less trivial in the context of software. So it's not about fixing one bug and we are done. We need to verify that everything is still working, there are no regressions, nobody is complaining. And the problem is gone. Here is an example session of GDB debugger that we set a breakpoint on a function named FunFree. We start the debugger, we see that the program has stopped. Here we can investigate trace of function calls. And this is the view of a traditional debugger like GDB or LLDB from a programmer or end user point of view. And to get such picture we need to investigate registers of functions of threads, stopped threads to retrieve at least stack pointer, code pointer. I when we retrieve code pointer we can check what are the functions next or before the interruption. With stack pointer register we can investigate park trace. This is a simplified point of view, but it is more or less about this. To improve the usability of a debugger we need debug symbols and for programs it means additional information that can help to navigate the code line by line or variable by variable. And also this mandates support for such language, original language of the tracet program in the debugger like C, C++ or Fortran. At the moment in LLDB there is support for clang languages like C, C++, Objective C, I think Objective C++, Swift, .NET, but there are also plugins in external sources. So we have an example program Hello World. And to create a debug information we need to use G option in GCC compatible compiler driver. It means that we want to add up and debug information and we need to lower optimization options like OG means that we want to get some optimization that is still able to trace a program. So here are examples of two computer program, one computer program and one core file. And I'm using a file utility to describe the compiled program for LBSD. There is a note about debug info that it's attached, it means that we are using G option to GCC and it's not stripped. It means that we can also, after the compilation, strip symbols and strip some elf tables to make the fine executable very small. And there are also core files, core files are in fact images that they are images of processes. In usual case, when a program is crashing, we can order the operating system, instruct operating system to stop the process and save its memory, save its state, information about the process and create a core file. And later we can open the core file and investigate the current state of the process. We can change it in very details. And yes, LDB is quite recent debugger, recent software. GDB is a very old project. It has support for many operating systems that are indexing. I think little people use develop today for maybe HP, UX, unique systems. And they need to support such operating system as a debugger. And old Unix systems had very primitive debugging API for the kernel. And it was not evolving to add more precise, more recent features to use better hardware, better performance, better correctness. And the development was stagnant for such platforms. And LDB is pushing for as modern capabilities of operating systems as possible. In fact, it's at the moment pushing for Linux level or Darwin, Windows, FreeBSD level of API features. And yes, this is what I told that we need very modern API, modern compiler with C++11 version and beyond that. And kernel introspection, modern functions, yes. And this enforced me to work on catching up after the competitors with feature parity in the Ptrace system call. And we are already there with machine dependent parts. And in the number of system calls or in the set of functionality. We are still not there with machine dependent parts. For example, we don't have accessors for XSafe, XSafe opt registers on modern Intel CPUs. To trace, for example, AlphaWix version 2 software that is using such instruction sets. At the moment I don't find the machine dependent parts as important to work on them now. It's scheduled for later. I'm focused on machine dependent code. And later we can add, I hope, quite simply all the machine dependent parts when needed. Yes, so I got some list of goals. So yes, as I told that this was the motivation to improve revamp the Ptrace system call. And we have also many problems with GDB. I think this common in other BSDs and other systems. And that, for example, threads are problematic in processes to trace them correctly. And in our electric working on Ptrace for LDB, I'm improving also GDB correctness. We are also heading for, there are also efforts to head for LL4M clean base system to replace the GNU toolchain parts as an option. So when possible, people can take GNU toolchain base system or LL4M base system toolchain. And we are still looking for LLD, so the LL4M linker porter. It quite works for basic things, however it needs more time of dedicated developer to get some hour specific or generic specific needs to get them right. And usable to at least build the base system and package source software. And also there is indirect positive effect on support for Dtrace and our HGDB feature. For example, Dtrace has option to trace kernel code or user space code. And for user space code, they are using Ptrace for some operations. For example, they put a trap on the entry of a Cisco. They check whether this Cisco is fork. If so, they remove all breakpoints, after breakpoints. They execute the fork and they put trap on the Cisco exit and re-enable, re-againset breakpoint traps. And this is a use case for Dtrace. And indirectly we improved the support for LibProc. LibProc has origins in Illumos Solaris operating system, because Illumos has approach to use, to rather use proc file system, virtual file system. Trace through the system, however, programmers are more familiar with CAPI. So they have wrapper for the proc file system. And the wrapper is called LibProc. And for better compatibility with Dtrace. And compatibility with Dtrace, the Dtrace version on Solaris and BSDs. We maintain LibProc. That is wrapping Ptrace calls in the BSD case. So, as I thought, there's no fight between two trains. At least this is not my goal, not my motivations. We need both. Some software requires this, the other this. Yes. Trace system LLDB, by direct LLDB benefits, like visiting the LFM project, that we are alive and they are concerned about NBSD support. I maintained, for example, NBSD build bot in the LFM clunk and LLDB project. I am obligated to not break it. Yes. Sometimes they need some support. Support, like looking for the correct solution for BSD and NBSD. Yes. In the beginning LLDB was developed for Linux. Linux i MacOS. Te systemy są bardzo inne, w związku z kernelu internalsą. To było dużo kodduplikacji. Jesteśmy pracowani razem z upstream. To jest też pewna próba, żeby poszukać komodne części w komodne library. Na momentu możemy znać dużo kodów z Linux i 3BSD. Darwin jest bardzo różny. Windows jest bardzo różny. Nie chcieliśmy za wiele linii z NBSD, tylko kodów, bo można oddać. Mówiłem o tym. Trzymałeś rozkaz ulicy LFM 5.0. Ulicy 4.0 do 5.0. Ulicy 4.0 do 5.0. Ulicy 4.0 do 5.0. Wydaje mi się, że mogłem znać rozkaz ulicy 1.0. Ulicy 4.0 do 5.0. Ulicy 4.0 do 5.0. Ulicy 4.0 do 5.0. Ulicy 4.0 do 5.0. Ulicy 4.0 do 5.0. Ulicy 4.0 do 15.0. Ulicy 4.0 do 0.5. Drosty tamto z lithu 6.0. Ulicy 4.0 dla 5.0. Ulicy 4.0 do 5.0. Podobnić- Attend웃음u inspection. To nie było zapewne, ale byli zapewne przez Extena Tool. Mamy tooly in-base, który nazywa G-Core. Można zrobić filmy o procesie. Możemy być inwestykowani przez inne propozycje. LEDB też opuszczy dla modelu klienta serwera. To znaczy, że mamy serwera, który prawie trafi proces. Klienta to komunikacja z serwerami. I używamy GDB Remote Serial Protocol Interface. Jest standardem GDB. Możemy też używać GDB Front End i LEDB Back End. Ale nie próbuję tego. Ale to nie powinno być trudne. I ten model jest też przyzwyczajny, ponieważ możemy złożyć serwery na pewnego maszyny, gdzie nie chcemy mieć source code, obiekt, debug code. Chcemy odnaleźć proces i komunikować wszystko, przez eg. serial line lub TCP IP stack, do innych komputerów. Możemy używać więcej silnych komputerów Front End, GDB, na przykład i trafić proces. Mamy też potrzebny rozwiązywania unika z serwerami, ponieważ system dupitry na unika używają sygnały, aby opłacić eventy o procesach. i używamy tego, że jest jedna w modelach klasycznych, że jest jedna w porządku sygnału. Tak, przez proces, więc, na przykład, jeśli chcesz integrować GUI klienta, to jest to prawie tracenie procesów. My mieliśmy problemy, że Gui-tulki, jak QT, na przykład, używamy też akurat w wtórzy, w lepszymi laboriach, i spadają się z tego modelu. Tak, i staramy', do fortuny nad PSD. to jest najważniejsze dla Linux z Android. Android jest większy z kompatibliwą Linux, natomiast są poddawione z niektórych legejskich interpezji, aby być kompatibliwym z old BSD wersji 4.3 API, ponieważ Linux jest przyzwyczajeni przez ten API. i to jest porozumienie, dlaczego my jesteśmy dość podobni między Linux i PSD. I możemy też używać debuger na bazie systemu, aplikacji. Mamy nasze specyfiki, jak USR, libdata, debug. To jest inox, freeBSD i może inne systemy. Mamy linux, USR, libdabug, bez data. A debuger może odnaleźć pasy, odnaleźć informacje, odnaleźć fily i pokazać potrzebne data o procesie. Natomiast w latach, w latach, w latach procesów przygotowania 5.0 wersji LFM i LDB, to było jedno mało miesiąca, kiedy debuger zniszczył NBSD, dlatego nie ma problemu z uspieszeniem. To jeszcze jest inwestykacja, o której będziemy rozmawiać w latach. I też mamy w lokalnej patrze support na core files z multiple threads. I mogę pokazać prezentację z aprilu, jak to działać. To jest skrypt, który wypracuje filmu o FDB na NetBSD. To jest ten przykład, który wypracuje kartkę z mnóstwą frecją na NetBSD. To coś zbrojone. To było w okazji w Aprilie, w ostatnim statofie na wersji LL5. I mogłem pracować na mnóstwo fredzy na WSD. Więc przedstawię wersję 5. Jest to skorzywanie w FF. Muszę również przejść do skorzywania w FF na nami. To zaczęło. Są 16 freds. W większości z nich czekają na coś, a fred 1 zniszczył ten signal segmentation fault. Jest też suport na interna GUI, coś podobnego do GDB2E. i jest w porządku, więc możemy przegrać przez kod i zobaczyć variables, spreads i zobaczyć, gdzie kraszczyło się. To było trochę innego, że jesteśmy zareferencjącymi nullpointer. Najlepszy kraszczyzny. To innego. Możemy sprawdzić lokalne variables, te values. Tyskota nie została obstrzymała, ponieważ miałam projekt na terenie regularnych procesów, i to wytrzymałem później. Podróż jest w porządku, w porządku, w porządku. Powinniśmy, że to powinno być rejestrowane do nowego wersji. Możemy też sprawdzić rejestrów. Każdy rejestr zamiast rejestrów. Możemy też wytrzymać się na porządku, żeby rejestrować rejestrów. Zmierzam to i wracamy do... ...przestań, więc zauważyłem, co jest wypowiedzi na systemie LDB. Jest to stak z LDB, stak z CLANK, LFOM, Librarium, Kernel i Hardware. Możemy zrobić coś z Hardwarem, możemy użyć z tym, jak jest. A to weekend alter kernel, when you did improve it, the same with user-run libraries, like libc, libfread or c++ runtime library, the same with LL5 and clack. I here's the impact on the kernel with the pitrace, debugging interfaces. We need to deal with forks, we forks. We had a pitrace system call to trace the orchestrate process. It's based on signals, so we need to wait system call. System call, we are so interested in exec system call. This means that we are swapping the image of the process to other process. It's the unit design that we are forking the exec and changing the process to other process. We need to notify the debugger about the switch and load new debug symbols for the new process. That is supported now on the BSD. We have also the proc file system, however it is not used for the LLDB debugger. We are slowly getting rid of this. We need this basically for the new compatibility. There is also this CTL syscall to investigate generic information about the process. It's now not debugger specific. We also have machine independent parts inside cores and machine independent parts inside registers. Like floating point unit, debug registers. Base system impact is quite small I think hopefully. LLDB is not requiring a lot of libraries. We just need good libc, epthread and correct C++ runtime library. We had problems with call once from C++. This was affecting LLDB in general. We had to fix this. So about LLDB clunk. So when I hit problems with the LLDB package, I decided to get the state of zero problems inside LLDB and clunk. I started to work on test suite for LLDB and clunk and get to near zero numbers of failures. There is still one ongoing effort on revamping the just-in-time API in LL4M to be compatible with packs and protect hardening on LBSD. Because we can't map in the same time writeable memory and executable memory. We need to work around this and we need to change the just-in-time library for this. And everything else is working correctly. So, as I noted, there is the ptrace system called that it's used for tracing. I was also already talking about this. It came from 80 and T-unix. It was adopted by BSD systems and by Linux. I will upload this later. And thanks to this, we can share a lot of ideas, use cases with Linux. And this is good for BSD. Yes, there is also sys-ctl system called that can investigate generic proposal. Generic information about process, its threads, like name of the thread. It's generic information. No, no debugger specifically. We can use sys-ctl for this. It's also KFEM, but at the moment it's not not in use by LLDB. It will be used in future for KGDB port. It's for reading kernel memory. So, we have also some security constraints, like mprotect. To debugger must or should be able to place a trap, software trap inside the process. And this means that the map page must be readable for a while. And we need to support this for debugger in the debugger's case. So, when a process is ptracing, other process, we are removing the restriction of no retable and executable memory, for example. And we also cannot trace every process. We need to be privileged. We also cannot trace processes out of the root. So, just a moment. Okay, so. I would not note that ptrace is based about reparenting. We have parent process, track process tracer, and there is a use case that we are swapping parent with tracer. And here is the evolution of ptrace. And there are some references for ptrace. And here is the list of new kernel features to host LLDB debugger and approaches. Yes. What's new in ptrace? I can share this after the talk. In detail. Yes, I'm also maintaining the build bot for the ptrace. There are around 1,000 tests executed for MD64 port. For for just ptrace and surrounding like weight. For debug reducers and many use cases. And yes, I'm working mainly in a packet source working progress. I put there my patches and I later upstream them to the LFM directory. Yes, I'm running LLDB build bot. And I'm also, I saw work on sanitizer, but I rescued this. Safe stack. It was in the talk about hardening packet source and diffuser. There are some also bugs. That's credits. Yes. So this, I was also thinking how to finish my presentation. And I was inspired by the opening talk about heroes. So I took a phrase from Gnus Pompeius Magnus Miner. It means Miner's younger one. It means that there was a history that there were domestic wars in Rome Empire. And the soldiers were out of food and they had to go through the sea. To do Rome and get food for the soldiers. And Gnus told that it's not necessary to live on that side of the sea. But it's necessary to sail through storms, risk. And maybe win the battle domestic war and get the food. I think this is also a big ability for NetBSD. That there are many problems, but we need to try to cross the sea and maybe win the war. Anyone with questions? Thanks for a nice presentation. But I was curious about the bit with package source hardening. You kind of glossed over them. If you could have a quick summary would be great because I'm quite curious about that. The hardening part is more about safe stack. And our developer Pierre is working on an introduction of hardening features to package source. So we can build packages in this particular case with safe stack. Safe stack is based on two stacks. We have safe stack for safe operation and unsafe stack for possibly problematic operations. And we need a lot of support for this and it was added within my work. And the process of implementation of this is up to Pierre. And I think this can be finished or done for LFOM version 6. So still we have some months to go. More questions? Quite simple question. Why it's necessary to patch LLDB to support multifredding on NetBSD? Was the problem or? Well the question is doesn't it support it out of the box? Well no, yes. In the beginning we had no support for multifredding in pitrace. It's also similar for OpenBSD currently. In old NetBSD version we had different model of threads. We had user space threads. It was based on the schedule activations. And we had libpthread that was doing scheduling. And we had special libpthread debug library. And we are trying to debug process through this library. And currently we have different model that mapping of user space and current space threads is one-to-one. And we needed to introduce new API, not on the libpthread level, however on the kernel level. So we can now start a suspend and resume a dedicated thread from the debugger site. There are also other problems. We have for example a mistake. A design mistake from the data that is shared for example with Windows. That we are using the same internal kernel property about stopping or suspending the process like the pfret is using. Pfret is using LWP suspend. And it's setting the same bit as the debugger. So they can config one with the other. And in future I will add a dedicated bit for just the debuggers. It will be invisible for other processes and for the process itself. And this is ongoing careful. And there are many bugs, stability bugs. And this is prolonging the work. And however this is very needed to move forward. And before resuming my libp user space work I want to get kernel part from multiple threads done. Thank you for your presentation.