 So good morning, everyone. Welcome to our workshop, automatic bug reporting for Dami's. My name is Martin Kutlak. This is Mirek Suhi, and we will guide you through this workshop together. If you would have a problem or a question, just raise your hand and Mirek will try to help you with it. And before we jump in, I would like to ask you how many of you know what's EBRT or ever. Nice, thank you, free hands. So at first I will tell you something about the EBRT and the components on the EBRT and after this theory I will try to show you some comments that you might not be aware of in EBRT and what can you do and later we will try to create an event, a custom event that can help you collect some information for your applications. So yeah, before we start you will need these three packages. You should install them before the workshop. I will use some of them. The wheel crash is our testing package that has some commands that you can create GordUms or Python exceptions and Java exceptions. Yeah, I can show you. So if you have the package installed, the wheel crash, you can just do wheel and there's like several different commands that you can try. For example, I can try the wheel sac fold which will create the GordUms. Yeah. And let's start with the EBRT stack. The EBRT consists of two components. One of them is client applications and the other one is services. The client application that you usually run on your laptop or server or somewhere and the EBRT or the client application consists of the EBRT. If you have desktop, you probably have GNOME EBRT and also EBRT uses two libraries. One of them is Libreport and the other one is Satyr. The services that EBRT provides is the EBRT analytics server, which was formerly known as FAF, but we decided to remove it to group together the branding and also the retrace server. I will talk about the EBRT analytics and the retrace server and also about the EBRT later. So first let's start with the EBRT. You probably know this logo. You might have saw some notification if something crashed on your system and the EBRT provides some command line interface for the EBRT. You can try it. Well, we can try it later. And also EBRT provides some demons. There was usually running background and try to catch the problems that happen on your system and also the stack traces. You can install some add-ons for languages and different runtime errors. EBRT can catch Python, C++, Java, Ruby or KERNA problems. EBRT also provides some data processing components and also the EBRT is integrated in the cockpit. If you have cockpit on your system or if you don't you can try to open the cockpit. If you don't know it's on your local host. Usually the port is 1990 and if you go to the logs and choose service EBRT notification you can see some of the crashes that EBRT catched and if you click one of the crashes you'll get just the basic journal information, but they are set up with the problem info. And that shows you what EBRT would normally show you with the information about the crash and there's also more detailed information where you can see for example the core backtrace from the core dump or different another different files. And also there's you can report the problems from the cockpit. I've already reported the problem, so it shows me that the problem was reported and there's actually a link to the problem. If I open it, I already have and also can you see from the back seats? Okay. Well, I think this is the problem actually. So when you report it the EBRT sends something which we called micro reports, which is just basic information about the crash. It's usually it usually looks like this is the core backtrace from the problem and it also has from which system it arrived, the architecture of the system that the crash happened and also the versions which have which the package from which the crash came and also you have if you are a maintainer of the package that crashed or the component you can associate the micro reports or the reports with the bug zealots. I've already did that and this is the bug zealot and if I scroll up the EBRT adds the link back to the problem and when this and then you associate this bug and try to or the EBRT catches the bug and you report it it actually sends you back the information that the problem is already reported bug zealot. Yes. Yes. I actually don't know that right now. Thank you. Yeah, thank you. Yeah, that was the cockpit integration. So let's continue. We have the GNOME EBRT. The GNOME EBRT is the graphical interface for EBRT. You probably know it. You can see the crashes that EBRT reported and see the details just as in the cockpit and you can report the problems to different platforms like bug zealot. You can also send mails or if you have sentos you can send the mails, the reports to mantis which is bug zealot for sentos. There should appear pop-up button from a notification area and if you click on that it will get you to GNOME EBRT. And that was the client part of the EBRT and then we have the services that EBRT provides and one of them is EBRT analytics. I've talked a little bit about the problems, but I will try to repeat it. So the EBRT creates microreports and sends them, if you have EBRT auto-reporting enabled, it sends them to EBRT analytics. EBRT analytics processes this micro report and creates what we call report. Later, EBRT analytics from these reports try to group them together by similarity and creates problems. If you open this link and I will show you some problem that I have already prepared, you can see this is already the problem that was created by EBRT analytics. If you scroll down, you can go to the different report tabs and if I just quickly go through them, I can actually see that the only thing that's changing is the version of the Java and there's also option to do a backtrace div and I can show you now, hopefully. Yeah, so these four lines change. So EBRT analytics try to group them together. So this these problems are together. EBRT analytics also provides you with the statistics of the incoming reports. You can see what's crashing in your application and basic overview of the component. What type of crashes do you have? We have this retrace server on Fedora project running, but you can also create your own instance. We have a Docker image available. If you run Docker, pool, EBRT and the FAF image and also EBRT Postgres instance from us, you can set up the FAF and report the crashes to your instance and see what's happening on your system and you don't have to send it to a retrace server and let's continue. Then we have the retrace server. The retrace server provides the remote retracing of core dumps. Usually when something crashes and it creates a core dump, the core dump contains just pointers to memory and it's not really human readable. And if you want to make it human readable, you have to download a lot of debug packages and it's not just for your application, but also the dependencies. So you have like huge amount of data that you have to download and find. So the retrace server tries to solve this by allowing you sending the core dump to our server, which then processes this core dump and sends you back the backtrace that's in human readable format. So it should save you time. You don't have to look for these different packages and also it should save you the storage or the space on your system because these debug info packages are usually huge and that was the theory part. Now we can continue with the workshop. If you have the ABRT installed, I can go to the terminal and the first one that I would like to show you is the basic command. If you have the newest version or the latest version of ABRT, you can do just the ABRT and it will list you the crashes that you have on your system. And this is just basic of what kind of crashes you have from which kind of components and when these happened and also the I can, is this enough? So if you want to try it, you can run the ABRT. Also if you have, if you want more detailed information, you can do ABRT list and this will show you what or where did you send the reports and also if if the report was already reported, it will show you that it was sent to this or it was if you reported to Baxila, it will show you the link to Baxila and you will know that you already reported this problem and ABRT should prevent you from already reporting the reported problems and I will show you later. Another command is ABRT info and if you type it just like that, the ABRT info, it will show you the detailed information about the latest crash, but you can also select it the problem by the ID. So I can, for example, choose this one and it will show me more detailed information. It's not old information, but it adds the command line and some more information like the package that it came from. Then there is a command which you probably don't know about and it's ABRT gdb and if you again select the ID which was this one it will okay, this one is not oh, it's java, so actually the latest one is gdb or coredamp it will start the gdb and if you have the back info packages, it will already show you the human readable format, but if you don't have if you don't have the back info packages, you can execute ABRT debug info install and this will try to install the debug info packages from the build IDs that ABRT collects from the coredamp and I already have the packages installed, so it also created the backtrace for me during this process. If you want to get the backtrace from the retrace server and you don't want to report it, you can execute ABRT retrace command I've already have the backtrace, so it will tell me that I've already the backtrace created and ask me if I want to show it I don't want now, so I will do I will force the retrace to execute and it will ask me if I want to do the local retracing, which will try to download the debug info packages or I can use the or I can use the retrace server and send the command to coredamp to retrace server I can try it now it will take some time, I already have I have it somewhere, yeah so this is then the output of the command will send the coredamp then process it and after a while it will generate the backtrace and send me back the human readable backtrace and then you can use the ABRT report command that will start too many sessions opened well, live demo but you can try it on your system, hopefully there's nothing broken in mine and it will it should show you where you want to report it and then ask you for bugzilla ID, bugzilla login and passwords and you can report the problem to bugzilla yeah, and if you don't want to or if you have the debug info packages installed and you just want to see what's what crashed you can run the ABRT gdb and then continue with the gdb directly and you don't have to you don't have to run the gdb and find the coredamp you can just do this and you have it ready for you to debug the crash yeah, that's information and now we can try to create the event, the custom event so if you want to try it with me you can create a new file you will need to be a privileged user so for example if you create file flockevent.com.conf which I already have or should I and I don't have it and paste this well, I will have to type it but first I can probably explain you what this does at the event it's a it's a word that ABRT tries to find in these files and it then executes when when this notify happens and the notify happens when something crashes and the ABRT shows you the notification there's also possibility to create the event during the post-processing of the crash but the notify is probably better then you can select something like component or you have more options you can actually choose also a package version or other different options and the the other options come from this problem directory you can choose package name for example which is will crash or if you want to match package version you can also do that and it will try to match this version but there's also opportunity for a regap regraps reek regular expressions and it will go through the file and try to find the matches in that file and then there's second part of it of this and that's the eco line if you want to put something or create some kind of file that or find some kind of information about your component and we don't collect it you can do you can for example copy some block files or execute some kind of command that will create a new file and it will create this file in the problem directory where the the ebrt collects these crashes and during the reporting it will add this file to the report and I've already did this before so when you send the bug report to the bugzilla and select that you want to send the send the file that you've created it will abrt will include it in the report and for example this one that I've created before is the flock file and it will show you the hello from flock flock in budapest so if you want to try it yourself you can do it now and you will have a problem with it I can try to assist you the the scripts should be written in bash and the first line the event should be uh there shouldn't be no space before that but the the following line the echo should be spaced or tapped or just comment it and it will then execute these commands in in order yeah this is the different different uh uh different times when the event events can happen one of them is post create then there's the notify and the selecting the values is the the first one is the key and wall it's it will try to match it yes so yes I forgot to mention so we collect just basic information and you mine as a packaging uh maintainer it might not be enough for you to debug the problem so you can create this event and collect whatever information that you want from the crash and it should it will help you to debug the the problems and continuous integration system about this uh this crash or you can send asmr being on a page or whatever whatever you need you can you can be using those kinds of events so it's both for for uh distribution packages and for local yeah by by default a vrt gather information about package about crashes in packaged uh and correct me if I'm wrong in uh when we are running in container it's opposite so we can uh because in container is most of the software is not packaged so we even send crashes from non-package container uh software but you can customize it in abrt config or really so so you can say whether you want to get crashes from non-package software as well and then you can upload it to to abrt analytics or to see your detection system Any more questions or is anyone trying to create their own event? Yes When you do this it will create the file and it will create the file in the problem directory you can see the problem directory if you execute uh ebrt info and this path It will show you where the ebrt stored the problem details Yes, I go and if I show the content what's the what the directory contains you can see the information uh and quite small and they contain all this information which by the way was very visible on that uh on the cockpit uh page so for example the cgroup contains all the information we get from the cgroups the online component so and here is the flock file and these events run under Privileged users so you should be careful what you do with the events Yes You got the files in ITC as well Yes I think yes, but I'm not sure so Can you repeat the question? Yes So the question was if ebrt is in Ubuntu as well Yeah, well right now the ebrt is only in the Fodera centers and the rail World but Terrorists or we have an issue open that we want to support also the different different platforms like the bn ubuntu and sus actually don't know Actually reporting is one thing which is useful, but I found Most useful are the micro records So And for that you have to run ebrt analysis service. So just having the Package available there is one thing but Then you go have to run the service So we run the service for Federa and Federa infrastructure Running it for Andriva that's Beyond our But definitely, uh, we provide What you mentioned we provide the container For ebrt analytics. So it's very easy to deploy your own container And Get your own questions, so if you are running your enterprise environment and You are operating Like that, you don't want to go your precious outside your company. You can run your own ebrt analytics server And Your precious in general So can you can you work more about those micro reports? How to enable them? and I'm so sure some examples from from How so There's a ebrt command called ebrt reporting, uh, sorry auto Reporting and if you execute it, it will Tell you if you have it enabled or disabled if you want to enable enable it You just type one or two or enabled And it should enable it and this one The ebrt auto reporting Means that the micro reports will be sent to the Location where you want to report them by default. That's the retrace server that federa project that orc slash fav and What did you want to show? So it's enabled by default. So, uh, actually everyone in federa if they have some crash It's automatically reported to ebrt analytics federa infrastructure And this is uh Some example so The the most interesting part is even if you are Maintainer of some packaging federa and you think your package is Working well Because you are not receiving any bugzilla. It's worth checking ebrt analytics server You can get your from federa packages and if you check like for the bash Fav it will navigate you to the Directly to the component of bash in Fav if you are in analytics What this is just for This is you have m2plug Yeah, and also and Even if you think you didn't have many bugzillas You can see that there is a lot of crashes where your component is involved. For example, we have like Uh, nearly 10 000 crashes In function vgp here And so on So you can see How many times it occurs on which architectures And in the statistics that you can even create some ensegraf. So it's even visible In which version this start happening And in which version it's still actually happening So for example for Well, we are creating internal reports using a vrt and we are Suggesting Developers that hey You have this bug open This is associated with this information in In vrt analytics And it's not happening. So you probably fix it A side effect of some other fix so you can wave it as as fix because it's not happening anymore So it can give you a really nice way how to solve some bugs Really cheap Those are those graphs Can I just ask you Did I convince you to try and create some custom report for your application? So if you already can collect it and help you With debugging So It's okay, it's okay. This program contains no Sensitive information that all you can give the profile. Maybe send it to me by mail Then I get the profile and in the meantime to mask as fast as Offer the profile is gone. So the usability of the reports has gone down And it will be so nice to again have the logs in the Barsilas the profiles, maybe maybe you could whitelist programs which contain no user information and third thing is that the barsilas are often Private because there's like key Yes option, which is very annoying. It's also annoying for users because users don't cannot see the reports And I have never never in my life seen a report that had any sensitive information Yeah, uh, can can you open and try to report some some crash from the ABRT The problem is that There is two things Against each other, right? What is usability of those reports where you need as much information as much possible? and on the other hand There is a privacy of Everybody involved And especially if you are reporting something from your company You probably don't want to leak anything suspicious at all So we actually have whitelisted Things which are sensitive like amazon credentials. So if there is in a Operating environment variables something aws underscore We will wipe it away. So it doesn't leak And we have quite long list of these whitelisted information which should not report But when you try to report it, uh, there is You have option to select and you see what's going to be reported including that private channel. So It's up to user to decide. Is this report private and if they check, yes Then yes, they think it's a private I'm not sure whether the right thing is like Uh, pop up several other dialogue boxes like You really think it's private? Yes But we don't think so Yes, it is private. Uh, but Yeah, it is private. So Just Okay, okay Also, also you have you give this, you know, lots of text that is not really easy to read and you ask users They won't ask to compromise your privacy and users will be like, well, no, right because it's it's hard to say and I don't know maybe For example, uh programs like System D resolve D I mean if the If the program crashes when it is resolving a domain name that is Sensitive like, I don't know Then it will be anyway visible in the backtrace anyway And So your privacy might be compromised just by the backtrace And apart from this there's no there's no no keys. Nothing sensitive ever And I would like to say just why at least any crash from from this service because there is no privacy possible, I mean Except for for some stuff, which is And this would It's always attached to profile and always I mean never made it by It makes it easier for user because they will not be asked questions Which are for answer and it will actually make for you make it possible for me as a developer of the program to make use of the backtraces and the reports Okay, I love it. I mean because users programs which are run by users as parameter session They often contain sensitive information, but system services I think they should not should never except for some select ones No, we don't because some of them can be huge and there's a limit in bakzilla What looks do you mean Oh Well, there was a problem that usually in the in the waterlock There or maybe it's that We have just basic We try to match the name of the Package or the component that crashed and try to find it in the lock and it's I think three minutes Back From the crash that happened And during that time, they're usually weren't that many locks or useful locks and Yeah, actually gnf or If we create the the event and So for example When watch logs are made services will crash because of watchdog timeouts and Three minutes or a minute and a half or something like that and very often this is caused by general machine overload and and If there are logs and the logs I see in the logs that the that the kernel is reporting Ior or something this and I get I bought a minute later And I'll be like, okay your hardware is busted Close the bar and there are no logs, you know You know, I can ask politely. Do you think that your machine is working correctly and this this is just First finished If if the if you can attach logs from journal, right And gnf sort of does this With this Again, so it's about if you collect all information from Systems uh journal cpl like all You risk that people will refuse to actually send it on that load So it's a gain in privacy. So so Year we'll I get just from the dnf Information So it's about configuration. So technically we are able to collect it Just about We come to conclusion that we want to all logs Or pass to ours fine So I think that this highlights the the issue. So this this command can potentially return An insane amount of logs because there's no time limit or boot limit or anything like that So maybe somebody has A hundred gigabytes of logs and this will run for 10 minutes reporting all the reports from the service since forever and Asking the the maintainers of individual packages to provide plugins for abortee to To basic load collection. It just feels like the wrong place to do it. I mean this should be built in Must be built in because otherwise there's what maybe 3 000 packages which Or more which could use custom reporting and If you ask the maintainers of each package to write a plugin like this That it doesn't scale You may don't know No, it's actually useful. It's a great suggestion. I'm really glad that you're getting this feedback Any other feedbacks As was already mentioned that there is a heuristics detecting whether it's like a report as private And this one is really overly sensitive Because it's catching some nonsense And then you really get three times asked Whether you want to submit a report as non-private And if you submit it as private by default, then you can even see the back in maxilla. So if it happens again Again, uh, something in the back Yeah Because for example, it is quite quite the thing that some ruby code is using And typically there are questions That's more question This is the issue and you still want to be reported But you don't want to be Aside in analytics to Well, the problem is that typically it's like It's reported against ruby, but would be executing some code which is typically in some user user home and the user is experimenting with such code and so on So I would love to be able to avoid these reports at all because it's like really if I am running My code and I'm doing this big I should get reported into into Some somewhere in maxilla because really The back itself exhibits In ruby, so it's like rightfully reported against ruby, but the issue is somewhere else You are getting This is such a situation you're describing Ruby is set fault or You get even tracebacks Yeah, it might set fault because there is some Involved numbering access or something Set fault is always the problem Yeah, it's like you are playing like You are not using if you are using the 8.5 via ruby you are not Using the ruby Facility Yeah, that's that's ruby ff My issues Yeah, it should not I'm saying that's Report against Yeah So And you are That's not true. We are using interpretive which is actually you are like running the strip which is run by ruby which is packaged in In federal but it's using some library which is third part library and it makes The ruby trace. So yes, yes, and you are last in the back trace. So we Played that it's your fault, but it's actually not correct. I'm aware of that. So I'm saying That it should be reported but not to you You can run It maybe it should not be reported because the user runs these homes somewhere from user And so I'm asking right there really specific the ffi is the most often report I get so if I know that ffi is loading I would love to ignore The box because this is really It's really dangerous I think it's actually possible to exit the reporting if you match it against some Against some string can Quit the reporting and it shouldn't be reported In the abr t should stop report the reporting process, but I've never tried it. So No Yeah Just I think that I should show Is there anything that I should show There's And if you have any issues or Problems you can send us an email or try to contact us on IRC Or just file a github issue And also we have a blog with some articles about the stuff we do so you can check it out Yes So I don't think that's possible because We catch the crashes or we try to catch the crashes from system d journal And But it won't be catched by ebi t Because the the first issue is the catching the and then there is the reporting and Yeah And It's a great idea Because To see that That's so I would like to This exception from from a system I mean just precious just from your application So I would say okay. We have all the reporting on the place Services in place. So You are not As I do this is about all application about the most tech You can't be sure the report from the application. This is exactly what you are asking for Say, okay, some quick exceptional situation All happened and I could write to you get the report So I'm sorry about the idea It's bigger some book which we will work this out. Oh, yeah. It's just an idea I think yeah, I think that's just trying to respond Yeah Just be careful to send all All but we will get a lot of report for the false So Maybe I did actually doesn't right now support flood packs. So yes, we have plans to Yeah, I think Ernesto has done the investigations spent some time on that Another block we have There's some entry about flat packs and we end up like reporting Four six issue against flat flat packs, uh, which needs to be resolved in flat pack first So if you want to read more about it, you can you can hear so There are some technical issues Yeah, that's one of the issues that we are trying to No, we wouldn't uh, we didn't figure it this out yet. We have plans how to or There was some investigation by Ernesto's how to catch these crashes, but Proceeding with the debugging fault. It's another issue and we didn't touch that yet. So in the future And there's no It will All right Human You don't need to even notice it because So smart that your application crashed Killed the boat And Transparent for you Five times during one minute, you may start getting problems So we would like to show some nice Actually you can turn on the notification if you go to the As my I don't think we accept packages from rpm fusion No, it's well we Pull the information from Federa project and From the pdc and it assigns you if you are the maintainer it assigns you to that component so you can Do these different like associating or archiving reports and Yeah, you count There's this IRC not to fire it you can configure Okay, it's a matter of federal notification Yes, yes, I understand Honestly, I have no idea I Yeah, no idea, but yeah, I think I've heard some Suggesting them because actually I'm crashing Problems During the building in Koji or in copper to the gray area We want to get But yeah It would be Yeah, no, sorry because I can actually show you we have like hundreds of them and going through them Yeah Using like We are using it Are not able to because that's just for the admins So I don't know what to do with this feature Just consume storage It's for nothing. So I don't wish to come with some idea I like to expose it to to users Again, but Well, the problem is that All it can contain the core dump with sensitive information Yeah Yeah, but but I already saw bugzilla reports with passports to bugzilla and And when you are upholding this If something If you think this in our tooling you can uphold it here And admins will check it and We will keep it private. No one will Not treating it And for that like investigators shoes So we either have to change your wording Move the community to handle this I'm not sure if you will be able to Participate in that Just for my like for my cases. Yeah, exactly. It seems like That it should be helpful to upload at least What he has Because it's like at least at the moment It's Also, there's still option for the user to just file the bugzilla and do this manually, but yeah, that's why we have the ebrt to send the report Well, I I know some people who don't send Core dumps to To the retrace server and use the retrace retracing on their systems But there are some issues with that as well because Sometimes some of the debugging force might be a still missing Question Includes the I don't think I don't Thank you for everyone for coming I hoped I hoped you learned something