 Welcome back. This is the last talk before the lunch break. Please enjoy Agata port with tracking ITP dependencies with graphics. Thank you very much. Hello everyone. So my name is Agata. I've been a Debian contributor since the pandemic started. Lots of free time. So I started basically in 2020. I mostly do Python packaging in the Debian Python team. Sometimes not Python but really rare. And this is my first time I come to a real-life event for Debian. And I'm here thanks to Bursary. So thanks DPL for approving that and the secretory team. Yeah. So one thing I've noticed when doing packaging is that with Python you have a lot of dependencies, as said before in the previous talk. So what I did is to because if you package one dependency, you have another dependency, another dependency, etc. It can go quite deep. So what I did, I did mark them in the bug tracker. So I've used BTS block command to track ITP intent package dependencies between the between them. The problem is this is what you get on the BTS. So you cannot really see the names of the package involved. You can see all you can only see numbers. So if I show you this, it's really not clear. You have to click every link in blue to know what this means. Yeah. Actually, we could do better because in the BTS, we already use GraphVis but not for tracking dependencies. We use GraphVis for this. So this is used in the BTS to track which version has some bug and which version it was fixed in. And this use GraphVis. Okay, nice. And I thought, well, I could do the same for intent package dependencies. And I did. So this is what it looks like. There's a small legend on the left side with some colors. And this is for only one package named Python DTChema which depends on two things. So you can see one ITP on the left side which is marked as done because it has been packaged, so closed in the BTS. And the other one is not an ITP but it's a classic bug. So there's no ITP in the beginning but there are some dots at the end. Actually, you have to be very concise because Graphs can be huge if you put the full name of the bug. So I did some heuristics. If it's an ITP, I cut down the WNPP stuff, et cetera. And if it's a bug, I just saw the name of the source package involved because otherwise Graphs will be huge. So how did I do that? Well, spent quite a lot of time on this. If you want to do recursive things with SQL, it's possible but it's quite hard. So at the beginning, you can see with the recursive, I don't know if anyone here has already used this SQL command, quite rare. And yeah, postgres people. It exists, yeah. So it exists but it's not recommended. You can imagine why. It works. So at the bottom, you can see the root node where you want to start. So where I want to start was I want to find packages which are in WNPP for finding ITPs with me as the owner and not done because I don't want to have everything. This is the starting condition and then you have to specify a stopping condition. That's why these go in a big loop. I think I crashed UDD once. So the request actually is a lot bigger than that. Of course, it has been abbreviated but I think it's like 100 lines. Okay. One other thing I did really quick was to add the status of the package in new because most of the time I didn't know if it was waiting or not. So for this with Python, it really takes a few lines. And also I have used, at the bottom right, you can see that I have used the assignment expression which is from very recent version but makes things really concise. That's why I would have to take more lines. And tracking in new is only known of course for ITP bugs because for other bugs, you really don't care about new or not. And so this was for one bug but actually I have a lot of software. So I thought I could make a dashboard by tracking every bug I had. And I did. So this is what it looks like. Lots of interesting things going on here. So on the left, you can see I think something a bit bogus because I intend to package OpenTracker is in new but the RFS is still open. Okay. So I think we should close that. In the middle, there's something interesting. I've learned with that graph that you can have merged bugs. But in the database, they're seen as different. So you have these crossings going on. But actually there are two ITP bugs for one package that have been merged but I didn't merge them in the code. So they are crossings. The last one on the right side, you can see the tree is going a little bit deeper. So it's getting interesting. And also for the red ITP, it's in red because it's waiting because it's not my ITP. If it's in yellow, it's my ITP. So I know this one I have to wait for someone else. If it's yellow, it's my job to do progress. But I'm stuck because of an other ITP. Also two arrows because I guess there is a dual blocks and blocked by commands you can send. And I think these bugs are both blocks and blocked by. So you get two arrows. But also this could be merged as well. And this is the left side of my dashboard. On the right side, I have some more basic stuff. So most of them in you and most of them I have to do the work really. So I plan to do a small live demo to show you how this works. Here we go. So I run this script. It will actually carry UDD and not the real database. Yeah, we can try it. I put time before because this is important measurement. So I run it. And what it outputs is basically graph commands that you can pipe to a plotter next. And actually it took like, it was quite fast. But yesterday it was taking one second. So I was quite a bit worried that I would bring UDD down if I do this a lot. Well, and this is the output. But actually if you pipe it to grab this dot, yeah, dot command. This will output an SVG. And the SVG file is actually very interesting because in SVG you can do a lot of stuff. So I will open it in Firefox. So here you can see what I've shown you. But there are more interesting stuff going on. If you put your mouse over the bug, you can see the full description. So this is useful. Also it can be clicked. So if I click on it, it will open the BTS bug. So you still have a small graph you can see, but you can access all of the information in the BTS. On the top right you can see nice graph vis, but for versions. So yep, that's it. Actually you have to scroll left, right. It's not really ergonomic. But graph vis only draws top, down or left, right. So you have to make a choice. Yep. And also, good. Also, yep, I wanted to show you the big SQL requests. So here I set UDT stuff. I have my colors going on. This is graph vis legend. This is a small function to go to new. And here it comes. Okay. Brace yourselves. So we start with recursive. We select what we want, but not everything, because UDT has huge lines. So you don't want to bring the server down. Also this here, we put true as root because it helps. Well, it's a constant, but it helps in the Python code to know if it's a root node or not. So it's interesting to set. And then the condition for starting, some union, some joints. It's quite big. And then distinct. Distinct was used to remove some of the duplication, but not all of them as we have seen. So I still have to do some distinct stuff inside this huge query already. And then basically, there are some heuristics here. So the first heuristic is if it's an ITP or ITA inside of WNPP, then you want only the name of the package. You don't want WNPP, ITP, et cetera, et cetera, et cetera. And also long text because I think I'm not sure. Yeah, short description is in the title of the bug, but you only want the source package. That's why it's huge. For RFS request for sponsorship, we can show them directly. And for the rest, it's considered standard bugs, not my problem most of the time. So it's read. And yeah. So that's it for the code. And if I go back to my presentation, oops. Yeah, evolution. So the dashboard I intend to keep, it's nice. It's available on Salsa. One thing I would like to do, but I'm scared off, is to show this dependency graph for each and every bug on bugs.divion.org. But I think it's spur. So I'm not convinced. I need to be careful here. And also I'm worried by the performance because the BTS is already quite slow, taking sometimes one or two seconds to display a bug. And if I add a recursive query against the real database, I'm afraid it will bring it even more down. So yeah, I could crash it. Ah, cache it. Yeah, actually, this is a good idea. I want to implement a cache on bugs.divion.org because I think each query makes a SQL query and it's really slow. So could be, could use some caching. Yeah, so this is the evolution I plan to do, but I need your feedback. So that's why. And finally, some resources. The query code is for the first link, which is the repository on Salsa with the source code. The second link is the schema of the UDD database you want to look at because there's a lot of stuff going on. So the third link is Python grab these library. I actually use it print in the beginning, but there is a library, so it's better. And the last one I highly recommend is a book I'm reading right now, designing data intensive applications. And there's a chapter inside about the recursive SQL queries. And it says it's not recommended. So yeah. So thank you all, and I will take now any questions. Thank you, Agat. Are there any questions? Have you ever encountered any cycles in the dependency graph? Up fully, no. But I think it may, well, when I created the SQL query, I did a recursive loop because I didn't have this root constant. So that brings down UDD. So in my request, I did some mistakes and it would be infinitely recursive because of my error, but I have not seen both in the BTS with this problem. If they are, I will probably bring the database down if I implement this. So this to check. Do you think that this would also make sense for regular bugs in tracking if there's a dependency chain for the bug itself? I don't know if there are cyclic bugs. There could be. Well, the query could, I think the query could not tell you it's recursive because it will recurse. So it will crash. You don't want to detect a crash to say there's something wrong. But I think it's another topic, but we have to be careful because if I implement this and there is a cyclic dependency, the query will crash us ever. So yeah. Any more questions? Okay. Thank you again. Thank you. We will now have lunch on the first floor at Cantina, and we will continue at 2.30, I think. So see you soon. Enjoy lunch.