 Okay, in this talk by Don Armstrong, he will talk about bugs in large packages. Strategies and improvements for dealing with large numbers of bugs. If you want to ask a question, please raise your hand and wait for the microphone. State your name and look at the camera over there. Okay, so this is a buff. And so its primary purpose is sort of twofold. One, I was going to spend a little time at the beginning doing what I didn't get to do in my talk and show off some of the changes to package report, just to talk about some of the changes. The most important thing that can happen in this buff though is working and talking with people who are maintaining packages with large numbers of bugs or people who are dealing either triaging or filing or otherwise dealing with packages that have more bugs than they're currently capable of handling by themselves using the things that they're doing. Both to change the way that Debugs is currently dealing with the packages and to sort of communicate ways that we're dealing with bugs and packages better and things that I can change or things that other people are doing that you may want to adopt. In order to help out, keep notes because I'm not going to remember everything that happens here and I probably won't have time to look at the video of myself talking. If you guys wouldn't mind joining the Gobi that I've got up there and helping to keep notes on what's going on, that would be really useful. I won't keep it up displayed the whole time but that's where it is. So you can join it at that IP. If you don't have Gobi installed, you can install it. It's pretty neat and pretty fun. I didn't actually check. Does anybody join yet? Oh, yes. People have joined. You guys can join and type away. Let me jump on to what I've changed really quick in package report here. Let's look at a more complicated package. Once Firefox decides to let me type stuff into my URL bar, thank you. So let's see, devian.org, okay this is a relatively complicated package. So the main changes that I've changed that you could see already are the fact that it's now using CSS or it auto collapses all the extra information. You can expand it now by clicking on these doohickeys and it tells you the same information that used to be there before. This is kind of useful if you have segments of bugs that are this long that you can actually see what in the world is going on with this thing. The other thing that I added are these bits which basically summarizes what's going on with these bugs. So like this is an example of a bug that's tagged more info, it's pending and it's merged with other bugs. So by default now all bugs are merged. So all merged bugs are displayed as a single bug in the page by default. It used to be the other way around by default. You can of course switch it, but I made that the default because almost always you don't care about the other bugs that have to be merged with. So it has some little cutesy smiley faces going on with it and frowny faces for it won't fix and stuff like that you can play with. Hopefully the mouseovers work for them. Well, I don't know why that was not working. There it goes. So the tooltips should all work so you can see what the tags are. This is configurable so if we add more tags I can easily add more things. The basic separation between the bars is between the severity. Obviously, severities are lower cased or uppercased or changed based on what they are and the things at the end are things like merged, forwarded, whether it's closed or not, has fixed versions and things like that. So that's one of the changes. That's sort of cosmetic but obvious to everybody. The other huge change is the options. Finally, all of the power of the select that I wrote a while ago is exposed in the options section. So if you want to do crazy things like select all the bugs that are in Debugs and all the bugs that are in bugs.debian.org, which are all the bugs that I really care about, just type that in there. Wait a while. I press return in case you were missing. And there you go. Those are now all the bugs in bugs.debian.org. Let me get a new terminal here so this thing is on this display. Or bugs.debian.org. So we can go down again. There they are. And if I want to do something else, like I only wanted to look at bugs with severity, I don't know, normal, for example. Then you could just add it in. And just like I was talking about with the SOAP earlier, these are bugs in packaged Debugs or bugs.debian.org and with severity normal. So different tags get ended, same tags get ordered. You need it bigger than that? Wow. Can't see anything. We'll have one word on the screen at a time. But anyway, so you could see that that's what it's selecting. It looks like it's selected more than normal bugs, but whatever. And you can, again, do more complicated things, like whether it's owned by you or not. The other thing that's here that was always available but wasn't, stop. That's great. If you want to know about William O. Duck's quotes, that's what it was. You can also limit bugs. So for example, let's go to a really crazy example here. Let's look at WNPP, for example. So it'd normally take forever. So we'll just delete these because we don't care about them. And we want to only include bugs with subjects that have, I don't know, what's a good one? ITA, okay? Theory, oops, it pulled severity, I need to kill that. It's not quite smart enough yet, I need to fix that bug. See if it'll work now, right? So now it's only showing bugs that are ITAs. So luckily in the case of WNPP, this makes it really easy because there's a concise subject tagging that allows you to do this sort of stuff. So these are only ITA bugs. And wow, are there a lot of ITA bugs that people need to rip and finish adopting? But anyway, so you can see how that works. The ordering here just reverses things, so you can do it by raw, the old view, normal or by age. This isn't complete yet. And so it doesn't do all the options, but it's something that needs to be done. Yes, include only and exclude means in addition exclude these. Yeah, it doesn't. So these things here are actually, I should probably box them or something, they're technically filters. So they take the set of bugs from select and then they apply some filter to it. So this is only include bugs and or exclude. Maybe I'll make it all the way. Yeah, I probably should, it needs more documentation. I sort of had the bit to the side here for documentation, but I hadn't really finished it yet. Is there another question or? Oh, okay. Which one? Oh, filter around it, you mean? Just instead of using the word include, using the word filter. Oh, yeah, so it'd be filter everything, because the reason why I call it include and exclude is because that's what the actual options are. So, but I'd put a box and make it more obvious that that's what it's doing. But anyway, those are sort of the things that you can do now more easily using the package report option. You could do all these before, but now it's easier. And then there's also these other things that you can show archived and unarchived bugs both at the same time if you wanted to. If for some reason you wanted all the information, you can click that button and now all of those things that were hidden are there so you can check out them all and be harassed with all that extra information. Okay, so that's pretty much it for package report. The changes that I made just recently. So I'd like to open it up for discussion and questions. So who here has maintains packages with more than 100 bugs in them right now? Okay, more than 200? More than 300? More than 400? Okay, so we've got people with a lot of bugs. How many, which of those people, or let's see, how many people have bugs that have been open for more than a year? More than two years? More than three years? More than four years? More than five years? More than six years? How many people have four digit bug numbers? Okay, so there's a lot of people who have experience with dealing with large amounts of bugs. So why don't we, what can I do to make things better for people who have such huge numbers of bugs? What are the problems that people, well, besides me sitting down and fixing them myself, that'll never happen. What can we change in Debugs? So I mean, this is a discussion, so I don't wanna talk anymore. I mean, I know I've got a few hundred open bugs in Debugs, but which ones are important? What are the things that are really bad now that I should immediately fix in the next couple days? What's good then? If there's nothing bad, there must be something good. I mean, yeah, go ahead. Oh, sorry. I really want to thank whoever made the forced marriage comment, that's really helpful. Okay, yeah, I'm to blame for that one. I got bored with trying to figure out how the hell the merge bugs, so that's me. Anything, yeah? I think at the moment it's very difficult to ignore bugs, to like say, okay, I've looked at that, don't show me that again for now, because I want to look at the other bugs. So you often get lost in the long list of bugs, even if you go through them. I mean, it helps a bit if you delete your browser history and so that you have different colors for the links, but that's just the heck. Yeah, one of the things that there's sort of support for that needs to be made better is ordering by Last Modified. So in theory, I don't know if it's working. I haven't checked it recently. But in theory, we can take these bugs and order by age, hopefully this will do the right thing. And it'll show us the bugs in the set once it reloads. Besides obviously, why is there a serious, oh, that's right, I know why, I don't mind. Hopefully I didn't click on that. Yeah, I did. But it should order them by Last Modified. It doesn't look like it's doing that. That's the idea anyway, so that'll help you, because if you look at a bug, hopefully you've mailed it or something and you can then start looking at the oldest bugs first or something. Another thing that may help with that is the user values so that you can set something like maybe a time at which you'd be notified. That might actually be useful. Don't let me know about this bug until after this epoch or Unix epoch and then at that time you would learn about it again. That might actually be good if somebody wanted to write an extra tool that would take a user value and parse it and then send mail out to people. That would actually be kind of useful. So an issue I had for quite a while which has got a lot better with the new layout is the number of bugs I can overview in a single page because it's silly. I mean, the more you can fit in a single page, the more immediate is the feeling you have of what needs to be done. Now it's way better, but if you compare with like, I don't know, track, in track you have much more concise overview of the number of bugs and I think it's just a matter of decreasing some, I mean, the title I think are ridiculously large. It's really, I mean, outstanding bugs and this kind of stuff is really large and maybe tabling the bugs. I mean, now they are more compact but tabling them so that they are aligned or something like that, it's one can get an overview of the bugs more quickly. Okay, so maybe having the initial, like at the top of the page, having all the bugs, numbers just smacked into a table or something. Yeah, for example. That may be worth thinking about. And hopefully some people are making notes in the Coby. Yeah, they are. Awesome. I'm sure there are very good technical reasons but one thing that annoys me is it's too slow to make changes to bugs, to control, you send a mail and you wait and at some point it gets processed and if you make a mistake, you have to resend the mail and wait again. I'm sure there are very good technical reasons but still it's, I use often other systems that have many advantages compared to the bugs but are very good at that, making instant changes to bugs. Yeah, that may be something that I've sort of started thinking that perhaps it would be useful to do authenticated some sort of web-based methods to control. The main problem right now though is the whole architecture is set up for, you put in a message into a queue and eventually the queue gets processed and something happens. But it may be the case that we make that even faster so that we can run the queue. I mean it'll obviously still be queue-based even if it's web-based but so that we can run the queue faster so that you could sit and wait on the queue results for your message. That might be something worth looking at. How many people would prefer or would use such a feature if it existed? Okay. So the idea is basically that for control changes you could on the web twiddle a bit, wait a little bit and it would come back with the results. Well yeah, at that point it wouldn't matter. It'd be some sort of SOAP interface with authentication of some sort yet to be determined. Okay, okay, so the people want that. Okay. What about using the Debian Curing to authenticate requests or something like that? Sorry, what? You could use the Debian Curing, your GPG keys to authenticate requests. Oh yeah, yeah, the authentication method, there are a bunch of different ways to do it. My main thing is whatever it is, it has to be tied to an email address. I'm not particularly concerned whether it's just developers or not. It just has to be tied to an email address because that's no less accountability than we have today. I guess if it gets abused then that's something else, but what else? I'm reading the comment on the screen. Does the fact that the mail goes through the anti-spam filters cause a delay in the process? Or it's just the queue which causes a delay? Yeah, so normally the anti-spam stuff is way faster. So assuming it hasn't gotten nailed by billions of messages, it'll process the spam almost instantaneously. It rereads the spam list that it has every couple seconds. If there's more messages it sends it off to the sub-processes. The main thing is that the queue is run every three minutes. So there are times in the day when it runs longer because it has to stop to update all the indexes and it's bad if things change underneath it, but for the majority of the time your messages should be processed within four to five minutes. I mean, end time, so yeah. Coming back to solving bugs, I think it would be nice to, just quite a different thing, to allow more users to solve bugs. For a user coming to a page which has 100 to 100 bugs is overwhelming. I mean, a user that wants to contribute with a small patch or something. I don't know what the solution would be, but I think that if we need help so we need people that provide patches because it's obvious that if you have 100 bugs you can't patch all of them. But for that to happen we need to let those that will be providing the patches like I don't know, see something that is less scary. I'm not sure how to do it, but I think that we need something like that. Yeah, I think that that is one of the problems that happens with big packages is the sheer number of bugs. Even for people who know what they're doing it's very, very difficult to wade through. I think maybe increasing the documentation on user categories may be taking an example of a couple packages and making a study of them and assigning user tags and categorizing out based on the workflow that package may be the way to go. But first time users to the BTS getting them to find a bug in a patch is always hard. I don't know necessarily how to make that better. Well, there's something in Bugzilla. Okay, Bugzilla is not very nice, but it has a few good ideas that could indeed help if you have too many bugs. We could have a summary field. One that we can update live so that we can, sometimes a bug starts with something and it ends up being about something else. We would update the summary and you could give us the summary in an expanded listing. And that does help a quick, oh, we already have that. Yeah, so your idea is a really good one. Let me show you how you can do that. Once I figure out where I put it, I did it in one bug and I just wanna show you the bug. I just have to remember what it was called. Do, do, do, there is summary, here we are. So this is the new summary feature that does what you're asking for. So this is the summary, it's kind of a useless one, but whatever, it was my test bug. If you were on www.list.dev.new.org, that's what was going on. I forgot that this was the www bug. But anyway, that's what all those messages were for. So basically what you do is you send a control command and you say, okay, I want message 15 to be the summary. And this is message 15. And so it tries to do the right thing and ignores the coded part of the beginning. It tries to ignore pseudo packages or pseudo headers, control. Things that look like they were control messages and then rips out the first paragraph it finds. And that's what's at the top here. The first time I want to set a summary, I'm probably doing this that with a single mail and I don't know yet the number of the message. So yes, so it has already support for being set at submit at time or bug number at time. I just haven't written the bits in process all to do that. But it is smart enough to figure that out. The only thing is that it'll be a little weird because the message it'll actually nominate as a summary won't be one that normally shows because it won't know what the file display message is at that point in time. But yeah, so yeah, hopefully the summary will help a bit for those hideous bugs. Another thing that maybe would be useful is for new users anyway is to perhaps have on the entry page more pre-made queries that display bugs that need help, for example. Bugs that are tagged help and stuff like that that are maybe sorted by age or something so that users who want to get involved in Debian have easy things that they can go look at or even maybe we need to automatically nominate packages that need help because they have large numbers of bugs. That would be another option too, so that might be useful. Another longstanding thing that I've been trying to do that I still haven't had time is statistics on the BTS. So if somebody wants to help with that or perhaps it's better in one of the databases, that would be something that would be really, really useful to display on the BTS so that we know which packages are doing badly and which ones are, you know, or the maintainers are overworked. So I don't know somehow we maybe could help if it was possible to tag a bug as needing user feedback and occasionally pinging the user to make sure he gets them back. Yeah, currently that's more info is the tag for that. Whether it automatically does anything, that might be useful, but yeah, Mr. John. Yeah, I was about to mention this more info stuff because while traging bugs, often you come back on old bugs and you try to re-contact the bug submitter. So the bug becomes more info, but at some moment, some teams want to close bugs where when they don't get more information. So for some other packages, we were using user tags to insert a date, kind of expiry date for the bug. After the date, this date, we will close the bugs. Well, we did that with user tags, but that's not really practical. So I could imagine some thing setting an expiry to the bugs so that we can view all bugs that for which we have set an expiry date and the date is passed for a while. Yeah, that might be a great thing for the user values to sort of play with. And it may even be that if this is something that a lot of people want, then I'll just add it as a normal field. So, I mean, how many people would like to have, I guess it would be sort of to combine it, a point at which the bug contacted you or the submitter or maybe an automated message that said, hey, whatever you wanted X to happen to this bug at some point in time with that. How many people would want some sort of auto notification for bugs after a certain time? Yeah, okay. Quite a few people, so. Okay, so I guess that's something that instead of just using user values, I'll just make it a new field. And so that it'll, I don't know, I haven't quite figured out or thought through, but whether it emails this the bug number itself or something like that, and of course, anybody who wants can make extra tools like this too, but it'd probably be useful in a cron job or something. Okay. Hi. Something I've seen in other systems, I don't know how easy or hard it will be to implement, just allowing some custom colors to be used so you can group bugs, you can even attach your own key to it, so for example, if we've got bugs that we would really like help for more easy things, we can throw at people, if we can actually make them highlight. Yeah, that might actually be useful because I've sort of started switching everything to CSS, so it would be possible to add CSS to some of those tags, like if a tag is help or if a bug is easy to fix or something, something like that, that might be useful. I mean, I don't know, currently there's no way to, for a package to say, okay, here's some extra bits of CSS that I want you to run on this page. I don't know if that's wanted or not, but it might be worth thinking about. Yeah, it's in the front, here we go. Some browsers already have that kind of support because you can do it on an individual basis, but a more widespread implementation of that would be better. Yeah, especially with all that, if anybody comes up with better ways to displaying, I'm by no means an expert in HTML, CSS, or JavaScript, so please, mock-ups, if you don't know how to write in Deadbook, that's fine. Send me mock-ups, that works too. I can't say that I'll do it immediately, but at least I'll be likely to figure out what I'm supposed to do with the things. Yeah, one problem I have with the user text feature is that it's very difficult to debug or use if you don't know exactly what, so things, it would be nice if I could have something, like show me all user text of all users that are currently in this package so that I can spot if two different list addresses were used or stuff like that, so that it gets easier to unify the user text for your own purpose. I don't know if, I mean, it's not actually private information or stuff like that in this regard, or? Yeah, I mean, one of the two features that people have asked for that I want to implement, I just haven't had time, one is to regurgitate user tags, so you can send a mail in via a request or something, say, okay, give me the user tags for it, and it'll give you the mail that you would send back to set your user tags and user categories and user values exactly as you have them set, so you could then edit and then send the mail back in and it would be done. So that would help a little bit for that. As far as displaying all available users, there's no way currently to do that, pretty much because you have to select by user, but it would be something that maybe worth useful or maybe useful is doing a reverse index on a bug, so okay, if you have this set of bugs, these are the users who have user tags set on it, and I would raise an objection to allowing people to query the tags that have been set by arbitrary other users because when that feature was implemented, it was certainly presented as being private, and I don't want anybody to see my submitter is on crack user tag. I'm sorry, my hypothetical user is on crack user tag. Yeah, okay, that might be something then that we consider encrypting or something. I don't know, because right now it's possible to get it because it's all mirrored and it's just a text file, so if you know what you're doing, you can get it and see everybody's user tags. So I mean, maybe if there's enough people where it, I know certainly if anybody starts complaining about the fact that somebody has set a user tag, I will immediately go and start implementing encryption on it, especially when I add priorities, if somebody starts complaining that oh, you've set this bug priority low, then that's it, then I'll fix it so that if you want, you can make them private, probably by some cheesy GPG encryption or something, I don't know what, but whatever. But other than that, you should sort of assume that they're going to be sort of public, so it's not like your email address is hard to figure out. Any, anything else? So who's been involved in triaging here? Do you have anybody who's been doing some serious triaging work? So that's one of the other things that I'm kind of concerned is that we don't have enough people who are doing triaging on packages, and that's something that maybe that's probably a fault of the tools, possibly we don't need better documentation or something, but it would be nice to get more users involved in triaging, because that's something that users can do really easily. I mean, figure out, can they replicate the bug? Can they reduce the, that's exciting. Can they reduce the test case for the bug? Stuff like that. So is there any ideas on things we can do to get more users involved in triaging the bugs? It's just a matter of documentation or what else can we change? I would say as a user for a long time, I was just didn't doing, didn't do any triaging because I didn't feel like I was entitled to modify a bug or merging or, I really felt it was like it's a maintainer's property and I shouldn't do it, which is. Yeah, yeah, I mean it's, I guess that's something too that sort of is changing because especially in big packages, the maintainers sort of have things, if the maintainer sets it, that's one thing, but I think, at least from my perspective, if the maintainer hasn't set it, then it's probably reasonable for a triager to come by and do it, but if the maintainer says, oh no, this is the way it should be, then that's the final thing. But yeah, I think that that is something that sometimes users are afraid to modify the bugs. I think it's also a usability problem, meaning that most of the user have no idea of how to play with the bugs. And so, I mean, mail is wonderful for us and we just reply to mails and that's the message for the control bot, but one which arrived to that page have no idea of how to play with the bugs. So I don't really have an idea of a solution, but I think it's part of a problem. Okay, yeah, anybody, I mean, do we, what would be, I guess, really cool is if somebody would write a set of tutorials, whoever is working on triaging, write a set of tutorials aimed at users, especially with aim to replace the documentation on bugs.dev.org, because I know it's technically correct, but not very user-friendly, as it were, unless you already know how it works. Something I haven't given enough thought, but I'm just mentioning, what about a web form that could create the control message for you? Not actually mail it, but web form that you just pick the severity, the new severity, the tag, click some boxes, and then just give you what you should be sent to the control bot. Margaret, did you do some work on, yeah, so Margaret, last year, did some Google Summer of Code work towards that, maybe she can comment on that. Yeah, I worked on something that sounds exactly of what people are suggesting. I couldn't finish it, but it's functional. It does send control messages. It doesn't yet submit bugs, but it can work to say close or change severity, or other stuff that you do with control, it works. It's in Alio. It needs some extra work, which I planned to do this year and never had the time, guess why? But I think it could help. It uses the SOAP interfaces stuff. Yeah, so that would probably be a good point for somebody who is interested in that, or to help Margaret make that so that it's part of the Debugs pages. And that would probably even solve some of the issues with new users getting used to playing with bugs, because it's easier, slightly more things that they're used to than writing out mails and control languages. So surely that can't be it. There has to be other problems. Are there any bugs that people have filed on Debugs that you want me to fix soon? So I can close every most of the other bugs now. I don't know, maybe you have already fixed it, but it's being able to remove user tags from an archived bug. Yeah, that's a bug that I need to fix. So I don't know if I fixed it yet, but it needs to be fixed. That it does some sort of sanity check that the bug exists and it's kind of stupid about it. So that needs to be changed. I think Luca, this week, some action where he unarchived a lot of RC bugs that were archived, but according to the BTS, not closed and testing. Or unstable, yeah. Or unstable. Is that a feature or a bug? I mean, can we avoid this in the future? Yeah, so the basic problem is that some bugs, when we archived them, they were close and unstable. Then, generally in the case of NMUs, the NMUs were not properly acknowledged. Where acknowledged, I mean, the change log from the NMU was not kept intact. So the next version that was uploaded was uploaded without the NMU. So the bug was fixed in the NMU. It was not fixed in the previous version from the NMU. And the next version that was uploaded to unstable didn't contain the fix, or according to the BTS, contained the fix. So because the BTS can't see forward in time, it doesn't know not to archive those bugs. And so it's a problem that that fixed by unarchiving them. Because now they didn't do anything else but unarchived them. So suddenly now they show up as being buggy. In most cases, the underlying bug probably has been fixed. It just needs to be marked as found, or sorry, marked as fixed in the appropriate version and then it'll be perfectly fine. But yeah, that's something that the BTS perhaps, maybe weekly, should be unarchiving bugs that are present in unstable. It wouldn't be a big deal for me to do that. But yeah, that's why that happened. Yeah, I knew about it before, so it was a bug that the list didn't have all the package information, but it should now. So you shouldn't get mails like that without the bug numbers on them anymore. Or without the bug titles, so hopefully. If you do, bounce me the message and I'll fix it because it's my fault. Okay, anything else? So if anybody else has any more ideas or thoughts, feel free to pipe up now, or go ahead, I guess. Yeah, you're good. No, it was more like, how can someone not very, someone willing to help on bugs.dbn.org, work on someone that help without having to get a very in-depth knowledge of. Yeah, sure, the major things that, it depends whether you wanna work on the code or not. If you wanna help with the documentation, it's all in WebWML. That would be really great to work on that. If you wanna work directly on Debugs itself, the source code for the running version, I'll show you where it is. It's, so we also have, let me show you first the Teams page because you wanna look at that. So this is the Debugs team page. Explain some stuff. Hello, scroll please, thank you. The source code is here. This is the branch that's actually running. I mean, actually running, there's some links to it. So if you wanna know why there's a bug in Debugs, it's in this code, hopefully. And then stuff that I'm working on is here. And that's the actual code that I'm developing. And usually they're pretty close, but if I've made huge changes that aren't ready to be running, then that's where they are. So if you wanna contribute, please check out the bizarre repository, make whatever changes, send me patches. If you make good patches, then we'll make you a Debugs slave too. Okay, yeah, go ahead. So how much of this is in the current Debugs package itself? None. Yeah, that's actually a big problem that I am planning on rectifying, hopefully by the end of the week, with an upload to Experimental, and eventually by an upload to Unstable. It's, the problem is that I've, the package is probably easier to install than the current version, but it's not really in a state that I would personally be happy with uploading it and trying to support it, so. Yeah, if I have enough. So yeah, that's one of the reasons why I haven't uploaded, and that's why if you looked at the change log for Debugs, let's see, let's open it up, it's kinda funny. Yeah, that's why the change log is that big. So, yeah. Okay, we're sort of running towards the end here. If anybody has any questions or things that they thought about later, you can always harass me on IRC. Most of the Debian team hangs out in pound Debugs on OFTC, or you can talk to me later on, I'm around everywhere, so just come and say, hey, this isn't working the way I'd like it, and we can hopefully make changes that'll make things work better for you. I'll also try to put together a bits from the bug people announced message with some of the new features too, to summarize what I talked about in the talk and what I hit on in the buff as well, so that you at least have a mail that you can point at when things go wrong. Regarding the new features, there were complaining on IRC about the new pop-up, which is counter-intuitive. Not the pop-up itself, the place where you have to click. They say it would be better if you go and close like a plus sign, so it's more evident that you have to click there. Okay, yeah, I had a plus sign originally, and then I took it away, and then, okay, I can put it back. Does really make a difference. Okay, is one more, a couple more people. Well, not so much a question, but just thanks, excellent work. Thank you. Well, thank you guys for paying attention.