 Hello. All right, I think we can get started. So this is contributing to OpenStack 201 for the not-so-new contributor. I'm Scott D'Angelo. This is Andrea Rosa. What we're going to do is kind of share some of our experiences as we move from being simple bug fixers and doing just a review here and there to get more involved in complex features, what it takes to really drive things in OpenStack. We also did a bit of a survey. We asked some people, a lot of core developers and a couple former PTLs and some other folks at about our level, just some advice, things that were perhaps annoying, the effect and influence of people's names, people's reputations. So we'll share some of that with you throughout this presentation. Welcome, everyone. Just a brief introduction of ourself. I'm Andrea Rosa, software engineer at HP. I work on Bristol Nova Team. I'm a middle-class OpenStack citizen since Diablo release. I work on HP public cloud. Then I moved to a new project, Helion OpenStack, which is a cloud solution based, of course, on OpenStack. You can find my details here. And I'm Scott D'Angelo. I'm also with HP. Work mostly on Cinder and some Cinder Nova interactions. Also started with the public cloud and continue on with Helion. And I'm Scott D'Angelo on IRC. So we define ourself as mid-class OpenStack citizen. What that means? Don't get me wrong. It's nothing to do with social classes. We don't want to give any bad connotation to this term. It's just a name, a name which found when we were working on this presentation. I'll try to explain what we mean by this term. So if you see the OpenStack community, we have three kinds of contributors. We have a newbie. So people just join the community. The community is very welcome for new contributors. They need to understand how the things work. They need to understand the processes, how to get it correctly. It's likely that the first contribution is about an easy back fix. Then we have Rockstar. So Rockstar, people who knows everything, they know the code, they know the processes, they know the workflows. They have amazing statistics in stack analytics. And between these two categories, we have the middle class citizen. Yeah, so this is where we found ourselves. Our employers asked us, hey, we want this feature in. We need to start driving things in OpenStack. So things get more complex. You have to start writing blueprints and specs. You have to know how things work well enough to even know if this feature is going to get in. You have to start using some influence with people to get them to even look at your patch. And so we felt like there was a bit of a gap between the welcoming for the new person and all the things you'll find in the documentation about submitting your first patch. And of course, people who already know all this, we had to kind of struggle. So that's kind of what this presentation evolved from is our experiences. So one thing we did was we asked people, what advice would you give? We asked, what things annoy you? And we asked about the influence that people's reputation has when you look at a patch and you see the name of the submitter. How does that affect your thinking? So that's the survey that we sent out. So it's all about sharing. We want to talk about good things, but even we want to talk about what we don't like or things that we think they are not really good. So what we don't want is to get banned by the community. How are these talk? Yeah, so and some of it is about sharing the pain if you haven't been through 50 patch sets of trying to get a fix in and you get a minus one on patch set 51, you don't know the pain. So I think everybody goes through that at some point as things get more complex. And some of it is just about saying, you're not alone, this happens to everyone, including experienced people and core reviewers. And finally, we know that technology doesn't give hug, but we do. So at the end of the presentation, if you want, we are here and give away free hugs. Let's go. So OpenStack is a big community. It's a lot like a traffic jam. People are helpful, people are encouraging, but people are busy. So in order to get things done, you kind of have to be as efficient as possible because you have to have this understanding that everybody has a completely full schedule. So there's certain things that you can do to speed things up and there's certain things that happen that slow things down. So the first thing is a bad review. I'm not talking about minus one because your pet has some issues you need to have some constant. That's how it's supposed to do, right? The review process is supposed to be used for these things. But we identify some minus one which we think they are annoying minus one and we think we should avoid to give this minus one because they are a problem. They really slow down the process which is already a long process. And we give you some example later. So the other thing that slows down the process is you, the submitter. I do this, I'll write some code, I'll work through it, I'll get the batch ready, it's getting to be the end of the day and then I hastily push it with get review but I didn't think about the unit test or I didn't think about the commit message and by not being thorough, it pushes the whole process back and you get a bunch of minus one. So part of the encouragement is just to think proactively about all the problems that you're gonna cause and avoid the things that you know are gonna give you minus ones or cause issues. Third factor, code reviewers. This is one of the risky things to say but it's a reality, right? They are at the end of the process, they are the people with the power. They can say, yes, this batch is good, can lend or say no, no way, this batch is not going to lend. And so try to understand who are code reviewers. Someone say code reviewers are the new unicorn, they live in a secret land and we know for sure that they are responsible for the dinosaur extinctions. Actually, we want the myth, the myth. We think they are normal people, very capable. We've a single big problem, time. They need time, they need to do reviews, coding, stay on IRC, following Medinix thread and they are usually a small group. I talk about NOVA especially, I think it's too small, right? One easy solution is to make the group bigger but I understand it's another of balance between the number of people and they need basically to be on the same page, right? So it's difficult but simply they don't have time, they can't cope with the high demand request of reviews coming from the community. Even other solutions. Yeah, so this is discussed and there's summit sessions about how to scale the teams and Neutron has created a system of lieutenants who have the power to promote core reviewers and delegate things. I don't know that we have any great answers there but that is what the biggest problem we see is that the bandwidth of the core reviewers is limited and at the end of the day we're struggling for their attention as people who aren't core reviewers. We sort of need their approval so some of the options have been discussed but nobody's really solved this problem yet. This is the first advice we want to share with you. We've got these advice from the survey. Basically say be prepared to stand up for your decision so when you submit a page and remember if you've got a minus one it's not the end of the world and remember they are not always right. This is too for core reviewers but for any reviewers in general. If you can clearly explain what your page does if you can prove that this is an improvement for the project or you are fixing a very nasty buy. I've seen in the past core reviewers change their mind from a minus one to plus two and yes there's another thing to say about this if you have some concept well try to be open mind and prepare to listen to the reviewer and try to address the concern as well. The second thing, this is an annoying thing reported. It's all about respect the other person so sometimes you can have a really rude review. To be honest it's not very common. Just let you know that if this happened you can report this to the community and again if you are on the other side of the fence don't be rude but try to explain your concern and try to give suggestion to the person. So one of our respondents when asked what was annoying had a great thing to say. He said nothing is annoying. It's really an opportunity to teach and to learn. And so I not only found that very welcoming and nice but I also thought that was a good thing for us to do is to kind of reach out to people when they ask questions on IRC and help them and instead of seeing some of this sort of negative reaction that sometimes you'll see where people say read the manual or this or that. See it as a chance to help someone along. So this was a former PTL and a fairly prominent citizen who responded this way and I thought that was really great to see. So we saw this over and over again in the survey. Bad commit messages or a commit message that doesn't do a very good job of explaining the patch's purpose. It's the first thing people see when they're reviewing. For me it's the last thing I write when I'm submitting a patch. So there's a little disconnect there because I don't put the effort into the commit message that I'm putting into the code which I've learned to stop that practice. So there was a thread in the mailing list. I don't know if you saw this where the complaint was if the commit message is over 72 characters minus one and the thread went on for weeks and weeks and weeks and is this something you should do or you must do? But it kind of shows how silly things can get. It's like, am I looking at this on my phone? Why can't it be more than 72 characters? But the truth is it's supposed to be less than 72 characters so rather than argue about it I mean at least know the rules and think ahead when you're submitting your patch because it's almost gonna see it immediately and tag it. By the way, I think it's still on this discussion. It's still going on. Another thing is that was annoying was that the commit message doesn't explain the patch. You know, as a developer you'll look at Git log and you'll see what the patch was all about. I mean the commit message ends up being the documentation for the patch. So if it's just sort of a brief thing that doesn't necessarily get you minus one but it doesn't endear you to the community. You do see lots of things rejected because of issues in the commit message when it's really just documentation. So do a good job I guess is the end result of people complaining about this. One more thing about commit message. Maybe you already know and if your change is implementing a spec or a blueprint it's required to have a reference to the blueprint. If it's a bug fixing again but it's a requirement to put a reference to the bug. But do not expect the viewer to go and open the bug description or open the specs and read through all the description. All the relevant information our viewer needs to get the context of the patch must be on the commit message. Or the fix for this annoying is quite simple. Just we have guidelines, read them, use them. We have put a link here just in case if you need. So while reviewing I found one thing I was reviewing that just kind of stood out. The author gives examples. He starts talking about what the patch doesn't commit message and he goes on and gives a bit of the code and it's quite a bit. But when you read it and when it's preparing you to review the code it really sets a nice tone. He found a sequel alchemy bug that he references so that you don't get caught up in that snafu. And then of course he adds his bugs and his specs and his blueprints. It seems like a lot but when you're actually reviewing the code it's a real pleasure to see that. So everybody likes to review his code and he gets a lot of attention and he actually became a core reviewer last release. I'll try to use this. Okay, you work on it. This is kind of gently reminder. We all know how important tests are about this guy. So yeah, good. The rule is simple. New future, you must have unit test. Okay, so test everything. As regards the functional test, probably they are not so mandatory and probably it really depends on your change but if you can write a functional test as well it's nice things to have. Okay, thank you. Yeah, so you may have seen this or encountered this but it's pretty easy. No unit tests, you get the minus one immediately. So once again it's one of those things where if you should plan ahead maybe write the test first because you're gonna write the test sooner or later. Another thing too is your test will fail Jenkins or your patch will fail Jenkins if it doesn't pass toxin pep eight and you can run it on your workstation but I can't count how many patches I see up there for review and then it fails and you look at why it failed and it was a hanging indent problem or lines don't differentiate. And some of those problems to solve I found difficult but there actually are tools like auto pep eight, VIM plugins, I think Emacs probably has a plugin that will fix it for you and it'll tell you what to do because I don't know about you but I've had the hanging indent one where you'd like subtract the character still doesn't work, subtract the characters. It seems impossible to fix but there are tools out there so. Now I have an amazing animation here. Oh, yeah. As we got backfixing. Amazing. We still think 99% of the time you have to write unit tests for backfixing so we have designed this workflow. Backfixing write a red test. So write a failing test. You can copy your test maybe and put in the back description, fix the code, make the test pass and if you've done it, yeah, job done, great job. We think this simple workflow can make the reviewer's life easier that means maybe your pets will lend as soon as. And we call this test-driven development. So you heard it first here. I mentioned at the beginning of the presentation we want to talk about annoying minus one. And so we think that probably some people think that there is always a reason to give a minus one. So if I open a snippet of code now, I'm sure we can find something we don't like, right? Another percent sure. But that doesn't mean it's a good reason to give a minus one. Yeah, so in this advice, I mean, you can't really do things about other people's reviewing and knitting and getting on you for a typo or something. But as reviewers, we can kind of make sure we don't do this. One of the things we started doing in Cinder is encouraging people not to give a minus one for a typo or something that doesn't affect the code. Just say, hey, next patch set, if you upload another one, please fix it. Otherwise the code looks good. You can give a plus one even if there's a typo in the code or you can give a zero if you don't want to give it a plus one. The thing about minus ones is it flags it to everyone else and everyone will not touch that code until the minus one is removed. So you gotta go through another patch set for some little typo or you've got an extra space there. It's a cultural change we're trying to do. I don't know how well we can affect all the services, but it's really pleasant to have gone a few iterations of this and gotten to the point where people who do knit on a typo or an error, they kind of get chastised. So it's starting to work. Yes, don't forget the power of zero. You can give a zero. I really like the idea of seeing the community to not give a minus one for a typo and especially for native English speaker like myself. So we speak English more or less. Scott speak slightly better English than mine. But don't forget we are an international community, right? So I'm not saying it's not good to give a minus one for a grammar mistakes. I'm just saying if you get a minus one on patch set number 25 because the grammar is not correct in a comment in your code, it's a bit annoying, right? So yes, please bear with people like me don't speak English very well. One more thing, this is a thing that happened to me about use conventional English. So I got a review from one colleague and he put a line, say something, this is super fish oil. Well, it took, I'm not joking. What did you? It took 20 minutes to me to try to understand what actually he meant and they gave up. And so I walked to his desk and say, sorry, can you explain? And apparently it seems that if you read super fish oil quickly, it sounds like superficial. He was obvious to him, he wasn't to me at all. So yeah, super fish oil is not superficial. So one more time, please use conventional English. We are an international community. This was one of the things both given as advice and pointed out as annoying is people who don't read the history of the patch. I was just talking with a core reviewer who was pretty angry today because he had flagged something, don't do this, you've got to remove this parameter. And then several patch sets later, the parameter came back and someone approved it and the code got merged in. So reading the history of the patch is important for a lot of reasons, but when you jump in as a reviewer, it's good to read the history of the patch because then you'll know why things were addressed the way they are and that you'll avoid that kind of mistake. It can be tedious to do, but it's one of the things that came back in our survey several times. So one thing that kind of worked for me and it was given out by someone as advice, I wanted to get more involved, I wanted to do more work upstream. So the thing to do is to find out what work is needed. There's certain areas I think that are either sexy or that they're very important to your employer and everybody else is doing the same kind of work because you're all storage vendors, for example, in Cinder. And then there's certain areas that are neglected. So I found myself kind of asking around what needs to be done, what's neglected. Because Cinder is pretty tightly coupled with Nova, there's a whole lot of interactions with the way volumes are managed with Nova and Cinder and nobody was really working on it. So I started in on it, started working on bugs, started learning about it. It just opened up more and more things that needed doing. And now I have more work than I know what to do with, but it's all very welcome work. So I think the key takeaway from this person's advice is that you'll be encouraged and welcomed and helped more if you find some point of pain in the project that's neglected. And the way to do that is just to ask, go to the PTL. And I've seen in IRC several people say, hey, I want to get started, what can I do? And the PTL will say, here's something that needs doing and point them in that direction. So a simple way to get involved and to get involved in more complex things. You can decide to play some strategy to try to get tension from reviewers. We have analyzed some of these strategies and we think actually they don't work. So we call these silly strategies. Give you some example. That's interesting, a Swiftcore reviewer told me that they have found sometimes people prefer to give a safe plus one after a plus two from a core reviewer, which is fine, it can happen, right? But if this becomes an habit, probably this is not going to help in growing your credibility inside the community. The other things, this is funny, if you put a typo in the commit message, the title of the commit message appears on IRC. If you have a typo, probably you get an immediate minus one that means immediate attention from reviewers. Again, it's a strategy which works, but well, it's not very helpful. So another complaint that I've heard people say is that the obsession with stack-olytics for people that, well, you're not a core reviewer, but you have aspirations. And if you look at stack-olytics, you can see your number of reviews and how do you rank next to your peer and what's your ratio of minus ones to plus ones? Because those minus ones are valued and that's why we knit and get an easy minus one. I get to give a minus one. But when you really talk to people, the core reviewers are doing lots of reviews, so they see your reviews and they know whether you're given a good review and you're really very thorough in giving minus ones because you find flaws in the code. And they also know if you're the guy who's racked up a lot of minus ones because you're the one who catch all the typos. So I don't think you're really fooling anyone and everyone I talk to sort of knows that and acknowledges that when they're at the core level. So I guess the advice there is to not obsess over stack-olytics too much. I only check it like three times a day. That's my rule. Just the advice we saw over and over again is review, review, review. You want to be reviewed. That's how you're going to get your stuff in. But then again, you need to review, not only to show you're participating, but that's how you learn. So that shouldn't be any secret, but it is kind of tough to just drop in and say, hey, I've got this thing and it's big and I want it done and you've never even really done a code review. So it makes it kind of hard to get people to pay attention to your stuff if you're not paying attention to theirs. Anybody can review. There's an onboarding process where you sign the agreement saying you're going to be a good citizen. You load your keys up, but anybody can review anything as much as little as you want. You can review any project, any time, just as long as you go through that onboard process, which takes 30 minutes or whatever, and you're set. Respond to review as soon as you can. So when you submit a patch and you've got some comments in general, try to get back. And it's true even the other way around. So if you are doing a review, please try to babysit a review from the beginning to the end. Don't just leave a comment and then disappear. All these things help to keep the communication going on. And again, we notice that if the communication can go well, this helps and gets your stuff merged sooner. So one thing that was labeled as annoying is people doing kind of bullying or nagging to get things reviewed. We see a lot of this in Cinder because we've got about 70 drivers for storage arrays in the tree, and there's always new people coming on board because their company wants to get a driver into Cinder. Well, a driver's two or 3,000 lines of code. It's usually somebody who, the first time your driver is, you're new to the community. And we have a deadline for it, usually the second milestone. And so as the deadline approaches, everybody's on there saying, please review my patch, please review my patch. And there's only so many 2,000, 3,000 line patches you can review. So the secret is don't wait until the second milestone to start nagging people about your patch and don't wait to start asking questions in IRC and don't wait to start getting involved in the project. I mean, if you just have 10 lines, some simple patch, it's not that big a deal. But you know if you have a major feature or even more importantly, something like a driver, it's just not gonna happen that you can kind of nag people into getting it done. And I've seen several blow-ups where it's like, stop bugging me about this from the PTL. We'll look at it when we look at it. One other thing that can help in speed up the review process is the co-authoring. Maybe you know you can work, more than one people can work on a single patch. This works very well, especially if the other author is in a different time zone because again, the producer, the round trip of the discussion addressing concerns. But we want to give you an advice. Be sure that you are absolutely on the same page. Be sure that you both know what you want to implement, how you want to do it. Otherwise, the risk is the co-authoring just instead of giving your game, just give you more pain. Yeah, and so one thing about co-authoring is you have to be prepared to jump in if your partner is not responding to reviews or if somebody says, here's a bug. I mean, you gotta change the code. You need to be ready to jump in, put the patch up and get it up there because especially if you're working for different companies or the lines of communication aren't great because you're on the other side of the planet, it's kind of frustrating to see someone put something up there that needs addressing and you know the answer and you have the answer and your partner's not responding. So just jump right in. Another thing about co-authoring is that we discovered as well we kind of met while co-authoring a patch and you know, with the patch and then when we finally decided to do this presentation you know, we used IRC, we used GitHub to share the slides. We used Google Hangouts and we met this week. We had never really met before. So you can do a lot of collaborating. We don't need Wi-Fi. Can you go to the next slide? All right, that's a neat. Okay. So yeah, so this particular patch was kind of one of the things that set us off for this. We were working on a spec with another person to get a feature into Nova. Changed the API, it was fairly complex. The spec kind of started in March. It went through patch, 53 patches from March. Then on to April, lots of people paying attention. Lots of core reviewers giving feedback. Lots of work. It went on into June. It went on into August. And then it got abandoned because at the end you know, the core team said we're not going to go in this direction. So this can happen. I mean, it was a lot of work. If there's any good news at this conference we kind of revisited some of the things. Found another way to get it done. Found general agreement at a session. There's still a lot of work to do, so it'll probably be two more releases before the work gets done. But I guess be prepared for this because not only does it sometimes take a long time to get things done, but sometimes it takes a long time and you don't get things done. And I don't know that there's any real way around it. I just noticed my comment. I'm a bit lost here. Yeah, so same thing comes back in advice. Simple things can take weeks. And complex things can take months or multiple releases. So even if you know everything, if you know the process, if you know exactly the code you are working on, remember it's never easy to get something merged in upstream. I'll give you an example. So it's something that happened to me a couple of years ago. I thought I found a typo. So an easy fix. As you can see, it was a single line code change. More precisely, I changed this field from update to update it. Say, okay, good. What could possibly go wrong? And that's the result of my patch. And, yeah, some numbers took me eight patches, 13 modified files, five weeks, and definitely not just a single line change. Yes, but got merged. So, you know, I guess that's why we do reviews, right? We think, well, this is just a one character change. So it's going to be simple. But during the review process, things were found. So that's yet another thing to think about is what you might be used to doing inside your company and like, I can get this done. I'll have my peers review it and then it'll be merged. And then tomorrow I'll start something new. Things are much slower upstream. Yes, you need to keep in mind, especially if your company asks something, we want everything now and for free. And you say, okay, I can do it. You make some promise to your boss. But remember that the timing of the community could be very different from the timing of your company. So you need to set the expectation very well. Otherwise you can end up like this guy, this is the review meeting. You are that one in front of the boss and say, sorry, it didn't land. We need to wait the next cycle. So maybe we'll have it in three months' time. So, yeah, be aware of this. Yeah. So another bit of advice about major features taking a long time. If you've showed up at the Design Summit here with an idea, it's not going to get into Mitaka probably. So that's just sort of one of the things that came back from advice is, you can start it on specs early. You can write POC code. The more prepared you are, the more likely things will get done. I had a simple requirement from my employer, Cinder Volume Manager. It's not possible to deploy it in an active-active configuration. We just needed to do a couple things to get it out of there. But as we delved into it, we discovered more and more things that needed doing and more and more features that needed doing ahead of time. So it's just yet another thing where what would have been simple if we did it ourselves upstream is going to take a couple of release cycles. But I was able to come at it prepared. I had some POC code and a spec ready a month ago. I was asked by the PTL, hey, will you have this stuff ready at the summit? I said, well, it's already ready now, and everybody's like, oh, what a surprise. Somebody didn't wait till the last minute to do it. So it behooves you to be proactive on these things. Yes, nobody did a very good thing in this cycle. They decided to open the submission and the review for the specs for the M release in Liberty. So the idea was, we start the review process and if you get something we need to discuss in more detail, just come to the summit, you can discuss it. I think it worked very well. Oh, that's interesting. We asked this question in our survey. How much are you influenced by the author of a page? Not all, but most of the answer we got is, yes, I am. I am. And I was disappointed at the beginning because I thought everyone should have exactly the same chances to get attention from a review. Then thinking more about it, it's just part of the human being, right? So if you know, if you have any kind of relation with a person, it's much easier to get in touch with this person. You can disagree with these things. Nothing you can do. It's how it works in the upstream at the moment, yeah. Well, that's it. That's right. That's why it works. That's what people said, you know, that they know a person submitting a patch has a certain expertise, or if they don't know them, then they have to scrutinize it more. So that's why they pay more or less attention to things that are known. But you can build your own reputation by doing things and paying attention in IRC and chiming in there and then people recognize your name and then when you submit a patch, they recognize your name. So it's, as Andrea said, it's a human thing. You have a reputation and you have to cultivate it. Yeah, so build your reputation. Don't be shy. Go out. No, no. Make a connection. It is a really good opportunity. You are here with Sami to meet new people. So this is my favorite picture, so I insisted that we put it in there. It's the Tower of Babel. It's from a 15th century painter, which kind of reminds me of Open Stack. You know, the story of the Tower of Babel where they built a tower that would reach the sky and they had to get more and more people working on it and then the people were all speaking different languages and they couldn't communicate and they never built the tower. But we're hopeful that Open Stack will continue to grow and yet at the same time we can keep the communication going and so I stuck the slide in. We have a little thank you for our people. We got the images from... And now Scott is taking all your questions. Yeah, if anybody has any questions, we've got a few minutes. Thank you. So you mentioned that you had to put in your first blueprint and your specs. What route did you take to figure out how to do it? Reading other blueprints and specs and reviewing specs because you can see what kind of things people look for. But you know, you can go into the Cinder Specs, Nova Specs, one of these repositories and read the specs from, you know, just read a couple of specs, especially if you're changing the API, find a spec that changed the API and see what kind of things were addressed. And then you'll at least have some examples. Just like when you rip off someone's code, you can kind of steal from someone's specs. And Faye, just put specs. Be prepared to get a minus one. Don't take it personally and iterate. Okay, cool. Thanks. Sure. How patient should you be when waiting for a simple review to get merged? That's a tough call because you don't want to nag. But you know, if you don't do anything, it gets starved out because there's no algorithm to prevent starvation of reviews, right? So I think one of the things that helps is know the process, you know, like Nova has a real specific way to look through bugs and they tend to do a pretty good job where Cinder doesn't really have a way. So it's a tough one. I don't know that there's a hard answer there. I think in IRC you got to find a buddy. You got to find someone who participates a lot and maybe is a core, maybe not. You got to sort of say, hey, can you have a look at this? Or even get people you know who aren't cores or whatever to review it and, you know, say, hey, this has five plus ones. Can someone have a look at this? So it's okay to poke and prod a little bit. I mean, you know, obviously, things do get forgotten. Yes, I think you need three things, patient, patients and patients. But you're going to have to do something besides just wait forever. If you go, for example, for Nova, in IRC you need a Nova meeting, you just say, okay, we'll have a look. Yeah, something that's probably worth being aware of with that kind of thing is that the projects, the PTLs do generate lists of patches and how old they are and how long a patch has been hanging around without being reviewed. So if you've been there for a very long time, you will pop out on somebody's spreadsheet as being this one hasn't been looked at. I don't know if all projects do that, but they do that. Yeah, in Cinder there's a dashboard that the PTL says, use this dashboard, it lists things that need more review, things that haven't had a review in five days. Whether or not people use the dashboard, that's a little tough. There's a balance there. Other questions? I have like five questions, but I didn't want to hog the microphone. We'll open it up between your questions. So I guess my next most important one would probably be when you're changing some code that doesn't have existing tests. So this question is from two sides. Like one is when you're the developer, should you just assume that you need to write the tests first? I mean I know generally that's obviously the case, but I'm talking about a situation where there's like a whole framework missing in that area and it would be like a massive amount, like another project in itself to add the tests. And from the maintainer side, because I'm in this situation that I've just started up this new project, but it's inherited a code base from somewhere else, and at the moment I'm the only maintainer, which is not ideal, but I want to change stuff and I want to add a test suite and I'm faced with the decision should I spend a huge amount of effort writing a test suite from scratch before I change anything, or is it okay for me to start changing stuff in the first place? I think it depends on the project. The example we put about my pet is for a single line change. Actually, basically because of this problem they found that we are not testing for that bit, of course, and they asked me if you want to fix it, just write tests. You have to write tests, and that guy said yes, he's a co-reviewer, he's one of the bad guys. You need more resources is what you need. You need... And it is a way to get involved and endear yourself to the community. People love tests, but they don't love to write them, right? And don't have time for it. The more you can contribute on the testing end, the more of a hero you'll be. I think we are out of time. Free acts now, thank you. Thank you very much for coming.