 So, today I'm going to talk about reporting MySQL bugs and maybe you are wondering why. I work for MariaDB Corporation and it's actually a technical enough conference so I would expect that of all the talks that I've suggested here, people would pick up something else like something about InnoDB, something about MariaDB or something about using GDB to study hashes, user variables or whatever. But they picked up or might talk about bugs and I think that's because people care to get contribution to MySQL and at the very beginning of this conference my colleague from Foundation told you that it's actually not easy to start contributing to the code but it's really easy to start contributing by reporting bugs and problems you have with MySQL. So, maybe that's the encouragement to divide the audience. If somebody wonders why me is talking about bugs, there are historical reasons for that. I worked at the bug verification team back in MySQL and was involved in bugs processing from both sides before the bug is accepted as real. So, the member of that group and then later at defining priorities for bug fixes. I stopped doing it at MariaDB but I'm still interested in this stuff so I have a blog that is called MySQL entomologist. I have MySQL bugs as my nickname on Twitter and everything so I'm watching bugs. Right now I'm interested as a community member in finding new stuff that was found not working well and at the same time historically I also try to care about the way bugs are processed, about the proper procedures that were once set up and I think partially they made MySQL popular. The fact that there is a public bug database and it's processed consistently. It's not like you put thing there and it's forgotten for four decades. No. And I would like to make sure it's still the case. So, it's not the first time I'm trying to represent this topic. It's the first time when I have more than 5 or 10 listeners about it and in such a huge audience and in 20 minutes. So, it's a bit hard but the agenda is quite typical. Nothing new here. I will just add a couple of words about reporting to MariaDB as well so that those people who work with MariaDB or who believe that MariaDB can do something useful to common code or for MySQL in general, still can get involved. So, my main message is simple by the way. Report bugs in public. You have these great public bugs database available for more than 15 years. So, it's a collection of interesting stuff. Some bugs are fixed. Some not. Some include walk arounds and comments explaining how people think when they solve MySQL problem. So, I say report bugs in public and I say this also specifically and not only to community users. I tell these to customers as well. To Oracle support customers, to MariaDB support customers. The fact that you can report your problem that may be a bug or not to your organization that you paid money for and they can deal with it privately with you is not a replacement. It's an option. It's a good option to have and you may get a solution but it's not a replacement to reporting a real bug in public. For two reasons. First of all, public bugs database is searchable. So, you contribute to community. We are all feeling out public founded and public, it's like writing at Wikipedia explaining stuff there. We are building a knowledge base and the fact that it's publicly available also help in a way that if your support provider fails for whatever reason, do not want to fix the problem or do not reply fast. By making your bug report public, you attract attention to other support providers, to other engineers, to community members who can solve your problem faster than your support organization. That would be not your problem. That would be your great result. It would be a problem maybe if your support organization, if they are not fast. But you should do it. Here are the links to public bug trackers to MySQL forks I care about specifically MySQL itself and a bit different software used by both Percona and MariaDB. We work on bugs in JIRA but these are still public bug trackers and I encourage you to use them. These slides are already available. Whenever you see on my slides anything underlined, it's a link so you can click and I care specifically for the link to work at the moment of writing. So, the talk was titled how to create useful bug report. I say report whatever problem you have. That's what people do. But if you just complain, something does not work for me. Or if you do not try to create your bug report as useful one, chances are high that it will hang around and be forgotten and you will not get any useful feedback. Neither from Oracle, for example, nor from other community members. So, what's a useful bug report? Historically, I highlighted many, many features of what should be a useful bug report. But for the purpose of this talk, I highlighted one. So, you should clearly state what your problem is. Not that MySQL hangs clearly in a way that it can be understood what the real problem. Not that you are badly affected. Not that it does not work. But you should try to explain what you did, what you expected and why and what you got. And then, in a useful bug report, surely you can provide a lot of additional information. You can do your own research and it's appreciated. But it's not mandatory. If you already can explain what your problem is, why you expected something and got something different, it's a good start. Then, the best you can do is just cooperate with those people who will try to ask you additional questions. So, it's already useful. Also, do not omit necessarily important details. Like, for example, I see a huge problem right now that, for example, Oracle does not mark real regression bugs. It means something worked before, but stopped working, worked in some older version, but does not work like before. These regressions are like not highlighted. Even though in bugs database there is a quite popular tag for this and it's called regression. So, do not omit this important detail like something worked for me in older version, but does not work in current one. This is important. If you tested more than one version, put it and got different results. Put it in your bug report. So, these are important details. That's like a message for you. What else? I told you it's a public bug database. It's searchable. So, what's really annoying for people who work on bugs is to get the same reports again and again and again. I don't know. Workbench crashed on specific VGA card or whenever I have a table without primary keys, row-based replication is slow. There are probably 100 reports like that. Row-based replication is slow. It starts like that. And very soon we can find out that there were missing primary or unique keys and recently there are some features to overcome this problem. But that's it. Whenever you have a problem, before you report, try to search at the bugs database. Try to use Google specifically put it onto bugs database. Try to use whatever search means you have. Another important thing, so duplicates is something nobody likes. Another important thing is surely to read specific instructions because depending on what kind of MySQL software and where you report, you may get some additional suggestions or guesses or best thing to have. In case, for example, of MySQL workbench, you should put some details about hardware that it helps you to generate in a special report. So do not omit that stuff because it's key of a key importance. And again, one old historical message from me on bugs is that every who report bugs would read a recruitment, how to smart questions at least once and try to follow the general guidelines of how to discuss problems, how to try to get help from geeks, from open source developers. It truly applies to MySQL as well. So as for search, it's as stupid as this and it works in majority of time. Just put it side bugs MySQL.com and whatever keywords you have. If you have a crash, you will get a stack trace in the airlock. Try to put some interesting unusual items from there. Not the entire stack trace. It would not help, but some function names. For example, closeFRM. Search for bugs and closeFRM. For example, MariaDB, if you prefer, you may go to another site and you will find quite specific heats about known problems. Please also do not ignore search tools. People don't like them searching in MySQL bugs database that is implemented in form. Bugs are split into categories. You can search for fields like SAIN name. You can put version here and you can get results faster than you do via Google. You will get hundreds of pages of bugs to review. While we are bug system specific search, you may get way less results. And I always say, if you use Google, please check carefully at least three, five pages or 30, 50 first heats. You probably know if it's some new feature, there is no point to look for a bug that is five years old. But otherwise, do not look only for three, five first roads. So what's also great? It's great to see what your problem is. Ideally, your bug report should get enough details for anyone in the clean environment. They are not on your hardware to reproduce exactly the same problem using the steps provided. So clean environment matters here, because if some crash happens only in production five months after the restart and not before, it's great to see, it's good to see a stack trace and whatever, but it's not yet a repeatable bug. The bug is that it's not repeatable is something that nobody will ever know is it fixed or not. It stopped happening, but we don't know how it happened. We never reproduced it. We do not have test case for it. So the fact that some work around happened, it's no longer happening or some company stated it's fixed because of some additional new feature or some other development or covered by some unknown bug and should work now. It's not something you can rely on. So I have listed some tools that first few like DB Deployer that was already presented or Sandbox helps you to create those environment specifically for trying to reproduce your bug before reporting or in the process. So the fact that something happens in production and doesn't happen in clean environment is an important fact to include in your bug report and that's what I would like you to do. The more you know your operating system, the better you as a developer, the more tools you can and will use to prove your point. So in the bug report, you should prove your point that there is a problem, not stated. So in some cases for crashes, I told already that we need core dumps, we need backtraces from core dumps. You cannot put the entire core dump into the bug. It might be gigabytes in size, but you can put backtraces. So some useful tools, the more you know them, the better. Now you should understand, you reported the bug and it doesn't get fixed fast. You should understand what state it is in and why. So it's a brief slide explaining basic most important states for my SQL bug. So it starts as open, then ideally somebody assigns it for analysis, then eventually if it's found to be a real bug and a person assigned knows how to reproduce it, it end up as verified. And at this moment in case of Oracle MySQL bug, it goes out of your DARS, community knows nothing what happens in between this verified state and eventually closed bug, closed state that you can see in release notes or whatever else. Because in case of Oracle, the real work on bugs happens in private database. And we know nothing about it. On the other hand, you still can see some public states that are important for you as a bug reporter. If the bug is verified, you know it's accepted as real. If the bug was already reported, you would see it as a duplicate. If there is some question to you, you will get need feedback status. And the bug that got no reply for 30 days and need feedback status automatically becomes no feedback. And these are true for gotten bugs until somebody will add some comment to complain. That's it. So these are all highlighted final states. You should probably not insist more after the bug ended up in some final state. So the decision is made. If you have new argument, add them. But these are final decisions about the bugs. So this more or less applied to MariaDB and to Percona as well, who use Jira in Jira. Depending on project, you may have different statuses. And actually, status is quite specific there. So all they are highlighted. So the bug is formally starts as open. Then it's usually formally confirmed. Then it's in progress. Then after the code is created, it's in review. If somebody stopped working and there is no progress, it will be stalled. And then it will be closed. What's interesting in Jira, the bug can be closed for different reasons, because it's a duplicate or because it's fixed. So a typical user of MyScaleBug database, if he opens Jira, or maybe a bit confused about bug statuses. But you should use a combination of bug statuses, so-called resolution field, and also depending on the project, there could be some labels, tags, and whatever. For example, explaining that we need feedback. It's a special tag, in case of MariaDB. It's strange, but it's real. On the other hand, you can click on tags and find all similar bugs. So it's also interesting. Given the time, I usually try to show examples of useful and useless bug reports. So there is an old collection, but gold is blind. I've recently liked three new. I just encourage you to click on each and every bug report. It will be opened. My comments, you know, what's good in case of hanks and crashes is good to get backtraces. In case of regressions like happened here, it's good to highlight this fact, and do not hesitate to report bug again if it's reappeared. And provide steps. Do not think that really what you need for a good bug report is MTR test case or docker image with all the commands that can be executed with building, adding data, starting virtual machine, starting replications and everything. It should be enough, and this example shows, to show the real point of the bug. So do not accept it as like a formal demands and if something is missing, it's no way. As for useless, I actually had a huge list of useless bug reports. Some of them ended up with a lot of fun. And with year after year, I understand that there are really very few useless bug reports. Because even if you do not state your problem quite well, eventually in comments, you can see a good example of studying the problem. You know what? It's called free support, and you may get free support. You are not entitled to it, but you may get free support in bug database. You should not abuse it, but you may get it. So there are really very few useless bug reports. I would like to add one statement about not useless, but a bit dangerous bug report based on discussion we had just today. So please take extra care to the information you put into the public bugs database. Do not put your IP addresses, user names, passwords, all stuff that belongs to you, and even more for example to your customers. In case of Oracle bugs database, there is an easy-to-use option to hide comments private. The bug may be marked as security, and almost any engineer can do it. In case of JIRA, it's way more complicated to hide some information as soon as it was publicly available, because of all the version controls and all change carefully tracked. So be careful not only with adding correct precise and needed information, but with information that may be confidential and not strictly needed to understand what the bug is. The last probably slide for today is the following. Okay, you reported great bug. It's repeatable, but it's not fixed, or even not processed. It still stays open. What you can do? First of all, if you got any request for feedback in the process and you are not replying, you are the problem. So please cooperate. If people ask maybe even repeatedly, please cooperate when it's reasonable. If you cannot, if you do not understand the requirements, if you are not happy with them, add your comments. Do not sit silently and pretend that some magic should happen to your bug report. No. If it's real bug, try to involve other people, colleagues, the more comments from different people you get, they may help to understand. They can create a test case for you and it adds value to your bug report. Do not open new and new bugs with the same titles. Try not to, you know, curse and everything. If nothing helps, if you know somebody and you already met a lot of developers here, or MySQL people, Oracle people, MariaDB people, try to talk to people in private and get some internal escalation. If it does not work, that's probably nobody in Oracle is happy about what I'm saying right now. Discuss bugs in public. If you have your blog, blog about it. Unfortunately, we do not have Jean-François Ghani here, but he is quite efficient with doing it in a polite way, in a useful way, but he's blogging about some replication bugs and they got fixed. If you have a support contract, surely open an issue. Surely. If you do not get free support, use your resource. And if nothing works, you know, just tell me. That's why I have my blog posts and everything. Sometimes magic happens. I will not tell you how it happens, but it happens. If you need to know other people whom you can ask, whom you should care about when you see their comments, please, in case of Oracle, it's almost shastry. He processes, preprocesses like 50% of all bugs reported and he participates, participates in important role in setting priorities for bug fixes as well. So follow his comments. In case of MariaDB, there is one person, Elena Stepanova, who used to be a single person QA. Now there are many more people involved, but she makes real decisions about bugs. You've seen Sergei Golubchik, surely he makes decisions about bugs. You've seen a lot of people with a number of commits. So these are developers who make things happen. So if you get in touch with them, please do. In case of Prokona, we have Sveta Smirnova here, who somehow involved in setting priorities. We have Lovrinas in a role similar to Sergei Golubchik and we have Royal, who is again a king of QA and he cares a lot. So if you would like to discuss specific bugs, we can still do it outside. Thank you very much. I hope I'm not that much out of my time. What you should do once again, please report bugs in public. When you see a bug, like my wife did, make some document recording of what it is about and report it. Praying Mantis is a huge, nice bug and it would be really great to see it here. So you do not make MySQL worse, but by explaining the problems of, or you found in public. If it's useful, MySQL bug report and almost all of them useful, you help community to gain more knowledge, experience and you may get your real production problem solved even for free, fast. If not, well, you will be notified. Thank you.