 Okay, we're going to continue. Thanks for closing the window wonderful. Camille is going to tell us a bit about Lucine upgrading in Jira 8. Okay, let's start. Good afternoon, and welcome to the talk about Lucine upgrade in Jira 8. With a catchy title, how we fought and killed the co-debt beast. My name is Camille Ciche. I work for Spartus in Gdańsk in Poland as a software developer working on Jira server. You probably know the company Atlassian, is the company behind products such as Jira Confluence, and Bitbucket, and other tools that let your teams collaborate more efficiently. Spartus is a very close partner of Atlassian, and we help them develop products such as Jira, bamboo, fisheye, and crucible. This talk will be about Jira. Does anybody in here use Jira? Almost everybody. That's very nice to see. So I don't have to really explain. It's an issue tracker and project management tool. You can put everything in Jira, and then you can manage your issues, saying what's in progress, what's been done, what's still left to do, and everybody is on the same page. Thanks to that. But as soon as you have more than, say, 10 issues, you need an efficient way to find them. And this is why search is so important in Jira. We even have our own language called JQL, where you can put attributes of issues, and then based on that, Jira will return you the result. And even this board, which shows you the sprint, this is backed by search as well. So search is the crucial part of Jira. This search is based on Lucine, and so we have an index, and we put all of the fields there. So we have system fields, which are the ones that come with Jira out of the box, such as summary, description, comments, who created that issue, who is assigned to it. But we also have custom fields, which can be configurable by admin, and they can choose from what Jira provides, fields like dates or text, numbers, but also they can be extendable by plugins. So plugin vendors can provide any type of custom field you can dream of. And apart, whoops, sorry for that. And apart from that, a very important part of the index is permissions because in Jira, you cannot see everything potentially because there might be permissions just for some people to see certain issues. And this has to be handled by the search engine because otherwise it would be too small, too slow if we went to the database to ask whether you can see this particular result or not. And let's move to the main part of this talk, which is Lucine upgrade because it's so important. We should be really up to date with what Lucine offers us. In fact, we tell our clients to update Jira regularly. We say get the latest version, get bug fixes, get new features. But unfortunately, we don't always pray as we preach. And if you look at Jira seven, it comes with Lucine 3.3, which was released in 2011. To make matters worse, it's dash Atlassian dash two version, which contains two crucial bug fixes ported from version four. So you might wonder why we are using such an old version. And this is down to what happened with Lucine between versions three and four. The API was broken in a backwards incompatible way. And this meant that the upgrade was non-trivial. I'm not saying it was a bad decision because the new API is better, but for us it required effort to upgrade. So it was put off for some better time in the future. And this time took seven years. With time, this upgrade project looked like climbing a high mountain. So we started here at the bottom with Lucine 3.3 and there was this shiny Lucine 7.2 at the top. But how do we get up there? First, we meet some dead bodies of people who try to upgrade in the future. They spend maybe a day or two to upgrade, but, oh, sorry, this clicker is very strange. Sorry for that. So they tried to upgrade. They spent maybe a day or two, but they failed because it wasn't doable in such a short time. And then there's lots of old code that nobody has touched for years. And then there were unknown gains. Imagine coming to your product manager and saying, we would like to upgrade Lucine. It's going to take maybe a month, maybe six months. We don't really know. We don't know if it's going to be any better. Let's do it. It's not an easy sell, as you can see. Then there was this new Lucine API, which we didn't know when we started because we were stuck on the old one. So we had to learn on the way. And lastly, very important is the impact on app vendors. So plugins can use Lucine Index in JIRA. And when we upgrade Lucine, they have to upgrade their plugins as well. And there is a big ecosystem of plugins on Atlas and Marketplace. And there are people writing their own plugins internally. And all of those who used Lucine in a more advanced way had to upgrade, which meant that we could only upgrade Lucine in a major version like JIRA 8.0, which is coming out in a few days. So we went for it. The expectations were rather low. Let's just make it not any worse than it was. And let's be more up to date. So then we can get some help online. For example, for Stack Overflow, if we have a problem, there are probably more people who know the current Lucine that those who remember what happened seven years ago. Effort-wise, it took two developers three weeks just to make the code compile with the new Lucine. Then another month to make unit and functional tests green. And then the whole team jumped in and we spent six months. This is not just to make it work. We took the momentum that we had and we decided to use as much as we could from Lucine to make search and indexing in JIRA better. So we wanted to catch the benefits of what the new Lucine gave us. So this is what we did on JIRA platform. So this is the base, but then there are products built on top of that, such as JIRA Service Desk, JIRA Portfolio, and also there are those plugin vendors and they had to put effort in as well. So you can see it wasn't a very simple operation. And do I think it was worth it? Yes, with the benefit of hindsight, I think it was worth the effort. Indexing is now two and a half times faster, which means when you set up JIRA from a backup or when you need to re-index because you changed the configuration in a major way, you get shorter downtime. And also you need less infrastructure to run JIRA, which means you need to pay less money for your hardware. Then index is twice smaller, then you need less disk space, and search is five times faster. These are the numbers we get from our testing instance. We have one million issue instance where we run automated tests, but that one doesn't have any applications on top of that and plugins. So your mileage might vary and the results might be slightly different for you. And I couldn't resist to take a very liberal approach to statistics. I took an average of these three numbers and in total JIRA is 3.17 times better than the old one. But apart from these numbers, that's not everything. Optimize is now gone in the new Lucene. And to understand how it worked in the past, we need to look at segments. So this shows us segments of Lucene. Once you index something, it becomes immutable. And the only thing you can do to it is to remove documents, mark documents as deleted. When you add new issues, you create new segments. Then you add more segments and when you have too many segments, you merge them. So after full index of JIRA, the index would look like that. But in Lucene 3.3, there was this method called optimize. And if I asked any of you, would you like your index to be optimal? Everybody would go, yes, let's go for it. So we did the optimize. What it did was merge everything into one big segment and that meant that you didn't have to go to different files when searching. So in some sense, it was a bit better. But the problem was when you added new issues or modifications, you had these new segments, but they were so much smaller than the big one, they could never be merged together. Then you started deletes from the big one, but it was still too large to be merged. And that was a problem. And if we now compare, because now this optimize is gone, we can compare JIRA 7 to JIRA 8 performance-wise. If you look at this, actually when we first measured, JIRA 8 was slower than JIRA 7. We took our one million issues. We run the test of searching by text, very simple test. And it took only 200 milliseconds in JIRA 7, but it took almost 400 milliseconds in JIRA 8. That was not what we expected, was it? But then we recalled that optimize and we thought, okay, JIRA 7 was merged into one segment. So what if we apply a week of work, say updates to the issues, commenting, creating new issues, and you rerun the same search again. This is what happened to JIRA 7. It degraded very badly while JIRA 8 is intact. And this is a big pain for JIRA 7 customers, because they have this optimized index, which means that the index over the weekend, it works fine on Monday, but by the time Friday comes, it really slows down. So they have a weekly routine to re-index JIRA, and now this problem is gone, and they can enjoy new JIRA. That was searched by text. If you search by ASAINI, which is simpler, they start head to head. But then again, seven degrades, well, eight stays intact. If you search by created date, JIRA 8 is three times faster, and that is because now we index dates as numbers, rather than as text, and doing range queries on numbers is faster than doing range queries on strings. And again, there was degradation in JIRA 7. Another thing that we got with the new LUSIN, actually we lost again, was field cache. Field cache was this data structure that answered the question, what is the value of my field across all documents? So for example, what is the value of the ASAINI in all my issues? The problem with this was two-fold. First, it was calculated on demand, so you'd have to scan your index, and put that, and the second problem was where you put it. You put it on heap, which meant that your heap usage was increasing over time, and as ever good cache, it was never freed in JIRA. So now this is gone, so we had to move to the new thing, which is dog values. It's more or less the same mechanism, but it's built during indexing, and when you need it, it's already there. You just pull it from the disk once you need it, and as soon as you don't need it anymore, the OS will swap it out for you, so there will be no impact on GC. Another thing that we got improved was resiliency. JIRA 7, compared to JIRA 3, is more resilient when it comes to abrupt failures of your machine. So in JIRA 3, when you were indexing something and the power went off, you might end up with corrupted index, and that would lead to, well, the index was corrupted, so you had to re-index it, and for our biggest customers, that would be a week of outage of JIRA, and JIRA 7 now doesn't have this problem, and there's a funny story that I heard from Uwe about Mike McCandles, who was testing this new Lusine and that it cannot be corrupted, so he set up two machines, and one machine was controlling the power outlet of the other machine. He was indexing something and pulling the plug and then making sure that the index was not corrupted, and this is something that you get when you update libraries. Somebody did the hard work and you get it, so it's always a good idea to upgrade, and we did work on our side as well. When JIRA has a new issue or issue update, we put that on a queue of issues to be indexed, but when your machine dies, sometimes JIRA lost that queue, and now we improve that, so everything will be in your index when you restart JIRA. And the last big change that we did was around permissions. Permissions, as I told you, is a big thing in JIRA. It's a very flexible mechanism. You can say everybody can see your issues, or you can say only people who are logged into your application can see issues, or only the project lead, the person who reported it and the person who is assigned to it can see those issues, or only members of a group can see that issue, or only members of a group which is set to a value of a custom field on that issue can see that issue, and all of that has to be handled by the index. So to understand the change, let's have a look at a very simple model of an issue. It has a project, it's in project two, it's a summary improved permissions, it's assigned to UVA, you will soon learn why, and then it has custom field one with value group two. And this is what we need for searching, but we now added a new field, project permissions, where we index combinations of these values, so we index P2 for project, P2 as any UVA for pair of project and as any, and then P2 custom field one group two for the pair of project and custom field. And now let's see how the search looks like. Say I'm searching for summary is like permissions. This is what I want to see, but on top of that, Jura needs to add a permission query which will limit the results to what I am allowed to see. We'll be building two permission queries, one in the old style, which will be a Boolean query and the new one which will be a term in set query. Term in set is a query which will match as long as there is one element in the query that matches one element that was indexed in the new field project permissions. So we'll be following my artificial project permissions scheme. So let's say I'm allowed to see issues from project one. In the old way it will be the project one and in the new way we add value to the terms in set query saying P1. Then I'm also allowed to see issues from project two which are assigned to me. So then that will be project two and as any equals Camille. In the new world it's one term which says P2 as any Camille. Then I'm also allowed to see issues in project two as long as I'm a member of a group in that custom field. So say I'm a member of three groups, group one, two and three. So the Boolean query becomes project two and custom field group one, group two or group three. In the new one we flatten that. We explore the Boolean query to terms. So they become single terms rather than combinations of a Boolean. And this one matches, you can see that project two and custom field one equals group two matches what we have in the issue. And in the new mechanism, I clicked but it didn't move. In the new mechanism you can see that the term matches what we index under project permissions so I would be able to see that issue in both cases. Then we can go for project three which has the same permissions and project four which has a different permission scheme. This might look complicated. In fact it's a rather simple query. It has four projects and I'm a member of three groups. Imagine in real life there would be a thousand projects and I would be a member of say 50 groups. The query would become huge. And then Dheera, sorry, had to go to his aunts and ors and ors and aunts trying to evaluate that against your issue and that was slower. And looking at this, to your eyes they might look similar in size but this one is much easier to evaluate because it's just a set of terms and you need to find the common value between these two and as soon as you find one, you're done. So this one is faster and in fact it's actually five times faster when we compared running search just before we merged that fix and just after it became five times faster. I cannot, we cannot take all the credit for this idea which moves us to the next topic which is partnership with Uwe. When we started our journey up that mountain of upgrading Lucene, we decided to get some help, external help from a consultant who knew Lucene. We reached a company named Flux and they directed us to Uwe. Uwe came on site and we spent a week with him. First he gave us lectures what has changed between Lucene three and seven and then we spent three days looking at our code to see how we can improve it. And some of you might be worried that when you invite an expert you need to have questions but the good thing about working with experts is you don't need yes, no questions. You can just show them your problems and they'll come up with solutions. For example, the new permission query is an idea by Uwe. We just told him our permission is slow. Can you think of something better? Also when you work with experts they will tell you what not to do. For example, every issue has ID, which is a number and we wanted to change that to be no longer a string but a number in Lucene. But Uwe told us that what Lucene has been doing for the past 10 years is searching for strings. So that's much more optimized than searching for numbers. Searching for numbers is actually a range query. So when you look for an exact value it's better to keep it as string even though it's a number. And this is something that we wouldn't have known if we hadn't spoken with Uwe and we probably would have fallen into that trap. In the future, because JIRA is much better now than it was but it's always a constant work to improve it. We want to improve the way we manage index in DC so when you have multiple JIRA nodes so instances of JIRA which shed one database but they have their own local indexes. We have a mechanism to copy the index between nodes if there is a failure of one node and we want to improve that. Also we want to increase the indexing speed because we identified some inefficiencies in the way we index. And we need to improve our APIs because currently a plugin can run a query in JIRA saying give me all one million issues sorted at once and that can bring the instance down so we need to find a way how to avoid that. At the same time, not stop the functionality of those plugins. And if you find any of this interesting we are always hungry for talents. So go to spartes.com and slash careers and I would like to leave you with three thoughts. It's always good to ask someone who knows rather than reinvent the wheel. So whether it's your colleague or a consultant that you need to pay it's probably worth the money. Paying off code depth pays off. So when you pay your mortgage you don't get anything new in return. You just stay in your house. But when you pay off code depth you can get more results than you actually expected. And the last one is never lose hope in your code. Even if you haven't done something for say seven years and you think there's no way you can get out of code depth there is hope. Our example shows that it is possible. Thank you. Thank you. We have three minutes for questions and I brought some t-shirts for those who ask questions. I'm asking a question but would it be any worth to use elastic search instead of lucent of what would the trade off be? Okay so the question is would it be worth to use elastic search instead of Lucene? It would probably be. We are considering that but it would be a much bigger change than this upgrade to the way the Dura is deployed and so on. So it would work with the multi instance Dura. For example DC where you have multiple nodes then you would have just one elastic search infrastructure but we need to see it's worth the effort in terms of the operation so this t-shirt is taken. Next. What cloud of experts? What about Dura server or data center? But what about Dura cloud? So in Dura. What's the biggest difference between how the index will work between? In Dura cloud they have a different mechanism based on Postgres because in Dura server we don't know what the database would be. Postgres might be Oracle might be other database so we have the index but in Dura cloud they have, they know it's Postgres so they can optimize the search on Postgres and they don't need Lucene at all. The question is how we sold it to the product manager? Yeah the question is how we sold it if it took six months of work. Well we told them it's going to take three weeks. But then we just said what are the possible improvements? How it can fix problems of clients which are running into issues such as this degrading index and so on so we told them it's not just our idea just to be better developers. We convinced them it's going to improve the product.