 He's the author of Jiggered in Action, the CTO of X-WikiSAS, and here to talk to us about advanced Java testing. Vincent, thank you. Thank you very much. Always nice to see a full room. Thank you for that. So today, we're going to talk about advanced Java testing and, more importantly, what's next. So I'm not going to focus on things that we've all been doing for years, but really trying to focus on what's next in the domain of Java unit testing on Java testing in general. So I propose the following agenda. I'll expect a bit the context of the talk, and then we'll talk about four domains of testing that I consider a bit advanced. I will ask the question to you whether you've done that already or are you planning to do that. So one of those is coverage testing and strategy associated with that. This is something very important for me. It's not just tools that are being used, but more importantly, strategies around those tools. How do you use those tools effectively? So for each of those, we will see strategies. So the second one that I want to talk about is testing for backward compatibility. When you've got APIs, how do you test for that? And then an advanced one on mutation testing. Maybe some of you have heard about that. We'll see that. How can we use that in our tests and what to do with it? And the last one that I want to talk about is environment testing. It's how do you test for various environments? So let's start. The context is, I've learned, and I'm using those strategies and tools on a project called XWiki. It's an open source project. It's 14 years old. This is to give you some ideas. 10, 15 active cometers. It's very extensible. And that's important because we have a lot of APIs. So one of the items in the list was to test for backward compatibility. We want to make sure that when there are extensions against XWiki that they keep working over time. So we've had to develop strategies for that. So we use Maven for our build. We have a lot of plugins that we use that we'll see. We use Jenkins for our CI. And we're using a pipeline, a pipeline libraries in Jenkins. In terms of context, another element is stamps. I'm participating, so I've worked for a company called XWiki SIS in addition to being an open source developer on the XWiki project. And that company is participating to a European research project called Stamp. And the purpose of Stamp is to extend the domain of testing by amplifying existing tests. And we'll see that the mutation testing that I'm going to mention is actually coming exactly from this research project. And the same for the environment testing. So that's the context. Now let's dive in. So today on the XWiki project, this is our testing status. So we have two figures here. One is code coverage. We have about 71% testing coverage overall of the whole code base. And we have about 10,400 something automated tests. So they take quite a while to execute, especially since we've got different types of tests. So we've got unit tests with Moquito. We've got integration tests. I said, sorry, I said Moquito, but yeah, some of them are using Moquito. Functional UI tests, and these are the ones that take the longest, of course. And we use Selenium WebDriver. So this is very basic. This is, I think, everyone probably is doing that on their Java project. This is well known, and we're all doing that. That's not what I want to talk about, but this is where we're starting from. So now we wanted to go beyond that. And we wanted to ask ourselves new questions. For example, we have 10,000 tests, but what are those really exercising in terms of the whole code base? How good are they exercising the code base? So that's the coverage testing that we'll talk about. Another question that I mentioned quickly is, how can I prevent breaking the users of my APIs? So that's the background compatibility testing. And very importantly, since we have a sizable number of tests, we want to know, are they really good? I mean, are they really testing something? Because you can have a test with no assert. The test would execute. You would get test coverage, but they would not exercise anything, for example. So how good are they in terms of exercising real stuff? So this is where the mutation testing is interesting. And more generally, XWiki is a server software. It can be installed on a lot of environments. So we'll see how can we ensure that it works fine in various environments. So I've put little, little ticks. Though the green ones, for example, are stuff that we've been doing for years on the project. Probably over eight years now. But the one with the exclamation mark, the yellow ones are new stuff that we're experimenting at the moment. So we have a bit less of years of return of experience on those, but we've started using them and having some strategies around them. So let's start with the first one, the test coverage. So we're using a Jacoco and Clover. We're using both. So this is not new. I want to see how many of you are using, I'm measuring test coverage for your project. Yeah, so we have, let's say, 30% or a bit less, maybe. So that's not new. That's quite well known. What we're doing though is, what is a bit different is the strategy we're using. We call it the ratchet effect. Well, it is a ratchet effect in the sense that we have a lot of Maven modules. So our software is divided in lots of modules. And for each module, we have in the palm.xml of the Maven build, we have a threshold that corresponds to the current measure of the code coverage for that module, okay? And when someone's, a new developer contributes code to that module, we fail the build if it's below the current level that exists. So in a sense, we want to ensure that new code is at least as well tested as what exists at the current time. We're not driving for 100% or 80%. We don't care about that. What we want is that it is marginally the same or better than what exists at the moment. We want to go in the direction, the upward direction, but we don't care to go too fast there, okay? So if you're a developer, you come in there, it fails, then you need to increase a little bit your tests so that they test a bit more. And if actually you go beyond the current threshold, you're allowed to change the value in the palm. So that becomes the new reference for all the other ones that will come after you. So we've been doing this for years. And recently, not long ago, I've actually measured, I was trying to measure, does this have any effect on the long run on the whole code base? So this is the graph you see on the right. So I've measured on one year. So I've took the full coverage on December, 2016. I took it again in November, 2017. I did some harmonization. I can explain why it's because I'm using Clover to do that and they don't generate exactly the same thing. For some of the code, some of the classes were considered as test classes. Some others were considered as production classes. And somehow I don't know why exactly. They mixed up, Clover mixed it up a bit. So I had to separate it a bit and recompute it. But after doing this, I noticed that we had a 1.9% coverage increase. So this is exactly what we want. We want slow, steady increase, always in the upward direction. And this is working very well for us. And so we measure this Clover thing. We measure it with the Jenkins pipeline that we run automatically every month. So it runs on all the whole code base. And so all the tests that are exercising code from everywhere, okay? Because the Jacob code thing with a ratchet effect is only on a single module. This one is a global computation for the whole code base. It's a bit more complex. That's why we're doing it in the Jenkins pipeline. So that's the first domain I wanted to mention. Second one is backward compatibility. So this is mostly interesting if you're developing APIs or SPIs. If you're just developing a product with no API, then of course it's less interesting. So the way we handle that is technically is using a Maven plugin called Rave API. We used to use one called clear, but it was not maintained. And this one is a new one that's been maintained. We've actually worked quite substantially with the guy developing this plugin to make it better and better. It supports source and binary compatibility. So that's excellent for us since we really want to maintain binary compatibility. So if someone develops an extension for XWiki, he's developed it several years ago. We want that when he uses this extension on a new version of XWiki, there is no need to recompile this code. It will just work as is. So the way we do that, we ensure this, is that we really break the build. Whenever developer introduce a change that changes an API or something that makes the code not backward compatible from a binary level, we break it. We want to raise attention for the developer that something is breaking something. Then this developer can decide and discuss with the community as to whether we really want to break this API. Is there a very good reason for doing it? But we want that to be an exception, not the norm. Because we want to be very conservative about backward compatibility. If it's the case that we really want to do it, then we add an ignore in our palm so that clear, sorry, not Rave API will pass. And at the end, when we do the release of the version of the software, that gives us actually a very good thing to put in our release notes, because we can list the full list of APIs that we have broken voluntarily. So that means all of those will be a list of APIs that we wanted to break, not by error. But we wanted to go, so this is what I've put on the right, is an example, don't need to read it. We actually generate it automatically from the palm, but that's a detail. Now, we wanted to go further than that on the XWiki project. We wanted to never break backward compatibility ever. That was more ambitious. So what we did is we used the following strategy. Whenever something is not good anymore, and we're replacing it with something else, so we start with a very standard strategy of deprecation, with the deprecated annotation. We all do that, that's very easy. Now, after this, we modify all our code that uses this deprecated API so that we have no more code using it. When this is done, we move the deprecated code to another module, another Maven module, separate. So that's easy to do if you're moving a full class, you just need to move it physically to another source tree. But if you just want to remove a method, a single method, there is no way in Java to doing that. So what we do is, that's why we use aspect J. So there's an aspect J Maven plugin also, so we just move this part in an aspect, and that other module, we call it legacy module, when it builds, it actually weaves those aspects onto the other code, the code that doesn't contain the deprecation, and it generates a jar that contains both the code and the aspects. And I've put an example here on the right, if you can read, this is just adding one method called check programming write to a class called API, and it just augments this class with this deprecated API. So this has several advantages. First, it means that for us developers, we see clean code. We're not tempted to use deprecated code, and we can't because it's some other module, and we actually enforce that, we don't ever have dependencies on legacy module. So if someone wants to add a new dependency in the palm to a legacy module, it will actually break the build. So we want to really make sure that it doesn't happen. So that's good for us. We see clean code, we don't have code littered with deprecated stuff, and we never know what to use, and people want to go the easy way and use the deprecated API because they don't know yet how to use the new API or whatever. So this avoids this effect. And on the other hand, from the user point of view, we release the legacy modules. When we release XWiki, it contains the legacy modules. That means if there are extensions on our code that uses those modules from the outside, they will continue to work. So this will win, win, and we don't need, since this code is moved to legacy, we don't care that it stays there for 10 years, actually. It has their tests have been moved to at the same time in this legacy module and provided they continue to pass and continues to work, it costs us nothing. So basically it costs very little for us to continue to not breaking anything. Now, for those who've tried to do this, there's a little problem, is when you create a new API, you never get it right the first time. It takes a very long time to create a new API because you have to tune it and you have to see the usage and by the usage you say, okay, I made a mistake here, it would be better to have this parameter, et cetera. And if you, so in the early phases of a new API, you really need something more flexible than what I've discussed. So this is why we've introduced an annotation called unstable and when we introduce new APIs, we are the developers are allowed to put an unstable annotation, but we don't want this to leave forever, that would be a problem. So we've also, so when you put an unstable, you also have to put a since, a notation with the version in which it was introduced. And actually if you put unstable annotation without a since, our build will fail because it will notice you don't have both, okay. So these forces to have them and then our build automatically checks that these unstable annotation cannot leave longer than one full cycle and for us a release cycle is one year. So we allow for a stable to last for one year and then they become mainstream APIs and then they go through the process that I've explained just before, okay. So this is our strategy for backward compatibility. I mean, is it someone doing something similar here? Your raise your hand, no? Okay, it has worked quite well for us over the years and we are, we actually extensions that were developed 10 years ago and they still run in X-Wiki, thanks to this thing. Vincent, do you want questions now or? I would prefer the end if possible. Thanks. Okay, next domain is mutation testing. So this is newer. So this one is we're using several tools. I'm going to start with two tools called pit test and decart. So I will explain the concept. So first, how many of you are familiar with mutation testing or use it a bit? Okay, a bit I would say, I would say 5% and maybe of the room. So this is pretty new. Actually, it's not very new. It's been gone for a long time but there are new tools being developed and I think it's getting more and more traction. To be, it was more in the academia and it's have the feeling that it's getting more and more out of it with tools that can really be used for real. So the concept, I will explain the concept. So imagine you've got code. You've got existing tests. So what mutation testing does is it modifies your code on the test. So not your test, but the code on the test makes some changes to it. For example, if you've got a less than sign somewhere it will change it to a greater than. If you've got an integer value it will change to another value, et cetera. You have other ones like if you have a method that returns a string it would remove it and return a null instead. Those kind of things, those are mutations. We call them mutants when you have a code that were to which a mutation has been applied to. And then you rerun your tests, your existing test suites. And if it continues to pass it means that your tests have not noticed the change. That means they're not good enough. And globally, you do a lot of mutations, several mutations and it generates a mutation score which represents basically the quality of your tests, how good they are in catching problems. So it's very similar to the coverage percentage. And I will show you some screenshots of Descartes and Pitt in action. Showing you both the coverage and the mutations scores. So Descartes is just a mutation engine for Pitt. Pitt already has several mutations and Descartes is an additional mutation engine. I will explain to you very quickly the difference here. So this is a high level schema. It shows that for running Pitt and Descartes you have to get all the compiled file, the source file, dependencies, everything. This goes in Pitt, which makes the modifications to the source files. And Descartes is an engine inside Pitt and it has about four extreme mutations. So one of them is that if you have a method that returns void it would actually completely wipe out the method content and do nothing. Okay, that's extreme. If you have a method that returns an int it will return a given value, a static value which you can configure in your POM. And for a string you can choose to return a null or any string value. And you see what happened. So these are really extreme one. We want to focus, because it takes a lot of time to run a lot of mutations. So we wanted to focus on the extreme ones because they are the ones that have the highest value for finding problems. So when you run that, this is an example of running it on a one small, very small module of X-Wiki. So this module has 14 classes. And this is the report generated by Pitt. You see it reports a coverage of 80%, line coverage of 80% and mutation core of 71%. And then you've got to drill down, break down by package. And you can go till the line number. So that's where you see the problems. On this example, this is an equals method for a Java class. And what it says here, it says that maybe those are the bugs won't be able to read. So I will describe it. On line 112, it says that the code, the test has covered this line, has called this line. So Jacoco shows that it's been covered. However, what Descartes and Pitt are showing is that even though it was covered, when those, this body was replaced by true, returning true instead of as a fixed value, it still survived. It wasn't noticed by the test. Okay, this is exactly what it shows. When it was replaced by false, it was killed. The mutation was killed. So the test was good for false test, but not good for true conditions or returning true. And if we look at the test, so the test is on the left here. Indeed, if we read it, we see that we're just testing for a setting true for comparing things, but we're not a setting false. And actually, we're not testing for inequality. So after I've run this, I've been able to modify my test to actually show that. So this is just a very simple example, but something simple to understand that it's able to tell me how good my tests are and what's missing. Now, there are some limitations. First, it takes times to look at all the reports generated and find the right places that are interesting to modify, because they're not all very interesting. So if you have one hour of time ahead of you, you may want to add new tests elsewhere rather than try to add new assertion in an existing test. It may have more value for you. So it's very important to find the important places. And there is a concept we call strong pseudo-tested method. Those are the worst because it says that no matter what the written value you change in the code, the test always fail. So they are the highest contenders. Those are the one you want to focus on and look at. So what the team developing, Pete and Descartes are working on is to modify the report so that they highlight those because I help you focus on having a look at those. There are some others, but they're less, they're more gray area. You can look at them, but you would take a lot of time actually to look at them. So another limitation is that Pete by default doesn't support multi-modules. That means that you can run it on a single module. It will modify your source code in that module. But if you test exercise code also in other modules, they're not taken into account. So someone has started as part of the STEM research project, another open source plugin called PeteMP for Pete multi-projects that allows to do that on a lot of them. The problem is when we've executed, I've tried it on X-Wiki, it takes a very, very long. For example, I've executed just on what we call X-Wiki rendering. It took seven plus hours just to execute fully on the whole code base. And for me, that makes it just unusable. So right now, there are things interesting. There are things that have limitations. So we've defined, I've defined a strategy that I want to try, which I think could work in real life. So the idea is I want to apply the strategy that works well for us about the coverage testing that I've explained, but I want to bring that to mutation score. So I would like to fail the build of our given module only when the mutation score generated by Pete and Descartes is under the defined threshold in the palm, same as the coverage. And when you improve, because it's exactly the same thing in the end. It's a measure of how good your tests are. The other one was a measure of how much of your code, there are tests for that code, piece of code. And this one is how good the tests are, again, for the piece of code, because it's also modifying the code. So it's exactly the same thing. But it provides, I have the feeling that it provides a higher value than just the line coverage. But I still need to try it, but that's the idea. And then progressively try to increase the mutation score over time. What I don't know yet is that going to be stable? I mean, is that going to work as well as the coverage that we've been doing it? So this is still an experiment in progress. Now, we can go further in mutation testing. There is another tool called Despot being developed also on the Stamp Research Project. So it builds on Descartes, but in addition, it re-injects the results and adds assertion to your own tests. So it augments, it amplifies your own test sheets by adding missing assertion automatically. Okay, so I can even generate new tests methods. This one is, it's more marketing than the reality because I've not seen it happen, but the assertion do work. It's still being developed, but it's usable and I'm actually have executed it on next weeky. There are some, okay, it's the beginning that is starting to work, let's say like this. So this is an example. So I've taken some code. We had an original test which you see at the bottom and you see this test is calling a method called escape attribute value on a string and we're testing, what we wanted to verify in this test is that some special characters like less than, greater than, et cetera, they are encoded. Okay, this is what we test. And we run this through this part and it generated a modified test and I've put in red the line that adds something that was actually not tested here. It automatically added a line for verifying the exact content actually of the escaped string. We were just focusing on the greater down or less than but it added this in addition. So for example, imagine that the escape attribute was also escaping the A, the A literal, the A value. We would not have noticed it with the test but with the addition that was done by this part it would have been noticed. So this amplified the existing state by adding new assertions. And this is a very simple example. Now the problem is this part is very slow, the very slow and that's a very typical invitation testing. That's the issue, many issues of slowness. So you have to define a strategy that would work. So my idea, the idea I have at the moment that I want to apply on the project is that we would not run it on developer's machine because it's too slow. We would run it on the CI from time to time only and the CI in the pipeline, for example, Jenkins pipeline, we would commit the generated tests like the one I've just shown you. The test generated by this spot, we would commit that in a source tree of the module where the tests are but in a different one, not in the same because I don't want to mix the tests that are generated manually by humans and the one generated by this spot. Because they are interesting to run. We don't even need to look at them. They're interesting because they add stuff that represents, they've been computed in a way that they represent exactly what your code is doing. So if you have a bug in your code, it will also actually, you will have a test to prove that the bug is actually the behavior. So you will need to be careful about that but that's not the goal. The goal is that in the majority of cases, it will just prove the behavior of your code. So in a different source tree and then you can configure Maven to have several source trees or actually test trees in this case so that it actually runs both your manual test plus the one generated. So that's my idea at the moment as a strategy to use these spots. There are some other ideas like you could imagine having a GitHub commit hook so that it actually does some mutation only on the diff on the new stuff that have been modified, not on the whole thing that would reuse the time it takes. Last one, environment testing. So this is quite useful I think for a lot of us. Actually I'm interested, how many of you have a software that needs to be executed or verified to work on several environments? I am not that many, less than I thought. It's about, I would say 20%. In our case, Xwiki runs in a surveyed container. It needs a database. Of course, there's an OS. It runs in the browser. So we have a big combination and actually it can even run in a cluster mode. So it can be clustered, not clustered. It can have an open office or a Libre office server attached to it if you want to import office documents in the wiki. So there are a lot of variations of environments and we want to like to make sure that we test those. At the moment, before we did this we were testing only a single one automatically and then from time to time people were testing manually stuff but we had no coherent way of doing that. So the idea is to use Docker here and to run the functional test that exists on various environments built by Docker. When we wanted to do this, we thought initially we thought about using the Docker integration in the CI, in Jenkins. We thought that would be a good idea because it has a good integration. But we dropped that and the reason we dropped it is because we wanted the developers on their own machine to be able to run the various environments. When there is a problem, when there is a bug, someone reports an issue on an issue tracker in a given environment, we want to be able to reproduce that locally to put the breakpoints and test it locally. So we dropped the idea of doing it on the CI. We moved to the idea of adding the Docker integration inside the build itself, in the Maven build in our case. And we found a very nice Maven plugin called Fabricate Docker Maven Plugin, DMP, short. And those are Fabricate guys. They have their own software but they're kind enough to develop some generic tools such as this one that works very well and which allows you to create, to build the Docker image from your build but also to start and stop it. So I'm going to explain quickly how we do that here. So we have a module, we have a first module in X-Wiki in our build where we generate the an image, a Docker image with this plugin for X-Wiki. So that's for example, in this image you have got for example Tomcat and X-Wiki deployed in Tomcat. So that's the box at the top. And then, so we generate this image. So when you run it locally, you run the build. That means in your local Docker registry, you will have this image built. And then when you go to a functional test module and you run that functional test module, we use this DMP plugin to actually start that container based on that image. We actually start by running an image for a database. Could be MySQL or Postgres or others. Then we run that image for X-Wiki in Tomcat. And let's forget at the moment the bottom. Just by having that already, we are able to execute the functional test against this running instance of X-Wiki that has everything it needs to execute. But we wanted to go further than that and be able also to change the browsers. Type of browsers testing in Chrome, in Firefox, in Safari, in various browsers and various versions of them. The problem is that we're using Selenium and it is Selenium that actually starts the browser. So this is in the client space, I mean in the build space. So we needed a way. So what we did is we created another Docker image that depends on the official Maven Docker image to which we added Firefox and Chrome and the various browsers, okay. And when we run the build, we start in addition to starting the two other Docker containers, we also start with this third one. And we're telling, so we are inside Maven. It starts the three containers and there is the container containing Maven itself. So we tell this one to actually execute Maven and run the test, so to run the test, so inside that container, okay, which has the various browsers. So the way to do that is we map, for those who have used Docker, we map the directory inside that container to the current directory on the host file system so that it can see the build. The Maven build, all the palm and all the test, they are visible from the container, okay. So and we run Maven. And what we also do is we also map your local Maven repository directory to that container because if you don't do that, then it will re-download the Earth and it will take forever, okay. So by doing that, it's a bit complex, conceptually, but it allows us to control all the environments up to the client-side environment with the browser. So there are some issues with this, I have to admit. I'm still working on this one. I have an issue where I'm getting from time to time a very slow execution when I'm running Maven inside Maven thing. I don't know exactly why yet. I have to debug that. And it ends up in some out-of-memory thing and I have no idea why. So if you've experienced this, I'm very interested to hear about it. So that's the idea. That's the strategy. So we've succeeded in having the first part and we're still working on the second part with the browser controlling the browser. And that allows us to have different profiles in the bill to run various configurations. So on MySQL, on Postgres, we see such container, et cetera. You can't do all of them because that's too long. So you have to focus on the subset of those. So that's about it. So as you see, we're experimenting. We're trying to progress and push the limit every year trying to discover new stuff. There are other things we're not testing yet. One big one is performance and stress testing. We're doing that in an ad hoc manner from time to time but we would really love to have that as an automated process and something that can be automatically checked regularly. That would be nice. Usability testing also is something not easy to do. We would love to work on something. So we do that with some experiments but we don't do it in an organized fashion and not automated way for sure. And I'm sure there are other things too. So thank you very much. And now the questions. We've got 15 minutes for questions. Thank you. Thank you. Many years ago, but I'm back on the dark side at the moment. I was just trying to see, I see an old friend of mine, Parasoft, got a C++ mutation to him. So do you know of any others? Obviously harder to do is you have to program a source code for it. Yes. Which kind of tool? Sorry, I didn't get it. Mutation one. Mutation one. Ah, no. Be sure you restate the question. Yeah, the question is, do I know other mutation tool that exists but in the C++, yeah. And no, I don't know this word at all actually. I'm sorry. So Vincent, did you explore Agitar as a commercial alternative to pit or decar? I didn't even know it was still existing to be honest because I remember I met the guys of Agitar 10 more, 12 years ago. Okay. And I tried to get a license there because it was not an open source project. I don't know if it is now. It is not. Okay. But the license was lasting only like one or two weeks. Like I didn't get to explore more. So I don't know. It doesn't get reintroduced. How do I measure, so the question is, how do I ensure that if we remove a deprecated method, move it out, that it doesn't get reintroduced later on. There is no specific thing except that we do code reviews regularly and it would be, I think quite quickly in our case, quite quickly notice if someone was doing this. But no, I mean, sometimes it could be a good idea. Maybe you've made a mistake and actually it was a good idea. It could be some good idea for that or by the way, so I guess there is no silver bullet there. Or actually I don't have one. Yes. The sources of which one? Yes. Yes, of course. I mean, XWiki is completely open source. So you've got actually, I'm now working on this in the branch of XWiki because I want it to be completely working, especially the part that I mentioned that where I have an issue. But yes, it's completely, I can put the link somewhere if you're interested or you can contact me. On the slide I have some information about me here. So if you get those slides, you can get them online from Penta Barf. No, it's on the main sources of the project in the branch. Yes. Yes. Yes. Where is the database that you spin out? Okay, yeah. So the question is in the environment testing, solution, strategy, maybe sometimes the database needs to be in a given state. So how do we handle that? In our case, we start with the empty database. But we provision it, I didn't mention it, but we provision it. Each functional test or each set of functional tests needs a set of extensions in XWiki for it to work. And they are provisioned in our build, actually. You see, when I said I created an extra docker image containing Maven and the browsers, actually when we run this, when we execute Maven inside that image, we actually also have a plug-in that we run for provisioning the database at the time. So it starts with a clean one and we provision it and we run the test on the provision database. We don't have anything specific. We don't have anything specific there, but if you were using docker, you could imagine saving the state in another image, for example, and restoring so that it be very fast. So if you have a test that changes the set of the database and you want to go back to the previous one, since with docker, it's very fast to actually start the container. You could actually save that image somewhere and re-initialize it as a container, for example, I guess. I've not done that. I have a question. You don't use stored procedures, do you? No, we don't. It's horrible because I want to restate the question. So the question is whether we use stored procedures, the answer is no. Is that you have code mingled with your test? Yes, yeah, now we don't do that, yeah. Yes? Sorry. Ah, so the question is how do we test upgrades, right? Upgrade path, yes. Sorry. Yes. Yeah, yeah, yeah, okay. Not an empty one. A real one with some data. So how do we test when we, I mean how do we test upgrades? Basically that upgrades for people are going to work fine. The answer is we don't do that automatically. Sorry, we don't do it yet. But that's something, we do it manually. We have a release managers when we do releases, so we test some upgrades, but we're far from being perfect there. We rely, I would say, I would say we rely on the community, the open source community to tell us when there are problems there and when we fix them. Look, it does happen, yes. It does happen that there are cases where it's not working, it's not perfect and we have to fix things. So it would be nice if we had something automated. I think that would be the next step. Yes. Exactly. Exactly. But first we needed to have this working fine first, because we need this as a basis in order to move to this level, I think. Yes, question? So, are you the author of it or? No, but I could say I have my hand dirty with this. Okay, I'm sorry, then I'm sorry. No, no, no. My, my deepest, don't do it. No. No. But something on the host. So, Any docker. So, you're talking about a machine with a docker engine running and everything running on that machine, right? Or is it working with what you can find on OZCI system like that or even with some docker and docker stuff with a remote docker engine that does not share the file system between your clients and your server? I'm not sure I understand fully what I can tell you. I don't know if I'm going to answer your question, but tell me. What we, before we were using docker thing, we needed the developer needed to have the browsers installed on this machine and we could run it because we were able to generate an environment, a very simple environment with an HSQLDB embedded and a JT container. So, no docker. But we see, so the need was the browser installed. Now we move these requirements to not having the browser requiring any browser installed, but having to have docker installed on the machine. That's the only thing that we need is having docker installed. Okay, so. Not yet, no. It's just a single virtual machine or parameter machine. Yes, yes. At the moment, yes, but maybe you are. Maybe we'll hit that problem later on when we want to test more complex stuff. And in that case, maybe it would be nice to move to the CI for that. Yeah. But the CI just as informed, the CI does run since it's in part of the build now. It runs locally, but it also runs on the CI, of course. Yeah. Yes. Yeah, I didn't want. Initially, I wanted to have more slides on explaining this part, but the timing was too short. I need to do that. It's a bit complex, yes. The best would be if you're interested, what I can do is I can show you some locations or else where you can get that information. It's a bit complex to explain. I will not remember all the details. It's basically you do the mutations and then you've got some oracle that will tell you whether you need to keep those mutations or not. And then you re-inject that and you re-run that several times. But it's a bit, yeah. I wouldn't be able to explain it properly right now. But I do have the information that I can share if you're interested. Yeah, so five minutes. Yes? Yes. Sorry, I didn't hear. You're running somewhere where private stuff does. Yes, well, they're started by the test, yes. Well, at the moment we, so the question is are we running on the multiple version of browsers inside this Docker image or not? At the moment we just, as you see, it's not working fully well yet. So I'm not at the level you mentioned here yet, but it's not going to be a problem because we can imagine either having several versions in the same container or having several containers, several images for various versions. We have the two possible options. And then you control, you will control which one you want to run simply with a system property that are passed. You're right, that depending on the version of Selenium you use, it has been tested with a given version of the browsers. Yes, and in the release notes of Selenium you will see this, they say, they say which browser and which version. So yes, you will need to vary, to know the, to vary the version, but we control the version of Selenium directly in our POM. With the dependency, Selenium is a dependency in the POM. And we have the version of that dependency. So you can very well imagine that having several of those with different profiles, for example. Yes, I agree. Selenium, we've experienced that also that sometimes it's depending on the version of Selenium, it works well, less well, or less good. And it depends on your test also with what they're doing. So yes, there's a bit of voodoo thing to select the right Selenium version you want to test, yes. In our case, we are very interested in testing the latest version of Firefox and Chrome and Safari, the latest one. We're less interested in testing all the previous one because they change every month or so. They move very fast. Yes, but we want to ensure, we want to make sure that it works with the, we want to work with the latest that is supported by the various browser vendors, the latest one, at least this one. That's the most important one. If we want to do more, then it's more complex. Takes more time. Yeah. Mutation testing seems like a candidate for massive parallelism. Is there any looking at scattering the mutation process across many, many servers in a cloud? I'm sure there are, but I haven't tested anything like this on my side, but yes, that's a good point because it's massive in term of changes in computation and CPU used if you want to do at a high level. What I've tried to do on my side is to try to see, with running on a single machine, what can we do, how could we use those tools and still derive some value even if we're not using it in the full way. That's part that exists. Still finding something interesting to do is then. That's been my focus at the moment. Yeah. On the aspect J-based deprecation technique, what's the impact of that deprecation technique on serialized classes? Does it alter the serialization? Is it? Ah, it's a good question. I don't think the... I've never seen a problem, but it's a good question. I happen to be on a project that serializes a lot and the deprecation technique is very interesting if we could... I think it feels not methods, but I am stretching my memory that's related to your idea. You might be right, yes. I don't know for sure. What I can tell you is that strangely, we haven't hit any problem ever in this domain and we do serialization a bit, but probably not to the level, not at the level that you're doing it. We're very small for us. Maybe that's why I've not hit the issue. I don't know. It's a good point. Oh, I'm sorry, Mike, could you restate the question? The question was whether the deprecation strategy and the legacy thing with aspect where you move to somewhere else, does this impact serialization? Because you've got the serial UID, the value, and it changes because if you're moving things out, then it can change. But actually, in practice... You change the serial UID, of course. Actually, yes, but actually it doesn't for us because when we do the... Thinking about it now, when we do the weaving with the aspect, we put it back. So it's the same thing. Allegedly, it's the same thing. But you can set a concept in serial UID in any way. Yeah. It's not gonna change. Yes. Thank you, Vincent. We have reached the end of our time. Our thanks please to Vincent, a hand for you. Thank you. Thank you.