 Thank you everyone for joining in, we are with Andrei who is here, thank you Andrei for joining us to share your knowledge on extending open source libraries on the example of Selenite and Selenium. Hello, I hope we can start. Yeah, my name is Andrei I'm from Estonia, Tallinn. And yes, I love developing open source project, maintaining open source project. I love creating automated tests. I'm working in company codeborne where we do use actively extreme programming, meaning that we as developers we create post production code and automated test for it, post unit test and like UI tests. And because of that, I, at some moment I created a library for writing like effective concise UI tests in Java. This library is called Selenite. It's based on Selenium. And that's why I have been investigating a lot of different issues and like technical troubles how to override some behavior on Selenium, how to customize some aspects of Selenium, and so on. And I wanted also to make my own libraries Selenite also like customizable and expendable. That's why I'm interested in this topic. And I hope you can see my slides. That's why my topic today is extending open source libraries on the example of Selenite and Selenium, but actually, it's also applicable for many other libraries like to unit testing G Makito, and so on, and so on. So the agenda is like this, but first of all, I will tell what is the problem that we are trying to solve in some real like code examples. Then I am going to show how the problem of extendability is solved in Selenite in Selenium. And some final thoughts in the end, some somebody. So let's start from a problem. Selenium, like imagine that you as a developer, like created some tool, which has some convenient API and users are used to it user can use it for example, if you create Selenium, you have API like this. So you're familiar with this, like, obviously, so you can create a driver, you can call some methods on it like the gates, click, check elements, something like that. And most of the time it's everything fine with this approach. Let me run this code for you. I like to show the code and like to run the code. So it's like, it gives the feeling that yeah, I'm not just reading theory, but I'm really like doing things and you are watching things. This is a typical Selenium test when you create a driver, navigate somewhere, find some elements, blah, blah, blah, familiar. And the question is, what if user wants to do something differently. For example, if user wants to find the elements not by name by ID, for example, most of the times it easily do doable by API parameters, for example, you can write by ID and use some other ID. Or you can write whatever. Instead of this displayed, you can write like it's enabled or something like that, do whatever. So most of the time customization can be done by API or methods. But sometimes there are things that are much, much harder to configure or overwrite. For example, you probably know that for communication between test and web driver. Selenium uses HTTP client, not native HTTP client nowadays. And for a long time, the HTTP timeout timeout for this HTTP client was three hours, which seems to me really unbelievable, unbelievably slow, long. What if something hangs between and tests cannot connect to the driver and like something hangs between there in network protocol, then your test will hang for three hours. It's like unbelievable, it's not okay. Okay, nowadays it's better, starting from Selenium 4, it's now three minutes, not three hours, but in many cases, I still think it's too long. So the question is, how can I change this timeout? How can I put like 10 seconds, for example, it would be okay for me. But no, it's not possible. How to change this timeout to some smaller timeout. There is no such configuration parameter. These three minutes is hard-coded somewhere in Selenium. That's a question, how to make such things customizable? This is a question. We will see it later. Another example, like familiar for me, Selenite, is a library created on top of Selenium, which is specifically needed for testing UI. Its API looks like this, in some sense it's similar, you open a browser, you find elements, you click it or do some actions with elements. But this API was made more sophisticated, more readable, exactly for tests. And again, let me show the code, Selenite demo. The code looks like this, again, we can run it to see that it really works. Again, it opens the browser, clicks some limits, checks some results, everything is fine. In this case, the test failed because I put a special error number here just to show how the test fails. In case of test failure, Selenite tries to generate a very, very readable, like very concise error message, which is readable, which is like expressive. Let me show, it says that, sorry, list size mismatch it. I expected 100 elements, but in reality you got 10 elements, and the real collection that was on the page was like this. It lists all the elements that you actually could so. It also takes automatically screenshots, save speech source and try down some other numbers as well. So for example, you can click the screenshot exactly right here and watch what happened. You need to get why the test happened failed. That's why this library is very convenient for tests. But again, the question is how to customize its behavior. In most cases, it's very easy to customize by parameters. For example, if you want to find the element not by name, but by something else, again, you can write by ID, whatever. Or if you want to click differently, Selenite supports quite a lot of different ways to do things. For example, you can click not by standard way, but by using a tricky JavaScript, which might click faster or more stable in case of like animations or moving elements. Yeah, so you can give some parameters like please click using JavaScript code. Also, you can say something like please click not exactly in the center of the point, but like in some corner, so you can click with offset. And there are some other options as well, timeout and so on and so on. So in most cases, again, you can change behavior of the library by parameter, but not always. For example, again, in this case, we saw that in case of test failure, Selenite takes a screenshot, but you see that by default, the screenshot covers only the visible area of the browser. But in many cases, what you really want to see is something out of the screen, something below the screen. There is some important things often, some disable buttons, some counters, whatever. So the question is, would we customize the behavior of Selenite so that it took not only visible part, but the whole browser screen. How to do that? There is no such an option. There is no such a setting or parameter auto customize sync like this. So this is our primary question today. Users always want something special, some special features that are not just doable by parameters or configuration parameters. So the question, essential question for me, if you have some library, some tool, how to make everything, every aspect of this library customizable. This is interesting for me, how to make everything customizable. And I had to investigate how to do it. And again, there are several levels. The most simpler levels are to allow users to do whatever by using API or settings. This is the simplest one. Another option is using dependency injection. This is a way of spraying and so on. And my option or Selenium option or Makita option is using some system of plugins or extensions. Let's review them. Making something available by API, by parameters is good in case of common wishes. In case of some absolutely typical common wishes, features that users often need, often ask for. It's okay to do it by parameter. And for features that don't really need some external dependencies. For example, we will see this with screenshot. But sometimes it happens that it's not sufficient to make just a parameter or just settings. And you really need to heavily customize the behavior, how to do that. Depends injection is next option, but it is only good if case, in case if your framework or if your project uses dependency injection framework. Like spraying or doing something like this, then it's okay to use dependency injection. But in many cases, you don't have dependency injection framework like Selenium doesn't have dependency injection framework or Selenite or JUnit. Because they are full of static methods, they don't have a single entry point, like start JUnit, like start spraying application. So the question is how to customize behavior if your library doesn't use any dependency injection frameworks, what to do then. And yeah, like all our favorite frameworks that we use in testing. And the answer is service loader. This was a concept introduced actually quite a long time ago in Java 6. Service loader is another like version of dependency injection, in some set it's simpler. How to do it, how to use it. I'm talking about service loader because actually both Selenium and Selenite use service loader pattern. And many libraries as well. How to do is, if you want to make something customizable in the library, for example, if your library has interface user repository, which might have different limitations. How to make it like possible for users to override user repository. In that place where in your project where you need to get user repository, need to use user repository, you are going to use Java till service loader. This is a class with static method load, which can return your user repository, some instance of user repository. This is like familiar to dependency injection. There might be several of them possible so that you probably in most cases you can get just the first one. And to say service loader, which would be your default implementation. You need to create a file in the folder in class path in folder meta in services, and the file must be named after your interface name. This is a full name of your interface class user package package package user repository. It's inside of this text file it should be a text file with just one line inside of this file, you need to write the full name of class of the default implementation of user service. Again, user history interface which has implementation default user history in our library. And in user code in any other project that one wants to use this library and wants to customize user repository override user repository. What they need to do they also need to create the same file this file with the same name in their code in their class path. Also with name user repository but which should contain the implementation class of their customized user repository. That's it. So in this case, in your like user project. You will tell the library to use your user repository and not their default user history. Let's try to do it in code. Okay, let's try to do it in code. For example, we have a project where we have a login service. Which wants to use user repository user repository is an interface. And as you see this class right now is suitable for a typical depends injection. It has a constructor which accepts a user history parameter. So it's okay to use this class in like in spring, for example. But what if we want to use this class like in tests where we don't have any spring any depends injection, what if we just want to create an instance of this. Or use like in case of Makita use some static method of what if so which server which user repository should we get. In this case, we can start using service repository. Let's create another constructor. And we are going to find some implementation. So as we saw, we are going to find to ask service loader to give us user repository class. But again, user repository is just an interface. We don't know yet what what would be the. We need to call find first. And if there is no one we probably could for exception or we could get. Well, it's up to us to as the author of the framework to decide which implementation should be the default. In this case, probably we're seeing that the default is postgres user repository. And let's print it out. That was real. Now we can run this method. By the moment only interesting in which repository will be really injected. In this case, we saw that it by default it injected postgres repository. And what if we want to in our project want to override this. And let's create, you know, we have a typical folder as said test resources. These files we are going to get into class but as you know, we created folder called meta inf slash services, and we can create a file named after this interface. We have a full package name and full class name here, and we are going to put some other implementation here. Oh, we already have some dummy users repository suitable for test. Let's put the full name of this dummy user repository in this class. And in theory, when I run the same test game now. Yeah, now it should be injected dummy user repository. This is a way that we managed to customize behavior of login service, we could inject sorry for all this. We could inject into login service some other implementation of user repository by creating such a file in full meta inf services. And why I'm showing it because you can write similarly many aspects of Selenium or Makito on the unit. We don't have too much time so let's quickly see how we can do the same thing in Selenite. How is how customization of behavior is implemented in Selenite again using the same service loader inside of Selenite we have static inject method which like wrapper for service loader. And we have a lot of default implementations in meta inf default services, you can look there and find many default implementations of different services. And when you want to implement some aspects of Selenite, for example, we want to take full size screenshots. Again, let's create a file called photographer. Why this name because Selenite has an interface photographer, which has a default implementation, like web driver photographer, which just takes a screenshot using web driver, a user web driver API, like a miller for you. But what if we want to take like full size screenshots. We just create another class called full screenshot photographer. And I have already and ready implementation of it. Let me just shell this to do it quickly. Now, we can look at this class. Now we see that we have in our project, not in Selenite, it only in our project. We have another class which implements the same interface photographer, you see, but which implements this method differently, totally differently. It uses Chrome developer tools, CDP protocol for taking full size screenshot. So, you know, you see an example of like huge customization of behavior of Selenite. It takes screenshots totally differently, like using some really tricky method. And remember, we have one failing Selenite test. When we run it again, and this test will fail again, it must fail because we put the wrong expected number here. Okay, the test failed as we expected, this was wrong number. And now look at the screenshot. Initially, we saw only a small screenshot. And what's now, I hope, when we now click the screenshot, we will see it much, much longer screenshot. Yeah. Now you see a much bigger screenshot, which includes not only the visible area of the browser, but the whole browser screen. Great success. So, we managed to override behavior of Selenite by creating some Selenite plugin, or overriding some Selenite plugin. This is how it works. And just to mention that Selenite, at some moment, I had to create like system of plugins or extensions for Selenite. And by the moment, we have free, like official plugins for Selenite, for Appium and for clearing elements in reactive framework. And actually we have a lot of other ideas to create some new plugins which are not implemented yet. Full screenshot is almost done. So, and so on and so on. Other brief introduction, what it means, Selenite Selenoid plugin, what it means. If you don't know yet, Selenoid is alternative to Selenium Grid. This is now, so this is a tool for running browsers inside of a Docker remotely. And nowadays this is very, very popular instrument. And the problem with this instrument for me is that many Selenite methods that work by default with Selenium don't stop working when people run browser inside of a Docker. For example, the test doesn't have access to Clipboard anymore. And that's why we had to extract Clipboard service like other plugin and customize this plugin in Selenoid plugin. And so on and so on. There are a lot of things that we had to customize in Selenite in order to make it possible to work inside of a Docker. For example, downloading files which Selenites can do in three ways. They have to rewrite it in Selenoid and do it like differently using Selenoid API. And so on and so on. We have a lot of other plugins like which customize Selenite behavior for Selenoid. You can find this GitHub project here by link. And another existing Selenite plugin is Appium plugin. Appium, if you don't know yet, I believe you know, Appium is a framework for writing tests for mobile applications. Appium uses Selenium interface, Selenium protocol. That's why Selenite is also possible to use Selenite for mobile applications, for testing mobile applications, but we had to rewrite some things. Again, we had to customize behavior of the default Selenite object factory and some few other things as well. For example, Selenite by default saves HTML of the current page when the test fails. But in case of Selenite, you know this must be not HTML but XML. That's why we had customized behavior of this page source extractor. And so on and so on. There is a lot of customization also, done and possible. We even had to customize click in Appium to make it more stable, faster and so on and so on. So this was a short introduction what are existing Selenite plugins. There are some few technical nuances. I must mention that every framework that uses standard Java service loader pattern actually does some customizations to it also. It's like funny. Because service loader in Java is quite low level and it doesn't cover all the details and doesn't answer all the questions. For example, the question about default implementations. In Selenite, we solved it so that we created a separate folder called default services where we have all the default implementations. You can look in this folder to get the ideas what are all possible services that you could implement or override. In Selenite, all the services are single tons and Selenite will always pick only one implementation of any service. Compared to Selenium, which can pick like all possible implementations in many cases. We will probably see it soon. And now getting back how you can customize, probably you wanted to hear it, how you might customize some aspects of Selenium. For example, as I show it, what if we want to override the native client timeout, which is three minutes right now, and we want to make it shorter. Let's try to do it. Yeah, Selenium also uses service loader but with some tricks, but with some nuances. So once again, now the Selenium, typical Selenium script works, it's green, everything is fine. But what if we want to customize the timeout? I found a way. We need to override this class at the client factor. Let's create a file called htpclientfactor. Let's put a different class name here. And let's create this class. Okay, let's create the class of such a name, class native client factor. It should extend as you know, as you know, it should extend htpclientfactor. Oops, I forgot what exactly it should extend, sorry. Let me speak on this. You should extend native client. Now let's override some methods. In this case, we want to override the only method, create client. And fortunately, it was easier to override the timeout by doing like this. We need to create another client config. And by copying the existing config and calling method, create timeout. In real life, it's okay to put timeout like 10 seconds, for example, or 5 seconds. But just for example, let's put really small timeout. Like 11 milliseconds, for example. Really small timeout. And let's use this method here. So, I hope that when we run the test again, we will get into this implementation instead of default. You see, this is a default Selenium implementation. And I hope we will get to our implementation and see a shorter timeout. I hope that my test should fail because this timeout is too small. And as you see, it doesn't work as previously. It was not sufficient to just create a custom native client factory and just to create such a file. It was not sufficient. Why? The problem is that Selenium does it like with Chicks. If you look into the default Selenium implementation, this is a Selenium class. You see an interesting annotation, HTTP client name. Yes, Selenium tries to apply some logic. It tries to find all possible client factories and pick one of them. If we call it, for example, fast, actually, it's not fast, but it's shorter timeout, but let's call it fast. So we will name our factory like fast, but it's not enough. Now we also need to say Selenium, please use the fast factory. I have a way to do it. It's a system property which you need to set to make Selenium use your factory. Let's try. Now Selenium has picked our factory and now our test will fail exactly because of too short timeout. It says that sorry, but I could not do it in 11 milliseconds. I could not open the browser. So now we can just increase this number. Let's see if now everything is fine. Yeah, now it can open the browser. This way we show how we could implement a custom service or we write a custom like some class in Selenium. So a short summary of Selenium, what we had to do. We had to set system property saying Selenium, please use our factory. It's because inside of Selenium there is such a code. Selenium gets system property to get the name of factory. Then Selenium tries to find out all possible implementations of this class, HTTP client factory. And to try to find only one of them which has this annotation which is a client name. This is a custom Selenium annotation and it tries to filter out one of them which has exactly this name that was in the system property. So it's not easy to find out how exactly could you overwrite this class. As you see it's not like trivial and I guess it's not documented anywhere. I just found it by erasing Selenium code and debugging it. So yeah, I mentioned that everyone, every library that uses service loader somehow customizes it. And in case of Selenium, Selenium uses additional annotations like HTTP client name to filter out the needed implementation. And Selenium uses auto service annotation. If I'm not mistaken, it's an annotation from some object like Google or Google Jewish or something like that. Using this annotation likewise Selenium generates this meta in services files during the build. Something like this, I'm not sure. Whatever. So yeah, these are nuances of Selenium that you need to know if you are going to overwrite some classes or to provide custom implementations of some classes. Yeah, once again, this is the same. And one major issue with Selenium as to me, it's like I would say even it's a bug of Selenium that there are many classes in Selenium that are declared as services. So you can find in meta in services you can find like default implementation of these classes. And it creates like impression that you could override them that you can but you can cannot actually let me show one pull request. This is one change that was made at some point in Selenium. As you see, like few years ago Selenium when Selenium needed to get an instance of blah blah blah Firefox driver service builder. It used to service loader to load drivers as builder. And again to feed them which will be suitable for Firefox driver blah blah blah. And if you look into Selenium charts, you will see that many of Selenium charts actually do contain such classes. You know again meta in services and this class and it creates like impressions that probably you might overwrite this class. You can create your custom implementation of this edge driver service builder, but in reality you cannot because as you see later this code was changed so that it does not use service loader anymore. But actually it creates. They need a class by just using new new driver factory new Firefox service and whatever. All these classes are not injected anymore so we cannot really customize them. No matter that they are here required us like customizable services. So actually in reality there are not so many classes in Selenium that you really can overwrite, not so many. It was even a little bit hard for me to find a good example for this presentation. Well, okay, I showed how could you customize behavior of Selenite in Selenium using information in meta in services folder. And you can actually just do the same with many other open source libraries like to unit Makito and so on and so on. So what might be a summary of this presentation. As a user you can always look into any of jars that you are using and look into the folder meta in services as I show it. In IntelliJ either it's really easy to look inside of these jars and look into meta in services and get hint what things of this library could you customize probably maybe if you're lucky. And many other jars also contains this folder. You also have possibility to write something, some classes, even especially Makito like Makito. It has quite a lot of such possibilities, but a little tricky one. Yeah, LiquiBase also, yeah, many other libraries allow doing that. LiquiBase as you see allows overriding pretty much any piece of LiquiBase. You can write really like anything and add your own implementations, customizations and so on and so on. So now we know that you can look into meta in services and play with it and try to customize some parts. Probably it helps you to fix some bug or investigate some production issue and so on and so on. Or just add logging to some parts. It's often might be possible, might be useful. And another summary is for those of us who create some libraries, some reusable libraries or frameworks to authors or creators or maintainers of some projects like reusable projects like Selenite or Selenium. Or just for any other developer, like usual developer, yeah. I think that it's always a good to sync like everything is a potential plugin or everything is a potential service. It's probably a good idea to make every part of your project, every part of the library like replaceable or injectable or customizable. It doesn't mean that you need to do it right now. No, probably it would create too much overengineering. It's not needed always, but it's always a good idea to sync in the terms of replaceable parts and injectable parts. It probably helps you to create a better design of your code, which works well with TDD. TDD also forces you to create like smaller classes, smaller methods, which are easily testable, meaning that they are easily replaceable by mock implementations. These things lead to a better code design, from my opinion. Thank you. I'm ready to answer any questions. If you wish, yeah, please raise your hands. That was wonderful, that was a wonderful practical session, Henry. Just one moment, let me remind that it's not always, you don't always need to do it in reality, because probably it's not really needed, probably it will just cause overengineering of your code. But if you really need, as me, I had to investigate some system of like plugins in Selenite, because people often asked, oh, how can I like take screenshot differently? How can I like change some timeouts and so on and so on. So I had to do it, especially for Epium, yeah, because in Epium, many things look like Selenium, but actually work really differently. And so, yeah, I had to customize pretty much things in Selenite. Yep, that's true. I mean, having worked on both of them, I can correlate that that's a great work. Any other questions from anyone else? Probably, it was a little bit too technical, I guess, probably. Not everyone does really need to know such technical nuances of libraries. Thanks a lot, Andrew, for your presentation. It was a great one. Thank you for sharing your experience and the nuances. How did you solve it? It was all great.