 Okay, so hello everyone, thanks a lot for coming. It's gonna be a bit of a tricky presentation for me because my Fedora doesn't support mirroring, so I have on this screen. So bear with me if I'll be just looking here, it's not that I have to read the slides, I just will have to look at the IDE sometimes. Okay, so the topic of this talk is docker-esque system testing with a dash of chaos. So I'll be showing a bit of demos on the way, but before we start, let's just state the obvious. Software is it in the world. Software is everywhere, starting from our computers, which sometimes fail like mine. Your kitchen appliances, TVs, smartwatches, everything which steals your time and let you procrastinate as much as you want. And most of it still is written by us human beings, even though DeepMind and some other companies which are investing a lot in AI are trying to have a deep learning, improving itself and writing software without our integration. But till now we do write software and we introduce mistakes to quote one of the, probably one of the most important computer scientists from last century, Edgar Dykstra, he said something like, if debugging is the process of removing software bugs, then programming must be the process of putting them in. And that's why I guess we still have jobs, right? So before we dive to the demos, let's step back for a second and let's talk a bit about the bugs themselves and what we can do to prevent ourselves from miserable lives, sleepless nights and hectic hugs to make software running as we would like to. So there were a few which are quite interesting. One of them happened to Tesla S. I'm not referring to the latest ones about self-driving cars, but there was also one which was pretty unfortunate. They have a software responsible for setting the clearance of the car. And on some highways in the States, apparently the clearance was too low and it happened to one of the guys that the car was set in fire because of touching the ground too frequently. So obviously what they did, they just sent over the air updates, clearance went a bit up by default, everything was great. Another one which happened not so long ago, believe me, even though it's Windows 3.1, it happened in November 2015. That was in one of the Paris airports. I think if my pronunciation is not wrong, it's only. So they had a system at the airport monitoring weather conditions. And for some unknown reason, it was running on Windows 3.1 and it crashed. The problem was that within proximity of let's say 100 kilometers, there were only three people still able to fix it. Whereas two of them were on holidays and one I think was already retired. But they managed to get this guy on board and he fixed it in two days. But before that, it was just a huge mess. They had to contact with the flight control over the radio, they were giving all the coordinates. But yeah, this kind of box happened. So the first one was a bit scary. If the first one was a bit scary with Tesla, this one was super scary, I would say, at least for the pilots and passengers. But there are the funny bugs as well. This one happened to the Swiss guy playing in Baden casino in Austria. I think it was a slot machine. And he, what he believed, he won $57 million. But the casino owner said, sorry, that was a software bug. You didn't win $57 million, but we can offer you 100 bucks and the voucher for free meal. So yeah, we can always blame developers that they introduced the bugs like they tried with Volkswagen, for example, with the diesel gate. So what we can do in order to prevent such situations from happening, except that maybe this one was intended, but we normally write tests, right? How many of you write tests? Okay, how many of you are writing tests first? One hero. Congrats. Okay, so automated tests are the way of at least proving that our understanding or our perception of what software should be doing is correct. And obviously, most of them are the unit tests. They are very fast, they are isolated. They give almost instant feedback, but they are not enough most of the times. The risk with relying only on unit tests and having people, for example, abusing mocks would result with having a system which unit tests are testing only in the way how the developer writing mocks was thinking the software works. So the problem with this approach is that we mock everything when component under test is entirely isolated from all the others. So it might end up that the tests are actually testing if our mocks are behaving like we set them up, which is what they call change detection tests. Like when you mock a lot of things and then you introduce something which by definition shouldn't change the internal behavior which is refactoring, your tests overusing mocks would end up with compilation errors or mock errors whatever because you defined mocks and interactions in the tests and this blows up the whole system pretty much. So unit tests are great, we should all write them but they are very frequently, if not done right, pretty risky to just rely on. One technique of setting us free from mocks in unit testing is using sociable unit tests. So we still mock, we don't mock all the interactions, we don't define implementation detail in the unit test but we define mocking behavior for some third party components for example whereas all the other things we wire all together and we just test a bit deeper network of collaborations, collaborators rather than just component and all the immediate levels are mocked. But this is still just a fraction of our system, this is still not giving us a full picture of how the real interaction would behave and we are not able to answer the very basic question, will this stuff work on production? Are we confident enough that our mocked tests are written in the way that we can easily deploy it and have fun and enjoy the weekend? In some environments they do, they have circuit breakers and all the other nice tools to gracefully handle errors but we can also do something on our own as developers before we really hit the production and these are the high level tests which we can run in production like environments. Obviously the proportion between unit tests and the high level tests might follow the testing pyramid so we have a lot of unit tests but we still have a bunch of those more sophisticated tests which can exercise the system overall as such. So when I was reiterating this talk, I guess I gave a similar one a while ago, I changed a bit the structure instead of just jumping to demos and showing you what you can do with the tool I'll be presenting today. I tried to ask myself question, why do we avoid writing high level tests? What makes us reluctant to do so? And there are several aspects or excuses or myths which I will try to prove there are myths which things I will go to show you will help to demonstrate wrong. So the first issue we always can say about writing high level tests is that they are not as easy as unit tests. They require quite a bit of boilerplate. They require us to set up all the interdependencies and pretty much handcraft a lot of bootstrapping code on our own and this is obviously not only tedious and boring but it adds maintenance cost to the overall code base. So there is a tool which I'm involved in, Arquilion, who has heard about Arquilion? Oh cool, good to see some people familiar with it. So Arquilion is not yet another testing framework. It's actually what we say a middleware for your test. This is the tool which takes care of all the heavy lifting, all the scaffolding so that you can still write the high level tests in the similar manner as you would just write unit tests. So you just focus on the business logic and all the noise, all the boilerplate is somehow handled by the magic underneath. And as I said, the main purpose was to fill the gap between writing unit integration and beyond tests which obviously are quite challenging otherwise this tool would not exist. It started as a tool for Java E, primarily CDI, but currently it's extensibility and the extension model lets us create tools which go beyond Java E which I think it's not the fact tool of choice for Java developers these days. There are some other things which are equally important. And as I will demonstrate today, at least a part of it, you can think about using Arquilion for any kind of high level tests because it will take care of all the plumbing but also gives you some nice tooling which takes care of Docker, takes care of configuring your UI test and so on and so forth. And obviously it's open source. So if you have some crazy idea, you can start with filling in the feature request or maybe you can actually hack it if you like. And then we obviously will be more than happy to help you. Okay, so just to show you that it's easy, I mean, if I would show Maven, it wouldn't fit on the screen, but that's why I choose Gradle. Just showing one dependency which have all the other modules already predefined for you. And then the first demo we will be showing with JUnit and Tomy, which is Java E certified container on top of Tomcat. But that's the only Java E part in this whole presentation. So regular unit test, not unit test, but the regular JUnit test which in this case is actually a black box test testing some rest resource would look like this. We would have some initial context where we say where is our endpoint and maybe we have some puff param. And then we want to call a get method on some certain resource. In this case, quite a handy endpoint for people traveling from Europe to the United States to figure out if the amount of beer they are ordering is good enough. With our Quillian, the test is obviously a bit bigger, but still the logic is as concise as on the previous screen. There are two differences and one benefit. So first, if we use JUnit, we just use the custom runner. In addition to JUnit, we also support TestNG which is quite popular for unit testing. We do support Spock, which is very nice groovy based behavior driven development testing tool. We do support cucumber and bunch of others. So we try to leverage the best of breed in JVM ecosystem and just give you a middleware for testing so that you can still use your favorite tools of choice and not be forced to switch to the other one. Then one thing which we need when we talk about deploying regular Java web application is the deployment itself if we talk about application containers. So in this case, we use Swingwrap, which is a spin-off or additional project in our Quillian ecosystem which gives a programmatic API to bundle any kind of archives. We obviously provide all the archives which are defined by Java ESPEC, but we also provide support for generic archives, zips, tarbles, and so on. So what we are saying here is that we want to create a web archive with all the packages where this class sits and that's gonna be a CDR archive. It has benefit of letting you think when you deploy such a service about dependencies and amount of other classes or modularity percent in your application. So having this construct, even though it might look verbose and tedious if your module grows, it's quite a good moment to step back and think if we shouldn't really slice our services or microservices better. But that's not a mandatory if you have already built a package from another module you can just refer to it. And then the test itself looks pretty much the same. As I said, there is one additional benefit that in the previous example we had to somehow know our end point, whereas in this case, our Quillian will give us back the URL or wherever we are deploying. That could be localhost, that could be OpenShift, that could be any kind of Docker container or what not. That said, let's try the demo. So I have two tests which are typical our Quillian test. So one is actually in container test, which means that our volume units converter, which is a CDI beam, will be accessible directly from the test. So the test will be actually executed from within the container. So I'll try to run it. You see it starts jambles with under to underneath and pretty much is as fast as unit tests. But if you would like to test from Blackbox perspective, I mean the deployment changes only slightly. We'd say it's not testable, which effectively means that we bundled pure archive as we want, whereas in the previous example, as we run the test in the container, that would add some additional our Quillian modules, which actually let you run this test in container and provide injection to it. Here it's a regular Blackbox test. So we deploy and we just exercise it from outside as we would exercise any kind of rest API. And again, it deploys, starts all the server, run the test, collect the resolve, and we are done. Breezing fast. Okay. So principles of our Quillian, which we had from the very beginning is first and foremost, the tests are portable. So if you looked at the, if you recall the previous example or the demo is such, there is no notion of target environment. So you don't have any boilerplate, scaffolding embedded container or whatever kind of targets environment you want. Everything is handled seamlessly underneath through SPI extensions. So pretty much if your tests are targeting, let's say, CDI, and you want to test if it's gonna work on Jbos, if it's gonna work on Tomy or Glassfish, the only thing you need is to swap dependency. And as long as the target containers are supporting your programming model, you're all good. No noise, nothing. The tests work as you would write them for Jbos, for example, or Wi-Fi. Then the other benefit is that you can execute them in the same fashion from IDE and from the build. So as you have seen from the demo, I just run control shift F10, which is a shortcut for IDE to run the test. Everything was started up, deployed, executed, results were collected, that's it. And for the build, you just execute the build, the same approach. You don't need to fiddle with configs, you don't need to fiddle with environment variables and so on and so forth, which is in particular very useful because you skip the build. So if you just want to test part of your application or, for example, some UI test, which is flaky, you can just debug it straight from the IDE. You don't need to start anything, you don't need to start application container or whatever, underneath our mid-testing middleware takes care of all of these bits and pieces. And as I said at the beginning, I would like to emphasize it again, we reuse existing frameworks. We don't train them, the wheel, we just seamlessly integrate and provide the tooling so that you don't have to write boring code anymore. It's flexible and extensible, which I'll show with next demos. We have a bunch of extensions which are not entirely related to Java E or Spring or whatever. And I truly believe that this makes writing high-level tests and hopefully you will be convinced at the end of this talk a breeze. You just focus on the testing logic and the rest is just a tiny little configuration because there is obviously a configuration, but it's fairly small. Okay, so another problem we have. So the first one was that writing a higher-level tests is a tedious and boring task with a lot of boilerplate which we need to handcraft. Another issue is that we want tests repeatable. So if you can recall, for example, running QI test, you run the build, it fails for whatever reason and then what's the next step you do? You run the build again. And this is insanity according to Albert Einstein because doing the same thing over and over again and expecting different results which what we are doing with running the build on CI is insane. So what can we do to minimize the risk of having flaky tests and environments which make our tests non-repeatable is to use Docker. I guess every one of you know what Docker is. It's pretty much the way of doing containers even though there is also Rocket and Open Container Standard, but this is one of the very useful tools to start with and we can actually use it for our own development purposes to write more reliable and isolated tests. And for that we came up with an extension which is called Arcadian Cube. So some of the extensions are named after Star Wars, sorry, Star Trek, which I'm not a fan of but some of the team members are super geeky about it. So Arcadian Cube is mainly created by Alex Soto, a friend of mine and a lot of contributors from community and it's steam after Star Trek, Borgs and all this kind of crazy sci-fi stuff. So what is Cube? First and foremost, Cube is a tool which lets you as a tester or software engineering test or developer essentially because we all should write tests that you manage life cycle of Docker containers. It also lets you orchestrate them. We created it before Docker Compose came up so we had our own format which we called Cube obviously which is also YAML based and we used it for orchestrating defining dependency between containers and so on. Now we do support Docker Compose. We also have the support for Open Compose, for example. And other interesting thing is that Cube gives a small abstraction on top of Docker containers or containers as such which we call container objects. So if you want to interact with Docker containers in your test, you can simply inject container object which you defined and for example take the logs or run some commands in shell and so on. So pretty flexible and powerful tool. We do support OpenShift for that. We work with several Jbos teams or Red Hat teams to bring seamless experience for that. We do support Docker Machine and yeah, it's as simple as that. You just bring in the test dependency and then tiny little config depending if you use Docker Machine, which kind of format you have and then where is the file for your services or containers. And then here we're gonna use Tomcat for blackbooks so we can use password. We have to use password to deploy. Kind of contradicts the usage of Docker because it should be all self-contained but just for the purpose of this demo we'll have Tomcat container to which we're gonna deploy. So don't do it at home because that's not the container way but for the sake of simplicity or showing you more flexibility this demo will be done to deploy to running Tomcat container. So then we have Docker Compose. No rocket science, regular stuff. We have two services. We have Tomcat and then we have PingPong and the test is simple. We have some HelloWords, Servlet which we deploy then Kube which can actually return us the IP of our Docker container then also the port which is exposed and internally defined to this and then we can do something for the blackbox test. I mean, this does nothing but just to show how it looks like. And another way of, or it's actually the pure way of using Docker containers which we call steel container less because we don't have a better name yet. This means non-application container containers. So any kind of container, let it be Node.js, Vertex, WhiteFly, Swarm which is not Java E container we can handle anything which runs your services like Go services for example, pure Docker container. And as I said, it runs everything which runs on Docker. Support, resupport, whatever you can think of as long as you can test it. Swingrub gives you a way of defining on the fly your Docker files and it's just simple config. And then what happens here is that we have a generic archive where we're here gonna create a Node.js application with Express.js and then it will work because it's JavaScript. Then what we have is the Docker file template because we want to generate this on the fly because your application as you develop will be changing. So you create your deployment here using this file which will be created by this method we will use Docker file template to edit and then we will start it. So let's see the next one. So I have pretty easy application which is just a simple get using the Express.js. Then I have a test which we have seen in the slide with again using the black box approach for rest endpoints which is rest assured and it's no more magic, that's it. I mean, here's a tricky part where we will have to invoke npm install. Hopefully internet will work. I'm just kidding, I have Node modules already created. So I have here, I watch on the Docker PS so I will show you, we can see together what's going on. It's gonna take a second I think but you will see one intermediate step which is npm install and then we will run the container on the fly. You see there's an npm start so everything worked. I think we have a green bar already. What happened in between is that based on our Docker file template there was created but deleted on the way the real Docker file and the archive which is based on this guy. So the test itself is creating the tar archive here which is called the strongest beer app. If I run it again, you will see that it's gonna be created in this folder. So this way you can work on your application. You have simple Docker file template. It's all isolated and it runs maybe with slower than regular unit test but it still serves the purpose. Except that probably you won't use Java for testing JavaScript but you still can. Okay, so next thing which blocks us from writing higher level test is obvious reason they are slow. So what you can do and this is not really our query on it just more an advice or observation. You can run your testing parallel for example by using your build tool of choice but you can also try to use Docker. So you can run builds in your Docker container as I show you in the previous example but you can actually parallelize them. You can run more than one at the time. And if you use isolated networks if you have some services then everything is configured the same way. This particular example I prototyped for our Krillian extension where we have support for several different databases and obviously we want to run different configurations like a matrix job and see how all this stuff works on different environments. So previous way of doing so was having profiles for Maven and run them one by one and it was taking like from three minutes up to 10 based on your machine. But the example I have here actually shows you that you can do simple stuff with just three Docker containers. The same conflict actually based on the same configuration I had for Maven and what happens here you see I have a post address, three instances of post address, three different application servers then some small Maven container to run my build. Everything runs in parallel. The ports are exactly the same but I'm using isolated networks so they don't conflict. And I have in three minutes, this is a speed up recorded CLI, in three minutes I have three builds done instead of spending nine minutes on the same thing. Fairly simple. Okay, so I guess we don't have to wait until you see that one is gonna fail and the others are succeeded but if you have similar setups think about using parallel Docker builds and think about using Docker as such to have isolated repeatable builds. Okay, and the last reason why people perceive tests which are system tests for example is that they are waste because most of the things I have seen is that people tend to duplicate the same testing logic or the same tests which you have already in unit test and test them using Arcudon for example. I've seen a lot of cases where instead of having fast unit tests with some boundary conditions and different execution paths with mocks people were just using Arcudon with CDI because they can. And this is not really the right way. I mean, it gives you a bit more confidence in the sense that you know that stuff works in your target environment but you shouldn't follow this logic. You can have just one or two happy paths verify that all the wiring in your environment works but all the other things pretty much can be easily done through unit testing. But what we can do with higher level tests is actually try to think about consistency of our system from higher level. So we can think about introducing some chaos. If you know Q from Star Trek whenever this guy shows up it's just a total mess and that's why we named our extension Q. So Q is essentially leveraging Docker but providing several different aspects of chaos engineering, I would say. So if you heard about Simeon Army, Chaos Monkey, all these things which people like to do on production you don't need to do on production you can try our tooling and see how it works locally before you deploy and somebody will call you that on the weekend Netflix is down for example. So we have container chaos which randomly will stop, kill or remove containers so you can see how your other services will handle these boundary conditions. For this we use Pumba which is a Docker container doing that. Then we have also system chaos which will randomly as you define burn CPU fill in the disk, kill the random processes and so on. And the thing I will demonstrate is the network chaos. This is used, we use it, we use the Toxic Proxy for that and this will let you manipulate on the latency bandwidth it can randomly shut down the connection or do the timeout so you can easily verify how fault tolerant your application is or your set of your services are. So it works fairly easy. Normally what you have is you have three services in Docker Compose for example which are linked to each other. What we do seamlessly underneath is we introduce Toxic Proxy, the rest stay the same but having this guy as a man in the middle let us introduce some mess and see how we can gracefully handle some strange color cases. So I'm gonna use the same simple ping pong application then I just add my Toxic extension and pretty much the test will look the same as far as the setup goes but here's the interesting thing. So we have a fluent DSL saying on container ping pong with this port I want to introduce a latency of 400 milliseconds and then in the lamp dies, not really a test but some interaction. So I will show you a bit more sophisticated demo where I have a bit more examples and then we will see on the nice dashboard how all of it behaves. So yeah, first step we have is just a regular get on our deployed Docker service. Then the next step is using queue for 15 seconds without anything so it's just a normal, regular happy path. The next step we have is introducing latency for half a minute, for half a second and this will be with 50% of cases that's gonna be applied and then we again will execute some interaction for duration of 15 seconds and the last example will be just to shut down the service and then obviously we don't catch the exception because we don't care, we just want to see how it works. So let me try to run it and we won't spend time looking at IDE, we're gonna have a look here. So I start hysterics, then you see all the other services which I wanted to have plus the Toxic Proxy and then I can actually start hysterics and you will see that currently everything works fine. I get requests coming, everything is green, nothing really strange is happening. At some point you will see there is a latency so this graph will slowly go down and this will again last for 15 seconds and then at some point it will just blow up, it's gonna be red because the last step of our test is just shutting down. So you see now everything is falling apart. So it's, as I said, it's not a test per se but if you use circuit breakers and any kind of agilogic which lets you handle gracefully some errors that was just bunch of lines of code which let you have the chaos introduced in the network of your services. So I think we are pretty much done all the, as you see no Docker containers run anymore because Kube takes care of everything. And let's recap, let's see what was the whole story here. So the main message is that the unit tests are still the fastest way of testing your business logic in isolation. This is something which you shouldn't replace. I mean, you can think about not writing tests at all but if you see a value in writing tests then unit tests are very important. But the higher level tests which let you do stuff which is not possible to regular unit testing is equally important. You should avoid trap of having to redo done tests which just duplicate but run against the bigger picture but you can use tools like toxicity and our query on Kube to set all of your dependencies and verify interactions and integrations between your services in a very simple fashion. When it hurts, hurts automate. So my case of using Docker for parallel test it was very painful for me all the time. So I just wrote the code which was very painful for me all the time. So I just wrote a pretty simple go program to run the tests in parallel. Our query on can be used for non Java projects even though it's still JVM tool but we are working on bringing it to the other environments. And you definitely need to prepare yourself for failures especially when we talk about microservices where your interdependent graph of everything is growing enormously. You should think about handling unexpected behaviors in the consistent way so you won't lose your users. That's the whole point. And we have plethora of other extensions. So if you don't write microservices yet or you have some services using database or you have some services which are user, if you have some web UIs we have very rich ecosystem of extensions which make it way easier than just regular tooling. So for persistent we have the tool which lets you define through DB unit very easily how you would like to fit your database and then do the verification. For UI we have drone which runs UI tests using web driver but it simplifies the configuration. It downloads drivers automatically you have on top of page objects composition of fragments so it makes the whole code base maintainable and easy to follow. And that's it, I think. Still have few minutes I guess. So I'm open for questions. My name is Bartos Maesak, I'm coming from Poland working for Red Hat in DevTools messing around with Arculeon and other tools. That's my Twitter. So let me know if it was not so hard to follow. That's where you can check what I'm up to because we do all open source so you can see what I'm hacking over nights and days. And if you have any questions I think we have still few minutes so I can address them. Cool, 10 minutes. Should we have a connection through database? Yeah, well I mean, so the question was how you can test the rest resource which also uses database underneath. So if you ship it in your Docker container you can have either a special one which will see the database for you or you can use the extension which will let you do that as well. So then you just deploy it, you will have all the setup like you would normally do, right? The only benefit here is that we provide you very convenient way of for example, sitting the database by bunch of annotations in your test. So that's how this would work. If you are doing a black box testing for rest resources I would rather think about having some import script as a part of your regular test deployment rather than using per test changes. But you can do both. Yeah, yeah. So the question is, how can we define one deployment and reuse it to several test classes, test scenarios? So originally and out of the box this is deployment per test even though you can put it to the super class and just extend from it, this will still mean that tests are isolated and deployment happens per test class basis. But there is an extension because it's all about the extensions which lets you deploy once for the whole test suite. This is convenient for example for UI tests because sometimes it just takes time like a few seconds, right? So if you have 100 test classes that's 200 seconds extra cost you need to pay. So you can use the suite extension that will deploy only once. The challenge here is to have tests which won't affect each other. So that you need to have some stateless architecture or test cases which are isolated in the sense of data they are interacting with. But it's possible. I mean if you do it then you need to be careful and think about what you are testing so that you don't affect each other, right? Okay, so there are no other questions. Thank you very much and enjoy the rest of the conference.