 All right So we are very happy to be here. Hope you had a good time at kubicon Today we are here to talk to you about metrics and then turn testing and how those things can be combined together But first let's present ourselves So my name is Jessica. I'm a software engineer at Red Hat I'm also a contributor and a prover for Portuguese content at the CNCF Glossary I'm one of the maintainers of the observatory and project This is our observability back end. We'll see more about it today And I'm currently working with Go. I'm also interested in distributed systems and also observability So, hey everyone. I'm Matei. I'm also working with Jessica at the monitoring team at Red Hat I'm a contributor and a triage member and at the TANOS project as you can also get from my cool TANOS socks that I got yesterday from the CNCF store and I'm also maintainer at the auditorium together with Jessica So yeah, they're usually interested in distributed systems observability and all of those things Cool, so let's go through our agenda today So the idea today is that we want to understand what our metrics. Why are they important? Why do we need them? Then we would jump into understanding metrics and entrant testing as a concept So like how this differ from the conventional entrant testing What are the advantages? Which patterns can we use? And we also go through different types of tests Then we will show how this can be used in real world applications And at the end we'll have a demo so we can understand how all of this can be used in practice But first I want to ask everyone Who in this room has your application instrumented with metrics? Cool, nice, nice. I see some hands I first want to share my personal story how I got Introduced to metrics on the first place was not very funny But yeah, I pushed a new service to production the code work the PRs were revealed everything was okay in my head at least I had the code there The tests were passing And then the service went live people started to use it I saw some logs. Okay. I got some requests, but I had no idea how everything was going, right? I had no idea how my application was performing. I was a new service So like how am I supposed to do that? Everything is fine, right? Like how many requests do I get like I had no dashboards Like how many requests per second how many arrows? And what about resources right like I don't if I have enough CPU or memory or actually if I Had more if I provision more so that's when I noticed that metrics are really important So why do we need metrics right so We need metrics so that we can get performance insights about our application. So for example You can have your application instrumented with different types of metrics like number of HTTP requests duration of requests number of arrows and Metrics help with making your application more resilient and gives you a better understanding of how your application is performed So why metrics also because we you can answer questions like how much load your application get how many arrows How slow is it right so you can measure your latency? What about resources as important to understand how much CPU and memory application is consuming so as I mentioned before Yeah, you can see if you need to provide more or less resources and configure your application properly So how do you instrument how do you do this right? I think in the last talk We also saw the importance of metrics and I was I think at the end of this was also an example of how you can set up your application but Ideally your instrumental application with metrics you expose slash metrics HTTP endpoint And then there use a client library that then we will help with your code instrumentation and once you have this endpoint with your metrics exposed then You have a server that then we will scrape those metrics and Possibly could store them somewhere. So for example, you can use Prometheus for that, right? Cool, like we saw metrics are important, but what all of this has to do with testing Yes So Jessica will test to like the typical use of metric that we often think about and we understand that this is what the promise of observability brings Right the engineers we want to be able to infer information about the internal state of our applications from the signals We receive and this is of course extremely valuable for our day-to-day work as Jessica was explaining when it comes to monitoring the state of our Infrastructure debugging or tweaking performance of our applications Inherently all of these scenarios and use cases have one aspect in common We're talking about some sort of life application, right? We're observing an existing system in motion whether we equate this to a sandbox environment or the actual production Deployment however when it comes to observability signals, we should not be limiting ourselves only to establish systems Although this is the primary use case that often stands at the forefront of the discussions It is not the only one How about monitoring our applications before they're actually running in a live system? Could this bring us benefits as well and as we were proposing this talk it can We don't often look at it this way, but tests are sort of systems as well If we think about it running a test equates to a creation of a system of components These can be like functions mock services or actual applications and subsequently the system is evaluated and asserted against the set of Conditions although these systems are only ephemeral it does not preclude them from benefiting from observability patterns So the idea we put forward in our talk is simple and powerful Let's take what we're used to from running our systems and apply it in the context of end-to-end testing We want to increase awareness of metrics as a part of the testing landscape and to show that as engineers We can reap immense benefits from incorporating metrics into our testing strategies And what is even better we already get this for free with the instrumentation for our life systems, right? Free things are the best things in life. We do not we do not need to put more in up front investment into our applications No code changes are necessary If you already instrumented applications the testing patterns we will talk about today are automatically applicable So I don't get more value in your testing experience for free Let's talk about this in more concrete terms Having metrics makes it possible for us to make more detailed test assertions Think of a more typical end-to-end framework you spin up a couple of binaries or containers and then you observe them and assert externally For example, if you wrote your tests with a bash script, you might base your assertion on the applications exit code So exit code zero mean success and other codes mean failure However, just asserting based on how applications are behaving from outside might not always be enough Sure the application exited successfully But can we be sure of all of our assumptions about how the application behaved during the test run in contrast? Thanks to metrics we can externalize the internal internal behavior of the application and assert on this externalized information Instead of checking the exit code We can check for example, whether we always responded with HTTP status code 200 whether we reached an expected number of responses Or whether the expected amount of data was processed and so on On top of that externalized data does not only greatly improve our capability to assert We also gain better control over test scenarios Thanks to metrics we can check our preconditions more easily Maybe our application for needs to wait on an event that will occur in some uncertain time in the future It is not trivial to check such preconditions without knowing the internal state of the application If you have metrics, this gets easier For example before proceeding with testing We might have to wait on an application to be connected to a certain number of replicas of another service If you have metrics telling us this value, it's trivial to leverage this for our precondition check This also allows us to fail fast if the precondition is not fulfilled Or on the other hand, we don't have to fail fast But we can wait until the metric reaches a certain value before we continue with our test Or if after certain time limits the The limit has not been reached. We can just terminate the test at that point and even better for more advanced cases We can for example restart our applications We can retry certain operations all together if our metrics are not still not showing the expected values So when we start to piece all of these patterns together We can see that we can create more complex scenarios easily and these do not have to be related only to internal state of a single application, but also to links between the applications Based on observed metrics, we can run one test with service started other test with a service stop and compare the results We can test how applications behave when a dependency goes offline For example, and then how it recovers when the dependency comes back online And lastly we benefit also from collecting various extra data points about our tests that we are again getting for free in this case This can be related to various metadata such as our application's version Or we can gain insights into applications performance characteristics since it's typical to collect for example data about About latency of our request these metrics can give us some idea about the performance and here We're moving a bit into the benchmarking territory About which we will also talk in a bit later Now before we move on to the practical application Jessica will talk us through some of the test types to which we can apply these patterns Cool, so let's go now over different types of tests Like first I want to talk about the benchmark test. This is a very useful test to test performance So it's the focus of this type of test. It helps you identify bottlenecks on your application For example, if you have any if you are introducing a new API to your architecture It can be useful to then test how slow this new API is right So maybe you can create a benchmark for it And the way to do it is that in general you have a function that is as executed multiple times And then for each time you test if the output of this function is compared to a certain standard Yeah, and I think it's nice if you are using already goal link you have this From the testing package by default. So it's it helps you to like just easily create benchmarks But the idea also is that you don't Optimize prematurely, right? So it's not nice if you try to run benchmarks for every piece of code you have Like so it's something to just to keep in mind I Think the other type of test this one I don't know if it's very well known at least for me before it wasn't but this would be the interactive test So this type of test the idea here is that you pause your application at a certain point And then you kind of play with the scenario in progress So like if you have a lot of services, that's maybe a good idea to then see how those services interact together And today, I think it will get more like we will understand better once we show a practical application with it Our demo would be also using exactly this kind of test. So I think it's a bit easier to understand Cool, so we saw a benchmark test interactive test I think those other types of tests. They are also very well known Which is the unit test integration test and entrant test or maybe let's just refresh our minds a little bit So the unit test the idea is that you test each unit of your application so like you in general you have a code base and you have a lot of functions and then You can create one unit test for each function of your application So it's really testing a unit. It's also a less expensive type of type of test. So then That's why you can also have them a lot right because also you have a lot of units probably You also have integration tests and the integration test the idea is that you you test how those units Work well together. So let's say if you have a function a this function a has a function b as a dependency Would be very nice to understand if those two functions are working well together and that would be the integration test The entrant test on the other hand, this would be really like spinning up all the components of your application So all of your containers So it's a more expensive type of test And I think also they don't ideally they won't happen very a lot So you won't have a lot of entrant tests on your application because yeah, like it's it's not not as the unit test for example and Again today will focus exactly on this type of test on the end to end and another thing I want to mention is also that you can have so I mentioned the benchmark test the interactive test But you can also have those tests Combined with those three tests here So for example, we can have an entrant test that is interactive or you can have a unit test That is also benchmark and so on and today. Yeah, we will see also an interactive test and also entrant test So we see how those how they can be combined together Cool. So let's then see how this can be applied in practice. Then I will hand over to Matthew Yes, so let's move on to the practical part We introduced the concept of using metrics in entrant testing in a theoretical manner But let's see how the how these ideas are put to practice What we want to show is that these concepts are universal and can be applied within any programming or scripting language But the concrete implementation will be looking at is the entrant testing framework written in go So this is a framework which is hosted under the umbrella of the efficient go project which is a collection of utilities and tools that are readily available for use by any go project and The end-to-end framework specifically is intended to run complex Workflow scenarios in isolation with help of go and docker The code of the framework was originally developed with the Cortex project and later was donated adjusted and moving to a separate Module mainly by our colleague Bartek Plotka We must have seen some talks from him because he had like 10 talks at the cube con. So he's very famous So the framework has few like defining features that are interesting for us It focuses on testing cluster workloads So meaning primarily cloud native microservices while keeping in mind test scenario readability The scenarios are running the containers environment with help of docker naturally the framework also focuses on Putting metrics front and center, which is the feature most relevant for our talk for our talk today The only does the framework support asserting and controlling test scenario test scenario flows via metrics It also provides self-monitoring out of the box by leveraging the C advisory utility You can monitor the test containers directly and gain even more control and insights over your test scenarios Lastly, it also supports on-the-fly containerization of your code So if you're not running a fully fledged application just a piece of code You can containerize it and run it directly through the framework and Although we refer to this project as an end-to-end framework There are actually multiple usage modes beyond plain end-to-end testing as Jessica has introduced some of these other types of tests The frame the framework can be employed to facilitate this as well The framework enables users to benchmark with the native go benchmarking package while Incorporating external dependencies or you could use it for macro benchmarking as it allows for running testing scenarios repeatedly and Test users can observe behavior and performance of their applications from a macro perspective Last mode is the one that Jessica referred to as interactive setup So this means that you can manually interact with the application and monitor them And this is also very useful for engaging new users because you can quickly spin up the whole setup just with one one command And it's all running on your local machine and you can You can also one you can also monitor it, right? Let's look at some real-world usages We will talk about two projects that me and Jessica both involved with that is Thanos and observatorium So it tunnels if you're not familiar with Thanos It's a CNCF incubating project for highly available Prometheus setup and long-term metric storage You could have seen us at the booth here at the cube cone as well And we've been using the end-to-end framework here to build and run test scenarios for all of our components Because Thanos is actually a number of loosely coupled components and they need to communicate together The end-to-end framework is perfect for satisfying such complex scenarios with a large number of components So it's not always true to ensure that the tests run successfully Because normally the components are long-running services It heavily relies on a certain on metrics as well as metric do and test run control Thanos project features a full-fledged interactive setup as well So this includes everything from synthesizing data for users to play with to spinning up all necessary components with monitoring and tracing in place as well second project that we will mention today and Where me and Jessica we said our maintainer is the observatorium just a TLDR an observatorium It is a one-stop shop for all observability signals It is a collection of tools which makes it possible to easily run a whole observability stack with some additional functions on top such as Multitenancy authentication authorization rate limiting and so on so feel free to check out the project if you're interested And here as well. We have been using the end-to-end framework to a great success mainly with our core component, which is called observatorium API and we use the end-to-end framework to create Complex test scenarios across all different signals that we support metrics traces and locks Because this requires spinning up multiple Dependencies right because we are providing authentication So we need to authentication we need to connect to authentication provider because we for example can use Open policy agent for authorization. So this is another dependency We're also depending on external service for for rate limiting that we need to communicate with And last but not least we also need to back this up by an object storage for which we use Minio and we again just run this locally through throughout this through this through the framework so having said that now let's look at the demonstration of all of the features that we set and For this we will use a more simpler example than than the ones in the real world And this will be just like just to demonstrate and see how this is how this works in the code and then how The actual tests are run. So it's just because mentioning we will see an interactive test I'll just quickly switch this to mirroring so that everyone can See what I'm doing on my screen Or not. Okay. Let me try one more time. I try couple more. I'm not sure now. It's not reverting back Okay, sorry folks This work, let me try to reconnect sorry about that Okay, sorry about that. I'm not I'm not sure if you're gonna be okay now. It's completely Okay, let's one less try So now now my whole setup is Yeah, so now it's not connecting any more I don't know if it's the cable I don't know much about Can you try yours? Yeah, let's try that Now it's the whole screen is wait something's happening Now I have three screens cool Okay, thank you Sorry about that Now it's just everything is a bit messed up My display is not there. It's not recognizing my screen I think Joey tried Sorry about that In case it's not working. There's there will be also video in the presentation So you can watch it for yourself and there will also link to the to the example test because it's it's part of our repository So, yeah, hopefully if you're not able to make it work, then you can still see the results But yeah, sorry about that Yes I Can so I can try to I don't think we have that much time, but yes, so as you can see just this is like a plain Go test is it's a go test file. We're using the native testing package as you can see we're simply we're starting by setting up the Docker environment and Then we have another method to To start our application. We're actually using an example app, which is a which is a simple Which is a simple HTTP server that is instrumented for metrics and we'll we'll be just observing The request these are coming in and checking the metrics on that so doing some additional things making sure that we can we can Access the service on our host machine, so we're exposing the port we're providing the image for the for the service that we want to Use in the start options and in it and then we'll go we we're starting and waiting for the service to be ready, right? second thing that we will spin up is the okay second thing those pinup is the is Prometheus you can see a simple Simple config for scraping we are providing the endpoint from from the app. We previously set up For for scraping so we will see the mattress in Prometheus again We are starting Prometheus setting the config waiting until it's ready And then what we're actually do is here We will see a pause in the test and this is intentional so we will be waiting until Prometheus is scraped Until Prometheus until Prometheus scraped something and we will do this based on the metric which which Check how many head samples have been appended We'll wait until this is greater than 50 and this shows how you can leverage this to use for checking preconditions, right? So before we move on we want to be sure that we already have some data in there We'll also then open the example application in our browser, which is another nice feature of the framework We have this end to an interactive package, which makes it possible to also directly open the service or the sort of the UI That you might have like a web web UI directly in the in the browser so that will automatically pop up for the user once it's ready and What we'll actually want to do here, which is the interactive part that we want to show is that we will wait and For at least five requests, right? So the test will not move on until Jessica has made five requests on on the in the browser on that example app We do this based on the HTTP request total metric We're not constantly hitting the application. We do this with a back off So each time the condition is not fulfilled We back off and we try again and we will also wait on missing on the missing metric So this option ensures if the metric is not there from beginning that we They will wait until the metric appears, right? Second thing that we'll also open up from it is see what has been scrapped actually which metrics we have there to gain some extra Insights into our test run and that's it then we're done. We're finished. We can close the test and now Jessica will walk us through the Since we don't have it set up on this computer, maybe we can Either show the video or No, but it's I can try here So Just a bit hard for me to see but like now we will run this end-to-end test, right? Like we created So let me just get into the right folder Cool. So now we will rate. We will run our make target It's just a make comment so that we can then run the test Cool. So now we see already like some logs. The example app is a started Exposed in the port 888 that and then it's mapped to this another random port Then we say already Prometheus being started. See some logs now We see that we are waiting right because we want to make sure that Prometheus as Prometheus already scraped something like on my Screen I already see the browser So it's just open after Prometheus scraped some metrics then we open in a browser And then now we are waiting for the five requests So I would try to then move my browser here. So you can also see So that's that's our example application very simple. Just print hello from example application And then as exposing some metrics Now we try to do some requests So I'm manually refreshing the page now and then now Prometheus. It's open, right? So I already reached more than five requests Let's maybe check like the metric right see if we if we get something Cool. So here you see the time series in the metric name is a TTP requests total all of the requests were 200 That's a counter metric, right? So we had of like 12 requests more or less And and then all of them were get requests Can also check the graph Like if I change this to five minutes so you have a better view Here you see the peak of the requests coming in you can also check the rate of those requests Like per second so I can check like for the last five minutes like you see that It was like about zero point zero three requests are like not so much Another thing I wanted to show also. I think we should have time Is this lash metrics endpoint? So this is the endpoint That from our Prometheus is scraping And then here we have all the metrics that are exposed so here you can also see The counter metric that we were querying the TTP requests total and also all the metrics And there's a last thing I also want to show How so this so as we mentioned this is the interactive test, right? So that's why it's paused But then if you hit this endpoint this address Then the test will be Finished why now I cannot Do that but I can also maybe Okay, so now I hit this the address Then if I come back to the test I See that the test also finished So like if yeah, if you want to automatize this you can maybe like set up something that calls this address Automatically and then the test finishes. Yeah, I think with that we saw how to set up an entrant test Using the entrant test the entrant framework like how can we easily? Spin up Prometheus instances. You can also spin up tunnels And then with that we had an example application that was exposed some metrics and in our test We could assert all of those things, right? I Also wanted to yeah, I kind of have these lights here But you can also check online like we uploaded these lights So we left the links for all of those resources for the entrant framework and also yeah Like for the observatory and project and also the Thomas project Yeah, thank you so much Thank you