 Okay. Good evening. Thank you everyone for coming to the meetup. My name is Lavanya Mohan and today I will talk about consumer driven contract testing. Before I get started with the session, I want to know what are your expectations from this presentation. Sorry. No, it's not something to do with blockchain. Nowadays API testing is becoming so common. So I want to know how it's been done or what stack has been used for API testing. Okay. So we'll cover that. Yeah. What else? I think one more thing is like if my application want to be integrated with outside world and if that is not ready, how do I make sure my application works integration with this outside world whichever the, yeah. Okay. I'm not sure whether that will be a part of this talk but maybe if I can get it then we can catch up offline and discuss that as well. Anything else? Okay, great. I'm pretty sure we'll cover touch up on that. Anything else? I want to know like, for example, if there is multiple workflow and multiple microservices are there, how can we handle it like? Okay. Yeah. Okay. We'll touch up on that. CI? Yeah. Okay. We'll talk about that as well. All right. So let's get started. I'll tell you a story from one of the companies that I worked for previously, a project from that. I'm not selling the project by the product. Okay. But yeah, we were working on an app which is an OTT that is over the top entertainment app. So something similar to your Netflix or hook. You could stream videos and watch them online. The things that we had were we could show users premium content which meant that only some subscribed users, like only the subscribed users get to see this content. But then there are whole bunch of free content which everyone can watch. So it was a premium model which meant free users can watch as well but they have limited access to content. And if you are a subscribed user, you can watch everything. And free users would see ads. So that was another way of monetizing. Apart from subscriptions, ads was another way of monetizing. The other thing that we had was offers. So even if you're not a paid user, but you've been on the system for quite some time, then maybe we would allow you free access to premium content for certain duration. Or maybe if it's your birthday, then you can watch the premium content for free. Things like that. I will talk about the architecture in a bit. This is a very simplistic view. We had two services. We had a lot more but let's focus on two. So it was a microservice architecture. We had a lot of services. Let's focus on two. Billing service, user privilege service and the app, of course. So now let's talk about the billing service. As the name suggests, it dealt with the subscriptions of the users, the billing. And there was one API which just returned whether that user is a subscribed user or not. So if it returns active, then it means the user is a subscribed user and has an active plan. Otherwise, it's an active. And it returns other information like what's the end date of the plan, when did the plan actually start and so on. Then there is this API service in between called user privilege service. This talks to the billing service and if the billing service says that the user is active, then the user privilege service will say, okay, now this user can see premium content, which means show premium is true. And we say do not show ads. So show ads is false. But let's say the billing service said it was an active user, then this user privilege service might say show premium false and show ads true. But then it's not just that. It has more logic. If it's your birthday, then it'll say show premium true, but then maybe you also need to see ads because we're letting you see premium content. But it's okay if you see ads as well. So those kind of combinations, all that logic was part of this service. And finally, we had our app, which just took the data from user privilege service and based on that, either let the users watch the content or not. Any questions up to this point? All right, so there was one day where we got a call late in the night and the customer support team said that all premium users, subscribed users are complaining that they cannot watch premium content. That's very scary, right? They are the top most customers that we had and they were unhappy. So we decided we have to debug this. Just knowing whatever I spoke about, like the simplistic view of the product, what would you do as a first step to debug logs? Okay, yes, that's one thing. What else? All right, if you're seeing logs, which service would you look at first? Would you look at billing or you would look at billing service, right? Yeah, so that's pretty much what we did. We first hit the API and we saw the response. And when we saw the response, it said user is active, so it looks like billing service doesn't have a problem, right? So next what we did is we looked at the user privilege service. So when we hit the API of user privilege service, it said show premium falls, okay? So basically what happened was billing service said, yeah, user is active. The user privilege service said, but user cannot watch premium content. And that of course meant that very angry customers, right? So what do you think went wrong? Which of these components has the issue? Billing? Yeah, that's... Why would you say billing, sorry? Oh, let's go back one step. So billing said it is active, right? And the user privilege service said that show premium is false. So you're saying user privilege service, right? So we also first thought that it would be user privilege service which is doing something wrong, and we checked what had changed. But there was no deployment for like three days. So it was very unlikely that something in user privilege service broke. And there were no monitoring alerts as well. So then we looked at billing service, whether there was any change there. And there was, what were the changes? It was just lint error fixes, nothing else. Now, have any of you used any linting tool? Okay, very few, right? So basically it is just a tool that shows you clean code, like code itself. So it might give you, based on what tool it is and what you configure, it might give you errors like your method is too long, your line is too long. Maybe you should declare this variable as private and so on. So in this case, for example, it said this particular variable should be final, right? And there were many such errors or warnings when we integrated, when we added this lint to the billing service, right? And the engineers just fixed all these code quality check errors and deployed the code, right? So then we looked at the response again, what was the previous response and what the current response is. So the one on the left is what the response was earlier, and the one on the right is what the response was after deployment. Can you see any difference? Status? Case, that's right. So status became from uppercase S to a small lower case S. And why would this have happened? Because well that linting tool would have said convention is to use a camel case with small letters at the start. So well the engineers fixed it. They fixed the code quality, whatever, right? That kind of an error and just deployed it without thinking much about the contract. And then that's why the contract changed. User privilege service didn't realize that it was getting status active and it started saying no, this user cannot watch premium content. Correct. That's the question. So why did the test not catch it? Very good question. Why would you? Can anyone just guess? So okay, let's first talk about what kind of tests we had. We had unit tests, we had integration tests, we had end-to-end UI tests. So ideally something somewhere should have caught it, right? Again unit tests you would test individual, let's say public methods almost. So ideally if your code was tested then this should have been caught. Did it get caught? Was it caught? Actually it was caught. But since the engineers were fixing thousands and thousands of lint errors they just fixed the test as well in the wrong way but then they just changed the test to expect something else. They changed the test to expect a smaller. Now integration tests, we had that as well. Okay, there is some difference in the way people imagine integration tests. So some people say that bring up one service tested from the controller layer to the service layer to the repository DB whatever it is that just take one service and testing it from the controller till the database that is integration test. Some other people think like imagine it to be a testing interaction between two services. So if user privilege service and billing service are interacting with each other then we start both the services and test from the user privilege service and check the response that actually billing service gives them. So spinning up both services and doing a test that's another type of integration test. So the first one we can call that narrow integration test and the second one which covers both services we can call that broad integration test. Which ones did we have? We had narrowed integration test. So we didn't have a test which would spin up both these micro services and communicate. We had just like a spin up billing service test from billing service controller and that's it. Did this test catch the issue? Yes it did, similar. Since we were fixing everything we fixed the test, right? And end to end test. So in end to end test you would have all your services up and running. You would install the app, you would launch the app and verify that way, right? So this should have caught it, right? What happened here? There is no test for that. That's right, we didn't have a test for that at all. So we found the error, we fixed it and we deployed again and customers were happy again. But we don't want such issues to happen again, right? So we want to add some test which will help the developers understand that there is really something wrong and not just fix it like how they did, right? What are the options that we have? Can any of you think of some options? What would you think you would do in this case? Change the DB value to the next date and write a test for that. Change the, sorry? For example in that case we can run the con job and change the date and check whether the API is retaining the status like... So that's more like an API level integration test is what you're saying? Yeah. But we already had that, right? And they just fixed the test. Any ideas? More test data. Sorry? More test data. More test data? I had, sorry? More test data. More test data. Would you want to test the same thing multiple times? Maybe that will just add up to the test run time as well, right? Testing each cases. At which layer would you have that? At the integration layer. Okay, maybe. So I guess what you're saying is more like have a broad integration test which would spin up both services, something like that. Okay, fair. That was one of our options actually. So we thought that maybe we could, instead of having a narrow integration test which just tests one service, have integration tests where we spin up both these services and do assertions. But the problem with that was one thing. Our microservices were managed by separate teams. They were written in different languages. And if we really did an integration test like that, then who is the real owner of the test was again in question, right? Plus, just spinning up these microservices and doing all that is a little costly. It's a little slow. So we thought maybe that's not probably the best option for us. The second option that we had was to add in the UI layer. But UI tests again, right, are like have all the negative points of a broad integration test as well. Plus, like if you had attended the previous meeting, somebody spoke about how flaky they can get. So this was really out of question. So finally we decided that we will go ahead with consumer-driven contractors. Now before I talk about consumer-driven contractors, let's understand a few terminologies. We have our billing service, right? And that is giving data that exposes an API and gives data to somebody else, right? So the giver of the data is called provider, right? Then we have the user-privileged service, which takes the data and uses it and does something with it. So that is the consumer, okay? But now if you look at user-privileged service and the app, then the app is actually taking data from the user-privileged service and using it. So in that case, the user-privileged service becomes a provider and the app becomes a consumer. So what it basically means is your services can be consumer and provider, but for different interaction points, okay? All right. So now the consumer-driven contract test is a test that is run on the consumer side, from the consumer side. And the advantage of that is that we do assertions of only those fields that matter to us. So in our case, user-privileged service cares only about status field, right? It does not care about the other three or other hundred things that were returned by the API. So the test would only have assertions on the field that it cares about, right? So if the provider changes, but it changes the plan and date, it changes the key of the plan and date, it adds new keys, or it changes the type itself from, let's say, you know, plan and date, we change the type from string to a long, then the consumer test wouldn't fail because the consumer is not really affected. But if we change the status from capital S to small s, then the consumer test will fail because that's what it cares about, all right? So for doing this consumer-side, the consumer-driven contract testing, we used a tool called PACT, it's open source, and it has support for many different languages, right? Now the advantage of PACT is that we can test the consumer and the provider independently, and this is done using a mock server. So I'll just show you that right now. So let's first look at the consumer-side verification, okay? We have our user privilege service, the consumer service, and a mock provider. So on consumer, we will mock the provider, right? And what we do first is we, the test will define interactions. So what are interactions? It basically says, like it tells the provider, the mock provider, that if you get a request for this endpoint with these parameters, these headers, then you should respond like this. With these headers, this should be a response body, all right? That's basically defining interactions. So that's the first step of your test. And after that, your test code calls your real code, right? It's like a JUnit test, if you think of it. It is like your unit test. So you would call a method in your code. The code would make an actual HTTP call, but instead of calling the real provider, it will call the mock provider. Okay, the clicker doesn't seem to work, right? So it will make an actual request to the mock provider. And if, like the mock provider will see if all the request params, as mentioned in the definition, are there, and if yes, then it will return the response back. And once we get the response back, we do our assertions. Okay? Any questions here? Yeah, so I'll show you that. Right, so if you look at the code, oh, okay, sorry, just before that. Now, why is this packed mock provider different from any other mock server, right? Is because it records all this in some ways and generates a file called packed file. And packed file is your contract file. So this file can be shared with the provider and providers can run their tests to see if they are meeting the contract. All right, so that's how it is different from a normal any other mock server. Now, to your question about defining interactions, so this is how the code looks like. So in your test, you would say, you would say it's a contract between your billing service and your user privilege service. Right? You would say if you send a request to the billing info endpoint with these query patterns and it is a get method, then you should get a response like this. Okay? So here in this case, we are, ideally in our case it would be just status. We don't really care about any other field. So that's all. It's a minimal response that we're expecting back. Right? And one important thing is, so in this case we are assuming that the user is in a particular state, right, is present in the database. So when we want to run this test on the provider side, we need to tell the provider that we're expecting this. So that's called state and we can define that as well. So this line which says given user one is as an active state, that's basically telling the provider that this is what we expect to be, like the initial preconditions. Okay? And so this is your defining your interactions part. After that you will have your real code which is very simple like any, so this is, okay, this is JONIT. If you're using any other language, it will be similar to your unit testing code. So it just calls, makes a call to the, it just uses the gateway method which makes a call to the provider mock in this case, gets the response, the code will do some deserialization, we convert it into an object and you just do a verification that whether the status is active or not. And this basically gets status method is just taking the status from that JSON response, right? It's just taking a status with the capital S. All right? Any questions? Okay, so this was the consumer side. Now let's look at the provider side. Provider side tests, so we share the packed file, assume we've shared it in some way. What will the test do? It will just replay that request exactly with those headers and parameters. And it will check the real response from the service. So this is a real service, real provider service and a mock consumer in service, right? So it will check the response it's okay if the response has 10 values, it should at least have those values which the consumer cares about. So it should at least have status and user ID in this case. All right? So that's the provider side verification. Now provider side verification actually packed handles on its own like a lot of things on its own. So we don't really have to say much except where to get the packed file from. So this could be a file itself, we can say where to get like a file in your file system some location. You can say that or it could be a URL or you can use pack broker. I'll come to that shortly, right? So this is the almost the only thing that you need. Apart from that, since the consumer had said that it expects the user to be in a particular state, that's something you have to do on the provider side. So this is in this case we're just adding that user into the database. So this runs before the test actually does the verification, right? Before the API call is actually made. Okay? So on the provider side it's just this much. You just would set your state, tell the test where to fetch the packed file from and it does everything on its own. All right? How does it compare with Postman? So multiple things. So here we are looking at contract testing. In Postman you would just probably do a post request and then verify the response on its own, right? Here we are driving it from consumer side. So the consumer says what it needs, what it expects, and then you do the assertions on the provider. Does that answer your question? Suppose if you're putting an API request, so there is a way where we can write some quotes that we are expecting this value in the response. Correct. And so, okay, one more thing. I think Postman recently also has developed something to do consumer-driven contract test. But okay, I have not really looked at that in detail. But otherwise the API request and response that you're talking about, that would be a normal integration test as well, right? Like if you, in Java world, you might use rest assured, which will call an API endpoint. It'll go through, do whatever, the code will do whatever and return the response and you will have assertions on your response, right? But here we are saying that we had those tests, but still the developers went ahead and changed those tests, even though they made breaking changes. So we wanted a way to have a test across these microservices, right? And we want a way to, for the consumers to say that this is what is important to us and this is what we expect. So if the provider changes, then these tests should fail and it should alert because there should be no way to do a deployment after this. Like the provider, if provider makes a breaking change, then they should not be able to deploy. As in they should not be able to fix the test on their side and then say it's good to go. If tests are driven from the consumer side, then either the consumer has to change or they actually fix the code, not the test. I just want to understand, so here what you are doing is like the developers out of the code, right? Then the deployment, then you are doing the testing, right? No, it's not after deployment, it's prior to deployment. It's like your unit test. It's kind of unit testing that you are talking about. So you have to give more importance to unit testing, right? It's like a test in between your unit and integration. I understand. Yeah, it will run much faster. Any more microservices? Yeah, you can have contracts for all those microservices. This is in Java. The demo code is in Java, but PAC supports multiple languages, Python, Ruby, many JavaScript. So you can use whatever you want. And your provider and consumer also don't need to have the same language. Because this PAC file which gets generated is in some sense language agnostic. It's a JSON file. And PAC will handle the testing part, right? And we can test those independently. So consumers, right? So consumers actually cares about these three fields, for example. That's the contract between the consumer and the provider. Now, provider can give 10 additional fees, and that's fine. But if the provider, basically you check on the provider side whether at least those three fields are set back, right? So consumer generates this contract on one side and then later this contract is shared. That is the PAC file. It's shared to the provider and the provider does tests on its side. So consumer and provider are tested at... Sorry? At the same time? No, they're tested independently. Time doesn't matter here. I'll come to that. I'll probably show you the CI pipelines and maybe that'll help understand. PAC handles it on its own. And it's a JSON file that gets generated, actually. I expect I can look for something like status and for some other things to respond, right? Take care of it. So when I create a list, who makes the list? As a provider, I need to maintain those responses. No. As a consumer, so you would have 10 different consumers expecting 10 different things. A provider? Right, you're a provider, you don't have very much. Consumers will write their consumer side tests and they will mention what they expect. That many PAC files will be generated and that will be shared. So the provider will just pull all those PAC files. Whenever there is change on the provider side, it will pull all those PAC files and do verification for all. Any other questions? Yeah. We can mobile app and user-provided service as a one test and we can have billing service and other things. In some sense, it is like that, right? Your consumer, for the... For the side test, you would define interactions, you would define different things and that would generate a PAC file, right? For your provider side test, you would just pull the PAC file and do that. So they will be on the same service layer, but it would be different tests. Does that answer your question? Do you mean this? No. So this is... Okay, this is the provider side code and here I'm using the repository. So provider side repository to actually save information in the provider side database, right? So this happens before the provider side test is run. So that when it hits... Like when the mock consumer hits the API, the actual code will run, it will fetch the data from the DB and then it will return the response, right? Whatever, serialize it and return the response and then the assertion will happen, right? Now, these are not deployed instances. They are just... Like if you're using Gradle, you would just say Gradle W test and whatever that does is what is here, okay? That's true. In our case, luckily we didn't have that many. But yeah, that's true. So there is Pack Broker which does a lot of... Which helps you in these things in a lot of ways. So it deals with having those contracts in place, versioning of the contracts and so on. So there is a tool which kind of helps you there. But then again, yeah, if there are 100 Microsofts, it's going to be not very easy, right? That's true. But this one solves another issue, like which we had. So our APIs were returning very big responses, okay? Like what would usually happen is some features would be prioritized, like API teams would do it and then it would be scrapped later. So the consumers didn't do it, right? Now, it wasn't removed immediately. And over a period of time, we didn't know which consumer was using what data. And it felt like a lot of these keys are sort of redundant, right? One approach is to talk to an android there and say, you know what, tell me what all you need and then I'll remove others. But the problem with that is one, you're looking at only the current code, this. So you don't know what has been released previously. Second, if you want... Like if your iOS app consumes different values, then you have to talk to that team as well. And there is that error in communication might happen. So in such cases, this helps because then you know what fields you can remove and then you don't have to worry about it at all. So that was one of the other use cases that we had. So that's why we went ahead with it. Okay, so let's move ahead. So we said that the consumer side generates that path file and provider side uses that path file to run its test to do the verification, right? So that basically means that you have to share the path file. One way is to share it via file system. So if both can access the same file system, then you put it in that location and the producer can just pick it up. Or through some URL, or you can even push it to the R gate repository almost, right? Something like that. Or finally, pack broker. So pack broker is again an open source tool developed by the same people. This does versioning as well. It handles versioning, it handles a lot of things. So you don't really have to build everything from scratch. So this is what I use for the demo as well. Now, if there is a change on the provider side, you don't upgrade the path definition file because it's still a key or whatever, right? That you, as a consumer, don't really care about yet. But if you, as a consumer, now start consuming that new key, for example, for a new feature, then it's when you would update it. Because now you care about that key. So you wouldn't update it on provider side changes. You would update it on your own side changes, like consumer side changes, okay? All right, so let's look at it all together with CI. Maybe it'll give us slightly more clear picture, right? So first is the consumer side test. So we would just run maybe a unit test and then the pack test and then publish the pack. Publish the pack file, which is generated, okay? So running pack test is also like running unit test in some sense. So you can say Gradle W test. Or here I just made it a different stage by giving, like, which folder it needs to look at. But it could still have been just one stage, okay? And publishing path, so we would publish it, whatever it, like put it in a file system or some URL, which we're going to use, like the provider is going to use, or publish it to pack broker. So to publish it to pack broker, there are certain plugins that we can use. Or it could be just a curl command, which does a put to the pack broker, okay? So this is your consumer side pipeline. And this is how the UI of pack broker looks. So right now there is only one contract between user privilege service and billing service. If there were others, then you would have many more. It just tells you which contract, which consumer and which provider have a contract, okay? Then second step is to have the provider pipeline, right? Now that you've generated a pack file, the provider would pull the pack file and run the test and check whether it actually gives the correct response. So on the provider side, again, you would just run the pack test, that's it, right? So let's say the provider, in our case, billing service made that breaking change from capital S to small s, right? So then the test would fail and the pipeline would go red, why? Because the pack file says I expect something in capital S and you're, I don't see that in the response. And if you look at the error, so this will, this is like your normal unit test error. It is very targeted. So it says exactly what the issue is, okay? Now, in order to fix this test, like in our case when we had the unit test or integration test fail, the developers just went and changed that s and the test pass and they pushed the code, right? In this case, to fix the test, you can't do that. Why? Because you need to change the pack file, that's one. Who will change the pack file? Like if you really wanted to go green with the existing provider code, you would have to change the pack file. But pack file is generated by the consumer, right? So then you would maybe go and talk to user privilege service and say, hey, you know what, can you change? And they say, no, we can't because we expect it to be capital S, right? So in this case, like even by mistake, your engineers cannot make a change like that and fix the test and push it. It kind of definitely blocks it, okay? Clear? All right, so this was how the provider side, like if they make an issue, then make a breaking change, then how it fails, right? But what if the consumer makes the breaking change? So let's say there was no change in billing service, but you saw the code earlier, right? User privilege service expects a capital S, correct? What is the... User privilege service team did linting fixes or something and then they happened to change this. They happened to change the status from capital to small. What would happen then? Correct, so two things would happen. One is if they run the test only by changing this code, then the consumer side pact itself will fail because the test earlier was expecting... And then the code will start expecting small S, but when we define interactions, we would have still said capital S, so consumer side test will fail. Now if we change the expectation of the response, like in the definition itself and change that also to small, then the consumer side test would pass, but it will generate a new pact file, correct? Because now the contract has changed, like consumer is saying there is a change in contract, right? So it generates a new pact file. So whenever a new pact file is generated, the provider side test should run, okay? That is important. So when the consumer says that I am expecting a change in contract, the provider side test should run, right? So the consumer will generate a pact file, it will be published to the pact broker. Now there are webhooks in pact broker where we can say that if there is a new contract, then trigger the provider pipeline automatically, okay? So we can have that, like this is one webhook which says if the contract content is changed, then trigger the billing service build, okay? So in this case, the billing service build will get triggered, it will go red for sure, because billing service is actually still returning capitalist. And fine, we're good. But is that enough? Okay, it's not really enough because what would happen is consumer side team would just push their code, but who's going to look at somebody else's pipeline, right? I don't think I would take the effort to look at somebody else's pipeline, right? So that's not enough. Just triggering the pipeline is not enough. You need a feedback back on the consumer side itself. So the provider needs to say whether it was, after the provider pipeline ran, it needs to tell the pact broker again whether it was a success or a failure. So in this case, it will say it was a failure to the pact broker. And that again, just sending that message back to the pact broker is also not enough because you're not going to look at pact broker UI either. The consumer pipeline needs to know. So for that, there is a Ruby gem called Can I Deploy, which is again developed by these folks who actively use pact. It keeps polling the pact broker and keeps checking for the status. So we can add that in our consumer side pipeline. So every time there is a change in pact, it will wait for the response from provider side. And then if it's green grade, it'll all go green. If it's red, this stage will fail. So now the consumer cannot deploy because their pipeline is red. So let's do a quick recap. We have consumer. Let's say the consumer makes some change. Then it will generate a new pact file and it will publish that to the pact broker. If there is a new contract, then pact broker will automatically trigger the provider pipeline. Then the provider pipeline will return the response, like the verification status if it's pass or fail. And there will be something which will keep polling the pact broker to know whether it is a pass or a fail and then will tell the consumer, yes, you can go ahead with the deployment or you can't. And if there is a change on the provider side, it will always pull the pact file and do assertions again. Is that fine? So I can show demo. I think I've already... I have overshot time. So maybe I will just show the video instead of showing the full code itself. Sorry, is it visible? Okay. So this one has consumer side changes. So here on the consumer side, we removed status and made it a small s. And we changed the contract as well. We are now saying we expect it to be a small s. Okay. And let's say we push this code. Now, currently everything is on my local system. So the pipeline won't be triggered automatically. So I'll just manually trigger the pipeline. Okay, it's not very clear, I guess. But yeah, so ideally when you would commit, like push this code, your pipeline will be automatically triggered. In this case, since it's local, I did it manually. So your unit S would run, your pact S would run. And if it's all green, it will publish the new contract that is generated. So here, when you see that, you can see that it was, like a new contract was published right now. And till now, there is no verification that has happened. Now, when a new contract is published, this billing service, the provider test got triggered automatically. And it will run verification on its side. Now, this will fail because it actually returns a capital S, but your pact is saying it expects a small s. Okay. Yeah, so it failed. And it will return that status back to the pack broker. And then your, can we deploy them? We'll get the status and your consumer pipeline will go red. All right. Clear? That's pretty much what I had wanted to show. I'll just show one additional thing. So pack broker maintains versions as well. So you can almost see which version of consumer, with which version of provider was green, which was red. So it maintains this history as well. Right? Okay, that's pretty much what I wanted to cover in the session. These are a bunch of references. So pact has great documentation. That's definitely worth reading. There's also video from folks, from Atlassian who've spoken about this. And it's pretty nice. Definitely must watch. I mean a good watch here. And I have also, whatever code that I used for the demo, I have pushed that to get. So you can have a look at that as well. All right. Any other questions? No questions. Okay. Thank you.