 Thank you for joining, showing off here, I am Gopinath Langote, I have been working as a software engineer at N26, we are based in Berlin, we have a couple of offices in New York, Barcelona, basically we are building our digital bank. Today's talk is about confidently releasing microservices with consumer driven contracts, I know there are too many buzzwords here, so let me you know set up some vocabulary around these words, the first one comes is a confidently, what confidence in this context means is going to production with the confidence that nothing will break, having 100s of microservices changing 1 microservices with small bit and pushing it to production having the confidence that not other 99 microservices will not fail or any existing functionality will not fail is the confidence here. Second is releasing, this is an ability to go into production, in short we can say it as a continuous delivery, right, so how confident I am to go to production or how is the ability of team to go to one service to the production, this is what a releasing means here. Microservices as you might be aware about this having multiple components, multiple individual units of pieces combined as a one product and going to your production, that is what microservices in short. Consumer driven, imagine a very classical problem of client server architecture where client being your consumer of data served by or pages served by your server, right, so in other way we can call it as a provider, this is going to be our new vocabulary for next of our remaining 40 minutes is consumer being who consumes your API or who consumes the data, the service which are consumed the data and the provider being who actually provides the API or the data to other services. Contracts, I mean any other contracts we write, I mean for our house contract, right, so the contracts are commonly understood way of saying something, for example there is a client and there is a server, client understand that okay my browser is old way and I cannot understand its mobile device, I cannot understand the big screen, just give me a short screen way and the server should respond with that specification or any other way for example a service having, let us say I am having a payment service and I need data from a user service and I expect that it should return me say name, email ID and blah blah, right, so in some specific format that is what a contract means here. As we now vocabulary setup, I will walk you through the classical problem that I have faced in one of my project and how we solved that using contract testing. The little bit about the problem statement was there used to be account service and there is a service called user service, of course we have been having 200 plus microservices in our organization and more than 20 agile teams, these two microservices were owned by different teams, account service is owned by payments team and the user service was based owned by growth team and account service used to the primary specification for account service was to generate some monthly account statement, your bank account statement or weekly you know in that matter account service used to create some PDF format file. For that you need to also print the user's data so that you can understand this transaction as by users, to do that account service used to call to user service, hey this is the API slash users slash give me the details of user one and in return user service used to return ID, user name and user email ID and everything was working fine, everything in production all those two services are compatible with each other and everything was working in in production. What happened is the genius developer like me thought of doing some refactoring, what I thought is why should I return user name or user email ID though I know this API is only for users, why should I write user always as a redundant prefix, why can't I just return ID, name and email ID because I know I am returning user's data, right. So I did refactoring and on what happened is you can see that in GIF what I thought of will not happen but everything broke as we had couple of our test pyramid ready in our existing organization having you might be aware about this traditional test pyramid having unit test at lower level then mocked integration test mocking your other service and behaving like the other service and then on top of that a functional service functional test which sometimes people call it as a system integration test where all those services and front end back ends coming together and working as a one application and you are running test on that or sometimes people call it as a behavioral testing and the last is of course the QA engineers manually testing your features. We had all of this and let's see how where we caught our problem of this particular refactoring. Talking about unit test as everyone you know it's just method level you are just testing a method behavior maybe some branches not even aware about more than your one single class maybe a multiple class but not aware about the other service, right. So having account service I am writing an integration test for it which calls to user service and in that integration test I am assuming that the class level unit test is not aware about user service at all right we are just mocking classes so that way unit test will never going to catch this problem of service to service communication so for us for this problem this is not trust what you to trust. Then we had second level of test which is like a mocked integration test wherein use account service used to write an integration test and not rather than using actual user service mocking the user service with some mock server right. So how we usually write integration test such integration test we set a first expectation saying that hey whenever there is a request for user one return may be this and you write actual test saying that okay call this URL you call the URL it actually go to your mock server not the user service and then mock server as expected will return you the same response and you validate the response all green right but the problem with this was the user service was mocked user service was not real right so user service will the changes in user service this account service is not aware about right even though integration test was in place account service is not aware about this refactoring and this integration test was not not trustworthy for us why because there is a note feedback look happening here there is account service in isolation is not aware about the user service changes that was the problem happening here apart from just checking the JSON structure and the serialization deserialization part this integration test was not more than that and then next level of test was a functional test wherein all the services up front end back end everything and your behavior user journey started doing right you actually user go login on your website click on download account statement you download the file and file will have your name and that was your whole integration test and definitely this functional test is going to take a lot of time because it needs all the services to be up and running with all the new versions and then you are testing it of course it come at the top of your test pyramid is going to take time but end of the day it was trustworthy it caught our problem right I mean actually the survey someone changing like me did refactoring and caught this problem in a functional test and definitely running functional test is going to take lot of time because functional test comes after your all the pipeline goes green and then you last day you run the functional test and then go to some higher environment right so it was taking longer feedback look we can see user service changes small changes account service got to know about it in a long way after a while running functional test at the same time functional test are going to take a lot of time should I write functional test for each service to service communication let's say there is no API there is no public feature I'm launching is just an internal service to service communication and there is no front facing application that we sometimes write and should I write functional test for it if I'm started writing functional test for each single communication happening between service to service having 200 plus micro services is going to take days to run this functional test right and of course the feedback look will be low much longer but end of the day it was trustworthy we found the problem and the last was of course manual test I mean we are we are engineers we are humans right we can make mistake and we can just ignore this small test to run at the same time if there is a single change in API should the QA engineer sit and write sit and do manual test for each commit going to production no right so the problem in the whole picture was getting having feedback look at very longer time some users some service changing some APIs other service are getting to know about it or way longer of way longer after going to production right so that was the problem which is happening what we wanted to have is no service should go to production without complying other services for example I'm a provider service is a user service or let's say authentication service which commonly is the pattern that people use and let's say authentication service change their API saying that okay okay if the rather than 403 I'm going to return in some cases 4x right say 402 or 401 in that case is account services just go to sorry authentication service will just go to production and other service are not aware about it to make other service aware it's going to take a lot of time when the other service going trying to go to production that time you'll understand okay functional test are not working and it's going to fail so it was longer feedback look and then we came to know about this term called contract testing which is sometimes people called as a microservices testing or T reading microservices in other hand so where where this contract testing fits in like when so as functional test was already solving our problem and unit test is not we are expecting to solve a problem the contracts test testing itself will fits in somewhere in your mocked integration test where you are doing which is little which is little heavier than your unit test is just not aware about just one class or one method it's more than that but at the same time it's little less than functional test because it's not aware about all 200 microservices it's just aware about me and the service I'm communicating to not more than that so that's why this fits in somewhere in contract test the level so which is the first problem got solved of having longer feedback loop and the second was having no service should go to production without complying with other and how that will get solved we'll look at into it at high level contract testing itself the paradigm itself can be divided into two parts one being consumer creating contracts in our case account service saying that I need ID I need user email ID I need user name not just name and email ID and I'll write some common format file saying that this is my contract and the provider you should be compliant with this before changing any API of you right so this is account service and user service and the contract between this the contract can be written in any commonly understood framework like for example XML JSON YAML anything that the two services can understand can be written in that same contract how the structure of your consumer contract test looks like is you have test it's exactly similar as your mocked integration test just on top of a few things first you register expectation hey whenever there is a request for user one with the detail please respond me with this particular JSON but this particular payload and you actually hit the request and rather than the request going to your actual user service it goes to your mocked server which is like kind of intelligent enough and written back you the same what you expected apart from this as mock integration test used to the same it also generates some file what you are interacting with this mock server this generates a file what we file we call it as a contract after you do interaction like account service was expecting some details it expected some detail like name sorry user name and user email ID it created a file and goes to production when there is a provider service which is a user service which wakes up and now tries to satisfy all the contract between each one of the consumer in our case there was only one consumer till this point account service created just one contract and goes to production nothing happened everything green goes green right next time something like me change the code and did refactoring and tries to push to production that time one of the your CI CD pipeline what you do is you take all the contracts of user service like user service who all are dependent on user service take all the contract and execute them against it saying that my current build satisfy all the contract that have my consumer have all green go to production but in our case ideally refactoring it will fail right so in that condition your consumer is independently going to production no dependency on production no dependency on your provider service and just creating one contract and going to production at the same time independently your provider service need not to worry about your contract when it there is a changes in your provider service it will just complain with all the contract all the contract satisfies which means service to service communication is good and I can go to production right so in this case it's good I am going to come to that part which in short some both needs to coordinate store somewhere and both needs to know that this these are the contracts where I stored so these are the two main flow of your contract testing paradigm one being creating contract going to production no worry second way okay I am the provider there are couple of consumer who has contract with me let me satisfy them if satisfies go to production otherwise no so in this way both services can independently go to production just satisfying their consumers right so one of the service which just talks to one service other service and don't need to write the functional test for it and wait for hours to run right only these two services independently can go to production which just take may be seconds I would say to just test and I can go to production this is how in short this contract test paradigm looks like let me walk you through the small code and then I will demo it how it looks like in actual coding I am using Java might be familiar with if not no worries I will just walk you through this the tool I am using for this contract testing is called PAT which is one of the de facto standard of industry currently what I am doing is the first step as I said in last slide is three state registering expectation requesting and your response the first one is I am just creating a mock server so that rather than actual user service my mock server will behave as a user service and this is how I am just creating is just like new pack provider provide as a user service and the target is this is just for other test tool I am using that that is J unit the second is setting up expectation actually if I call you user one what should I get written response if you can see this if I can create a body here I am expecting ID I am expecting username and I expecting user email ID not just name and email ID right and if I can see the builder if you just read it is very straightforward that okay builder upon receiving a request for a user with the path slash slash slash with method gate and it should respond maybe the headers which is like application JSON as a content type status should be 200 and body should be the body I expected and just to pack in this way you're just creating some expectation on your mock server which behaves as a user service and the last is as like normal you write a test you have a gateway rather than in the gateway if you see here rather than using actual user service host and port I'm using mock servers host and port and I'm just calling gateway dot gate user of ID one I'm expecting some user should be one ID one username should be Bob and user email ID should be me address email dot com and I said it equals that whatever I expected my mock server to respond me and this is actual should exact match this is how in the three step what we can write the test for maybe it's let's keep make our hand dirty with the code can you see it I just walk you through I have two services here let's see your account service and I use a service if I go to user service I have a controller here which which is a spring boot application upon asking for ID it just written me statically as of now with Bob and me at the real gmail dot com a new user if you see here user has ID user name and user email ID right so the old way of specification and this is my controller looks like and at the same time I go to account service and there is a user service gateway upon asking for a gate user ID one it just generate some URL and just make the HTTP request and get the response and using some Jackson to deserialize and at the same time on a account site account service also which is our consumer which expect the specification of ID user name and user email ID right so both looks same so let's look at a contract test how it looks like so as this might be the answer to your question saying that where this contract should lie where should I keep as of now I'm just doing it on my local machine so I'm just keeping at some file directory or system file directory in my machine as of now but this can be stored in S3 or cloud storage also I will come about it about in bit later and in this line I'm just creating a mock server which is behaves as a user service I'm creating some expectation about account service talking to user service with request a path method and will respond with whatever specification I specified and the last part is your actual test which I'm creating a user service gateway class with rather than using a user service host and port I'm using my mock servers host and port I'm just calling the gate user and expecting the user and expected user should be same let's run this test apart from just going green because everything I set up it right it's nothing no other services in the picture apart from it it just generate a contract for you if I go down here if you see here the folder I have specified it just generate the contract for me so this is the JSON contract which any service can understand easily and the content looks like very very straightforward saying that okay who is the provider user service is the provider consumer is account service what all interaction between these services as of now I have only one interaction the first interaction is a request for a user I mean this can be a feature for your API also right or get statement get account statement how the request specification look like is this is the path and the method is gate and what should my server respond with the status 200 with the header content type of the JSON and the body if you can see here a user email ID ID and username now this is the contract that I generated the account service generated now provider needs to be compliant with this in order to go to production right and there are some metadata which is the tool specific which need not to be same for other tools that you are using which currently the tool I am using which we just capture the specification that I am using for example the pack specification I am using 3.0 and which JVM library I am using is that one right so I have generated a contract now and now the second parts come now account service generated a contract go all green go to production right now actual refactoring came and now we are trying to do verification against the consumer the second part of it which get the contract from your consumer and verify against your current build all goes green go to production otherwise no right so how it looks like I am using gradle tool to just do verification need not to write an extra code for it as you can see here which very straightforward with the pack tool I am using who are the service providers here in my whole organization only one service product let's say as of now it is user service how I can contact user service is using HTTP what are the host which currently I am using local host it can be your pre-production environment which can be your production environment in that matter if it is a little data what is the port I am talking to and what is the base path of my service user service has packed basically has contract with other services in our case only one which is account service and where I can find this contract is this directory the directory that I have stored the contract in now this can be a S3 location like Amazon S3 or Microsoft Azure some cloud location or a SFTP server any of this right this is how simply it looks like and all that as we have created contract now let's run those contract against or I'm currently I'm running this contract again my local machine this can be a happened using your pre-production environment or production or staging environment right I'm just great running my application locally so that I can run all the contracts locally application is up and running and if you can see here it's just a gradle command saying that gradle user service is my service and just verify the pack that I have with all my consumers it just takes the contracts from this location that you have shared the files that you have if there is no contracts happen it just goes ahead right as you can see here log it's very straight forward saying that verifying a packed between account service and user service where I found the pack is in this file what are the specification a request was there for a user which responded with 200 status codes which matched headers matched okay matching body okay everything goes green both services in production now now the actual parts like the developer like me did refactoring let's do refactoring and let's see how it catches the whole flow let me go to the controller which I have which uses this user now let me change this to let's say just name right as we have said and user email ID to just email ID right now let's go here now I did refactoring right now let's run the new build of my application and I'm trying to run my whole application and it should feel now right this is the user service with the new API which reads respond with just name and email ID and now if but the my account service is not aware about this change and still satisfy still expect that the older way of contract exist right now let's run the pack verification one more time which we have changed the now contract and now this should definitely fail because I'm expecting some else Jason you're responding with mail Jason if you can see here it it shows me here body status matched 200 okay headers matched okay but the baddie matching failed why failed because there was some differences minus being someone thing I was expecting but not happened which is like user email ID username not matched but instead of that I got email and name which I'm not expecting right so this is how your contracts this change now now team can decide what to do with this now now maybe if the current circumstance and you can say that hey user service please do not go with this API to production because we are breaking we don't have time to implement this or just implement one more version of your API and make your API back out combat other way around account service also said that okay this makes sense of course I mean user name user email ID is redundant name and email ID make sense to us also and let us do also changes which we are going to do it right now and let's see everything goes green right so this is gateway I'm expecting a user and let me change the user I'm expecting to a name now and let's change our contact test also right so we were expecting a username and user email ID let's make it email ID only right now let's generate a new contract which satisfies the new way of specification that we agreed upon both the teams agreed upon and if I look at this Jason now all all all things are same as previous only thing change is the body that we had both agreed upon that hey make sense name and email ID make sense to us also which looks good now let's go try to go to production right because user service up and running with new specification client also change the new specification and all goes green right so now with this way if you see both services can independently go to production just having some contract between them right and this contract can be generated and verified independently if for example there is a all contracts all green rather than provider service changing let's say consumer service thought of okay I'm not expecting this Jason from you I'm expecting some else Jason which user service is not aware about and trying to change the contract when trying to go to production user service will say hey no this contract doesn't exist because there is no API like this exist if you want to change contract let us know we will develop this API will give this API to you so these both services can both agile teams in other way can understand and depending upon the problem they can either service can come back or other service can go up and just make is each one of them compatible with each other so this is all your end feedback loop is completing right and without knowing apart from this services to services no other 98 services are aware about this right and similarly there are hundreds of thousands of call going on between each services they can independently run their contracts and they are not aware about all what's happening in other world which is good which is not happening in your functional test but if you remember in the contract that I was expecting something like this user email ID ID or username or for that matter just name and email ID which is a static data right me at the redgible.com but let's say account service expecting some user one should have this data how come user service will aware about that there exist a user one right so I mean someone needs to create this data in order to get this from API right so this is what we call as a provider state in this contract testing consumer can say that okay before running my this contract make sure for example in our particular case before requesting for user one make sure that there exist a user one and this is how we set up a contract a provider state as I mean in the gradle tool previously I can use the same thing and I can say that okay I want to change the status of your service to have user one and what is the URL is this is the URL and this is the body just create one user for me after creating user one and then request for it all goes green this is what we call it as a provider state which is a very common pattern in contract testing so this is this is that like very small level local machine everything goes green right what are the challenges that we have faced last three years I'm doing this tool I'm doing this contract testing and what are the challenges that I have faced in my teams and how we overcome about that what are the paradigm to choose which to choose or what that will talk about in in a bit the first one is a contract test I mean how should I write the contract test like I use the Java as of now but what are the paradigms there what is what should I use for that what tool I use for that one of my project we used to write a test as a source code as you write unit test as you write mocked integration test same way we used to write the contract test like what I wrote currently in my demo is source code as your source code and write a contract test for it with this everything was going good because we had only 10 15 microservices one of our project which was okay but then later our organization grown and we had a couple of teams now where QA engineers used to see differently not with the development team and QA engineers sometimes used to write contract test and they said that okay we are we developers have their own choice and using multiple languages to write code at the same time QA engineer said that hey we can't write code in that much languages so is there any way that we can satisfy we still want to write a contract test but is there any way that we can do it so that way we decided to write the contract test as part of different repository and different language where QA engineers and developers both can understand rather than using your source code as a contract testing source of truth that way you got a flexibility of your working teams in the QA engineers can independently write test independently work can work with the older way this days to know about all the codes that developers is writing and every single change developer changing we used to communicate to QA engineers say we change this can you change the contract test and blah blah blah so this way we came up with the other option that okay why don't why can't we write the contract test as a separate code separate repository in this what extra thing we have to do is we have to actually call to the service right is just the source code of your gateway needs to be overridden that was the only bit that you use to write twice which was okay with the given organization structure we had so this was one of the lessons that we had second was as you mentioned how to share these files of course the local machine is not going to scale right it's it's a big team big organization so local sharing is not going to work and then we came to know that cloud storage why can't we store on s3 Amazon s3 and we started doing this on Amazon s3 over the time 200 plus micro services each service to other service let's say 10 calls which is like a 2000 calls and 2000 files on s3 constantly getting updated there are versioning over API like thousands of file on s3 and each time we have to get all the files filter one of them just to between these two services was taking too much time right 200 services keeping same file same location and if I want to know the files between these two services I have to filter all those go through all 200 for 2000 file and filter only these files right so taking too much time so that what we thought of rather the community thought of to come up with something called as a broker what broker means is is something which is intelligent to keep your file along with some metadata and some capability to you what it do in nutshell is the problem that we have with cloud storage one being if there are 2000 or thousands of files or lakhs of file we don't care because broker is intelligent is backed up by some persistent system and if knows that ok service a to service b give me all files it knows the where these files are and just will return you those files which is intelligent enough second what about if I run against a service to be service and it failed where should I capture this where should I keep all the result of which service is compatible with which service previously we used to keep it in some XML report you know documenting and all those stuff verses contract broker started giving this feature of you can just see matrix of service a is compatible with service b or not it's easy to visualize all your 200 micro services which service is compatible with what service third benefit of contract brokers is your versioning sometimes people write version a of a API right github if you say facebook this organization right if you say version v1 v2 v3 of a API right so how to maintain this API I want to test service a with service b with only version 2.0 not less than that not more than that maybe version 3.0 is a testing version and they are just testing some feature I don't want to test against that I just want to test exact same version which is not possible with the cloud storage because cloud storage is just a dumb five storage and that intelligent logic I have to write on top of it which is a common logic and that's why this contract brokers started giving this feature of versioning thing and the last thing is can I deploy how can I how can I be sure that service a is a provider and there are 10 micro services which are dependent upon a are all the services are compatible with this or not right so can I deploy service a having 10 micro services dependent upon this can I deploy this and I can just fire ask broker on my CI pipeline hey service a version 2.0 I want to deploy can I deploy and what can I deploy will do is run our against current build to all your consumers and says that okay you can deploy or no 8 of your micro services are good but 2 of them are not please make sure they are good with you right so this is what contract brokers come up with and the last one and is a pretty bit tricky is your verification how where I verify this now we have generated a file we stored it somewhere and where to test it where to verify it the first was just write a like consumer test just write a normal Java code or your language code to test this file get from some URL take the file hit your URL and check that everything works I mean of course if there if you have only few micro services it can kill scale at the same time if you have big micro services too many micro services too many teams is not going to scale and that's why of course you have to use some external tools like something that I was gradle which is language agnostic and which need not to worry about your code or your services and this way your infrastructure engineers or your system engineers or your QA engineers can write this script to test it which means your develop it's not dependent upon developer or some repository or your source code and the last bit and very important bit is where should it I put it in on your CI CD pipeline right where should I put it in continuous integration the first option we did was running against my current build every single build going to production used to test this verification and the later we realize ok this service is not changing that so frequently the API but changing some background thing which in that case I did not to run against my existing bill I mean every single build let's say I'm just testing some API I did not to run all those stuff in my current build right so that's why we decided to run against stable build which is like UAT pre-production environment which was working fine couple of tools available there in market about this one is packed which is like a kind of de facto industry standard if you just Google it comes contract testing you just get the pack why because it has all the language implementation it has specification you can implement in your own language all good documentation of course if you want to have your different ecosystem like everything is in spring boot in your ecosystem you can go with spring up cloud which is a good tool I would suggest for you if you are using Java or spring system and there are couple of like Janus which is like closure if you're using Pacto which is like it started by ThoughtWorks which is one of the pioneer project of this type contract testing which later Pacto said that okay we are migrating to Pacto and we are not no more maintaining it and VCR for Ruby if you are Ruby programmer you can go with Ruby VCR as a tool you are still not convinced with why to use contract testing or at all Pacto you can definitely go to this page which is pack.io and FAQ convince me which is definitely they will convince you with all the possible option in the industry and why the contract testing is so important if you want to go look at some resources my code whatever I've shown is available on GitHub which is like Pact.io implementation there is a Kotlin implementation that I've already pushed there the Pact itself their main website which has good documentation about this contract testing and if you might for if we are following in Martin follower here couple of article around microservices testing and which are you know easy to start and very good resources to start with that leads to if any questions function definitely I mean we used to write each service to service call we used to have functional test which was to take 2-3 hours to run right and then we started doing this service to service call and then we end up writing only functional test only for user's earnings okay only if user try to download something okay that is the real testing not the if one service call goes to 10 microservices I'm not going to write a functional test for it all all all the back end infrastructure we are in the phase of writing a front end to back end also which is also going to solve our most of the problem we are in the phase of that also yeah I mean it's a it's a problem you have to choose waiting for functional test to run for 3 hours versus some writing some mechanism for 1 hour and getting it then ready and we have to choose it at the same time for us once you have your infrastructure ready with this what we have to write is one more test and that's it that is the more extra benefit that that is the one more extra efforts you are putting not more than that but putting up some infrastructure is going to take some time definitely yes but time wise yes I mean a user journey we have to write user journey test for a lot of this cases which used to take less time to write test but used to take a lot of time to run it at the other hand in contract testing we have to write a lot of test because each services calling to other service thousands of contract testing right maintenance is big but at the same time the running the feedback loop is faster ability to go into production is faster right in order in the other hand the other way functional testing writing is easy with which was easy but running was too much which is to take hours if I go to production which have to wait for 3 hours to run my all functional test in this case I have it will take little time to write contract test but going to production is easy the loop is fast definitely I mean if I am writing here as a body if you see the string value I can say string matcher and I can say this reggit should match it should be int I can say int type I can say Boolean I can say anything I mean it depends upon the tool which how much of the tool is but of course all the tools have this basic functionality yes I can't do that right it's a little different paradigm we use verging in for that concern I would definitely be face this problem that couple of services used to have some different format takes place to expect where in Google also do this this different way of responding things we did it using rear verging and the tool that the broker used to help us ok among 10 consumers 8 of them use v1 just run the packed verification against this 8 of these and do the v2 against these two and the verging helps us so you have multiple contract contract file definitely for different version yes one single I mean you can but it's a pattern that you have to change whether you want to do it because each consumer needs to get some isolation and they're moving way of writing one right tomorrow if they say one service said that ok I once in different format I mean do I need to change all of nine of others right no right so it's a different paradigm that you have to choose let's say for example in our case user service which definitely not going to change contract for one service in that case I can just write one contract but in the specification like as he mentioned where each service one their own versioning one way of getting response in that case this might not scale that's the best question actually I mean we do use third-party services more than like 20 30 third-party services and we ensure that we write the contractors for them also and we against this them their new attunement we and if we ask first of all can you also write the contractors for us right if they are consuming our API we also ask them to if you can write the contractors for us that's the best part if not then we write on our side right that has to be differently I mean user there is we definitely not cannot ignore with the consumer test contact testing you cannot write functional test for each single integration happening between services right a functional test can be a just one happy path may not be a bad path how about bad parts how about corner cases how about API returning 4x how about if you are returning 500 sometimes we also write how about API returning 4x x 5x right how my service will behave in that case so which is not possible in functional test which is possible which is going to take lot of time definitely that's it thank you