 The CTK is a great idea. It provides a convenient automated conformance verification mechanism for API implementers. However, when we tried it, we noticed a few major loopholes in the conformance verification because of architectural issues. And we believe the CTK can leverage specmatic to address these problems. In this video, we will be going over major issues with CTK 5, which makes it ineffective in testing the conformance of API implementations. Let's begin with a quick demo. This is the party management API version 5 on the TM forum website. I have downloaded the open API specification, the CTK and the sample implementation code from here onto my laptop. Here I have the sample implementation code running on my laptop. Let's execute the CTK and see what happens. The tests are passing. The CTK ships with an open API specification for this API. Let's open that in postman and try to execute one of the requests. Here's the post slash party API. The base URL has been configured in a postman environment variable to point to the application that's running here on my laptop. As a sanity check, let's just send this request to the application and see what happens. Okay, we get a 404. And that means that the application has not implemented the slash party API. But that means that the application is in fact not in sync with the API specification that we just imported. But how can that be because this is the specification that comes with the CTK and we just ran the CTK and we saw all the tests passing, which should have meant the application is in sync with the specification and has implemented all the APIs in it. But very clearly it has not. So to understand how this could come about, let's spend a little time digging into the architecture of CTK 5. TM Forum provides an open API specification, the first source of truth on its website, and the CDK for conformance testing. CDK v5 at its core leverages Cypress to test the API implementation. CDK v4 used to use Newman. There's a report generator that generates the report published for the conformance process. But it also contains its own copy of the API specification, which is another source of truth. And this is the first problem. There are now two sources of truth and they can diverge. In fact, just a minute ago we saw how the CTK specification for the party management v5 API was out of sync with the sample application. Now let's compare the specification on the website for party management with the specification that shipped with the CDK. This is the API specification for TMF632 v5. If you look on the left, you'll see APIs like Hub, Individual, some listeners, and organization related APIs. Now let's take a look at the API specification that ships with the CTK for this API. And this time you see a very different set of APIs. The individual and organization APIs are completely missing. And there's a new pair of APIs here for party, which was not there before. It's clear that these are two different sources of truth and that they've significantly diverged from each other. Now, coming back to the architecture, we realized upon deeper inspection of the code that this is not the only issue. The next piece of the puzzle in the CDK is the changeme.json file. This is where we define the application URL and also provide the payload sent in the request. The request that we provide here is not validated against the open API specification and hence can be out of sync with the spec. This is problem number two. The Cypress test directly sends these requests to the application and only verifies the response against the schema in the CTK's copy of the open API spec. Now, the Cypress test itself is a third source of truth because it does not generate tests based on the open API spec. It has actual logic in it. As a result, the test implementation duplicates a lot of the details that are already present in the open API spec, which is problem number three. All this means in the original open API specification, only the response schema would be verified. All other aspects like request parameters, error responses, property such as optionality, nullability are not getting verified at all. Substituting the Cypress setup with Specmatic can solve all these problems. In our proposed architecture, the API specification on the website would be the primary source of truth. The Cypress tests in the CDK would be replaced with Specmatic. Specmatic reads a configuration file called Specmatic.json and gets from it the link to the specification on the website. This would be the only specification in use, thus addressing the first problem of duplicate specifications. The CDK would also contain a directory with test data.json files containing the payloads for making requests. Specmatic would validate these payloads against the specification before using them, which addresses the second problem of unvalidated responses. Then for every API in the specification, Specmatic would run contract tests using the validated payloads and would validate the responses again using the specification. Now, while the Cypress tests set up hard codes, the API definition, thereby duplicating the OpenAPS spec, Specmatic instead generates the tests based on the OpenAPS spec in a completely no code manner, thereby totally avoiding the duplication and in the process immensely simplifying the architecture. As a bonus, Specmatic can stub out other dependencies of the API so that the conformance of the API can be tested end-to-end in isolation. Specmatic can even do fault injection to make sure the application can handle timeout and other outages from its downstream dependencies. Let me now demonstrate the issues that we've pointed out in the CDK and how Specmatic can address them in a side-by-side comparison. We'll be using the OpenAPS specification CDK and sample implementation for TMF 642 version 5 provided on TMForum's website. Let's say a developer makes a change to the application, which is incompatible with the specification. To show what might happen, we've modified the TMF 642 sample application to require a new key called recurrence. Here's the create alarm controller, which handles the post slash alarm API, and here is the code that throws an exception if the recurrence key is not found in the request. Here I've got the application running on my laptop. Let's run the CDK and see what happens. Here 22 of 73 tests have failed. Now if this key is really needed, the right way to fix this is to get the specification updated. The wrong way to fix this is to directly update changeme.json and add the key, which is what we've just done. We'd expect the CDK to catch this and as a result, tests should fail, giving the developer feedback. Let's try running the CDK again. Okay, we have 73 of 73 tests passing, but clearly this means that the CDK has not realized that the request payload does not match the specification. And then this in turn means that the application is out of sync with the specification despite passing all the CDK tests. Now let's put this same payload into this file for specmatic.json to read. It's the same payload from changeme.json, including the extra recurrence key. And now let's run specmatic contract tests. There you go. Specmatic validated the payload against the specification and immediately complains that the recurrence key is not in the specification and thus the test fails. Specmatic simply does not accept a payload that diverges from the request schema in the OpenAPS specification. Another issue that may not be obvious at first glance is that the CDK only verifies two execs responses. The four execs response schema goes completely unverified. Since the CDK is already passing against this application, let me run specmatic directly with generative tests switched on. With this feature switched on, specmatic actually sends specification invalid requests that are intended to trigger the four execs responses from the application. Let's see what this looks like. Let's take a closer look at the test that triggered this error. All right, note that type here was passed as null by specmatic. Specmatic did that because it knows that type is not supposed to be nullable. It's just a string not nullable. And so it actually created a request with null in the type. And so the application has returned a 400 bad request. But there's something wrong with the response. Let's make a note of the keys here. There's message and there's errors. And the concern is that at type in the specification was not in the response, code was also missing, reason was missing. And there's an extra key in the response. Key named errors was not in the specification. So let's turn to the specification itself to see what 400 responses are supposed to look like. So let's look at this. Right enough code and reason are mandatory keys they were missing in the response. Let's check here. Yes, right enough at type is a mandatory key as well. And it is missing in the response. And the errors key in the response was not in the 400 response here. So specmatic has correctly identified the problem with the response. And it has picked up on this because it actually triggers and validates 400 responses, which the CDK does not. Now for this next demo, I've switched off request validation in the sample application. Here's the code where I've done that. Now let's run the contract tests with generative tests switched on. We see expected 4xx status but received 201. But the question is why would specmatic be expecting a 4xx status? For that, we have to look at the request because 4xx is an error that means bad request. So we can see here type null. And we know now that this is a specification invalid request because at type is a non-nullable field, as we have seen before. So the application should be returning a 4xx response, but it's handling it and returning a 201, which makes it out of sync with the specification. And this is even worse than returning 4xx that does not match the specification, which is what we saw earlier when validations were on. Specmatic discovered this through the attempt to trigger the 4xx response, but the CDK misses this issue completely. Let's also review the API coverage summary provided by specmatic after the contract test ran. This is a list of the APIs in the specification with their test results. And here's a count of the APIs that have not been fully implemented. And here's a very crisp summary of the test results as a single value total API coverage, which represents the percentage of APIs that adhere to the specification. In addition to these issues, the CDK today verifies a subset of paths for TMF 642 and others that we've checked, and it does not verify aspects of the request such as optionality, nullability, etc. So to recap, the CDK's core responsibility of verifying conformance suffers because the source of truth or API structure and behavior is duplicated several times. TM Forum can address all of these issues by simply replacing the Cypress API test setup with specmatic contract tests, which we believe is the more suitable tool for this use case.