 Okay. Hi everyone, my name is Vili Deshref. I work as a Node.js developer at Trilayer in Berlin. I'm also member of Node.js Foundation and Open.js Foundation, and contributed to some open source projects. So today we are talking about validating even driven architect with async API. How many of you already know async API? Okay. So yeah, not a lot because it's relatively a new project, but it's like very useful for anyone who is using Kafka every time you MQTT your web sockets. So how many of you use any of these technologies, Kafka? Yeah. So the LDR, it's Swagger for Kafka. So you'll get the idea of what it is. So it's a schema definition. We'll talk about it in details. And let's start. So my name is Vili Deshref, and how my colleagues pronounce it is like this, valid. And yeah. So yeah, that's the only joke I have in my slides. So you can laugh as much as you want right now. The rest is pretty boring. Okay. So I work at Trilayer. We are an IoT company based in Berlin. Our clients, we provide customized solution to clients where we receive data through MQTT or Kafka through hardware and sensors. We process them on our cloud, then we show them in a fine fancy dashboard. We send them alerts, emails. So just to get you an idea of what kind of industry we're working in, our process is like we have some sensors on machines, then there's a gateway, and then we ingest the data in our cloud where we do processing, analytics, storage, and then we have some like dashboards and mobile apps where the client can see what's happening with their devices, et cetera. So that puts us in a situation where we have a lot of, we do have to deal a lot of asynchronous communication from the devices, which are sending data through MQTT, Kafka, or maybe through even STTP protocols. So a little about OpenAPI and Swagger is, Swagger was initiated in 2010. It came, the schema definition came part of OpenAPI in 2015 because it got so much value and used by everyone in the industry because that was the only way, or one of the only way you can validate what's in your STTP request, what you can send in the response, or how you can define the schema. And the other part was you can get a nice fancy documentation for all of your STTP communications. And then we have Async API. The project started around two years ago. It's been used by Slack, SAP, some Salesforce, some companies, but it's not very popular right now. So the need for this is because even the messaging, the Async communication we do between the microservices is a contract between two services, like one is the publisher and the other is the subscriber and they both need to know what kind of messages one is publishing and other is describing. So they can validate it, they can, when they receive it, they can check the body, and on the other side, the producer should know what data should send or not to send. The Async API also gives you an ability to create documentation which is understandable by humans. It's not like JSON, but it's actually a nice documentation just like Swagger. You can read about what channels or topics you have, what payload do they have. Yeah, there's also tooling around how to use the schema definition of your Async API for testing and validation. Like you can test on runtime in your production environment when you receive a Kafka or Swagger, Kafka or MQTT message that is it valid according to your definition you have defined, and it gives you ability to create nice documentations. So let me just show you how it looks like. You can go to this website, playground.asyncapi.io. You see here I have defined a YML, a schema in the YML file, just like the Swagger, and here I have a nice view of it, just like we see in the Swagger files. So it has few attributes, like you can define under the channels, you can define your topics or your channels for MQTT, it's the type of, Kafka is the type of the message, like I have defined user events, and then message you can just tag with the ref, just like in the Swagger, and you can define what properties it will have, what all they're required, what's the type, and here you can see a nice example where it also generates automatically generated example, and you can see the schema itself, what are the keys, and what are the restrictions on them. Okay, let's go back, now you have seen an example. So the problem we faced with event-event architect was that we had around 30 or 40 micro-services written in Node, and all of them were communicating internally through Kafka. We had more than 50 types of messages with different payload, and not everyone knew what's inside, so whenever you were debugging or creating something, adding some key, you had to ask someone, a developer who has already worked on that type of message to see, or go check the code by yourself and see what's inside, or if you wanna change it, if you want to add a new attribute, if you want to remove it, and it was a mess. No one knew everyone, and we also have some services in Scala, where Scala people are working on them, and we don't communicate the different teams, the backend teams don't communicate much, even the front-end team don't communicate much about what's in the payload, and every time they have to check, they have to look at the code, or we created a markdown file, just read me to keep it up to date, and then problem with such documentation is it always gets outdated, no one cares about it, so you have different problems. And that's, I think, happening in all the micro-service driven architects where you have multiple services and they are communicating to each other through some of these protocols, and then you don't know how to have one source of truth for all the communication between the services. So we had something like this, we had client devices, which were sending data through MQTT to our cloud, and then we had a broker which converts the MQTT message to relative Kafka messages, and these are our services communicating internally through Kafka. So yeah, there was a lot of communication going on, there still is, and we wanted to have something to one for the documentation, and another for to validate all the messages if it has the required payload or not. So in my company, there are different teams like backend, DevOps, front-end, and all of them needs to know the content of the messages, especially the backend teams. And we also reside in different offices, some are in Munich, some are in Berlin, and different people are working on different services. So there's no one documentation, and also we have MQTT protocol for the devices for the client, so we need to tell them also what you should send to our cloud in a nice documented way, and also validate if they are sending it right way or the wrong way. So previously, we were doing something like this. We had a readme file markdown, and someone would know the code base and know what's in the payload, and for the testing, we had hard-coded schema in the services where we just write down all the properties and match them using the Azure library or something. But this gets outdated because, and whenever you want to update, you have to write it down as a hard-coded message. With ASIC API, we created two documents, for one for MQTT and one for Kafka messages, and we would define all of our events, just like I showed you for device on, off temperature, and for internal communications, user-created, user-updated, user-deleted, whatever. So it solved our problem for the documentation because it gives you a nice view, and it also generates an output file which you can easily share. Let me just show you how it looks like in one of our surveys, which is public. This is a bit old UI for the ASIC API, but this is what you get if you want to generate a HTML document out of it. So you see these are the topics on MQTT you can define, and then we have defined all the payload and the messages inside it. And this document we can share easily with our clients and they'll see what's inside. And we can also validate using this schema inside our services. So yeah, you can see it's a very nice way, it gives you all the messages, the schemas you have used, and the topics you have. So now they come about how to validate and how to test it if the message is right or not, or what property it is missing or not. So we wrote this library, it's open source, casing API validator, it's a key value based schema validation, you give it the key of your message and you pass the payload and it will validate according to the schema definition you have defined. It uses AGV module for JSON comparison, maybe some of you already know it, also browser compatible. So I'll just show you how it works. So the project is already on MPM, we use it widely in our services. It's a validator. You can go to the GitHub and just check it out. So I'll just show you how it works. So I have required the validator on top and then I need to pass the schema, which is this file. The schema is same as I showed you in the example for the user event. So I have defined the schema. We have these properties on the message and the message key is user created, let's say. So I load the schema on this line and then I ask the validator to validate this key of the message and the payload. And then I log the result. And let's see how it works for this. So it says data user name should match format email and user creator is the key. So if I go and check the schema or I think it was the user name. So the format is email because these types are defined as like normal JSON schema definition. So you can just have it as the swagger definitions, all of them like type, format or a pattern. So if I just change it because I think it's missing an at and then just run it again. Okay, so it's passing. And let's see if there's any required property. So the ID is a required property in the message which is mostly if you use Kafka or some similar protocol. So let's say if one of the message didn't send an ID and the rest of the payload was okay. So yeah, so it will tell you that yeah, this property is required. Just like you have on swagger definitions if you use it for validations. So yeah, this gave us a really good opportunity and ability to validate our messages which are coming through the client side because we were facing a lot of issues when we didn't know what's inside the message and how to debug or how to tell them what was the missing and also give them a nice documentation which they can use. So let's move forward. So we started using it in production. We made a private package of our async APS schema. So like Kafka and MQT schema, we push them into a private repo and on Vodacio and we edit it as a dependency in the services and then use the validator to validate the messages. So we also doing it on runtime in production for all the messages which are coming through the client side. But for the messages internal communication like Kafka we only do it when running the unit test or when doing integration test between the services not on production live environment. So the flow was like this. So we have, we consume the messages and we forward it to the, previously it was like this, we forward it to the relative service. So when we started consuming, we started validating also validating the message rule. And if there's no error, if the message schema is right, if it's not valid, we just log the errors but don't fail it or don't send an error black to the client. Just log it and see it in the logs if there was something missing. And if it's a valid message then just forward it to relative service or inside the cloud somewhere. And even if there's a error, it's not a valid message, we'll still forward it to the service. So this was the flow when we started using the async API how to get it inside the running production environment. So we just validate, log it, and the rest of the use case remains the same however it was working before. After like going through one or two weeks of process and seeing all the logs, we saw that sometimes it was our schema which was not right and sometimes it was the payload the client was sending was not right. So we made tweaks in the schema definition and sometimes we communicated back to the client and asked them to change the payload. And once that was done, everything was fine. We started sending the error events forwarding. We just stopped it. So if the schema validation fails, we throw an error back to the client or don't let it go into our system. So this move was pretty easy for us to adopt the schema inside our production environment. So the use cases for us are pretty obvious it was the validating of all the communication we did for documentation. We use it for our system tests. And now we also introduce this process of if a developer wants to extend a message, remove a key from existing messages, or want a different type of channel or topic, you first open a PR in the schema definition repo. Everyone approves it, the team approves it and then you start working on actually making the changes in the services. So it gave us a really nice way how to implement or make changes in the existing messages. Otherwise it was previously, someone would be working on the code itself and then reviewing and people would see okay, this properties removed, this properties added, these are the checks on it. So now we first get settled on the schema definition, how the messages will look. Once that's approved, you start working on the code itself. So this saved us a lot of time in reviewing the PRs and actually developing new kind of messages. It's just like if you do for the swagger, first you change the swagger and introduce a new endpoint and then everyone is okay with it, then you start developing further on. So for the external use case, it helped us a lot for the clients because previously we were managing a document by ourselves to share with them. And now it also gives them a really nice error about what key is not a valid key and what type of properties it should have, like it should be an email or it should have length less than 64 or something. So it was easy for them also to fix if there's something's broken. So when we started working, I also started contributing to Async API, the schema definition. It's getting very widely used in a lot of companies like Slack and we just released version two of schema which have a lot of different properties. You can define custom bindings for protocols. It's sports different kind of schemas and you can do like channel based validation. So we also have a Slack channel for Async API. We also have bi-weekly meetings on YouTube which anyone can join and yet it's open source. You can contribute in any kind of way. And we are also working on a lot of tooling around the schema definition right now. Just like this validator, we are working on generators which will generate the code from your defined schema. So if you define your schema for different messages, you can generate JavaScript or Node or Java code in different languages through the schema definition. Just like you have support for Swagger which you can generate. And so yeah, there's a lot of opportunity for tooling around this right now. So yeah, that's it. If you have any questions, let me know. So you can reach out to me on Twitter. You can check these links if you have any questions. And just let me know if you have any questions.