 So, hello everyone, we are going to talk about AI ready microservice architecture today My name is Ali Oak. I work for Red Hat. I'm a principal software engineer here, and I'm a Knative contributor Hi everyone, I'm Pietrangelo. I'm an engineer at Red Hat and I'm Knative event intact lead So for this talk, we actually want to talk about something that we think could be tangible So we're going to give you this example. So let's give, let's consider a brand detection use case where you would like to study your brand logo, for example, in images, in social media, for example So to do that, you need to be able to have, you need to have some kind of AI system to kind of recognize this logo in the images In this talk, we are not going to talk about the Digestion of the images, that's another topic, but more, we are going to more talk about the AI model serving part of things So Overall in very high overview, this is a very simple use case, you know You have the browser or your client which can, which could, you know You can, you can manually upload images or you can have a client which Systemically, which automatically uploads images to a logo detection system It looks very easy and I'm not going into detail about the logo detection itself right now But the issue here is You cannot really see what's going on in the logo detection system Because the AI systems that we use today are very opaque, you don't know how they make their decisions So you need something, you need to evaluate their, like, how can you evaluate them? So for that, we thought you would start with Storing the images and the predictions your system generated so that you can actually do some analysis later So this is very simple use case, again You upload the image, the logo detector stores your image in an object store because it's a binary image It doesn't make sense to store in the same database with the prediction And then you also store the prediction, but again, yeah, in a separate database The prediction is basically just structured text, could be XML, could be JSON, things like that But here we have this issue that your logo detector system is actually talking to two systems And these operations are not atomic, like they're atomic individually, but now the atomic has a hole So the store image operation could fail, for example And then you would end up in a prediction where you don't have the image So this is like very basic problem that we know from, you know, resiliency topics Okay, now we thought maybe we can actually add another level of feedback so that we Level of input so that we can do more analysis And we thought maybe we can ask users to send feedback, like thumbs up, thumbs down, things like that And again, you need to store this in the database and later on with an analytic service, for example, you can Correlate all this information and show them in a dashboard Again, the issue here is that there are always issues in this first part of the slides And the issue here is that you have shared data stores. So you're actually Affecting the core part of your system when you're doing your offline analysis Obviously, you don't want to do that So to kind of talk about a little bit more problems You probably noticed logo detector system. The system has too many responsibilities So it's talking to too many components too many other systems And when you would like to change it, you would like to add a new system You'd need to change the code or the Configuration within the system And as I mentioned also scale we are scaling the database for two separate use cases And Now we'd like to talk to you about a better alternative and Perangelo will do it All right, so for like we saw a bunch of issues So let's kind of recap what are the main use cases that we have in the system So we we need to be able to kind of detect the logos in like in the images that user provides Also, like receive feedback from the user as well as eventually offline analysis for kind of correlate the predictions To the model version as well as the user feedback So to do that we kind of split each use case in different services and That helps us with you know adding more functionality without changing the existing services and For each use case we have a bunch of like one or plus one or more services and And for the first use cases, which is like detecting logo in uploaded images We have an upload service which just receives image and stores it in the object store and then the object store is like wired to an even broker, which is our mechanism to kind of communicate with other services and So the logo detector is like the same services we had earlier. It's like just focused on detecting the actual logo in into the images and what it does is basically receiving an event a notification from the object store through the even bar the even broker and It's just responding with a new event. So it's publishing a new event Which contains basically the bounding box of the image on where the logo is and also the confidence That we are going to talk about that later on So we need to we need some way to respond back to the client and We the asynchronous architecture like is is like you need some way to kind of push the information to the user and To do that we we have a new service, which is the reply service, which is connected to the user Client or browser with you know, WebSocket or whatever other protocol can push like HTTP to and push the prediction to the user so that we can correlate the uploaded images to the prediction and So with that we have solved the first use case and the goal with the other use cases To add new services without modifying the existing ones So we did the second use case which is receive feedback is you know We have another service which is the feedback service is simple receiving feedback thumbs up or thumbs down or like a score and What it does is basically publishing an event to the even broker in this case we have separated we have a separate even broker in this case and Is the even broker is going to store the feedback for us and So with that we solved the second use case. We are able to receive feedback and We are not yet kind of storing that in a way that is like usable for offline analysis And so with that we are going to introduce a new service Which is like analytic service, which is our service that is aggregating the information from the predictions And as well as from the feedback And is storing them into a database whatever database we like And so with that we solved the offline analysis of the user feedback and so Now we are gonna have a little demo of the this system this So I'm gonna pass it off to Holly. Thanks So To do the demo, I'm gonna switch over to this browser tab Can you how well can you see it? so We just in this demo We just use k-native primitives like brokers channels sources and things like that and k-native services So that everything is scalable all these little services that Perangelo introduced They're all scalable and we are using k-native eventing for Delivering the event part. So that is why this is we call this event delivery event driven architecture So This is a very simple application. It's it's actually a bit silly because it's you just upload an image And it tells you if there is any If there is any k-native logo in it, sorry about that We need to find the good images that we'd like to show to you which we Kind of missed They were on my Mac and My Mac doesn't work Today is a funny day because we had a Yeah, okay, I'll tell you that story later. So This first image is Very simple image. Oh, come on. No, I can't find it. It was working five minutes ago. So We did something wrong. Is this the wrong URL correctly? Okay We don't have a backup for that but anyway So the idea is that you upload an image and the system will give you some feedback and Based on that feedback, you're able to actually The system will also ask for user feedback and with the user feedback with the prediction and the image you can collect all these information using a k-native broker In a single place where you can do the correlation and show some analysis like this is our model prediction That's a model confidence. These are the images that didn't really work well, etc. And then and Then you can make the decision of Retraining your model. So here is what I was going to show here And then with a new model of the with the new version of the model We were Going to show that an image that fails was now is now working. I think we should go to the takeaways okay so the the takeaways are like The usually like the the AI model itself is like a fast-moving target of your system is it's gonna need a ton of iterations and to kind of eventually get to the point where it's like Working fine and it kind of usable for the users too And so you you need an architecture to kind of support that it can The fast-moving evolution of the AI system and we think that you know even driven architectural like the key to to that part and The other one is like Marffi's law, you know anything that can go wrong will go wrong and you know the demo went wrong And so The like the Operations if you have operations that are not atomic you need to think about them and think how you can solve them in Obviously like a synchronous even driven architecture helps with that because you decouple the operations between different services and The system is usually eventually consistent and so at the end of the day, there is like no server bullet like there's gonna be like Advantages and disadvantages for each approach and There is like the first option is always like you have I complexity in one single module as opposed to the the other alternative with the complexity like spread out and We believe that usually like tools in the cloud native space are helping with you know with managing that complexity into having multiple services and Also the the even driven part as well Which is you know, you have like more resilience with you know, ending retries of operations Automatically without like having to code them into the services and so with that I guess if We have done the presentation We'll be around the kinetic kiosk and so if you have any questions So this was actually the quick QR code for the demo that we would ask you to also give it a try But now that's failing. Yeah, there's no need to Yeah, exactly. So we will bring the demo live and we'll be will be around the kiosk if you if you're interested, please come visit us and We can then actually show you what we have built Thank you very much