 So next is William Arias and he's gonna talk about deploying the machine learning bots like a boss using CICD Will William Arias you work for GitLab is that correct? Hello. Yes. I work at GitLab. Yeah, perfect. So you're the expert No, so right. Where are you joining from by the way? I am I am from Colombia, but I live in Prague in Czech Republic. Oh, right So it's a good time zone for you now, huh? Yeah. Yeah, okay. Excellent. Okay, so please start sharing your screen and then you can take it away Okay. Hello everyone. So my name is William and today I want to share with you some of the learnings I had since I started like back in 2016 working with developing bots and I am a big fan of NLP and while they use cases of a chatbot and in this in this talk I want to introduce you to how you can step up your game using principles of continuous integration and continuous delivery using RASA the framework for chatbots and GitLab CICD So why why why I decided that I wanted to share these these learnings with you because When you start working on this as a hobbyist or as professional I have done both Usually what you do is that you start behaving like a one-person band so I was reviewing the agenda of the talks today and If you have been following as well, you can see that there are presentations that they are teaching you how to Create a Docker file how to deploy a model using docker how to serve it so you have to create an API Besides you have to use other tools for to create a model itself And at the end we start behaving more or less like this like like I don't know if you like the symptoms When I was doing that, I remember this guy because we are a one-person band. We are playing all the instruments So you start becoming so or you are expected to deliver An application that contains that has machinery embedded you more of less have to behave like a one-person band and know a lot of instruments and to Learn how to play these instruments. There is a learning curve that might be frustrated. I was the other day Teaching assistant Yeah teaching assistant in some online course where the students They were able to create the model for a chatbot But then when they were asked to please deploy to AWS and show us give us an endpoint where we can connect a channel and use the bot Many of them they start struggling because that's not what you are supposed to necessarily be aware of You know the models, you know, how to try how to use different libraries to create the to classify intense to classify entities But not necessarily all the tooling that you are expected to more of less Don't have some some some some knowledge to be able to surf a model. So example Let's say you use tensorflow for something for To create some some sort of some classification problem to solve some classification problem So your prototype in Jupyter notebook and once you prototype in Jupyter notebook You you if you want to store and probably you will need a database So you need to know which ones should I use should I use something like a Sandra? For example possible my sequel and then as as as I have to say before Then if you start digging into this you will For sure you will find okay So I should deploy it as a container and then cure and it is and how do I deploy it? Should I create a virtual machine or should I go directly to the platform as a service or should I just use an API so for the service whatever and Next to serve to the other world. So all of these things They are more of what they call today the fully stock machine learning engineer or data scientist. I don't know and I Want to share with you What was the approach that I follow and how today if your purpose or if the challenge that you are facing is I Want to create? I don't know my own startup in which I will build chatbots for whatever business and You want to be fast and you want to deploy in a very very like close to a standard professional way, but also With iteration in mind and being fast. So I want to show you how I did it How am I doing it? And this is not like written in stone This can change and all these workflows you if you know the principles then you can adapt it to your own workflow So how is the how I started to to divide this this problem when I say? It's taking me too much to deploy a chatbot to production It works in my machine the typical one it works in a machine is very easy to run it in local host But I want to deploy it and and if you are doing this for yourself as a one-person band or for a company You will find these these challenges. So first I started thinking a bit like what they are calling these days Machine learning operations that this could be there a workflow of that This this talk is known as Israeli about ML ops, but it borrows some concepts from there So first you are aware that the model that you are creating Has that some some life cycle in which you have to go through all these steps that? These are the ones that we are we are expected to be knowledgeable in how can I ingest the data? How can I version it or label it and run the validation test training model fine-tuned high-pay parameters Analyzes reading all these learning curves and then deploy and then get feedback So this is the only the model But one of the things that I learned that is very important when you want to scale these solutions And when you are not only developing for yourself, but this even helps if it's for yourself Because when you move to the process view of what you are attempting to do the first person that will be grateful Three weeks later of a month later. It's yourself. You say oh, that's so good that I put it in this in this view Because now you understand all the steps and more or less how you can Split the responsibilities And I put it here that one of them is a dialogue contributor because one of the things I have found while you're working developing chatbots with Russell library or others is that you Might be the let's say expert. I don't like that word But you might be the most knowledgeable person in how to build it The both but you are not necessarily the subject matter expert of the topic that the boat is supposed to provide a service And in many cases the subject matter expert is a person that they are no developers or they are no They don't have any Computer science on foreign generating background, but they are the ones that they know what should be the training data that you feed to the boat They know what type of questions there the end users will will ask So in this way, you are so it can start enabling other type of profiles in how can I incorporate their Contributions to my process to create a better boat. So this is a process view that it contains many of the steps from the model life cycle view and What are the benefits of this when you when you do this first is a favor that you do to yourself But also you have a common understanding of this process for everyone So if next time there is an iteration that something should change now It's very easy for let's say some some new employee or for you one month later to Retake and say, okay, this is this is what I was doing here Is where I am and and when you use this process modeling as well You can identify bottlenecks or where the things that you are doing might be failing And also this enables you with some standard that will enable iteration So you can improve with the pass of time the performance of the chat and now like how does this looks so the components view Here This is the high level diagram of what I have architected that it works to a very decent level in which this this could be the Pilot or prototype, but very decent not not POC hackathon level very decent pilot that I will go Step by step and I will show you starting from from this part. So what it's in the in this part in the left The dialogues or Python code. This is your development environment This is where you have your by charm or this was to your sublime text. Whatever here is where you have the your code Someone in dialogues in the certain format for your boat So the only thing you do once you invest time learning principles of CI CD is that you work in your machine Once you are happy with your changes or with whatever you did in your local environment You do a git commit and get pushed to remote So what I am doing in my workflow is that I create Mega change or in this case for this example I am adding more dialogues to the boat so the boat knows knows more combinations and is able to have better conversations This push will automatically and from here everything is automatic Back in the day, let's say four years ago when I started before using this You have to use the command command line and know each one of the commands that for example The raza framework provides to do raza train raza cross validate or raza Test data all of these things. So here what it's what is happening is that once I commit the code Automatically using it love CI it will start. It will start some runners or Containers or that was called runner that each one of these Will run the steps that you usually will run manually So this is not nothing new for someone who is who has been working in traditional development for years But this is more tailored for people that maybe they come With a math background or they are breaking into machine learning and they would like to build boats This is not necessarily knowledge that they have at least for example I didn't have it four years ago when I started these these good practices of software delivery. So Now that I discovered that's why I said, oh, this is this makes my life easier. I want to share with you So these these these steps that are usually Executed manually they start running in different machines. So one one one runner is going to validate that the data that you push To to the repository. It's okay. The format is is there with the one that the boat requires Then that the the data is complete And it will train the model and it will test the model and by test the model here I'm is the concept of testing in machine learning this with you can you can either create a testing Test this data set that you will run against your model or you can do for example in this case of my demo I do cross validation With the data that I have available So assuming that each one Of these steps It's successful then the model gets created. So you have an artifact now And the the artifact is your model And if you are happy with the performance because when you test the model You will get some metrics of the performance of your model And if you are happy with that you say, okay So now I want to do a merge request and and I want to put my boat into the main branch And this will deploy and this will deploy to production or staging and preproduct And here this part of the paper tray. This is this is one of the the key key key steps In machine learning operations that I am borrowing and bringing into this into this pipeline of of serving tablets is that Whatever you do, whatever you you change you make In your dialogues or in your code this by running these pipelines. This will create a paper tray What does it mean? You will have a record of For example, this is snapshot of today the model that was created today What type of the training data it used? What was the code at that moment? So in other words, this is version control that is getting into a record That I post in a wiki and if someday you work in a company that they will get some audit And they have to show show me why your model Was executing these predictions so you can easily go to the history of commits and say look This model had this training data and these were the parameters and here is the code and here is the data that we used to trade So this is the paper tray that this is required in in in the enterprise And also if you are just playing this paper tray helps you to keep track of your experiments and have some sort of report in which you can Compare if you if whatever changes you were making was improving or deteriorating the performance of your model So once the model is ready, and I'm happy with everything and all the steps and the testing From the software engineering point of view and machine learning is successful and I'm happy with everyone everything What I do here is that using this uh raza framework. It provides api rest that you can deploy A model server the model server in Whatever you want in my case. I'm using it in world cloud and then you can use this api api rest api to put And make it active and once the model is is Is there Then it's it could be available To be tested by real users or or other people that they can serve as testers of your both of your dialects And but it doesn't finish there because until this point I could be Quite happy that the only thing I had to do was Let's update the knowledge base of my bot Let's just do git add git commit git push. It goes automatically through all of these steps I don't have to worry about putting manually Executing commands and checking performance metrics because I can build a logic that if it's above certain threshold It's okay But once you are there and the model is served In this model server and it's ready to take conversations It's very important to close the loop with some feedback and the way that I've been using this and I will show later is that I am using the same rest api that provides the raza framework to extract Record of dialogues or conversations that the bot has been having with real users So what what but you can do here is that when you expose it to the outer world and the bot is interacting with real people using different Ways to say hi or to express an intent you can bring this back to to to To your environment and some sort of data playground and here you can do what you like the most that is okay. Let's use pandas python libraries to analyze To to to the nlp in other words and from there you can extract insights to later Feed it back into the bot and retrain it and it's our continuities. What I say before This loop that enables interage iteration to improve the bot performance so more of less what I'm telling you is that You you could be a one person Band but you can also bring a conductor to your orchestra Someone or something in this case technology that will help you orchestrate all of these manual steps And at the end of the day by using these principles of continuous integration and delivery We'll step up your game creating bots. I'm very important Really, it will make your life easier. I am a testimony of that I used to to Before as I was saying with the the other With the other talks I I had sometimes when I wanted to make some change and the change was minimal like adding just one more sentence or changing it happened to us building a bot for this topic that Have it here Have us here online. That is When we were building a bot like two months ago for frequent ask questions about coronavirus One day we were the bot was supposed to answer No, don't use mask because mask should be used only by health professionals one day later. No, no The bot should reply. Yes. Yes use use create make your own mask and please use it So it was only one single change in one sentence of the bot and it will imply that you have to maybe jump into the Dokey rematch makes a change of many things here. The only thing you have to do is Just put it in your in your pie chart in your local environment update it give commit give push and this will take care of that so This is how in this case i'm in powerpoint keynote so everything works in keynote So I want to show you a video of how it could look like so This I will share with you also a link. This is public on youtube So here typical developer workflow. I'm just creating a branch in which I want to work on my changes So once I create the branch i'm using the I'm this is the raza format to To add intense or knowledge to the bot. I'm just adding two words Okay, I am I'm just adding two words here Get add get commit and and here after I I commit the changes and I will push I will push them here is where the where the magic starts because I'm pushing to the remote and in the remote which is good lap is where I have configured all this orchestration with my new newly hired conductor for my orchestra So typical typical let's say developer workflow This is great in a merge request that when I come here And I these started automatically once I did the push Automatically this is started to run this pipeline One of the steps are ensuring that when the bot is ready is because it's really ready to be deployed to the production And each one of these steps what they are doing is what I described in the diagram before Is testing that the format of the template that we need to feed the the training data to the bot is is correct It's testing It's creating the model And once that it's creating a model is testing the model So typical developer workflow with the good practices So here what I'm showing is that I'm going one runner by runner And these are the the different machines that are executing Automatically the steps that I that if I don't have this I will have to do manually in command line. So by using these runners I am orchestrating automatically all these stages Following good practices of development and here. This is how my my my workflow or pipeline Looks like this is what the the bot The what I did what I provoke with this good commit is that is as I mentioned before these are the stages That it for this branch, they will end up writing a report that the report is this is a state Is this what we show me that The different artifacts that were created like these ones Different artifacts that get created when I train the model such as I don't know a histogram showing the prediction confidence distribution Or a json file that is showing me also some some sort of report of how the bot Perform based on that but this json is pretty helpful. But it's not Nice, let's say for a business person that they just want to show me show me how it looks like in a nicer way So when I this is step that is called write reports, what it's doing is in same good lab is writing in the wiki The report of the experiment where you can out to generate this thing that you can later revisit Or or or keeping some track of the changes of your experiments of the or of the model that is in production So once I'm happy with everything I can merge The the changes to my bot and here when I start merging. This is a view of the raza raza model server in which as you can see that moment when I when I took this video The model that was active was a model of last sunday But here this was Wednesday. I'm running a pipeline that is merging the changes into the main Main branch that once this this pipeline will will will finish What it will show show show us here. It's about to to finish the deploy stage There right there. What is going on is that my my pipeline is Talking to the model server to raza server and it's saying I am ready the model went through all of the steps Please deploy Once this is successful as we can see there I will refresh this page and you can see that now the model is ready was automatically deployed and made active And from there I can use tools like the ones that raza offer to create a link to share this newly created bot With external people so they can test it and I can also leverage an approach of conversation Driven development because from here before Deploying before following these steps that I just followed and deploying to production You can have more feedback of how the bot is performing So as if you appreciate the the beauty of this you were working in python You made two or three changes your git commit git push and all of these things started automatically and to finish the to close the loop I want to show you the last part is that in this case. I am I am using the api from raza and I am bringing Okay, we have five minutes and I'm using the the api from from raza to bring some of the Dialogs of conversations that they both had so this bot was newly created only for this event So it doesn't have too much too many Too much data, but this can give you an idea that here the sky is a limit you can say, okay I want to filter by number of intents, but the the Confident highest confidence of the best intent or what is the type of token that people are using the most? Are they saying hello the way I thought or are they saying hello? I want these are they using to move the intent in one sentence And for example in this one I was just extracting the last word and this for example Could give me a glimpse that look if the people are saying i'm a bit mad or i'm quite furious It's probably the bot is not it's not Working in the way that I expect so to finish that I show you how it looks like and if you adopt these continuous integration and delivery Practices it will allow you to have an orchestra for yourself with some conductor That will help you to make your life easier if you scale these after a company this will increase communication and collaboration You can have a more reliable solution and at the end of the day is setting you up to grow more from there You can grow from the manual development to the pipelines that I just show you to another point where you can Think of fully automated processes like instead of using this what I was showing you an api To call the rasa api to bring the conversations. I could automate this job every day apm and run some preliminary scripts And yeah, so this is like the the next step if you want to know more I have some links here and I will make available this is started project that you can fork and from there you can see how I Orchestrated all of this use it using git lab and rasa from this project You can just copy change to your point it to your own server and it will be ready to To replicate what I just did or I just show you in this demo video So thank you. I think this is I have three minutes of questions Thank you very much, William. That was a very nice talk very interesting indeed. So we have one question The question is that what are the cons and pros of git lab? Cicd versus having a Jenkins server okay, well If you already know Jenkins, so what are the pros and cons cons the learning curve if like me you came to this Cicd without knowing Jenkins. So good lab is very easy to learn So that's one thing learning curve and one one one pro is that in git lab you have everything under the There's in a single app. You don't have to switch environment. For example here It is I have in one single view. I have the pipelines here I can define everything and what I define in my project that will be my pipeline Is automatically triggered and read by git lab here, for example so pros and cons pro If you don't if you if you are comfortable with learning this tool because it's very easy and everything is Pre integrator, uh, pre integrator and as I said will make your life easier So you don't have to know to think about okay, so I need to learn a new tool I need to learn Jenkins plus raza plus my development here The reason I like this a lot is because here is everything Everything is here under this single application and whatever I push to my to my to this repository It will trigger a pipeline in my In my git lab ci So that's what that's something I can tell you from a point of view of Git lab fun But also personal point of view if you already know Jenkins learning curve Okay, go for the things you already know because at the other day you just want to make your life easier and deliver faster Yeah Okay, thank you for that answer Um, there's there's a second question. I don't know whether we have time, but uh, is there a difference between git lab ci cd and github actions Well, yes, there are different products Uh, yes, there are there are differences that We can take it offline because I'm this is moving me into the conversation of competition that Uh I can I can I can in the in the room. Sorry in the room because all of this is is public for for for git lab We have a competition page that if you google Git lab versus github actions in google, this is public and you will find it there It's not like it's no Possible for me to talk. No, you can do it and it's so transparent that you can have it only if you search for yourself But it's a quite long topic So, okay, so let's take that maybe offline into the into the talk chat We posted the link on the on the talk chat in discord and and so you can then find the talk room for this So thank you very much areas for the very nice talk. Let me give you your applause. Thank you. Thank you