 My name is William and I am Technical Marketing Manager here at GitLab. All right, let's talk about ML Ops and DevOps and why is this important? Well, it is and will be more common to add machine learning to applications. To enable those workloads, the data scientist role is becoming more relevant in the software development workflow. Machine learning workflows also need data, of course. Here is where we learn features and parameters. And on the contrary, with respect to the code, data changes fast, varicing volume, velocity, and even veracity. So this combo of data plus code is the one thing that brings us new challenges, such as how can we track large data sets and models? How can we quickly allocate specialized hardware to automate training and run machine learning pipelines? A possible solution for those challenges I mentioned earlier resides in adopting a DevOps platform to help us keep track and bridge the lifecycle of data and code. These two usually follow and grow in different paths. I want to show you today how can we reduce the distance between these two, enabling teams to experiment and have visibility into the code that is associated with certain versions of the model and training data. This is the architecture and steps I will guide you through during this demo. In this case, we have a deep neural network in a Git repository and GitLab project. Every time I commit a change to the neural network code, I want to build the model graph, train it, and upload the train model to a model registry. Test the model and the code, and if everything goes alright, trigger a CD pipeline deploying the model and code to an user facing environment, where we can run predictions. In this case, we will use the classic MINDS dataset and the end users will be able to draw in the app and try it. Okay, let's see it in action. Here is my GitLab project, where I have the deep neural network code. Let's check it out. Pay special attention to the dropout, because we will change it and see how the neural network will perform. As a data scientist, using planning tools, I want to work on this issue that's open and requires to change the dropout hyperparameter. I click on the issue and create a merge request, so I can add the changes to the neural network code. This is a quick change. I will use the web IDE. Alright, here is my script. Let's change the dropout to 20% and commit this change. Cool. This will automatically start a pipeline. We can see it from here. Let me click on that. And here it is. So this pipeline is right now building our neural network. Using the dropout, we just set. And this change will create a report for me right in the merge request view. And if I want to see how it looks like, let's open this job. We have a log here of the training process. We are towards the last epoch of the training. And we can check the accuracy here as well. But we want to see it in the merge request. We also can see the log of all the parameters and information that has been uploaded to our model registry. Cool. It has finished. Let's go back to the pipeline view. The right report job will automatically bring model metrics into the merge request we just opened. Let's see it. It will be published here anytime now. And here it is. This hyperparameter change, the dropout, give us this accuracy, batch loss, and also an ID of the model. The model candidate to be deployed to production. So I'll be taking over from William for a second to explain this section. We saw earlier that we were displaying a table of summary metrics from previous model runs, as well as metrics for the candidate model within our merge request. In addition to those metrics, we included a link to Comet's single experiment view for the candidate model, which is what you see here. The experiment view provides rich details about the data logged from your ML experiment. Different data types related to the experiment can be viewed in these tabs that you see in the side panel. We are currently in the charts tab that allows you to visualize metrics reported over time, such as the various accuracies and loss metrics you see in this example. It is also possible to customize these views. For instance, if I want to compare the training and validation accuracies for my model, I can create an empty view and add a chart with the relevant metrics. Let's move on to the code tab. Comet will automatically track source code from a run, so that you have an idea of the code that was used to produce these results. And as you can see here in line 19, this is the change that was committed earlier from GitLab. If we head over to the hyperparameters tab, here we can view the full list of parameters used in this run. These include training parameters, like batch size and learning rate, as well as model parameters, such as dropout and layer size. In the Output tab, we will capture the standard output and standard error from the run, which can be very useful if you'd like to inspect issues from your run without having to SSH into your machine or container. The next set of tabs showcase some of the advanced logging features available in Comet. For instance, in the Confusion Matrix tab, we can take a look at the model predictions on our test set. Notice how clicking the cell with a misclassification will bring up a model with examples of this misclassified data. Next, let's head over to the panel section. Here we can get an overview of the whole project and all the previous runs we have executed. This view is meant for easy comparison of multiple experiments and allows us to get a higher level understanding of the relative performance of a model. For instance, in our case, I can sort the experiments based on the validation accuracy and identify our top performing model so far. Okay, now let's go back to GitLab. Now, let's say I want to get a review from a senior data scientist. So I select Cesar as my reviewer. Cesar could go through the changes to the code, see what we change and read the model metrics. Let's assume everything is okay and merge these changes to the main code base. Alright, so this creates a new pipeline that as we can see, will build my production model. Right, it's associated report and register it to the model registry. Taking over from William again to talk a bit about Comet's model registry. The model registry provides a system for logging, registering and versioning machine learning models that are intended to be deployed to production. I want to note here that Comet makes no assumptions of what composes a model. A model can consist of any number of files or folders in the file system. In our example, we have a single model binary that we intend to deploy to our application, but if your model consists of multiple files such as a binary and a manifest, these can also be logged to the registry as a single model. In the model registry page, which is what you see here, you can see that in addition to logging the model assets and the current version, there is also a link to the experiment that produced this model. So you can always trace back the lineage of your model to the experiment that produced it. This allows the registry to serve as a link between the experimentation phase of the model development process and the production phase. You also have the option of tagging the latest model with a stage of deployment. For instance, if you'd like to push a model to a staging environment before deploying to production, you have the option of tagging it as such. Now let's go back to the pipeline execution. We want to keep a human in the loop. So the final step to deploy the model and build a UI for our digit recognizer application will be manually triggered to illustrate the update to our model registry. Let me show you the actual and latest model we have there. This is my model registry and we can see here we are in version 1.8. Let's go back to the pipeline. The jobs are still executed. All right. The registers to model registry has been completed. Let's go back to our model registry and refresh the page. Great. We have uploaded a new version of the model. We are in 1.9 now. All right. Let's do a little recap. If we go back to our Merch Request page, we can see the pipeline is still awaiting for manual execution and the metrics and performance look great to Caesar. Cool. Given that all the jobs completed successfully, let's deploy this model as mentioned before. We want to keep a human in the loop. So this job is manual. I will trigger it now. By triggering this job, I am starting a multi-project pipeline. The one that will be in charge of importing the model from the registry and build a UI of our application. We can see here the steps outlined. We started the downstream pipeline and we have import model, build model, and deploy. If we click here in the downstream pipeline, it will take us to the project where the UI of our application resides. We can see the pipeline executing that this is a different project. And if we go to the repository, we can see that it's a different code which has and contains the logic for the UI of our digit recognizer application. Here we see that the model has been imported and now we are building the application. We are building this application by reading a Docker file that is in this repository. Okay, the application has been built successfully. Now it is deploying. Let's see how it goes. And it will deploy successfully as well. All right. So let's go back to the pipeline view and we can observe that all of the stages have been completed successfully. We have this arrow in the middle because of the nature of the multi-project pipeline where we built the model in one project and we triggered the UI in a different project. Let's go back to the environments because I want to play with my deployed application. So clicking open and it's loading. And here it is. So this is our classic minced application. Let's draw one number. Let's say number one. Click on predict and it is correct. That's cool. Looks like the model is working well with once. Let's try a slightly more difficult number. Seven and predict. So it works. We were able to tweak a hyperparameter to drop out in that case and automate the deployment of the model to a model registry and then deployment of the UI where end users can interact and try the model. Back in GitLab, you can also see here that we can monitor the health of the application and the number of instances in my Kubernetes cluster. In addition to that, I have a view of the different environments that my application is using. I hope that by now this sequence makes more sense since we walk together through the changes to the neural network code. We saw how the neural network built and got trained. The performance metrics it generated in the Merge Request page, its interaction with the model registry, keeping the model under version control as well, enabling collaboration, choosing reviewers and finally being able to automatically deploy the digital recognizer to an environment where end users can quickly play with it.