 Welcome to our talk on Applying Machine Learning to Get Offs. My name is Wuxin Zhang. I'm a Associate Consultant at Red Hat, and today with me is Yipsian, who is an Architect at Red Hat. First of all, what is Git? Git is an open-source version control system that tracks code management changes. It applies changes to the desired states as version control commits. Git is a tool for managing source code. It allows collaboration between developers by tracking changes in commits history. What is DevOps, you ask? In summary, DevOps is an approach to culture automation and platform design. It speeds up the process for an idea to go from development to deployment in production. DevOps is using CI CD to automate builds and deliver iteratively. DevOps is using tools like Jenkins and Argo CD to apply a version control of application and infrastructure code. DevOps is automating operational tasks. This can be achieved by integrating OpenShift. OpenShift is the enterprise-ready Kubernetes platform for managing containers and providing automation support. When we combine Git and DevOps, we get GitOps. GitOps takes DevOps best practices for app development and applies them to infrastructure automation. GitOps is developer workflows for operations. The goal is to automate operations. GitOps uses merge requests for all infrastructure updates. GitOps is automating builds with webhook triggers. GitOps also uses a Git repository as the single source of truth for infrastructure definition. For example, this is a GitOps workflow. At the top, we have changes made to the source code. The output of that goes into an image repo like Docker Hub. This piece is called continuous integration and can be automated with Jenkins. At the bottom, we have configuration changes driven by Git commits. We put the files in Git and synchronize them in Kubernetes. We can use Argo CD to continuously monitor running applications in the cluster and compare them against the target state in the Git repo. This enables Kubernetes to automatically update or rollback if there's a difference between Git and what's in the cluster. This continuous GitOps cycle allows for higher velocity. As Co gets iterated on, pushed and merged to Git, Git maintains the source of truth for what is running the cluster. Anything pushed to Git causes an alert to occur to indicate in this match. So why machine learning? Machine learning can speed up each step in the GitOps cycle. It reduces maintenance in build, test, deploy, monitoring, and security scanning. Machine learning automatically generates tests for validation, machine learning can also augment human engineers and intelligence. First, let's see how machine learning can aid in the build process. Traditionally, GitOps uses information extracted from build metadata to determine which packages to build when there is a particular code change. By analyzing build dependencies in code, GitOps can determine all packages that depend on the modified code. However, it is cumbersome when there is a change to one of the lower-level libraries. Because you will need to rebuild all dependency packages on every project that uses that library. This graph here shows the large tree of dependencies. But with machine learning, we can train the system to select the builds that are most likely to fail for a particular change. We do that by using a large data set containing results of packages on historical code changes and then apply standard machine learning techniques. During training, the system learns from previous code changes to come up with the build selection model. The system learns either the build passes or fails based on the features from the previous code changes. So then, when the system analyzes new code changes, it applies the learned model to the code change. For any particular build, we can use the build selection model to predict the likelihood of a build failure. Machine learning is also used in security scanning. For example, JFrog X-ray is a security scanning tool that integrates with Artifactory to identify vulnerabilities and security evaluations before production releases. Security scanning of artifacts is an important step of the CICD pipeline. The idea is that we can use historical data to prioritize artifacts scanning and scan the artifacts that's most likely to fail. This allows us to fail fast and recover fast. As developers, we work on large projects that has constant flows of pull requests. We can apply machine learning to explore a model to filter out unwanted PRs. To make predictions about PRs, we need to create a model for the predictor. So the steps for creating a predictor are define relevant features, gather data from data sources, do some data normalizations, train the models, choose the best model, and lastly, evaluate the chosen predictor. This diagram shows the steps for building the predictor. As you can see, we have a GitHub project. We start by identifying the relevant features from the project. We gather data from the repo and normalize it through the ETL process. The ETL process consumes REST GitHub API, then we train the data and choose the model. Lastly, we evaluate the predictor. Here's a list of possible relevant features just to name a few. Description complexity is a number of words in a description and title. A longer description might lead to a more complex PR. This integrator is a Boolean that indicates if the author of the PR has merged the PR before. Has tests indicates whether the PR has tests, which might increase the chances of approval. Success rate is the acceptance rate of the author's previous PRs. Requested reviewers indicates if the author requested any reviewers. SocialCon is the number of followers of the author. And lastly, Creative Friday is an interesting feature. It is possible that requests created on Friday are more likely to not be accepted due to the weekend. As I mentioned before, we use the ETL process to normalize our data. ETL stands for extract, transform, and load. The process is a set of tasks for loading and normalizing data. As you can see from the diagram, first, it gets all commits and batches all PRs using REST API. In the end, it merges all the data to generate the training data. Machine learning is also used in IDE co-completion. This is an alternative to the standard ranking mechanism. Machine learning can learn from your interactions with co-completion UI to provide more relevant co-suggestions. In this picture, we get a glimpse of what is possible with machine learning-assisted completion. We can use machine learning to sort completion suggestion. As you can see from the before and after comparison, the after image shows how machine learning re-ranks completion suggestions. With website deployment validation, we could also use machine learning to do validation to run against tests on the website post-depointment validation, defects and the occurrences that may generate any dependency, and deployment could return a success status when the defect was not identified in the website. So this is an example where during a web deployment, you may have an iFrame and different HTML code or third party provider or JavaScript library would inject different HTML into the iFrame. So this is an example where machine learning would be super useful in terms of identifying whether the website deployment passed the validation. So to do the website validation, we basically created 10,000 different images of the website. Some of them include defects, such as incorrect images, tags, and layout. The images will be used to train a data model to identify and discover defects. Once the model was completed, we could use the same model to check for many other copy of the page and look for defects. We leverage and supervise learning algorithm to take a set of training data example as training set. The learning algorithm is provided with training data to learn from the desire function. The validation of the learning algorithm is compared with the set of test data. And then at the end, the process of learning from the training set and validating against the test data is called modeling. First, we need to select the algorithm we want to use for learning and training. And then on the left side, we have the training data being feed into the model. And on the bottom, we have the test data that also got feed into the model. The output of the model is being used to apply validation logic to the surface on the right-hand side. First, we could capture the entire web page as an image. Image then gets subdivided into different UX component. The division of the UX component generates different training and data set and test data set to feed the model. With the model, we can test any new UX component across the browser with different resolution, different dimensions by feeding the image of the UX component to the model. The model classify if the UX model component passed the desire qualification criteria. This is the process of making a decision of a particular image based on a specific cloud. It's called classification. So this is an image website. We take an image of the entire site. And then the second step, we do different segmentation. You have the header, you have the body, you have the footer, you have the main scope, and so on. All of these different segmented images will get feed into the new network on the third box in the middle. And then at the same time, historical data of the 10,000 images will also get feed into the new network as a training set. You would make a decision based on probability. You will say whether the website deployment passed or failed the validation state. So each image was classified with a score between 0 to 1. Score closer to 0, meaning that the model predicts a potential test case failure, such as a UI defect. A score closer to 1, meaning the prediction of a test case meeting is qualification criteria. And we need to set up a cut off the threshold. Any value that were lower than the threshold is going to be predicted as a defect. These are some of the smaller images that got captured by the model. And you can see each image have a different classification. And furthermore, each classification will make some sort of prediction of the image, whether this is a defect or not. If this is a defect, what type of defect? What is the severity of the defects? Alert and monitoring is another main concept in GitOps. We want the DevOps engineer to get alert when a production issue happen. But oftentimes, the production issue happen too late. And when the alert will trigger, production autos has occurred. This is too late. Engineers are reacting to the issue when it happens. By monitoring with early detection, we could identify simple production issues, such as this space using Omiphias Alert Manager to report the problem. We could predict when a problem would likely to happen. And we could use that to notify production team and production engineer to look at. So this is a graph. You can monitor the dis usage of one of the production server, different data point, a different time and different time frame. Using machine learning, we could predict the trend of the usages as shown in the blue line. And we could potentially say, this usage is going to happen within one hour. So this is an example of a predictive dis outage. Dis outage make a judgment to the algorithm in production. The result got feedback back into the program for analysis how to apply its rules based on the rule that it has worked on from the past. As you guys know, the predictive analysis always start with two steps. First, you get the historical data. You apply to some predictive algorithm. And you generate a model. And you use the new data that comes from production. Apply to the same model again. And this is getting your prediction. How could we do regression testing using information extracted from the build metadata to detect which test to one or for a particular code base and how to prioritize this test based on the prediction? Because we believe each test is a little bit different. Some tests are more important. Some are less important. By analyzing the build dependency, we can find the test that less dependent and find the test that are more dependent. If you change one of the lower level library, we don't need to want all the tests in sequential order. We can want the test that are more important. And if any of one of these tests fail, we can do an abortion and fail all the test pipeline. This is a dependent tree that we are showing. Source file at lowest level, right? This is the source file that were developed by developer. And then in the middle, you have some intermediate module that were using this different source file. And on top of that, you have different tests. Each test could be a unit test, could be an integration test. They could be testing different co-paths, testing different components, testing different integration points, testing one, two, three, four, and five, six, six different tests. And traditional regression testing, you need to test all tests in a sequential order. So let's see how we could use machine learning to improve this process. The machine learning will predict the probability that a given test find a regression with a particular code change. It will make informed decision to rule out tests that are extremely unlikely to cover an issue. It will open up a new and more efficient way of selecting tests based on priority. Let's say you have used a large data set that contain different test results from the historical code base. We can apply to the existing machine learning model to make the prediction. We have the training on the top. We have historical code change over the test. Then we could apply that to make a pass or fail prediction. Based on that, we would come up with the selection model. Once you have the selection model, we could use that for making the prediction. With the new code change with the test coming in, we would apply the test selection model and that would generate the probability of a failure for each single test. So during the training, the system learned a model based on the feature from the PBS code change and the PBS test. When the system is analyzing the code change, it applied the learned model to the new code change, finding a defect with the specific test. With this technique, we could find and assign the probability of finding a defect to each specific test. The test one has a probability of 99%, which is really high. And then test two only have a 9%. Test three has a 32% and test four has a 14%. And so on, right? You can recursively apply the same model and do the prediction for each test. So in this situation, we would execute test one and test three first. And also we need to deal with a lot of unstable tests. When the test was not generated result constantly, while some tests will pass and fail unpredictably based on timing, based on dependency, based on data, data source, those tests would not be good to use for doing the predictability. In this case, you need to segregate out unstable tests. If you do that, you would be able to generate a more consistent outcome. Deploying non-deterministic application also require machine learning. We need to evaluate the quality and risk of the latest deployment and evaluate the deployment according to acceptance criteria. We need to prepare to support those assertion in a statistical manner, if you assert true or assert false, but it may not be always true or always false. So you need to do it in a statistical way. And the confidence percentage of the deployment will produce an answer in a different range. Deployment would likely to pass in a range of, you know, zero to 70%, for example. And you can use that range to make a determination of whether to start the next deployment or not. Having a high-level understanding of CI-CD pipeline is also very important. Efficiency can be tied to a particular component when you have a component that will not function well. Acceptant criteria for deployment, when we deploy, we need to set up the acceptance criteria in terms of the number of defects, number of the types of defects, and the severity of defects. And acceptance criteria should be used in terms of statistical likelihood of the specific certain range. So in this meeting, we have covered a lot of application using machine learning and how machine learning could be used to improve the efficiency of the GitOps lifecycle. We could use machine learning to make the evaluation and prediction quickly. And if there's a defect, we can use machine learning to fix them and we fix the code and we anchor the code back to the CI-CD pipeline. Machine learning also eliminated the unnecessary computational resources and capacity when ML predicted the deployment is going to fail, we don't need to wait until the end-way for a failure. So it also solves some of the limitations many organizations are facing in terms of how to shorten the release cycle and fail fast and save fast. When the deployment results are dynamically changing, we could use machine learning to make the best decision and prediction of the outcome. Thank you again for attending this meeting. It was a pleasure talking to you. If you have any questions, please reach out to the web-ad consultant team. Thank you. Thank you.