 So, hello everyone, welcome to this video. In this video today we will talk about Dependence Monkey and this video is from series how to beat Python's PIP and more precisely we will talk about Dependence Monkey and how Dependence Monkey is checking Python dependencies of a TensorFlow stack. So if we take a look at TensorFlow, we can pick one specific version to simplify things a bit, then we can have TensorFlow in version 2.1.0 installed from the PyPI, the Python package index and we can see 23 dependencies. These 23 dependencies are in a form of dependence name and version range specification that needs to be satisfied in order to install TensorFlow. You can also see some environment markers that means that these dependencies can be installed conditionally, for example based on Python interpreter version. Okay, so we have these 23 dependencies. Let's create a dependency graph. In this case we have just one let's say direct dependency that is TensorFlow in the specific version that we mentioned and these 23 undirected dependencies that we saw earlier. We can create a node in this dependency graph that directly corresponds to dependency name and version range specification that needs to be satisfied for all the dependencies in the dependency graph when installed specific package. So this version range specification is resolved based on resolver and that basically checks what dependencies are available, for example on PyPI or any other Python package index. And I did that. So for example, if we take a look at ABSLPi, we can see some additional nodes that correspond directly to specific versions that satisfy the version range specification for ABSLPi as mentioned in TensorFlow dependency listing. The same can be applied for other dependencies basically for all these 23 direct dependencies of TensorFlow. Okay, these are just direct dependencies and to inspect the wall dependency graph we need to take a look also at transitive dependencies of TensorFlow, meaning dependencies of direct dependencies. I checked just few, I provided a link to a GitHub repository where there is stated the wall dependency graph in case you are interested. And here you can see that, for example, at Opt-Ansum depends on NumPy, PsiPy also depends on NumPy, but different version range specifications in comparison to the dependency that is stated for TensorFlow in version 2.1.0. Which version of NumPy is installed? Well, that depends on the actual resolution process. If we install these dependencies using pip, pipen, .3, then the latest possible NumPy will be picked. If we create some smarter picking, like which NumPy versions work very well with our TensorFlow release, then we can gain some higher quality software in comparison to the software that has, let's say, the latest, but not the greatest, dependencies. This, which NumPy dependencies will be, or which NumPy version will be installed, that also depends on other packages that you have inside your dependency listing. If you are using TensorFlow as a library in your application, you can use, for example, different version of PsiPy or something else that includes a different version of NumPy, and then the resolution process can create a software stack that is formed with a different NumPy version than stated here. Also, this depends on time. When you install or when you resolve dependencies, then it depends on which versions of NumPy can end in your software. Okay, and this all can be applied also to other direct dependencies, so let's take a look at it. If we check TensorFlow 2.1.0 as a wall, we can see NumPy, that is, a dependency of H5, Py, OptanZoom, PsiPy, Keras processing, TensorBoard, and possibly others, but I think I stated all. You can find the listing in the GitHub Gist, and if you take a look at direct dependencies of TensorFlow, we can construct something like 3 multiplied by 10 to the power of 13 possible combinations of packages in different versions that can be installed. Of course, this version varies based on the actual resolution process, and if you include also transitive dependencies, things get more complicated. So, you can see that NumPy is introduced into TensorFlow stack by different dependencies, a direct or transitive ones, and if there is an issue in NumPy, then there can be some effect or some bug in the software, and you can break the application, it can misbehave, and the overall quality of software that is shipped is, let's say degraded, or if you install proper version of NumPy, then the quality can grow. We can also touch overpinning, underpinning in these cases, but that's a different story. We will talk about it in the upcoming articles. If we take a look at the TensorFlow dependency graph and check how many nodes are there, you can find something like 17 million nodes in the dependency graph, and this number can grow over time, so you can, there are new releases of NumPy that can satisfy version range specifications in the dependency graph, and in that case, dependency graph grows. It's worth mentioning here that this number corresponds to a number of packages in specific versions coming from a specific Python package index. So it's, for example, a node in this dependency graph, in this case it's NumPy in version 1.7 coming from the Python package index. So you can see that's pretty huge dependency graph. If we go more into the theoretical direction, let's say that we have an application that depends on two packages. One is called SimpleLeap and the other one is AnotherLeap. We consider these dependencies as direct dependencies of our application, and also these dependencies do not have any transitive dependencies that would be introduced into the software stack. Then we can explore the state space that is created out of these two packages. So let's say we have SimpleLeap and UnderLeap, and we are just checking combinations of different versions for SimpleLeap and UnderLeap. Then we create some scoring function that will explore how the software behaves when it comes to its quality. You can imagine quality being how performance the software is. This is just a brief introduction, and you can find more information in the linked article. This is just to give context what we are doing with the dependency monkey. So as stated, we have SimpleLeap and UnderLeap, and we have some scoring function that tells us how good the software is when we change versions of SimpleLeap and UnderLeap. We are in a discrete phase, and here you can see these purple dots that correspond to the final score that is computed by the scoring function. It's not that intuitive, so let's try to interpolate this function, and in this case, we get this surface that corresponds to the interpolated discrete function for scoring these combinations. And as you can see in this artificial example, our software can have different aspects, different quality depending on versions that we installed. Okay, now let's say that we pin SimpleLeap to some specific version, and what we will do, we will change some versions of UnderLeap, or we will change, or we will check what versions of UnderLeap can be installed. In that case, we are basically mapping these two-dimensional space where x-axis corresponds to versions of UnderLeap and y-axis is the score. This function tells us how the software behaves when we have SimpleLeap in a specific version, and we are changing these versions, or possibly, for example, builds of AnotherLeap. Then this scoring function can look like this as an example. Okay, this was a more theoretical background, and let's take a look at a resolution process that can create different combinations and satisfy version range specification and resolve dependencies considering the dependency graph. So our ultimate goal will be to resolve TensorFlow software stacks with different NumPy versions, but still respecting version range specifications on NumPy, so we still satisfy version range specifications that are put by different libraries such as TensorFlow, SciPy, and others that use NumPy. So we will create a reconfigurable resolver. This resolver is created of pipeline units, and these pipeline units basically form a programmable interface to the resolver, and you can think of a pipeline unit being something like do not submit already or do not check already seen software stack. So if we resolve some software stack multiple times, that can happen based on dependency graph resolution, consider just first resolution and do not create multiple duplicates. Then another pipeline unit can be in all versions except for some specific library, in case it will be NumPy. We can do this also on the direct dependencies, but in that case we are directly saying that these dependencies have to be present in the dependency graph, but it's not something that needs to be true. Like if we've been, for example, TensorFlow in a specific version inside TensorFlow stack for all the releases of TensorFlow, TensorFlow does not need to be present in the dependency graph, meaning the dependency graph can look like different because there are no restrictions or emerging range specifications that would be introduced by TensorFlow package. And you can think of any other pipeline units. These units are basically some programmable interface, and they do one thing, and they do it well, and when grouped into a pipeline configuration, then the pipeline can resolve software stacks that have certain aspects and are suitable for our needs based on what we want to see. So, for example, we want to see different versions of NumPy in a TensorFlow stack. Okay, if we take a look at dependency monkey as a component, we already know the pipeline. So, in green, these green rectangles are pipeline units. They are grouped into a pipeline configuration, and this pipeline configuration is used by dependency monkey and resolver that can resolve software specs that are respecting Python packaging requirements or Python packaging guidelines and can resolve pinned-down software stacks with certain aspects, with certain quality. The input to dependency monkey is a vector. This vector can be formed out of direct dependencies. These are required, so stating what needs to be resolved. In our case, it will be TensorFlow in version 2.1.0. Then information about software environment, hardware environment, meaning where the software will be run, which operating system, on which hardware, what CPU, all these things that we want to consider when we want to run some specific jobs that verify quality of software. Another input can be library usage. In other words, API calls to different libraries, like TensorFlow, so if you are using convolutional network or convolutional layer, you can form or act differently in pipeline configuration or pipeline units. And then there is another parameter, this decision type that also parametrizes dependency monkey in some way, we will not go into details. Then all these things in the input vector are taken. The pipeline configuration is created, this pipeline unit can have parameters that state how these pipeline units can act. And the resolver and dependency monkey itself uses knowledge base to resolve software stacks. So one of knowledge that it uses is how dependencies are structured. So if you are following our series, you know that pipeline dependencies can be resolved using solver. And it basically states what dependencies correspond to which version or any specifications of packages. So that's one of the knowledge that we have and is required to actually resolve software stacks. Another knowledge can be information for pipeline units, so pipeline units can act differently. So if we do not want to resolve same software stacks across multiple dependency monkey rounds, we can code it into a pipeline unit that will query the knowledge base and ask for already seen software stacks in the previous rounds. So we do not need to inspect or check or resolve software stacks multiple times and things like that. Dependence monkey is a component that is living inside a cluster. So it can be run inside a cluster. In that case, the interface to trigger a dependency monkey run is an API server. So we can directly go to an API endpoint, pass in all the parameters, and dependency monkey will be run. That's just another interface. In the demo that I will do, there will be used dependency monkey locally, so it will be run from command line, but that's just another interface how to interact with dependency monkey. Once dependency monkey resolves software stacks, software stacks are sent to Amun to check inspection, to run inspections, and these inspections verify software stack quality. So if you are following our articles, you know that these inspections check how the application behaves when there is a metrics multiplication done, what is the overall performance or how the application behaves when convolutional layers are created, or it can be done on model level, not on operation level. So Amun creates inspections, and these inspections create, or the output of the inspections is a document which can be subsequently reviewed, how the software behaves, where there are any build failures, where there are any runtime failures, and based on this review create, again, knowledge for dependency monkey, and later for resolving high quality software, as we will talk in the upcoming articles. Okay, so let's dive into a demo. And we have switched to terminal. As stated in the presentation, terminal is just one interface to interact with dependency monkey. I will type directly commands to interact with dependency monkey, but this all can be accomplished also inside a cluster where API endpoints would be used, and these API endpoints accept JSON, and that JSON is subsequently transformed to parameters that are used by dependency monkey. Okay, so let's take a look at first command that we will trigger. So in this case, we will run dependency monkey. The implementation lives in TOTS advisor, because it's reusing resolver that is used in advices. We will talk about this in upcoming articles. Dependency monkey accepts PIP file. So if you take a look at PIP file, it's under PIP file as produced by PIP end. You can see that we are requesting TensorFlow in version 2.1.0 running Python 3.6. So that's PIP file. Then there is a recommendation type. Again, I'm not going into details, but that's just another parameter that is passed to advisor. Then we are using predictor. This predictor guides resolver how to resolve dependencies inside dependency graph. So package combinations is one implementation of predictor. This predictor accepts parameters. If you take a look at these parameters, they state what package combinations should be resolved. So in this case, we are resolving TensorFlow in version 2.1.0 with different combinations of NumPy. Then another parameter is account that says how many softwares text we want to resolve. Then another parameter is called AMOON context. If you take a look at it, it's the JSON file that is sent to AMOON. This is used to describe how the runtime and build time environment should look like and how the build process should look like. It also states script that will be used to verify that the application runs correctly. In this case, we are using UBI-8 Python 3.6 that is compatible with REL-8 Python 3.6. You can see run requests and build requests. So that's about AMOON context. Another parameter is seed. This seed is initializing random generator so dependency monkey can run in a completely random manner or if you explicitly say seed, then the random generator is initialized so multiple dependency monkey runs produce the same results. So I put it in there just to have similar output across multiple runs. Then another parameter is runtime environment. Let's take a look at it. So runtime environment states for resolver that we are using REL in version 8 running Python 3.6 and platform is x8664, so Linux. This is input to resolver as dependencies that are installed can depend on the environment. When we are running Python 3.6, we can end up with different dependencies that are installed given environment markers that are supplied to packages in various parts of the dependency graph. So this needs to be done. Okay, then we have pipeline configuration. So let's take a look at it. It states what pipeline units should be present inside pipeline configuration. The implementation of advisor implements multiple pipeline units and these pipeline units can be of different types. For example, boots, thieves, steps, strides, and reps. They serve their own purpose and we will talk about them in the upcoming videos. So for us, the most interesting ones are strides. So here you can see multiple strides called one version stride and the limit that each package should be present in a specific version or in one version that is resolved. So in this case, we have ABSL Pi, GAST, Google Pasta, and all these packages in one specific version. And then I commented out one pipeline unit that says one version of NumPi. That's something we don't want. We want to resolve multiple versions of NumPi for multiple combinations that can occur inside a TensorFlow stack. Okay, so this is pipeline configuration. Let's take a look at what we have. Another parameter report output that just says where the report of dependency monkey run should be stored. So it's a JSON file that captures all the relevant inputs and the outputs produced by dependency monkey. And then there is output. In the first run, we will use output directory. In the second run, we will submit the first stacks to Amun. So Amun really builds them and verifies that they are running correctly. Refer to the previous article from Python Beats, how to beat Python's PIP series to get more information about this. Okay, so let's try to run this command and what we should see, we should see the dependency monkey starts and it starts resolving TensorFlow stacks. As you can see, there were produced 10 stacks in six seconds, something more than six seconds. And these results were stored inside output directory. So let's take a look at them. These outputs are PIP file log files. So standard PIP and files. And as you can see, they differ with NumPy version. This is all considering the dependency graph and version rank specification, as stated for all the dependencies inside TensorFlow stack. So for example, that SciPy TensorFlow and all these that were stated in the presentation. Okay, so now let's try to submit them to Amun. So we will remove commented out URL to service and let's run it. As you will see, it will be slightly slower because there is networking overheads. But if we take a look at it, now we should see in Argo workflows new inspections that are running. So each inspection creates image stream, then creates build config build. So the application is built inside an open shift cluster and subsequently there are run inspection drops that verify that application runs correctly. So again, refer to the last article about Amun to get more info about this. Okay, so that would be the first commands that we run, our first run of dependency monkey. And now let's take a look at second commands that we will take a look at. So again, we are running dependency monkey with same PIP file with same decision type, but we switched predictor to random walk. This predictor can randomly walk the dependency graph and randomly resolve some software stacks that are valid TensorFlow software stacks. We will produce 500 software stacks. We will use same configuration for Amun, same seed for the resolver and the random generator, same runtime environment, same report output, and we will adjust pipeline configuration. So if you recall, we had specific wraps, sorry strides that removed or prevented from creating different combinations of packages that we didn't want. In this case, we removed them, so we are not limiting to some specific combination of packages. Okay, so let's give it a try. Again, first, we will run it for results inside an output directory that is called Out2. And in a second, we should be able to see that software stacks are resolved. As you can see, they are resolved much faster because the resolver uses pre-computed information about dependency and dependency graph. This was already mentioned in one of articles from previous How to Beat Python's Beep series. Okay, so as you can see, we generated 500 TensorFlow software stacks in 30 seconds. That is something like, sorry, in 16 seconds, and it's something like 30 seconds, 30 stacks per second. As done before, again, we can submit these dependency monkey run to Amun. So Amun again verifies that randomly sampled software stacks TensorFlow built correctly and that they run correctly. So we can do that. Again, it will be slightly slower because there's networking overhead to submit these results. And as you will see inside Argo workflows or Argo workflow, there are created new ones. So why do we do this? We basically randomly sampled the state space of all the possible TensorFlow software stacks that can be resolved. And by doing so, we can spot issues that can occur when resolving some packages inside dependency graph. So an example can be resolution process when you install TensorFlow together with other packages like Flask. So Flask introduces its own dependencies that can be shared with dependencies of TensorFlow and they can affect how the resolution process looks like and how the resulting software stack looks like, meaning different versions of packages. Okay, so by doing this random sampling, we found issues. An example can be an issue in TensorFlow 2.1 when specific URL lip was installed together with TensorFlow 2.1. We got this import error when we were trying to import TensorFlow. So here we can see, we tried to import TensorFlow but we observed import error when there was import range from package 6. This issue was caused by bundled 6 that is shipped together with URL lip 3 and the initialization of the importing logic in URL lip 3 collided with 6 package imports. So we observed this import error. That means that TensorFlow when installed with this URL lip will never work, will never start. So this is one observation that we have. You can find all the files on GitHub that are used in this demo and in this presentation. I will link them in the description down below. So that was our demo. I hope you liked it. Everything you saw is made in a project that is called project.tot. Project.tot is a project in AICoE, AI Center of Excellence that is a team inside Office of the CTO, the Red Heads Office of the CTO. You can find us on homepage.totstation.ninja. All the source code lives inside a GitHub organization that is called totstation. We have also Twitter handle. So if you are interested in updates, any progress we make or any quality that we conduct, feel free to subscribe to our Twitter handle. And we have also YouTube channel. So that's the YouTube channel where this video is published. So feel free to click on the link and feel free to subscribe to be updated when we have anything new. This way I would like to thank you for your attention and I hope we will see each other next time in the upcoming articles about project.tot and about how to beat Python's PIP articles. So thank you and have a nice day.