 Good day, I'm Eve and I'm Josie and we're here from the Google open source security team in Sydney, Australia to talk to you about dependency management. Before we start, we wanted to give some background on who we are. So even I on the Google open source security team or ghost for short. And our aim at ghost is to make open source secure at scale. Ghost works on and contributes to projects like the open SSF school card, salsa, OSB, and also depth.dev, which is more specifically the project that Eve and I work on. And we at depth.dev are particularly focused on being able to understand your open source packages and their dependencies. And just as dependency management can be quite hairy, the depth.dev mascot pictured here is also rather hairy and you'll see him feature throughout our slides. Open source continues to grow in almost every way. There are more packages, critical projects and attacks happening in the open source supply chain than ever before. An integral part of preventing future attacks is dependency management. And in this talk, we're going to highlight some of the ways that the way we see dependency management has changed in our time working on the ghost team. The things that we learned include that resolving dependencies is a difficult problem. That dependency graphs can be quite big. That dependency graphs can change quite a lot from day to day. That issues in indirect dependencies are much harder to fix than issues in your direct dependencies. That how you specify your dependency requirements can both cause and solve issues. And finally, that effective dependency management requires scalable solutions. So let's start with one of the very first lessons that we learned when we started looking at how the projects that we built depend upon open source software. And that is how complex it is to go from the set of open source dependencies that I declare in my package.json, go.mod or cargo.toml file to the actual concrete packages that get built into the final software artifact. Let's say you're making a web app or some other project represented by this package A and you want to depend on another package. That could be your web framework, your testing framework, or whatever it is. And let's call this package B. You specify your dependency on B in the form of requirements. So here my project A depends on package B at any version greater than or equal to version 1.5. And that requirement might be something that you've specified manually or just what you get if you run npm install B. We can see here that there are two versions of B available, version 1 and version 1.5.1. It's the responsibility of what's called a dependency resolution algorithm to take those requirements, look at the available versions, and determine which one should be included in your dependency graph. Each language ecosystem has its own dependency resolution tools, some of which you might be familiar with, like go.mod, cargo install, newget add, pip install, and so on. And it's the job of these tools to resolve the requirements to versions. Here there's only one version of B that matches my requirement greater than or equal to 1.5, and that's version 1.5.1. But the dependency resolution algorithm has to take into account not just the direct dependencies of your project, but also their dependencies and their dependencies and so on. And you'll notice that we've been talking about a dependency graph, not a dependency tree, and that's because your dependencies can have common dependencies. Here A has added another dependency on X at any version greater than or equal to 1, and X also has a dependency on C. Now, rather than just taking a single constraint on C into account, the dependency resolver has to satisfy the constraints from both B and X. Just like with B, there are a few different versions of X that could satisfy the constraint A has on X, and each of these versions has a slightly different requirement on C. Let's walk through each of them and see how bringing in that version would affect the resolution. Say the algorithm selects the oldest version of X, version 1. We now have both B and X depending on C with the same constraint, greater than or equal to version 2. So that's a pretty easy task. The resolution algorithm can just select version 2 of C like it did before, and everything's fine. What if our algorithm selects a slightly newer version of X, version 1.1? This new version has a dependency on C at any version greater than or equal to version 2.1. Now when the resolution runs, the algorithm has to select a later version of C that matches both the constraint from B of greater than or equal to 2, and the constraint from X of greater than or equal to 2.1. Again, we can find a version of C like version 2.1 that would satisfy these constraints. But you'll notice that the constraint that is introduced by X is actually bumping up the version that gets selected in your graph. And now what happens if the resolution algorithm selects that latest version of X, version 1.2? Now X is depending on C with the requirement of any version of C equal to 1.0.1. And that directly conflicts with the requirement B has of any version of C greater than or equal to 2, and this is the diamond dependency problem. So the resolution algorithm may actually not pick that latest version of X, version 1.2, to include in the dependency graph, because it will introduce an unresolvable conflict in the constraints on the package C. But actually not only can the resolution algorithm choose between different versions of X, it can also choose between different versions of B. And there might be a different version of B that has a different requirement on C that doesn't conflict with the requirement introduced by that latest version of X. Or maybe a version of B that just doesn't depend on C at all. So practically I hope that at this point you're beginning to understand the depth of the dependency problem that the resolver is trying to solve. And in fact, if you're someone who did computer science at university and you think back to your algorithms class, dependency resolution is actually an NP complete problem because it's a form of the constraint satisfaction problem. So what this means is that any true solution to this problem must sort of brute force. A true solution can't do any better than go through all possible combinations of package versions one by one. Check whether the combination satisfies the requirements. If it does, the algorithm stops and you found a solution. But if it doesn't, it moves on to the next combination and the next combination and so on. You can imagine that this can take a long time. And it does. Indeed, we at Depth.Dev have seen some package dependency resolutions out in the wild that take hours to complete if at all. So to get around this, some resolvers change the definition of the problem. So for example, in NPM, the resolver allows for packages to be installed multiple times at different versions. So in this case, rather than do the hard work of realizing that these requirements are incompatible, multiple versions of a package are allowed to be installed. And you can see here, there are two conflicting constraints of C greater than or equal to two and C equals 1.0.1. And the result here is that there are two versions installed. See it two and see it 1.0.1. This may be the first time you've heard about a constraint satisfaction problem, but you're probably already familiar with one. And that is the puzzle game Sudoku. So in Sudoku, each cell holds a number from one to nine, but not just any number. The rules are that if a cell has a number say five, none of the other cells in the row can be five. None of the other cells in the column can be five and none of the other cells in that square can be five. Initially, only some of the numbers are filled in and to solve the Sudoku, you need to fill in the rest of the numbers. And usually when you start solving Sudoku's, you start with the easier ones where many numbers are filled in initially and everything can be inferred from those starting numbers. But as you start doing the hardest Sudoku's, you reach states where doing simple elimination doesn't help you get the whole way. You actually need to start guessing numbers. So you guess a number, fill in all the numbers implied by that guess, then see if you come to a contradiction. If you do come to a contradiction, you erase everything that you filled in since that guess and then guess a different number and so on. But how is this the same problem as dependency resolution? This nine by nine Sudoku can be a lot to visualize. So we'll show you how it works with the simpler example of a two by two Sudoku. So in the two by two, the rules are a lot more simple. The only numbers available are zero and one. And the only rules are that the number in a cell cannot be the same as any of the numbers in the same row and column as it. So if there's a zero in the top left cell here, which I've labeled as A, the cells in the same row and column, so that's B and C, cannot be zero, which means they must be one. And the final empty square cannot be a one as there are ones in the same column and row as it. So D must be zero. And that's how the two by two works. We'll see how this translates to dependency resolution and package version requirements. So the problem started off with the top left cell labeled as A, and that cell contains a zero. And if we think about the cells as packages and the numbers in the cell as the version, this is like having a constraint on a package A that specifies the exact version zero. So B is in the same row as A and cell C is in the same column as A, so they can't be zero. So in dependency resolution terms, this is like saying that the package A at version zero has requirements of not equal to zero on packages B and C. And the constraint not equal to zero leaves only one other possible number because in the two by two only two numbers are possible. So version one is chosen for packages B and C. However, versions B and C at one also have requirements on cell D. And those requirements are that D must not be one. So the only option we can choose for D is zero. And we pick version zero and we found the solution to our two by two. The same principle works for the nine by nine Sudoku. It's just bigger and has more constraints than the two by two, but it's still the same constraint satisfaction problem. So in that cell on the top left, which we've labeled as A is equal to one. The constraints of the package AA at version one are that nothing in the same row can be one, nothing in the same column can be one, and nothing in the same square can be one. At depth.dev, we've had some fun with this fact that both dependency resolution and Sudoku's are actually the same problem. So we've re-implemented some resolvers. So since because we've re-implemented some resolvers bug for bug in the go programming language, we've had to write unit tests for them. And the nine by nine Sudoku is actually a test case that we have in our code base for our PI PI resolver. We constructed a test case with requirements on 81 packages with nine possible versions each, and the resolver was able to solve it. Nine by nine Sudoku's are a little bit more complex than two by two Sudoku's, but what's even more complex is real dependency resolution. Sudoku's are actually quite easy for dependency resolvers, and that's because in Sudoku only nine versions are possible for each package, and there are only 81 packages, but in real life dependency resolution, there can be hundreds if not thousands of versions for a package, and there are often hundreds of packages in a dependency graph, more on just how big the average dependency graph is later. But for now, the point here is that dependency resolution isn't as simple as looking at the requirements file. It's complex. The resolver is doing a lot of work behind the scenes to produce the final graph that gets built into your software. So when you're trying to manage your open source dependencies, it's important that you're not just looking at the requirements file, but also at the concrete set of dependencies that it produces. But dependency resolution isn't just complex. The dependency graphs produced by a change over time. And this is another lesson that we've learned building depth. So that dependency graphs can change quite a fair bit from day to day due to open requirements. Let's have a look at what open requirements are and how they can affect your graph. So we put requirements into two broad categories, pinned and open. Pin requirements specify exact versions, and we've got some examples of pin requirements on the left. Different ecosystems have different syntax for pinned requirements and different behavior if the exact version doesn't exist. On the other hand, open requirements allow room for change. So for all these requirements on the right, there are multiple versions that can satisfy them. With the exception of the example for MPM, these are all examples of open requirements that mean accept any version greater than or equal to version one. In MPM, the carrot operator is the usual way of specifying open requirements. And it means any version with the major version of one. Or in other words, any version that's greater than or equal to one, but less than two. When there are multiple valid versions that satisfy the requirement, what should the resolver choose? So here we've got the requirement on B of greater than or equal to 1.5. And there are two valid versions here that satisfy that requirement, version 1.5 and 1.6. So the question is, should it prefer the oldest version possible or the newest version possible? So there are some resolution tools like the Go Resolver and the Nougat Resolver, which prefer the oldest version that matches. In this case, 1.5. The idea there is that the oldest version is the most stable and won't automatically pick up vulnerabilities or bugs that exist in later versions. But there are other tools like the NPM Resolver, which prefer the newest version that matches. So in this case, that would be 1.6. The idea here is that the newest version will have the latest bug fixes and the latest features. So which is the better choice? There's no obvious better approach here. There are just pros and cons to each. An example of the effects of the interplay of the version selection strategy and open requirements on a simple dependency graph. For this example, we're going to say that the packages we're looking at are in the NPM ecosystem. So again, I've got my package A that depends on B with the requirement Carat 1.0.0. And remember that that's any version with a major version 1. And as we just explained, the NPM Resolver defaults to selecting the latest matching version. So I want to try and figure out what my package A depends on. And I run my resolution algorithm and I get the following dependency graph. So both B1 and B1.0.1 satisfy the constraint Carat 1.0.0. So we end up picking the latest version, which is 1.0.1 to include in the dependency graph. What happens if B introduces a new version? Let's say version 1.0.2. Now again, I want to figure out what my project depends on. I rerun my resolution algorithm. And the default behavior of the Resolver, remember, is to select the latest version that matches the requirements. This time when I run my Resolver, the version of B that gets selected is version 1.0.2. This might not seem like a huge change, but it can have immense flow on effects, especially in really large dependency graphs. For example, version 1.0.2 of B might introduce dependencies or delete all dependencies. And this isn't just a hypothetical situation. So this graph here is the percentage of package versions within each ecosystem. And we've got the cargo Rust ecosystem, Go, Maven, NPM and PyPI. So this is the percentage of package versions within each ecosystem that had their graphs change over the month of August this year. So you can see that most ecosystems kind of are hovering around the 10% mark. And that means 10% of package dependency graphs in those ecosystems change each day, which is quite a lot. The only outlier is the blue at the bottom. And that's Maven. And that is because in Maven, the norm is to specify requirements as pinned rather than open. And we'll see what effects this has later on the ecosystem at large. This churn is important to realize because it causes vulnerabilities and licenses to change every day. Really 40,000 of the 22 million MPM packages we track see a change in their license set or vulnerability set every day. So that could be a new vulnerability or if you're lucky a vulnerability that's been patched. And that means if you're someone who is working to manage that license or advisory set in your project or organization at this scale, you're doing a pretty massive job. So dependency graphs, they change over time. And what that means is that when we think about our dependency graph, we need to be conscious that it's something that we need to maintain. It's not enough to vet our dependencies one time and then forget about them. But not only do dependency graphs change over time, in terms of the number of dependency packages, these graphs tend to be quite big. And this is because your dependencies depend on other dependencies, which depend on other dependencies and so on. So for example, in MPM, the average package has only six direct dependencies compared to an average of 110 indirect dependencies. And we see this play out across many of the ecosystems that we track in depth.dev. For each ecosystem, the blue is the average number of direct dependencies and in red is the average number of indirect dependencies. So you can see that dependency graph size blowout comes from the red indirect dependencies. So to point out a few in cargo, that's the Rust ecosystem. The average package is six direct dependencies, but over 60 indirect dependencies in Maven, the Java ecosystem, it's about an average of six direct dependencies versus 33 indirect dependencies. So after looking at these statistics, we can see that this dependency graph that we introduced earlier is probably not very representative of the average dependency graph for a package. A typical dependency graph probably looks something more like this. This is the dependency graph for the MPM minimalist web framework express. So you can search this package on the depth.dev website, click the dependencies tab and see this graph visualization there. Although if you're really unlucky, your graph may look something more like this. This is the actual dependency graph for the Go Kubernetes module at version 1.28.1. And this obviously looks a bit ridiculous. And yes, there may have been a clearer way to visualize this. But the purpose of showing you this graph is to give you a sense of the complexity of dependency graphs in the wild. Because all the ecosystems that we analyze have packages and really important packages that have dependency graphs containing thousands of package versions. And in MPM and cargo, there are package versions with tens of thousands of indirect dependencies. So when you go to add another dependency, think about it as adding not one dependency, but 60 dependencies in cargo or 100 dependencies in MPM. And also again, take a moment to appreciate the people in your project or organization who are actually having to deal with this problem and deal with these huge dependency graphs. So your graphs are probably quite big and they probably change with time due to open requirements. But why should you care about the graph itself? Why do you need to pay attention to what's in it? Well, paying attention to the graph really comes to the fore when there's an issue with one of the nodes in the graph. For example, if there's a security vulnerability affecting one of the packages in your graph, how do you go about identifying the vulnerability and updating what is possibly an indirect dependency to a newer unaffected version? Or what if there's an incompatible license in the graph? Where is it? How would you go about cutting it out of the dependency graph? How would you even know that there was an incompatible license in the first place? Which brings us to the fact that issues like license incompatibilities and security vulnerabilities are really hard to fix when you're affected indirectly versus directly. And this is pretty important because in Dev so Dev we've seen that 98% of the time when a package is affected by a security vulnerability, it's affected indirectly. But why is it harder to fix when it's indirect? Well, to understand that we need to understand how requirements are typically specified in an ecosystem and how this affects remediation. For example, let's take a look at how projects in the Maven ecosystem specify their requirements. So of the nearly 67 million requirements we analyzed in Maven, we saw a pretty clear preference. 99% of requirements in Maven specified an exact version number. So we call that pinning. So an exact version number in Maven, say on 1.0, means select 1.0 and if that version doesn't exist, then pick a later version, which means that in 98% of the cases, they got the exact specified version. So in almost all cases, explicit action needs to be taken by maintainers in order to move off of versions that contain vulnerabilities. Let's compare what remediation looks like when you're directly versus indirectly affected. Here is a project that depends on an affected package version A and to pull in the fix, a newer unaffected version of A needs to be published and we call that the fixed version. And then you need to update your requirement file to specify the newer version. Sometimes these packages don't fall as simple and they can break your tests, but let's assume that this package does and you're just able to bump the version. And let's compare this with how indirect remediation works. In this situation, the package A is pulling in a package B with a vulnerability, which means you could be affected indirectly by the vulnerability B because in your graph. So in order for you to pull in the fix, a newer unaffected version of B needs to be published. A needs to publish a new version that specifies the unaffected version of B and then you need to update your project to specify that newer version of A as a direct dependency. So the important thing to see here is that in order to remediate that vulnerability, you actually depend on the maintainers of A and B to remediate that vulnerability first. And that is the key difference between direct versus indirect vulnerabilities. Indirect vulnerabilities require you to depend on package maintainer action. And you can imagine what this would look like if the vulnerability is more packages away from you in this case for this would require new versions of A, B, C and D to be published. And your project A, B and C need to update their requirement to require the unaffected version of its child package. So you can see that that's quite a lot of work for maintainers. And the overhead that's required to do indirect vulnerability remediation has real world effects. The log for shell vulnerability was a really great example of why the often indirect nature of vulnerability introduction makes remediation so difficult. Many of you will probably have heard of log for shell. And if that's the case, I hope this doesn't bring back too many bad memories. But just in case you haven't log for shell was a zero day remote code execution vulnerability in a popular Java Java logging framework called log for J. It was discovered at the end of 2021 and the incident affected more than 35,000 packages, which amounts to over 8% of all the packages on the Maven central repository. From analyzing the security incident in the depth dot dev project, we saw that one of the reasons this vulnerability was so difficult to remediate was because it was affecting packages so deep in their dependency graphs. To break it down, of all the packages in the Maven ecosystem that were affected by the vulnerability in log for J, only 20% of them were affected at depth one. That means that 80% of the time log for J was introduced indirectly. So in fixing log for J, we at depth so dev saw that after one week about 13% of packages of the initially affected packages had been fixed. 10 days out that number was about at 25%. Few months later, it was 40% of packages fixed. And according to sonar type at the end of last year, there was still about 30% of remaining packages still left unfixed. But it's important to know that this is pretty amazing because this was a lot of manual requirements to bomb. There was a coordinated effort across the Maven ecosystem to move off of affected versions of log for J involving thousands of people. So based on looking at the Maven log for shell case study, you can see why indirect issues can take longer to fix than issues in your direct dependencies. But you may be thinking that there are some other solutions to this problem. So for example, in Go, rather than needing new versions of all these packages to be published, you can directly specify the patched version of D from your project. So that would look something like this. No new versions of A, B or C need to be published. Your project just specifies D at the patched version. And because in Go, you can't install dependencies at two different versions. C uses the patched version of D in your graph. Another solution in ecosystems like NPM is to use open requirements, which are the norm in NPM, particularly the tilde operator and the carrot operator. So in this case, rather than needing versions of A, B and C in this scenario to be published, the flexibility of the requirement on D from C would automatically pick up the patched version. So let's look at how using these open requirements can affect your graph. Across all NPM packages, nearly 74%, essentially three quarters of requirements are open. This means that a newer version of A will be used when one is available, when that is when there's a patch number increase. You can guess that these trends within each ecosystem are very dependent on what the default requirements are when you install a package. So let's think about what log for shell would have looked like if it was in the NPM ecosystem. Three quarters of all log4j users would have automatically pulled in the newer patched version. Log4j happened in Maven and not in NPM, so we'll never know what it would have really been like. But we can look at a similar incident that did happen in NPM and see how that played out. In early January 2022, the developer of the popular NPM packages Colors and Faker intentionally published several releases containing breaking changes. These were picked up rapidly due to the programming norm we saw in JavaScript of using open requirements and the way that the NPM resolution algorithm preferences those recent releases. The graph here shows the dependency flow in the ecosystem over the 72 hours where the action happened. So on the x-axis we have time and on the y-axis we have the number of packages in NPM that were depending on the various versions, some affected and some unaffected. You can see about half, that's the green half at the bottom, remained using the unaffected earlier versions. But in the top half we see some movement. At the time of the incident more than 100,000 packages most recent releases depended on Colors and around half of them, 50,000 packages, had a dependency on an affected version. In fact shortly after the first affected version 1.4.1 was introduced, another similarly affected version 1.4.2 was also introduced. And many packages quickly moved on to the new version that same day due to the way NPM is choosing the highest version that matches the requirement. But we can also see this jagged curve where more people are moving to unaffected versions of the Colors package, in this case usually 1.4. People were beginning to pin to unaffected versions of Colors or to just move off the package entirely. For comparison with Log4Shell, this whole chart from vulnerability introduction to remediation takes place within just three days. So you can see that this vulnerability was mitigated really quickly. But why did it take months for Log4Shell, but only a few days for Colors? How were the packages remediated so fast? The key lies in the number of maintainer actions needed to mitigate the issues. Just a little over 1% of the affected packages that depended on Colors actually made a release during this time period. But their work resulted in 43% of the total affected packages mitigating the issue. And that's because the open requirements were allowing fixes to propagate really fast to the ecosystem. So does this mean that all project owners should change all their dependency requirements to be open and that all resolvers should choose latest versions? Unfortunately, like everything else, it's a little bit more complicated than that. As a counter example of how open requirements might cause problem, let's take a look at the event stream attack. So event stream is a helper package for working with streaming data, but it hadn't been in active development for a couple of years. It's author had a lot of things on their plate, so they let maintenance of the package fall to the wayside. So when a user contacted the author and offered to take over the project, the author of event stream understandably saw that as an opportunity to shed some responsibility and so handed over ownership to this person. The problem was, as you may have guessed, that the person offering to help was a hacker. So they used ownership of the project to make a new release of event stream that contained a malicious payload. The latest legitimate version of the package had been published about two years ago and was version 3.3.5. And the malicious actor published the new vulnerable version as version 3.3.6. And as NPM is an ecosystem that automatically updates to latest versions, projects picked up the latest vulnerable version 3.3.6 by default. The hack actually wasn't discovered for a couple of months and it was only eventually discovered because of a small error that the attacker made. As event stream is a popular package, by the time it was discovered, the vulnerable version had been downloaded millions of times and had been executed millions more. If this had been a Maven package, where the dependencies are not automatically updated to latest versions, would the hack have had such a broad effect? So we can see that there are good and bad things about pinned versus open requirements. Pinned requirements mean that you have more control over your dependencies, which means that you don't automatically pick up vulnerabilities and you don't automatically pick up breaking changes. The graph doesn't change under your feed, it changes on your terms when you manually update the requirement. However, there are benefits to having it the other way where dependencies automatically update. Open requirements which facilitate this allow you to automatically pick up the good stuff quickly like bug fixes and vulnerability patches. But of course it can also automatically pick up the bad stuff. So one way to get the best of both is to use pinned requirements with a bot like DependerBot or RenovateBot. That creates PRs to update your pinned requirements when a new version is available. So this means you get control over your changing graph and you also get automatic updates. And this is a particularly good solution if you're in application rather than a library. Cara and Pedro gave a really great talk about scorecards discussing the solution a few days ago. I would really encourage you to see that if you haven't. But any solution that you choose would have its pros and cons. So it's always a good idea to scan your graph continuously for issues because it's big and it changes. It changes sometimes due to open requirements but it can also change because a version that's good today might not be good tomorrow. You might ship your product today with its dependencies and have done your due diligence by scanning for vulnerabilities and other things and found none. Only to have many vulnerabilities reported on it in the following weeks. This brings us to our final lesson learned which is that effective dependency management requires scalable solutions. Any form of dependency management we're going to do needs to be scalable. These are some tools outside your resolver that can help to manage your dependency scalably. And we'll take a look at a few of them. That's OSB scanner, open SSF scorecard and the depth.dev API. The online tool that gives you a summary for any known vulnerabilities on packages that are listed in your log files, mod files or whatever those are in the language that you're using. Here I'm running OSB scanner on an old version of memos, which is a popular self-hosted memo hub. And we can see from the output table that this version is affected by several vulnerabilities in both Go and MPM. And more than that, for the Go vulnerabilities, these are split into called and uncalled vulnerabilities. Where called is identifying that the vulnerable code is actually being executed by your project. And OSB scanner is using the Go Von check tool to determine that. So from here as a developer, I have a really clear understanding of what my next step should be and where I need to prioritize my efforts. And good news, there'll be even more language ecosystem specific support for called analysis coming soon. And even more good news, those vulnerabilities have all been patched in the newest version of memos. So that's an example of how OSB scanner can help with scanning for vulnerabilities. But vulnerabilities are just one of many things that you might want to be scanning for. You may want to keep scanning for licenses, general health package checks and so on. The scorecard project runs fully automated checks on more than a million repos across GitHub and contains this package and collects, sorry, this package health information for you to be able to look at at a glance. These checks include things like whether the tests are running continuous integration, whether there's at least two contributors from different organizations, whether they sign releases, whether there have been any recent commits and so on. We display these scorecard checks on depth.dev among other information like dependencies, dependency graphs, dependency licenses and dependence. We serve this information to our website, but also through our API. In addition to things like vulnerabilities, dependencies and licenses, the API also serve scorecard checks and the API is completely free. So for example, if you wanted to build a tool that prevents any packages with a failing scorecard check, like no code review from being checked into main, you could build a tool using the API to check for this specific scorecard check for all packages listed in your log file, then fail the pre-submit if any of the packages fail that check. We also make this depth.dev information available through our BigQuery data set for ecosystem-wide data analysis. And in fact, this data set is how we produced a lot of the statistics that we've included in this presentation and maybe especially of interest to researchers. And lastly, we wanted to say that if you are a maintainer of an open source project, thank you for all the work that you do. Even I really believe that in order for open source to be sustainable, maintainers really should be getting the critical recognition that they deserve. And here are just a couple of programs and projects that work on recognizing maintenance. So today we've discussed the following lessons that we've learned from our time working on Ghost. That resolving dependencies is complex. It's not as simple as just looking at your requirements file. That dependency graphs change? The dependency graph that you're managing changes? That dependency graphs can get really big when you're adding a dependency. Don't think about it as adding just one dependency, but adding many dependencies. That issues in your indirect dependencies are much harder to fix than issues in your direct dependencies because multiple layers of maintainer action is needed? That how you specify your requirements can both cause and solve issues because it can mean that vulnerabilities or features and fixes are brought in quickly or slowly. And finally that effective dependency management requires scalable solutions. So scan your graphs continuously. Thank you very much for listening to our talk. Here's some links you can check out. I don't think we have very much time for questions and I don't want to get in the way of anyone's lunch plans. So we'll stick around outside if you want to come and chat to us about anything in the presentation. Thank you.