 All right, I guess we're on time now. So yeah, thanks everyone for coming here. It's been really exciting being here the last few days, seeing all the talks, interacting people, and talking a lot about dependency management and supply chain problems. So my name is Joseph Hyderup. I'm a software engineer at Endolabs. For those of you who don't know Endolabs, it's an early stage startup that works on improving the dependency management lifecycle where we try to improve with selecting, securing, and maintaining open source dependencies. And apart from working at Endolabs, I'm also a PhD student at the Duff University of Technology in the Netherlands, where I've for the past five years been working on researching, studying dependency management, problems in package repositories, and also what kind of leads a bit to this talk. I mean, things related to static analysis and program analysis. So the talk is titled Going Beyond Metadata, Why We Need to Think of Adopting Static Analysis in Penalty Tools. And I thought before we dive into dependency management static analysis, it's sometimes good to always take a step back. And I'm going all the way back to the 1980s. And I think one thing that's been often talked about is with security, with dependencies, maintenance problems. But we also have to look back and think about why are we using these tools in the first place, because we're doing it for software reuse mainly. And I was very curious to know how did we use software reuse back then. And I found this very interesting guide from the US National Department of Commerce and also what I like with the cover of it. It shows a little bit like a dependency tree, probably already thought about it back then. And in this guidebook, I found some interesting definitions of what software reuse is and the ideas around it. And there are two things that captured my attention. So one thing was with the productivity aspect, reusing well-designed, well-developed, well-documented, software improves productivity and reduces software development time costs and risks. And the other aspect is quality. Improvements in the quality of software developed from well-designed, well-tested, and well- documented reusable software components. And if you also think about it today, this is kind of what we want. And if you also see how far have we come with doing this, we have basically starting using, not surprising, packet managers. And maybe it's probably my little bit like academic OCD or something. So I mean, there's a lot of talks about packet managers, but I think it's important to distinguish the parts of package management that are interesting. So usually the way I see it, there are three main parts. So one is, for example, that you have the gateway to thousands of libraries and frameworks, like the package repositories. And then the other aspect is it's also like a way of distributing packages, like for you to easily publish and finding them, et cetera. But I'm going to focus more about this really on the dependency management and the complexities around that in general. And I think for many of you, know the key problems in package management or the key challenges. But I think it's always good to touch base and be on the same ground. So one of the main problems is that you usually come up with very complex and large package compositions. So for example, last week I was installing a very simple Rust project. I think it was about 150 lines of code, but I ended up trying to install 307 dependencies for that. And that's quite a lot. But it's not only about installing these dependencies. The other aspect is that, for example, like in MPM or in Cargo, you can have the same dependency, but different versions. So for example, if you see in the image here, you have accepts 1.38. And then another one that is 2.80, because here Cargo wasn't able to find a single version that can match the dependency specifications in that particular program. And then on your left, we have the typical package manifest. So we have the list of dependencies that you're using, but also the version constraints. And here we have the other aspect of temporal properties. So for example, if I install the dependencies today, we have these type of versions. And then three days later, we have a different type of version. So that's also a challenge that becomes difficult, especially when you want to build dependency tooling or understanding what's really going on with your dependencies. And then the other very common aspect, which is more like from the global perspective, which is often being very much talked about and always pops up like in news feeds, that there's a lot of things going on, like for example, the notorious left-back incident or with the event stream user trying people, trying to hide bit-concealing code, et cetera. And that's also a big concern for organizations, software developers in general. So these two are the kind of main two aspect that's kind of what you have to think about when you're starting to create some kind of dependency tooling or thinking about what way to try to understand and deal with these type of issues. So kind of looking at what is the general approach to trying to understand these problems, and with respect. So the first one with the temporal properties, I think here is kind of like what you can do. You can sort of opt out from version updates, et cetera, by basically doing version pinning log files. And I think here we kind of have a set of tools on how to deal with it in general. But then for everything else, we have to start building dependency analyzer, both plugins, and able to track monitor problems that might exist in repositories or within your own dependencies. So what is the usual workflow? So you have kind of like symbolized with a bug, but it can be anything like vulnerabilities, updates, auditing, quality, deprecations, et cetera, right? So that's kind of like the starting point, and usually you would know the exact or the range of packages or the versions that have these type of problems. And then comes the second part, which is in the middle here. So you have, let's say, your package manifest, which I was showing a bit earlier with the temporal properties. You start building, let's say, the relationships between them. Like you resolve the version, you get to your direct dependencies, and trustive dependencies. And then, for example, if, let's say, the most top-bottom package here has a problem with the red line circled, that's like how if, for example, we do a reachability analysis. And that's for the most part how a majority of, be it open source, even industry tool works, that you operate on the level, like what I'm calling the metadata perspective. And then we see to the right that, although this is very useful in general and also give you some idea, but sometimes it becomes too much, which many, I would say, online complain about this alert fatigue. You get 200 warnings, or you get 50 updates on the panda bots. It's not very easy to operate with this type of information. So while this is a well-known problem, I kind of wanted to go a little bit beyond that and look into what is really the actionability, but also precision, because we shouldn't fully downplay that metadata is terrible, we should look into better solutions. But it's always better to try to have a more pragmatic approach and see what are the trade-offs, what are possible benefits, or downsides, et cetera. So that's kind of where I, let's say, oh yeah, took more like a research approach, but okay, yeah. Before going into that, it's always good to revisit before, so I kind of highlighted on the productivity aspect, so looking at the kind of landscape that we have today, of course, this is, I wouldn't say, very conclusive things, but if you look from very, very short-term, if you do software use, we can easily reduce, let's say, time costs at the very short-term, long-term, that's a different question. But one thing that we haven't, let's say, really graphs very well, in my opinion, is the whole kind of risk aspect, or profiling in general. So that's something that we need to address more, in my opinion. So yeah, so kind of going back to this whole sort of classic alert fatigue thing, so it's kind of become more like something that we're blaming a lot on. But I kind of want to, let's say, focus on the quality of analyzers and going into the whole trade-off aspect, what is better in which use case scenario. So the first rule that I'm saying is that metadata is not source code. Because when we have a package manifest like this, what it's only really telling is only the declaration, but not the usage of a particular pattern or package or why we are having theory in general. And then going to the second rule is that we should try and make code first-class citizens in analysis, which is basically derived from the first rule itself. And you can see here, so this is an example here. We have the dependency tree. It shows the packages, et cetera. And then we have the other aspect with the call-graph perspective. So call-graphs is basically the functions and the calls between them. So it also shows basically from the project and their dependencies. And then the last one, for example, if you know that a particular function is vulnerable or has a bug, et cetera, you can basically rule out which functions are the ones that are impacted by a particular problem. And here if you see from a very developer point of view, if you know that version 1.0 has a warning, versus knowing that function bus, like in version 1.0 it has cybernabilitizing, which are being called, that gives you a much more clear perspective and makes it also more easier for you to understand what is problematic and what is going on. Because as a developer, we work more on a source code more than the manifest information. So that's kind of also what I want to highlight here, that these two different representations, like they give very different information and actionability. And the third part, so the kind of main criticism I usually get is that it's really cool to do program analysis, static analysis, right? But it's often usually very expensive. Sometimes it's that people who have special knowledge is also not scalable, because usually when we think of program analysis, we do it from a project, and not a project together with all the penises. And think back to the example I was giving to these 300 Rust crates, imagine we have to build a call graph for all of them. And luckily as a researcher, I can freely study these kind of questions and issues. So for one research paper, I looked into trying to build a call graph for every single crates and release on Rust. So this was like back in 2020. And yeah, so I was able to, from 23,767 packages and 140K releases, I was able to build a call graph in 10 days. And also like, this was let's say like 67%, but in reality it's actually 80% because many of those releases were actually broken, for example, like from 2014, 2015, that's not really used. But it was fairly quick to build it. And I also like built a call graph for Java. And that I can also do in like, let's say like a couple of seconds as well. So the general thing is like, try not to aim for like complicated static analysis. Start with something that's very lightweight, for example, like using a call graph to understand like limit what's happening in like, say like dependencies and so forth. Yeah, and also coming in other sort of concern is like, yeah, it's always like a bit of overkill for me to add program analysis. And then the typical question, what about Python and JavaScript? And then you also have like this aspect with, like program analysis suffers from false negatives, like my security customer won't be really happy about it. So there's no point for me of adding program analysis. So knowing all these kind of like different like possibilities and like opportunities and also like the kind of problems. Let's try to like understand like, what would really be like for a very simple question, like counting number of dependencies? What would it be? Let's say like if we do that using metadata based versus using program analysis, like call graphs to count the number of dependencies. And yeah, this maybe a little bit too much to like sort of grasp, like this graphs, et cetera. So I would say like the details are not too important. So like I basically like calculated from different standpoint, the time point, the number of direct dependencies. Of course, like from earlier snapshots, like from 2015 and 2018 of the Rust ecosystem, there's a lot of differences between number of dependencies. But if you look at the recent snapshots, and if we like so mean to like the median here, we can see that they sort of like on average, relatively like approximate each other. And if you think about it, right? So when we use, let's say like the direct dependencies, we also like when we declare them, we also very likely to use them as well. So in those cases, like, if you're not doing like any fancy analysis and just want to know, for example, what is like the number of dependencies that you're using, it can be well enough to just use the metadata and not really having to do static analysis. And here you see like, there are basically three different networks. So like the present one is like, yeah, basically like using call graphs. The creator is like the metadata and doxorize basically compile verified metadata. Now that we turn on to the transitive dependencies, they have a totally different story here. And we basically have no close approximation. So if you look even in like, when I studied this like in Feb 2020, the number of transitive dependencies like an average project might use in Rust is 17. Whereas when I do the call graph based perspective, it's six of them. And then when I calculate the mean as well like the average, I also find that 60% of the transitive dependencies are also not called in general. So this is quite interesting. And that was kind of maybe like very curious, like how can it be such a huge difference between those two like representations. And that's also when I kind of did like a more collective analysis by doing like a diff between like the metadata based, let's say like dependency tree or like a conventional one versus like one that is based on let's say like function calls. And, you know, studying like a project together like with dependencies, not like a fun thing to do, especially for large ones. And I did this for like 34 cases. And this is like basically the main differences that I was able to find. So in the first case, we just like, okay, so those cases were not only like transitive ones, but also like direct dependencies. So I think out of those 34 cases, we had like seven cases which was not totally unexpected. There were for some dependencies, like no input statements, but also in others, there were like four input statements, but no actual usage with usage. I mean, like the basically, you know, data structures imported, no function calls or any of that. So like, it's not totally unexpected. I mean, this can happen like on open source projects. And then the other difference, and I think this also shows like when you start using static analysis versus like metadata basis, that you start seeing those features that kind of matters in code. So sometimes like you might, for example, like have this like conditional compilation flags or in the source code, you might have like a specific section that if you like set this flag, it will start compiling this code. And if you just use, for example, like the default one, you will see that, for example, one dependency might not be used because you didn't add this extra flag to it. But then on the metadata base, you don't really have this information. So you would directly assume that you're using that dependency anyways. And then the other thing, maybe this is more rust specific, not the case for like every language, but it's at least like a good case to think about that not all dependencies are like, just like runtime libraries are part of like APIs that you would directly use or like might also be for code generation because in Rust, you have like this derivative macro libraries, but from the metadata information, I'm not sure if it's fixed yet, but there's no tag that says that this is like a regular library or not. And this is specifically like with the Sergi library that, you know, when I build a call graph for it, I cannot directly see like call connections for all functions because they are code generated. And then another thing which I would say like would be more like something of a linting category thing is that you have like the test dependencies. So if you've wrongly declared it, that's like in your default, like where you would have like your regular dependencies. Yeah, like you might like just put it there and you can basically use it for testing, for example. I'm not sure if that's changing Rust, but yeah, it was shown like that. So that's something that I would say like you can kind of rule out those kind of things. But then this is like the main thing. This is like kind of coming back to the all with transitive dependencies that there were like 60 cases of non-reachable transitive dependencies. And what do I mean here? So let's have an example. And before I explain this, so this is like now like a quiz question to see that you actually understood everything of my talk. So is anyone brave enough to tell how many dependencies in app version one we have? Three dependencies. So live one, live two, live three. So that's correct. That is from like the metadata perspective. What about it from a cold perspective? How many dependencies do we have? Two, exactly. And as you can see here, like so I have like denoted the cold relationships. So we have that main calls full and then full calls bar and live one. And then bar calls use and live two. But then we see here that we have a function called unused and live two. And that is called a live tree used, right? And this is where I mean with the non-reachable transitive dependencies because we are not really using the, let's say like unused function in live two. But in the metadata base, we don't really see this perspective. And what it really shows that context really matters because what the kind of assumption we're using when you analyze, let's say like package data or metadata is that, we are explicitly making the assumption that we're using all direct APIs, right? Of direct dependencies. And then all the direct, sort of all APIs of transitive dependencies, which is, I mean, doesn't really make sense like in real life because usually let's say like, if you have a library with 20 APIs, we might use five of them. Or in other case, if it's like a very important, let's say like library, we might say like 80% of the API. But thinking about like what the APIs are being used and then further sort of see what it goes down to the transitive dependencies is like really important because that's also like where we can eliminate a lot of the false positives as well when it comes to doing analysis in dependencies. So now I'm going like to the practical like trade-offs. So when we look at the clear dependency like, yeah, so when we look at let's say like direct dependencies, based on like the analysis and research I've done, we can see that if you don't have like a very specific case, we have to like pinpoint anything. Using, let's say like metadata is like enough to approximate what call graphs or like static analysis would do. And that's like really true for the case of direct dependencies. And of course, like in the case of security and let's say like case where you need like very strong, like soundness in general, it makes a lot of sense to use like the whole package-based perspective, sorry, metadata database perspective or in general, let's say like you can also like think of like doing like a hybrid solution. But an issue with let's say like using static analysis and it comes particularly for libraries that they use a lot of code generation, let's say like or dynamic features, you might be missing a lot because you cannot capture those function calls in general. And then yeah, so those are like the pros and the cons. On the other hand, is that yeah, you miss out things for example, like if you have a library that has like no input statements or like yeah, and you want to look at maybe some specific APIs, et cetera, you kind of have to go down and like understand like what I'll say like the APIs are being used, et cetera. So then you kind of still would need like the use of call graphs, but also like you don't have to specifically use call graphs as well. Like even simple, let's say like code scan, finding input statements, et cetera, already gives you some idea about what is actually being used as well. And that's something I didn't like mentioned earlier. And yeah, the other point that you cannot eliminate the dependencies that are solely doing code generation. And then looking at the practical implications trade-offs for transfer dependencies. Here I would generally like advocate that if you're gonna do something that really captures the whole dependency tree, or at least goes beyond direct dependencies, trying to like thinking about, what kind of static analysis tool, program analysis tool, or any code analysis. I think it's very important to kind of take this information account because here context matter a lot. Because like one product might use their direct APIs very differently from another project and that also affects what kind of transitive dependencies are also being used as well. And I think especially when it comes to transitive dependencies, like here like the actual ability could also be like, yeah you also get better of that as well because you can directly pinpoint on problems. And I think that's also the thing as well because like if you look at the direct dependencies because you have a good idea about how we're using your direct dependencies, you can also directly filter our problems. But for your transitive dependencies it's way more difficult because you don't really know how it's being used or like how it plays a role in your project. And then like the con series of course, like false negatives may not be ideal for security applications because that's like an inherent limitations with program analysis techniques. And then the other point is like, if you let's say like have to analyze libraries that are like dynamic in nature use a lot of code generation, et cetera, and yeah, that will always be very difficult to do with program analysis. Because sometimes like you might focus on let's say like a subset of the ecosystem where most of like your libraries are based on that, then you're not gonna get much actual ability from using program analysis. So that's like a very important thing to know about. And then we go into like, yes, like in general like program analysis itself, like precision recall like the toolie. And so when I've like been analyzing package repository or like dependency management in general, I've actually not focused so much on precision, but I focus more on trying to like have high coverage of language features. And the reason why is that I feel even though let's say like you might not get, I mean, this particular program comes, let's say like for example, when you have to deal with dynamic dispatch. So if you have, let's say like an application and all the dependencies and you want to find, let's say like all the implementations for like a dynamic whole, usually it happens that you also link with like other libraries within your program, et cetera, that has like no connection to this interface. I mean, that's of course not great all the time. But at least like you get, let's say like a more clear idea what is being used. And that can be very useful. I mean, I find that even, I mean, it's maybe not ideal, but it's still better than not having this information by looking at metadata in general. But in cases that you need to have, let's say like more precision. So like you might have to consider for example, like points to algorithms, et cetera. But then you had the problem that it's not super scalable. It might work for your project, but the moment you start going down to like your dependencies, et cetera, and trying to do this analysis, it can take really long time. I think in one case of years ago, like I was trying to analyze like a couple of Java projects with their dependencies. I think in some cases like took just two hours to build like the core graph with points information. In some cases like it never even finished. And that's the thing. So the general implication is like when you want to do program analysis in the context of dependency management, you have to think about scope of the analysis, which is your project and its dependency tree. And the second part that package repositories are not like homogeneous collection of libraries. So like one thing especially like if you're doing research like you read a lot of static analysis papers, the assumptions are usually made around like some ideal set of packages that has this particular language features. But the moment you start go to the ecosystem or starting to like analyze things in a while, the situation looks very, very different. And that's something like very important to keep in mind because I've seen like also talk like a lot of other projects, et cetera, that you kind of directly heavily invest in program analysis. But once you sort of put it in production, you see that, oh, it didn't really work for this customer or that customer, et cetera, because they're using this set of libraries. So it's very important to kind of have that. Yeah, I mean, think about that aspect as well. Yeah, so I've been talking a lot about like program analysis in general, but now looking at this like tooling after because many might, for example, use the pen about sneak, et cetera, for like the pen analysis. And what I'm like really glad to see is that more and more like moving towards program analysis. That's like a very, very good thing. And yeah, so, and I think also in general, like we're gonna see more and more tooling doing that. I think the main thing, like, especially like in the open source space, like we need to have, let's say like better, maybe tooling support that makes it more easier to like use, let's say like program analysis tools in the pen analysis, because for example, you might use a tool like Cargo Audit, et cetera. But thinking about how to use, let's say like a Rust core graph, et cetera, it's not that trivial or easy. And that's also something that I'm thinking maybe about how I can, for example, like contribute this type of information or like guidelines on how to actually implement it. And yeah, so kind of going to Dan, so like, yeah, this is also a little bit like what my research has been about. So another like possibility, so I've been talking more about from, let's say like the user perspective, like I mean, using package manager, what about from a community perspective? So this slide, a long time ago I was showing us on how we would do, let's say like building like core graphs for Rust. And I have actually done a paper where I used the whole, all the core graphs to build like a huge core graph network of Crayster.io. And what is like very interesting here is that when we know, let's say like the correlations in packages in the ecosystem, we can start like learning about, apart from general question, like, you know, what are the most used packages? You know, like how many packages have, let's say like dependencies to outdated packages? But we can, for example, know that if there's like a particular security vulnerability in the ecosystem, we can directly find out through call paths, which packets are directly affected and not showing, for example, like 40% of Crayster.io is impacted by this, but by having this call information, we can similar to what we're doing, let's say like, on the application level, also eliminate false positives as well, let's say like that, only 20% are actually impacted. The other thing as well is that we can also start learning, for example, about what the APIs are being, let's say like more co-used together. So for example, like analogy, like in Python is that usually like you have a set of pandas libraries together if non-py libraries are often used together. What if you, for example, like, club them together? And see basically like how would the ecosystem look like from that perspective? So yeah, yeah, this more or less like concludes my talk, so does anyone have like questions on like, yeah, static analysis, program analysis in general? I'll start, yeah, I'll start with you first. Is it your possible target for this kind of analysis? So I think language that are like, usually they say like compile this languages, they are relatively quite possible to do, but then of course like, where tooling around static analysis have come, that kind of really varies a lot. For example, like in Java, which is like very mature, you have like very robust, let's say like static analysis tools, et cetera, that are like very easy to do this type of analysis. Also in the case of Rust, like when I like had to do this analysis, there were no core graph library in the first place. I have to like do some kind of trick with LLVM to get the core graph, but then I realized that was not that great, so we had to eventually build our own core graph generator, but the challenge really comes when we have to deal with, let's say like Python or like JavaScript, because here we cannot really use, like I mean, we can use some static analysis, but it doesn't really give like a complete picture. So we'll always like have to complement with using, for example, like dynamic analysis. There are like some research on where you're trying to sort of use like a crowd sort of base approach. So you know, for example, like you have one JavaScript library and you know like which ones are using it and like on GitHub, for example. So if you run like the test suite of those projects and you can see like what kind of things it's calling on, let's say like in your library, you can sort of like build like a core graph on that. Yeah, I mean, I don't know so much about like a goal specifically, but yeah. So I think there was someone on the back there or, yeah, sorry? Okay, oh yeah, hold on, yeah. So how do you mean nested dependencies? Yeah, okay. So if I understood the question correct, so like the question was like, what is like a nested dependency and like how does it like, yeah. So the question is about like in relation to like nested dependencies and trusted dependencies. Yeah, not sure exactly like the difference by it, but from let's say like a program analysis point, so like when you have like a let's say like dependency to something like that exists in a package repository. So the way I would do it is like, I would usually say like have to download the code of that and then start let's say like building the core graph of it or sometimes you can adopt a tooling to like expand the scope. So for example, you can do this in Java. So you have to basically provide, let's say like, yeah, like the source of bytecode from your project and those dependencies together, and those dependencies together and then start doing the analysis, which, yes. Yeah, so yeah, okay, yeah. So the question is like how do we do this type of analysis? Like how do we actually get like the source code, et cetera? So like in the case of like Rust, even Maven and some other package repositories, you can easily use some API that allows you to download this code and you can start doing analysis directly on it. Yeah, any more questions? Yes, I could repeat again. So the question is more like with the static analysis tools, like are they only looking at like one particular point of time, let's say like of your source code or dependencies or like do they also look at changes over time? Yeah, actually that's a good point. I think the majority of the tooling, it is what I've seen, they don't actually look at that. I think definitely that's like would be very interesting to have this kind of continuous aspect of like how things are changing and basically like build some knowledge around it or like decision making around it. Yeah, that's a good aspect, yeah. Any more questions? Yes, yeah, so the question is like what are like the source of false negatives? So this would be the last question here. So in the case of like, let's say like in Java, for example, you have like reflection, for example. Or like, for example, also when you need to do like dynamic code generation. So when you analyze the source code, you will see that is for example, like maybe loading some JSON file where it has to create, you know, like certain functions, et cetera. But like the program analyzer cannot like detect that. He only knows like you're reading a JSON file. That's it, but it doesn't know the code that's being generated. So like that is less like a main source of like false negatives. I mean, there is less like active research that is like working on trying to improve on that. Yeah, so yeah, I think this concludes my session. So if you have any more questions, feel free to like come and ask me or reach out to me in general. Yeah, thanks a lot.