 Good morning everybody, thanks for coming to the first session. I hope you've all properly woken up because when I suggested this session, I made a little proposal talking to you about a few composer best practices and kind of talking to you about, you know, how to best use composer and Timothy Lennon who was emailing with kind of said, hey, there's lots of people in the Drupal community, you have plenty of knowledge about composing how that works. If you are there, you should really talk about something a little more advanced. So kind of a little warning, there's going to be some fulene logic, algebra, math today. There's going to be some, you know, this stuff to really wake you up this morning. But I do have a ton of content, so I'm going to try and go through this relatively quickly to make sure that you all get a chance to ask questions in the end. I'm Neils Aderman. I started a composer project in 2011, so it's been a few years now and yeah, I'm sure that you've all worked with it. If not, this is probably not the session for you, based on what I just said. And I really, I hope to speak to as many of you as possible during the rest of the day. I'd really like to hear from each and every one of you individually in the hallway, if you see me somewhere, like how do you work with composer? Just what does your CI pipeline look like? How often do you do updates? How do you do that? I'd really like to find out more about how composers actually use it in the Drupal community. Simply because the way that we usually find out about how composer gets used is by somebody creating a buck ticket when we break their workflow. We don't have any idea how people actually use composer. The only way we find out is if we make some change that happens to really annoy a few people. And it's nice if you actually have an idea and can avoid that before it happens. So yeah, today I want to talk about composer behind the scenes. And I think one thing I'd put in the description is I want to answer the question of it, like why is composer 2 so much faster? Like it's been around a few years now and I hope nobody's using composer 1 anymore. But I think it's an interesting question to answer. And looking at that will give us an opportunity to really look into how some parts of composer work. So there's some benchmarks, put a couple of links there. You can see things like composer install got 30 to 50% faster and composer updates really ranges from 30% to 90% faster. A drop in memory we use from off 70 to 98%. So there are cases where you can run a composer update. It uses 2% of the memory that composer update used in composer 1. So the easy answers to this are yes, composer does parallel downloads now, so that's faster. It's using HP 2 features to reuse connections to do multiple things over one connection. We paralyzed archive extraction, so it extracts a bunch of zip files in parallel, so if you have multiple CPU cores that'll help. We improved the network protocol, the metadata format as more efficient now so we don't have to transfer as much data. Yeah, so all of that stuff makes sense, kind of obvious whether that would speed things up. But that doesn't really explain the improvements for update and particularly the memory usage. So that's what I kind of want to look at now. And to do that, we're going to have to kind of take a step back to just like the commands you typically use. So I'm sure you're all familiar with composer require, composer remove, composer update, and composer install. And I think I use this particular slide in every single talk I give on composer because I think it's so important for people to understand how this is supposed to work and like what the meaning behind these commands is. If you run a composer require command, it's basically the same as editing your composer JSON file to add a line in there and then running a composer update. A composer update is the thing that updates the log file based on the composer JSON file and a composer install only reads the log file and then puts stuff in your vendor directory. A composer update, if you actually run the command, automatically triggers the composer install afterwards for convenience because usually once you're doing an update, you also want to have the files locally. Yeah, but a composer update's purpose is to update the log file based on metadata it gets out of the internet. It doesn't download any files. It only downloads the JSON metadata. And then composer install is not supposed to read the JSON file at all. It's only reading the log file to generate what's in your vendor directory. Yeah, these convenience commands that composer require remove are really just things on top of that. They're not things that do something different. They're just running an update. And I think the part, you know what I put in the headline, that's important here is that we're all about declaring state. It's not a package manager in the traditional sense where you go and you install a thing and then you remove a thing and you kind of have this state that's manipulated over time and you have to kind of inspect like what's currently there to remove something or add something but instead there's always the full definition of everything that's supposed to be present on the system in your composer JSON and the process of the composer update or a composer install is just to match the state that's in the file that they're reading from so the JSON file or the log file with the system that they're writing into, right? So like an update will just look what are the differences between my log file and my JSON file and make sure that those are in sync. And then an install will synchronize the log file with your vendor directory. Yeah, they're not kind of going at like, oh, I want to add a thing here. And oh, maybe that's already there because it's a vendor directory and some other developer system, right? Like you don't want to have those weird state mutation issues. So a big thing that happened in Composer 2 is that we properly separated update and install. So you can actually see this in the output now if you pay attention. So an example is, I'm using like symphony examples here because I'm way more familiar with that than Drupal. Sorry, but I guess you all kind of use symphony too. Let's say we locally tried upgrading to the latest symphony 635, but we ran into some bug, like it's not working for us. So I'm kind of resetting my local system to the log file that I could get reset. And then I'm back at 5.4.28, which is the system, the version that I was trying to upgrade from. So I'm editing the Composer JSON to say, hey, I just want a 6.2 version because the 6.3 thing didn't really work yet. And then I run a Composer Update. And this way you can really see the difference because in our vendor directory, we still have 6.3.5, right? We never run an install. So the log file is in a state where it contains 5.4.28. The vendor directory see 6.3.5 and the JSON file 6.2. And so the update first modifies the log file to synchronize them, which says it's upgrading from 5.4 to 6.2. So the log file now also contains 6.2, like the JSON file does. And then it says installing, right? So Composer Update triggers an install. And that one then does the downgrading from 6.3 to 6.2 in the vendor directory. So the operations on a log file and a vendor directory can be the opposite of each other if you're running a Composer Update. Because the state in the vendor directory must not necessarily match what you have in your log file. This is the case where if you do something like get operations that modify your log file, but it's most commonly the case if you're transferring a log file to a different system, right? So if you're committing a log file, somebody else checks it out. The vendor directory is not in the state that the log file is in. If they then manipulate the JSON file, you get this point where you get three different states that you have to synchronize. Okay, so how does a Composer Update work, right? The idea, pretty basic. We've got our Composer JSON that requires a few things. Then we have packages and packages.org, we're on group.org. Then we've got this magic dependency solver thing. And out comes the Composer log. Then we download and install all of that stuff in the vendor directory. We're done, right? Pretty easy, basic concept. So this is the reality in Composer 1. As you can see, this makes a lot of sense, right? So first of all, there's this issue where the code that's actually doing the installing is kind of the same function as doing the updating. And so even though, like, logically we're kind of thinking about it in this way that I just described, code-wise it's a bit of a mess. And then we... I think I'll explain that in a little slide, but yeah, basically the dependency solver reads from the packages while it's solving dependencies and the vendor directory is used as input to the solving of dependencies. Yeah, it's a huge mess. So there were two ideas that we had in Composer 1 that led to this chaos. The first one was the solver should really only load what it needs to get to the point that it's at, and then it can kind of, like, let's do, like, lazy loading, right? Like, once it figures out, hey, we need, like, another thing here, let's load it then. And then ideally, we don't spend as much time downloading things and we're already working processing on things. So in the end, it turns out as kind of nonsense, because really it just means that the solver has to wait at a later point because it's not like it can keep solving things until, like, it's done down. Like, it's got to wait for the download to happen at that point. So you might as well just download everything up front and then solve things. Like, it doesn't really make any difference if you have to wait for the download to continue thinking. It means that it's impossible to reduce the set of packages we're looking at before passing them into the solver, right? Because we haven't even finished loading everything by the time that we're already starting solving things. So depending on, like, if our idea is like, oh, hey, the solver is probably faster if we kind of remove a few metadata things that we probably are not going to need in the end, we can't really do that up front because first the solver is already running and then we can't really, like, retroactively remove things from it. Doing something like parallelized network access becomes really complicated because this is kind of spread all over, like, some other code. So yeah, terrible idea. Next one was we want to avoid downloading metadata unnecessarily. And we want to make sure that if you, I don't know, some package goes away and you still have a copy of the metadata in your log file and maybe the files in the vendor directory, you can kind of keep it. So our solution was, like, hey, we can just use the data from the vendor and composer directory also as part of the packages available to you when resolving dependencies. And then if, like, those don't exist on the red mode server, you can just use the local copies that you still have. There's at least a lot of problems. You end up with duplicate metadata because you actually have the versions of all of these places, which makes figuring out which one you want to use a lot more difficult. Also, like, if the remote system updated the metadata, like, just changed, like, I don't know, URLs in it, like, it's still the same version, but, like, the project moved to some different location, do you have to, like, use that thing now or the local thing and then you're transferring this to some other person's machine, which has, like, different state in the vendor directory and then their update command has a different effect than the one on your machine because there's other metadata in their vendor directory than in your own. Leads to really bizarre bug reports that took us forever to figure out. Yeah, you just get, like, really inconsistent behavior. So, right, this was the idea. All right, one thing that I added with the reality is this platform packages thing. Like, that's the thing we're going to have to add to the idea. Like, that's the thing we actually need. It's things like your PHP version, just the thing that we had to add to this, which we didn't really think about originally. And the reality in Composer 2 is a much nicer one. Like, it actually looks like this. There's, like, this one little thing in between called the pool builder and in Composer 2.2, there's a pool optimizer. So, there's, like, kind of, like, this is, like, the actual thing that we wanted to build. So, what are those things? So, when we talk about a pool, it's, like, a pool of packages. This is the input to the solver, which is basically just an array of packages coming from different sources. The pool builder is the process that collects this information from all the different sources. So, it looks at the different repositories. It looks at your local system to figure out which PHP version you have and represent this as a Composer package. And it's doing so in a, like, kind of already optimizing way where it looks at constraints that those packages require because it has to kind of recursively go through your requirements to download those packages. But it keeps track of, like, which version ranges you've requested or required and then only tries to keep those version ranges and already minimizes kind of the amount of package versions that come out of this into the actual pool. And then, in the 2.2 version, we added this optimizer, which then even looks at this pool and tries to, again, reduce how many versions need to even be considered. So, it does things like figuring out that some versions really look identical. So, this happens a lot when you do minor bug fix releases, where all the dependencies remain identical. And then, if you're doing dependency resolution, you really only need to look at the latest one of these because as long as the dependencies of that package haven't changed, you can just use the latest one. There's not going to be a conflict by using the latest one if there isn't going to be a conflict by the other version. So, you can kind of throw away a bunch of these minor bug fix releases because they'll never get picked in the end anyway and it just reduces the amount of metadata that you need to keep in memory. Remember, memory was our big problem. And the amount of data that the solver later needs to analyze. There's a few other things that it does now, and I'm sure there's lots more improvements possible in this optimization step. And it's really the key part is that we've moved all of this, generating the metadata that goes into the solver into a separate step that we can now try and minimize this as much as possible before the actual work begins. Yeah, thanks to Jason Woods who's from the Drupal community. It's kind of cool to see Drupal have a good impact on this. Who added two things to this optimizer or provided the ideas for one of them. Yeah, I'm not going to go to too much detail on this, but yeah, basically it's just trying to figure out ways of identifying specific versions that have no chance of getting picked by the solver later on and just throwing them out before we even look at that. So, what's in this magic hat? Hey, what's the dependency solver? And why does reducing these package versions matter so much? So, we're going to start. This is the part I warned you about, Boolean algebra. So, I hope you've all heard of this before, slash use this word, I don't know, went to university at some point. Yeah, there's like things like war and not. You've got a bunch of laws. I mean, this is five of them. Those are probably like 25 of these. But it's basically like things you can logically deduce from one formula that you can, of course, get to the other one. There are basic things you know from other math, like associativity, commutativity, distributivity. There's some special logic ones like absorption, where if you have something a or a and b, then clearly this is true as long as a is true because the other part of the or doesn't matter. There are things like complementation where, yeah, if you have like one thing or the negative of it, then yes, this is always true because it's got to be at least true or false. Based on these types of laws, you can do something called conjunctive normal form, which is basically a way to represent any Boolean expression as a series of clauses. So in the conjunctive case, these are or clauses, right? So you put a bunch of literals. I mean, I've got the names here, right? So let's do this from the bottom up. So we've got our variables called atoms in this context. We've got literals, which are basically the atoms, but also they're negations. And then you can put a few of these together with ors as a clause. And then in the end, you're doing ands between all of these individual clauses. And I'm not going to go through like the actual proof of how you can do this, but any Boolean expression you can think of, you can reformulate into a form that looks just like this. No matter the depth of your nesting, you can always use the laws in the previous page to kind of reformulate your expression to look like this. So yeah, what's in the dependency solver? A SAT solver. SAT stands for Boolean satisfiability. And it's a tool that answers the question whether for a given Boolean formula, like the one you just saw, is there a set of values for the variables so that the entire expression evaluates to true, right? So can I, I don't know, if A ends up being true, B is false, C is true, whatever, like at some point if I put all these values in, the whole thing becomes true. Yeah, so I got two examples. Like A and B is satisfiable if A is true and B is true, or A and B and not A is satisfiable because A cannot be both true and false. It's just automatically true. So that's the kind of thing that a SAT solver tells you. So why are we using a SAT solver to do dependency resolution? So it's a bit like we looked at like other, like how do other people do this and the origin of the solver in Composer is lib zip or the tool zipper in SUSE back in 2011, which internally uses a SAT solver. And the theoretical background for that is a EU project. Unfortunately, its own website doesn't exist anymore, but there's like a follow-up project that has some background information, links to a few papers. They basically prove that package resolution is NP-complete. I'm not gonna, like whoever knows what that is, great to everywhere else. I'm not gonna go into that because that's gonna take us another two hours to explain even on a surface what that means. But yeah, for the ones really interested, you can read about how you can encode any three-set problem as a Debian resolution problem. And kind of really did like the formal proof on how you can convert package management into any random satisfiability problem in vice versa. So yeah, how do you do that? Like how do you do like our requires and JSON files and represent these Boolean expressions? So the basic idea is each version is one of these atoms where the, and being present, it's a positive value or the negation if it should not be present, right? We're looking at state. We're not saying install this, remove this. We're saying in the final state of the system, should it be present or should it be not present? And then you can represent different rules. Like A1 requires B1 is a, either we don't install A1 at all or if A1 is present, so the first part is false, then B1 has to be true, right? So we need B1 if A1 is present but if A1 is absent, B1 doesn't matter. A1 conflicts with B1. At least one of them has got to be negative. C1 and D1 provide a virtual package B1 and A1 requires B1. Can be represented as A1 either not being present at all or one of the two available implementations being present in the system. Or C1 replaces B1 and A1 requires B1, so replaced always means an implicit conflict, so that means C1 and B1 can both be present at the same time. But if A1 is present, then B1 or C1 has to be present. So you can go through all these rules and you just make a gigantic Boolean expression. I really mean gigantic. Because you have to consider this for every version of every package that composer has to look at to think about whether it may possibly be used for this. And this is why having less package versions makes this way more efficient, because you don't have to look at quite that many of these rules. Also just an example of how this actually works in practice. Yeah, so we started with something at the beginning. So we have a project that requires A, so A1 is just a clause by itself. A1 requires B and C. B requires C as well as... Yeah, B requires C and then A requires C is at the end. And so you can go through and you just go like in the first step, clearly A1's got to be true. So let's put true in for A1, which means for not A1 is false. We can put this in and then like, okay, so false or B1 is just B1, or false or C1 is just C1. And then you go, okay, let's do B1 is true. Sure, like fill this in. We get to like, oh, C1 is... Okay, this C1 got to be true. And we've got our resolution. A1, B1, C1 are the three packages that we've got to install. Easy. It can happen the other way around to where, let's say, B conflicts with C instead, right? Same case, just B conflicts with C. You just start doing kind of the same thing. B1 is true. Fill this all in. But then you arrive at a point where you've got not C1 and C1. And so if you assume the first one, like before C1 is false, then the last one, like that does it, like that rebellious to false. So there's no way to complete this Boolean expression in a way that all three variables have a value. And it evaluates to true. So that's a conflict. And the composer will tell you like, hey, this doesn't work. You can't install these two things at the same time. So that was all easy because you feel like always straightforward go like, oh, yeah, sure. Like based on this next thing true, next thing false. In reality, there are free choices. And there's a thing in the composer called a policy which determines the precedence of solution attempts for these three choices. Meaning we have multiple things that we can pick from that could possibly be selected as true. And we don't know which one to try first. And this policy is kind of where on one side we implement the three choices. And this policy is kind of where on one side we implement things like, oh, the latest version is what we want to prefer, like we want to have as updated versions as possible. But it's also where flags like prefer lowest get implemented. Prefer lowest as an option for composers usually useful for CI and libraries where you want to install the lowest compatible versions to kind of check if your library still runs with the lowest versions in your ranges that you claim to be compatible with. And then, yeah, similar example again. You have some package A that's required. A requires B in any version. Now we've got two versions of B, B1 and B2. And only B2 requires C1. So A1, same thing as before. And then we get to a point where you've got either B1 or B2. And this is where the policy comes in. The policy goes, hey, try B2 first. B2 is true. And then it continues from there. It goes, okay, B1 or 2 is true. The part in the back needs C1 has to be true. So C1 is true. And it decides, fine, this works. First try, very nice. Usually it ends up in a conflict and has to backtrack to the point where the policy made a decision, changed the decision, tried a different one. This is where things get a little more complicated. But in this case it's like, yes, let's pick the latest version of B2 even though that means we have to install a separate package C1, which if you pick P1, we probably wouldn't have needed. The difference for composers is we want the latest versions. Yeah, implementation, if you ever actually want to work on a composer, look at how this works. Basically each package object gets an integer ID. Oh, that's in this big pool that we've generated. And then, yeah, the ID is the absolute value of the integer is the ID. And we use the sign as the negative or positive thing for the Boolean value. Yeah, so we've got a pure integer representation of all our packages that we're working on. And then you've got this thing called a rule, which is the representation of what we call the clause in Boolean logic before. It just contains an array of these things, like an array of integers. And then you've got the solver solve class which actually implements the SAT part, which again, I'm not going to get into how that works exactly, but you saw me kind of do it on these formulas that's kind of what it does. And then it generates lock file states from that. So it takes like, oh, these are the integers in the end. Let's look at the current lock file. Let's see, like, which packages that each of these integers just ready to represent. What if we change the lock file for it to look the way that we want it to look? And there's an implementation of the policy called default policy, which is a bit of a joke because it's now the policy for everything because it just handles options, but I guess we're never getting away from that name. But yeah, it didn't implement these free choice decisions. So options like prefer lowest or prefer stable, other things. So this kind of generally gives you an idea what happens in there, but like, again, how do we manage to make Composer 2 so much faster? So if you look at this with more real package names, I mean, I kind of went through this. Like if you have something, Fubar requires fast books, but then you have like a bunch of different versions, 20, 201, 21, these rules get longer and longer, right? Because you've got lots and lots of versions that are compatible with each other. Or same with the conflict. You've actually got to create a class for each individual version that you're conflicting with, each individual version that you're conflicting with. And then PHP is this thing where you can't load the same class name twice, right? There's no dynamic naming. So each version is always conflicting with each other version of the same package. We don't allow you to install two different versions of the package at the same time. And so you've got conflicts for each version with each other version of the package. And you've got packages with hundreds of versions. Yeah. And so that's the point where we get to like extreme growth, like for a N is the number of versions here, you end up with lots of rules in this example. Like if you have something like 500 versions, you end up with 125,000 rules on conflicts, just for this one package internally, right? We're not even talking about dependencies with other things. It's just that package alone. And this is where as composer grew and people started publishing more and more versions of their packages, things got slower and slower. Because this, even though we're just using integers, this starts eating up RAM. You've got to represent all this stuff as like objects in memory. And so one big change in composer two is that we figured out a way to represent neutral exclusion rules. So kind of a set of conflicts that are where all things conflict with each other as a single rule in memory. This was a lot more complicated than it sounds at first because you've actually got to change the entire logic of the SAT solver to kind of understand what these rules are and how they work and how to backtrack inside of them and remember which parts it's tried or not. But we eventually managed to figure this out and this made a gigantic change to the memory use. That's it for the math part. Let's get back to a bit more practical things. So one thing that came up here earlier was that this is the platform packages. And I'm sure you've seen these, right? Like I said, you've got dependencies in PHP versions or extensions. And it's just kind of this implicit additional repository. It's kind of like you have like a Drupal.org repository. There's just one that's the platform repository, which is always there. And these packages cannot be updated by Composer. They can't be installed, removed. They're just there. You can look at these by running Composer Show Platform. You get like an idea of like, ah, this is all the stuff that Composer recognizes on my system that I can do requirements against. And that's the thing where if you, you know, put like a requirement PHP 7.1 and you run it on PHP 5.6, Composer Update will tell you, nope, doesn't work. And that's how it works. So it's the same kind of conflict as you would have with a package. But now you may take multiple projects for different platforms that happen to run different PHP versions. Or you would like to try out if your project would install in the different PHP version. How do you do that? Because you can really only do this with like your local PHP version that's on your system, right? And that's what platform requirements are for. So one old way of doing this is just Ignore platform requirements, which basically removes all of these requirements as well as the platform repository and just does resolution as if there was no platform concept. And then that works, but the problem is you don't really know if, you know, you're trying to upgrade to 8.1 here. Like do your dependencies actually resolve on 8.1? No clue, because we just removed the platform. So instead, you can actually, in your config and Composer.js, define overrides for these values in the platform repository. Say like, hey, actually resolve is if I was on PHP 8.1, and I was running 7.4, and pretend I have the extension installed even though I don't. And then Composer Update works, gives you a log file that works in the system. You try and run the code, it'll probably break because you're on the wrong version. And that's what happens if you deploy this to broad, right? You just run this, you deploy to broad. Broad was still 7.4, you thought you'd already upgrade to 8.1, but, uh, and stuff breaks, and it's kind of, you notice this a bit late. So there's a different command that's still heavily used, I would say, called Composer Check Platform Requirements. Uh, which is something you should run on your protection servers as part of the deployment to verify that the platform that you're actually running it on now matches your configuration. Like, so if you use a platform configuration to say which version you're trying to do dependency resolution for, use this command to check that the deployment platform actually matches that. Um, there's another thing I think I briefly mentioned earlier called Partial Updates. This is when you run Composer Update with arguments. So you say, like, Composer Update, and then Package Names. So you only want to update those packages. That's also what Composer Require does. So if you do Composer Require to add a new requirement, like, when Composer Require foo, it'll run a Composer Update foo. Like, it won't try to update everything, just that new package that it's adding and its dependencies. Uh, so yeah, let's say we have, like, some package called Zebra, package called Giraffe, Zebra depends on a horse, duck, sorry Giraffe depends on duck, horse on Giraffe, and then we've got a duck, and then we've got a project that wants to use Zebra and Giraffe. So this is what this looks like as a graph. Way more understandable than the JSON. And now let's pretend each of these packages releases a version 1.1. So if you run Composer Update, Zebra is Zebra. By the way, Dry Run, a great option. If you want to just try out what an update would do without actually doing it. There's also no, uh, dash dash no install, which makes Composer only update the lock file and not actually run the install afterwards to update the vendor directory. Just random thing I think of. Um, so this will only update Zebra. All the other packages will remain the same. Then you've got an option called with dependencies, which will update horse, but it won't update Giraffe because Giraffe is part of your actual local root Composer.json. So it'll stop at anything that's manually defined by you. And then you've got, if you're sure you could do like Zebra and Giraffe, then you wouldn't get the other two. Or you just do, uh, yeah, Zebra and Giraffe, with dependencies, then you get all of these, but that's kind of inconvenient, because now you have to like list all the root ones. Uh, and if you want to really just get this, you can do with all dependencies, which then includes the ones that are in your root Composer.json as well. And that would get you to update all of these. Um, right. Just comparison with dependencies stops with all dependencies continues even when it reaches a root dependency. Um, so I was asked to talk a bit about upcoming features and future plans. So if I think about future plans, the first thing that comes to my mind is to not change anything. Um, and it's, and I think it's just, I mean, it's a tool we're all kind of sort of happy with it. I'm sure there's like some things we could improve, but overall, the most important thing I think at Composer at this age is just, it should keep working. Like I don't want it to randomly break on me. It's annoying, right? Like I have to keep modifying things. And once in a while, things become necessary. So we added this thing about like a, allow plugins configuration, for example, which was a bit an annoying thing for developers. But there's good costs for it, because it does have security implications that we never really thought about to that degree before. Uh, and I think that was a good move. And so once in a while it is necessary, but in general, we try to do this as little as possible. We do not want to break BC. Maybe someday there'll be a thing where we say, okay, this is worth it, but like we try not to. So one thing we do want to do is like really get people off of Composer One. So we're going to, at some point next year, probably there's going to be an announcement to make Composer One metadata read only entirely. So you will not be able to get any new versions on Composer One anymore. It'll sort of keep working in like a read-only state for a while. We'll see how long. But yeah, if you still anywhere run and then we make like small improvements or we notice like, oh, there's the common workflows that people could like have like a little more convenience to something. I'll give you an example of that. And the other part is just we try and work quite a bit these days in like improving users' projects security and like giving them tools to improve that situation. So example for small improvement is a new flag probably coming in 2.7. There's still an open pull request. It's going to be called either minimal changes or minimal update probably. And the idea is we just looked at this whole thing with like partial updates. But what if I want to, I do want to update just the one package and all I want is for everything that's really necessary to update to be updated. But everything that I can keep in the current version I want to keep that way. So I think that's actually more sane default for Composer Require where like, yes, if necessary I have to update a couple of packages. I kind of want to keep everything as it is because I'm really just trying to add this one new thing. And the solution is basically to do partial updates with dependencies but try and keep them at the lock file version where possible. Back to the example from before, Girav, they all do like a 1.1 release but Girav now requires, still requires duck 1.0 and all the other ones actually have a requirement for their 1.1 version. So that means if I run with all dependencies they would all get upgraded. But duck 1.0 is actually still compatible because that requirement wasn't changed to 1.1. Like the other ones were necessary but that one wasn't. So with minimal changes we want duck 1.0 to stay duck 1.0 because you don't have to update it to keep things compatible. So who could follow the earlier part? Any idea how to implement this? Just a word? So basically we still want to make the update happen as usual as if there wasn't anything in particular. But we just make the policy pick the locked version number first. So any time that we have a choice instead of using the latest version we just go like, hey, this is the version that we have in the lock file. Let's just try that one first. And that way we end up with using the lock file versions wherever possible. Because if that runs into a conflict it will then go to the latest version of the lock file version. Yeah, that's the link to the PR. Then I mentioned, yeah, improving security for users. We've recently added Composer Audit which I hope you've seen or the second default thing that runs as part of Composer Update now you get like a little summary report that says, hey, there's so many vulnerable versions in your project, you should probably look at this. I think in the future in Composer you should have the ability to block updating to vulnerable versions in the first place. We can already do this by requiring one of these two packages. There's like a general PHP one and a Drupal with a specific one which basically define conflicts with all the known vulnerable versions. And so by adding this in there you just make sure you can't install these but that's probably a flag we got to add to Composer at some point where like some configuration we haven't quite figured out how exactly that should look if it's built into Composer directly. We want to add things like public UI access and probably also notifications about changes to the Audit and transparency log that currently kind of exists internally but isn't public. Not because we don't want it to be public just because, hey, somebody want to help? There's not only so much we can do in a day. Yeah, we want to shift built-in S-blum support like software build materials. If anybody went to the supply chain talk yesterday you've kind of heard about this already. Currently there's already a couple plugins it's not that original, like there's ways to get those to work but I think it's something that Composer could probably provide you without of the box. Yeah, the next part. Will, I just do like a quick thing. Random things you should know about packages.org. Packages.org only serves metadata. It does not provide the download files. Like over 99% of packages are hosted on GitHub and that's where you download your packages from. You're not downloading them from packages.org. Yeah, packages.org is JSON only. There are no checksums for GitHub stored packages. The reason for this is that the way that you download packages from GitHub, the zip file you download changes over time. They kind of regenerate it depending on which server you hit and the time of day you hit it. So like those zip files are never the same so providing checksums doesn't really work as long as we're doing downloads for GitHub. There are no signatures. There's no way to upload code to packages.org because I mean we're not hosting any. This is a nice part for security because it removes an entire class of security issues because you can't even put any code on there. But tags can get recreated. That's something to always keep in mind. If people do delete and recreate tags, do not ever do this. Yes, please. Anybody in this room who's heard me say this now, never ever delete a tag and recreate it. You can always make a new version and make people update. This is an absolutely terrible practice but it is a reality that this happens. So if you have dependencies, you've got to be aware that they may decide to do that. And this has implications, right? Because the thing that you're downloading may change even though the version number has not changed. And there's been security issues with packages.org still. There was a case in May where somebody managed to find passwords for a few accounts that hadn't been used in 10 plus years but were still listed as maintainers on a few important packages, just like the Doctrine ORM. And they managed to swap out the URL for a different URL that you would install from. Fortunately, in this case, the person just added a little message and it was more pointing out that there was a problem. And we took measures now to prevent that kind of thing from happening again but nothing's ever perfectly safe. Who knows what the next thing is going to be that we didn't think about. So one big thing that people always ask about is signatures. Why aren't there signatures in Composer? The first problem I ever mentioned those archives that we're currently using to download stuff aren't stable. So what are you going to sign? A changing target doesn't really work. So one solution would be we'd have to actually start hosting the code. We've got to make stored copies of everything that's on GitHub where we want people to download front packages. So we kind of open us up to this entire kind of worms of hosting code but maybe that's a necessity. The other part I'm a little worried about is the amount of manpower required to moderate content because suddenly you're hosting like any random zip file and people always find a way to do this to hosts like, I don't know, movies and this kind of stuff just generates support work. Then another idea was we could sign the contents of the archives only. The problem with this is it's like non-standard like just throw away all the metadata changes like the timestamps and whatever that's in the zip archive and just look at the actual files in the contents on them. It's more difficult to implement because you can't just use a off-the-shelf solution because this doesn't really exist. And then the other problem is the archive metadata itself can actually contain exploit code. There's cases where people use the zip archive metadata to run an exploit in the system extracting it. And so if we were only signing the contents then somebody could replace the zip file with one containing the exploit and our signing check and wouldn't even notice this. So you'd have to be very careful about which parts of the archive you have to sign to prevent something like this from happening which, yeah, good luck. This seems kind of tricky too. Anyway, so these are like problems to solve but yeah, specifically in Drupal there are people working on this. There's a session this afternoon if you're interested in this where they hopefully talk a bit more about this. I'm meeting a few of them today as well. I'm looking forward to hearing more about what the process or state of things is. A few of these things are a bit easier with the Drupal repository because they do host the code. So it's like one big challenge removed. But it's still an awful lot of work. I'm really curious to see where this goes and if this is going to be something that we can broadly deploy across Composer. But at the same time I always like to point out signatures are not the holy grail and suddenly every package you ever download is secure. They don't solve important questions like can you actually trust the maintainer of this package? It's like yes great, the maintainer signed the package but there might still be exploit code in there. That doesn't tell you anything about what's in there. And there are cases where there's one particular thing that happened with a JavaScript library event stream in 2018 in which a developer trusted somebody else who's been making very useful contributions to his project and eventually gave him direct access to his GitHub repository and then that person added a perfectly fine looking other dependency that was doing something very useful in a repository of his own and then a while later made a commit there which was basically trying to steal crypto wall stored in the same system as you were installing this thing. And in this case like I mean yes you trusted the maintainer, the maintainer trusted this other, like trust is the important thing and signing wouldn't have helped you with something like this at all. And we already do like TLS with GitHub, right? Like you're, so you're already getting a lot of like transport layer security. It's not the same thing, they're like different. There are some aspects that signing can still protect you from in addition but it's not that gigantic a change. It's not like suddenly everything will be safe if the composer gets signatures. Oops sorry, this was, I don't know. Yep, I think that slide I meant to delete. Yeah, one other thing I'd like to see kind of running out of time aren't I? Kind of wanted to do some Q&A. I'll like script to these and I'll at least do a couple questions. Something like maintenance support levels defined in packages would be great. Something I'd like to see better support for tooling and composer kind of resolving dependencies for tools like PHP unit that may conflict with your own dependencies and just figuring out a way and how could we do that in the future that would work better. Yeah better support, patches working in a better way or a composer having better support for patches. I don't very much like the way that the patches that are very commonly used in Drupal currently work. They circumvent a few of the composer defaults of using repositories to be able to mirror or proxy packages. So if you do something like private packages, the company that I work for that helps support composer, if you can't put your Drupal patches into that because there's no mechanism by which it would override those URLs. So I think that's some place where a composer could improve a bit and I don't either those plugins or maybe we can move some of that stuff into composer. Yeah that's something that I'm really interested in right now is like with private packages we're building something like automatic updates like what you know from the pen about a renovate bot but we're trying to build a version of that that's like a lot better specific for PHP and that's another thing if you're interested in this like come talk to me I'd really like to hear like if you already use something like a renovate bot depending on what kind of problems you typically run into with that like I'd love to hear your feedback and kind of take this into account of what we're building next. So with that, thanks very much and I'll try to take a couple of questions even though we're kind of at the end of time but yeah, feel free to leave if you want to. Anybody in the room? I think there's some stuff on the laptop I can read out. How does composer audit vulnerable versions? Where does it know that a packet is vulnerable? So packages.org now provides a database of vulnerabilities that is fed from both the GitHub vulnerability database as well as the Friends with PHP repository which is basically a PHP project for tracking vulnerable PHP libraries and it aggregates this information and as part of the audit output you'll actually see the source of this information so basically use this existing databases and tries to merge them into one to expose them through a API on packages.org that anybody can consume to get this kind of metadata. How does composer audit fix conflicts after procure highest or lowest? Let's just try the second highest version, something like a binary switch that was like earlier. Yes, it just tries the next version currently it just goes in order because we do want to have like the most highest version. One thing to keep in mind, we've got this pool optimizer now so that means that if the next highest version is very similar that won't even be in there. The next highest version is the next meaningfully different version. How much does composer use NPM for inspiration? For composer and NPM to collaborate I think we've definitely used it for inspiration like the composer audit command is kind of a direct copy of what NPM has been successfully doing. The JSON file originally comes from NPM there's always some stuff we look at there isn't a lot of direct collaboration it's more like in the entire package management space we all kind of look at each other's solutions what's working well for cargo what's working well for bundler and kind of copy things out of each other that makes sense. Anyway, thanks very much for coming talk to you all.