 Composer can figure out where all the modules are. So if you run, we ran composer-required Drupal slash path auto earlier. The reason it could find Drupal slash path auto was because of this repository. If you don't include that, it won't be able to find it because it's probably not on the main packages repository. It'll just tell you that it can't find a package called Drupal slash path auto. There are a few more packages. They're not relevant right now. This is a required section. And this is actually a requirements. And we are saying there are a bunch of requirements that are necessary to make the composer setup work. But you can look at the last few lines and that's where we actually mention our modules. Like config installer. That's a profile by the way. Migrate plus, migrate tools, migrate upgrade. I took the beginning of the composer file actually. Later on you can see things like bootstrap and rules and panels and so on. But this is how it looks. There is very little apart from this in the composer file. Most of the composer file is just to get the Drupal setup working. And all of this is included in the template so you don't have to worry. Before we move on, any questions so far? I think you should take the mic. I have an existing D8 installation. How do I get this to work on my existing one? I don't have a web directory and some of my directories are being tracked. Like you had the contrib is ignored. Mine is being tracked. How do I set up my installation to this? Okay. Is your existing website already using composer? Okay. Okay. It's not. So there is this little point in this exercise. I think if you are going to use certain modules then you must invest time to just translate it over like this. Most of the things won't change because the code is still the same. It's just the way you're getting your code. That's the only difference over here. So you can transparently start placing your composer JSON files. You probably can't use the shortcuts we discussed earlier like not like composer create project and all that. Or you can do that and then copy just the composer files in your project's directory. And if you have a web directory in your project, you're good to go. Otherwise you need to make further customizations to your project. There is no easy way to transition from a custom setup what you have to this. It is possible. It might need some effort. Any other questions? With this process working with composer is it possible also to do the updates for core and also updates for their models? Actually that's the very next topic I'm going to talk about. When I talk about version constraints we'll see how updates become possible. Okay, I think let's move on. Yeah, okay, please. Also when you have the opportunity because this is kind of a tweaky sort of format linting tools would probably be a great thing to do just to make sure that you're not breaking your composer.json or some of the files like that. Especially since you can't really comment them. Well, there is a handy command called composervalidate. You just run that command and it will lend the file for you. So whenever you make a change you always run it through composervalidate which will check if the format is correct. And anyway, if you're testing a change to composer or json you're probably going to run a composer update or install anyway which will tell you if something is broken. So how do you maintain the custom modules and the dependencies for the custom modules? Yeah, that's an interesting thing. I would suggest in simpler cases you would just put the custom modules in the repository. Now there is another thing about the auto loader over here which is more of a development-related topic. We'll actually see an example later on where how do you include the dependencies of your custom module as well. We'll see an example later on. This is more of a developer topic. But yeah, that's an interesting question. As far as the custom modules themselves are concerned I would say for simpler sites just put them in the same repository. That's still your code, right? That's still the application you're writing. Of course, go and put it in the repository. If you want to make it more reusable across different websites you can actually put it on a private packages. There are a lot of options over there. All those are actually kind of advanced composer things. I'm happy to discuss that but it's not really in the flow of the presentation. Okay, so moving on. Let's talk about semantic versioning and versioning constraints. You brought up the point of how do we take care of updates. Semantic versioning has actually been brought up in almost every keynote since Drupal 8 is being worked upon. It basically defines a convention of how the version numbers are supposed to look like. You might be familiar with Drupal 7's version numbers. We had from 7.1 until 7.54 now. All we are doing in most cases is just adding bug fixes. We're just fixing bugs. We're not adding features or at least not major features. We added some features in 7.50 but that was very small. We wanted to change the whole thing with our Drupal 8 release cycle. You might be aware that with every six months we are releasing a new minor release. We had 8.1, 8.2, we just had 8.3 and in six months we'll have 8.4. We're adding features over there. In each of these cycles we are having various patch releases. We had 8.2.8 recently and 8.3.1. Those are bug fixes. You can't add features in there. When we come to Drupal 9, like we saw today morning, the Drupal 8 modules won't readily work with that. That's a breaking change. A module written for Drupal 8.2 will work with 8.3 but not with 9.0. Not necessarily with 9.0. It might work. Like we saw if it's staying up to date with API changes. And that in a sense is semantic versioning. It just defines in a more technical way that this is how you should name your number, your versions. If you're adding a feature, make it the next minor release, 8.4. If you're breaking something in backward compatibility, make it a new major release, Drupal 9. That's all it says. So how does this help us? When we're talking about updates, you might want to just stay on, let's say, 8.2 line. Of course you want the patch updates. There was a security release couple of weeks back, or just last week. And you want to make sure that you update to that release with minimal effort. Or you might even want that. You're happy with updating to the next minor release. When 8.3 came out, you're happy to go to 8.3. With Drushmake, we specify the exact version we wanted earlier. In Composer, we can use something like this. These kind of version constraints, which make it easier for you to specify that these are the versions to be used. So you see in the first line over here, Drupal slash core 8.3.star. It just says that I'm happy with any 8.3 release, starting from the 8.3.0, 8.3.1, and so on. Or you can use a special syntax like you see in the next one, Drupal slash color box, and that's till day, till day 1.0. That basically says I'm happy with 1.0, 1.1, 1.2, but not 2.0. Until 2.0 comes, I'm happy with all 1.x releases. It's just a fancier way of writing that. And this... Okay, let's just talk about a few more different kind of version numbers. You also can write... Suppose you want a dev release for some reason. So you can use a syntax like that. 1.x-dev. Or, you know, you want to specify like 8.2.8 came out recently, right? You want to say, I want at least 8.2.8, but anything in 8.3 and onwards is also fine. So you would use a syntax like that. These are little exotic, you know, you'd probably never use them. The most common is the one we saw, like the star 1 and the till day 1. That's the one you would see most. But you can actually get even more control. You can exactly specify these are the versions you want. Or in this range, like you can see, greater than equal to 1.0, but less than 1.5. Or something like that. By the way, this... The second example is identical to the previous slide. Now, how do you use this? So when you have this composer.json file, and when you run composer install, it actually matches this version constraints with all the possible versions and installs the latest version it can, depending on the constraints. So in case of... If we're using Drupal slash code with this constraint, it'll install 8.3.1, in this case. If, suppose, for some reason, you want to stay with path auto 1.4, you don't want to go to 1.5, you would use a constraint like that, for whatever reason, you know, until you're ready to move on, you use this constraint. And you run composer install, it always picks up the latest file. There is also something called composer lock, which is a caveat to my previous point, that when you run composer install for the first time, it generates a lock file. And lock file, you'd probably never edit this. That's the reason I don't have a screenshot over here. There's no reason you need to look at it. Composer handles it entirely by itself. This lock file contains the exact version of the dependencies you have installed. So, at the time of development, if you have installed, let's say, 8.2.8, your version constraints allow anything to be installed, even 8.3, 8.4, and so on. And they are released. But if your lock file specifies 8.2.8, it'll always install 8.2.8. And this is because you want your application to... You have tested your application with a particular version. And, of course, everything is supposed to work in an ideal world, right? But we're not living in an ideal world. We are confident with what we have tested, and we want to stick with that, until we are ready to run with updates. And that is why we have a command. When we run composer update, it'll ignore the lock file. It'll look at the version rules all over again, and update whatever versions it can. So, in the lock file, you may be using 8.2.8. But if there is 8.3.0, you'll update to that. Does that answer your question with this? You would run composer update whenever you're ready for updates. A thumb rule over here is that, if you're building an application, as in you're building a website, you always commit the composer.lock file as well. Because the next developer that picks up, the developer would run composer install and get the identical environment running. If you don't commit the lock file, it'll always, it's open-ended. Your version constraints can allow, and should allow, a range of possible versions, as per the rules of semantic versioning and all that. And also, composer install is faster. Composer update, if you've used composer by now, you would have noticed composer, the first composer install or composer update are quite slow. That's because it actually checks the range of versions that are possible and everything. Composer install, it's fast. So this is the simple usage of composer so far. Now, more specific usages in terms of patches. Somebody asked a question related to patches. How do we use, no, sorry, that was different. So how do we patch? In Drushmake, there is where we specify patches for a certain module. We can do that in composer.json as well. And this is roughly how it looks. There is a section in composer.json, and you just sort of follow the format. There are examples for this, and I'll share the slides so you can always take a look at that. The important thing is that you include a package called c, I'm sorry, I'm not sure how to pronounce that, slash composer patches. And that makes it possible for all the patches to work. And you probably don't need to include it because the composer template I mentioned earlier, it has it by default. So it works very similar to how the patches work in make files. Over here in the patches entry, you specify each package, and then you specify the label for the patch and the name of the link to the file. You can't see that over here. It's chopped off in the screenshot. But the link where the patch is stored. And yeah, you can use local patch. And there was a question related to custom composer.json. So this is how you would use... There is a component called merge plugin, which lets you, like it says, it merges a different composer.json into the existing one. I think Drupal's core uses it, but I couldn't find the example. But if you have a custom module which has dependencies, like for example, this module I wrote called contract tracker, it needs a third-party dependency, which is on packages. So I include the composer.json explicitly. It's the whole path to composer.json. And it will resolve all the dependencies in the custom module as well. So this is, again, important because composer is also an autoloader. And that's... I think I'll digress if I go too deep into that. It basically means that the classes, which the classes, the code in the third-party components referred to by this composer or json, it won't be available in the code flow unless you do this. Again, this is something that developers have to worry about. So if you're a developer, you probably know what autoloading means. But I'm not happy to talk about it outside the session. Again, like I said, this is not built-in to composer. It's available through a plugin called wikimedia.com, composer merge plugin. And this is, again, present in the composer template. That's actually a cool thing in the whole package is opening up to the different things. This plugin, we use it for Drupal, but it was actually not written by Drupal community at all. It was written by the wikimedia community. And we are benefiting from it. This is the cool thing. This is the reason we want to get everyone on the whole PHP, you know, like the larger ecosystem bandwagon. Because we are now sharing a lot more code and we are much more standardized across the whole spectrum. This is a UI tool I've come across recently and this is actually the only UI tool I have found that lets you manage composer JSON. And personally, I think it needs a lot of work. But for any reason, if you are not comfortable with terminal, I think you can start by looking at this. That's the link where it's available. It's available as a Windows and OS X application. I think Linux as well. And you get an interface like that. When you open up your project, you get an interface like that. It just lists all the requirements you have and you have buttons to install, update or validate the composer JSON file. There is very little it does which you can't do in terminal and you'll probably be more comfortable in the terminal anyway. Like I said, this UI needs more work. And if you have any suggestions for the UI or for a different UI or for a different way to manage composer.json, I want to hear about it. Okay, so that's actually the end of the bulk of the presentation. Any questions? Is there, for instance, if the database was updated on the model. Well, there's two questions. If the database was updated in the model, do you have to run the update of the database with composer? That's the question number one. Question number two is, for instance, if you grab a model and then put it on rupa.org, how it gets sent to the packages? I'm sorry, what is the second question? The second one is, if you grab a model and then put it on rupa.org, how do you share it with packages? Okay, so I'll answer the second one first, because it's quicker. It doesn't get sent in the main packages.org. There is a service running on rupa.org itself. The link I shared earlier, packages.rupa.org. If you go there, you'll probably not see anything. It's just a JSON output. That's where it gets updated, and that is why you need to put the repository in your composer.json file. So all the Drupal modules are available through that package. It's not the main packages. Does that answer your question? So, for instance, it's an automatic process that syncs? Yes. So if you're a module contributor, and you're contributing a module, and you create a full module, it's immediately reflected into the packages.rupa. And it is intelligent enough to take what kind of module it is, because the modules, the themes, the profiles are handled slightly differently. So it actually takes care. So if you're creating a module, it will go into the modules directory. It does that job. I'm sorry, what is the first question? The first one is, if the model, the components update the database, and you download or update the model with the composer, but you still have to run those updates in the database. Yeah. Do you run that? You mean, brush update DB? Yes. Right now, composer does not do that. There is actually some effort in figuring out how best do we do that. Because composer, like I said earlier, it's more of a PHP thing. And there is no concept of enabled or disabled modules there in composer. Drupal has that concept. So I should have mentioned earlier, when you run composer require, it does not enable the module, at least as of yet. It just downloads the module. You still have to run drush en to enable the module. So we are not entirely replacing drush or Drupal console. We are just replacing the drush DL part of it. And like I said, the reason is that you better handle your dependencies. But we are working for that. This is definitely a concern we have, and we are trying to figure out ways how we can make all of these more seamless. So any suggestions you have, I'm happy to hear. Hi, and thank you. You had said that the composer.json is sort of comparable to drush make, but more flexible. Can you expand on that? Sure. So drush make, it was written for Drupal, and it actually did a lot of things very well, like specifying patches and all that. There was one limitation in, well, I would say two limitations in drush make that was kind of a, not a deal breaker, but a big disadvantage. One of them was that you could not specify version constraints like we saw. You always specify the exact version that you need. So if there are updates down the line, you have to go to the make file, update each and every line with the latest version. So you have to actually figure out what's the latest version, and then run drush make again. That was one. Second thing is more subtle, and it happens if you're going to use dev releases. So let's say you have specified a dev release as version in drush make. Now what would happen is that at the time of development, you would have run drush make, and you would have got the latest dev release. But when the next developer is checking out, there could actually be an updated dev release. And when the next developer runs drush make, he or she gets the updated dev release, and that might not be what is intended, right? Because the code might have broken, the site might have broken with the update. You have no way of knowing that. And Composer solves both those things. It lets you specify a range of versions, or a more flexible version constraint. And with the log file, you can actually lock it to a particular commit. So a dev release in time, a particular commit hash, it locks it to that. So whenever the next developer runs, and there could be 100 dev releases after that, but it will still pick that commit. And also, like I said, drush make is just a way to define your dependencies. Composer is also an autoloader. It's an autoloader. I've been using Composer a little bit with Drupal already. And one issue I've run into is that sometimes when I update Drupal core, it changes the composer.json file, sometimes wiping out things that I had been adding to it for my site. What's the right way of handling it so that that isn't an issue, I guess? So this actually is... It's almost definitely a bug. It's not supposed to change a Composer.json file. If you're using commands like ComposerRequire, that's the only ComposerRequire, ComposerRemove, and so on, those are the only commands that are supposed to test a Composer.json file. Nothing else should. So if something else is not working this way, I think it's most definitely a bug and we should probably take a closer look. But the right way to do this, I would suggest, if you're not comfortable editing the json file yourself, is the ComposerRequire series of commands. So the command we saw earlier, ComposerRequire.drupal-slash-path-auto, that's how you would specify your dependencies. Certainly. Okay. And ComposerRemove. So I didn't demo that command. I thought it was a little out of the way. But if you want to remove a certain, remove a module, you would run ComposerRemove-drupal-slash-path-auto in that. The cool thing about that is that it'll actually track that what is a dependency and what is not. So you know when we ran ComposerRequire-path-auto, it actually installed token and C tools. Now when you run ComposerRemove-remove-path-auto, it'll even remove token and C tools. Unless something else is using it. Thank you. So I'm a long-time user Composer, big fan. Thank you for having this talk. Everyone needs to be using Composer. I agree. So my understanding is that Drupal has its own, you know, package.drupal, because Drupal modules do not follow semantic versioning. So this private package serves as sort of a translation between Drupal module versions and proper semantic versioning. So my question is, if I wanted to set up my own private, for my organization, you know, something, a custom private package just for our own personal Drupal modules, is there anything special I have to do? Well, there are two ways. If it's only once in a while kind of thing, you know, that only once in a while you would use a module this way, push it to a repository. And in your Composer.json, you remember the repository section I mentioned, and there were like few more. There were like four lines which were hidden over there. Actually those were the four lines. I was calling in custom repositories. So you can actually do that. But as far as like setting up that customer repository, I'm wondering if there's anything that has to be done to like translate module, you know, our own private module versions to... You don't have to, you don't have to. There are ways to transfer. So if you're using this method, you specify the version numbers inside the repository's entry itself. So if we follow semantic versioning internally for modules, there shouldn't be a problem? Yeah, it wouldn't be a problem as far as the version constraints go. I believe you would push code to your custom repository following that convention. But your package entry needs to keep updated. So the package you have in Composer.json that needs to be updated. But if you're, it sounds like you need a more elaborate thing, you know, I mean of course this is a lot of work and it is suitable for one-off kind of usage. But if you have, you know, if you need like more flexible usage or, you know, a lot of modules, then it's better to use something like Torrent Proxy or something like that. That's, like you have GitHub and GitHub Enterprise, you know, it's something like that, like you have packages and you have Torrent Proxy. It's built by Jodi again and I think it's not very expensive. It's probably like 10 bucks a month, something like that. It's a good way to support Jodi as well for his work on Composer. Thank you very much. I think we have a multi-site use case being handled because my understanding is that Composer is a PHP dependency manager for, or dependency manager for applications. Multi-site is multiple applications. Sure. Say you have differing module versions and sites all in a specific site. Yeah. So again, personally, I've not tried a lot of it. What is definitely possible is that you can specify each module to go in a different directory. So you might, for example, there's a particular module that you don't want in sites all, but only in a specific site, site slash example.com, let's say. So you can, there was a section we saw a little earlier, over the installer parts. I didn't mention this earlier. But using this section, you can actually mention where exactly you want to place that. All right. So are there any other things that you would like to add? Yeah. Yeah. Yeah. Yeah. Yeah. So that's something I'm not right. I'm afraid I don't have an answer for you right now. Okay. Cool. Thank you. I've got a question about the tracking the contrived modules. You're not, you're not tracking them. You're not tracking them from the branch. They'll get an error because they don't have that module installed. But that's the workflow. So whenever the next developer picks up this, picks up the website, the first thing the developer needs to do is run Composer Install. And since you added that module, you would have added it in the Composer.json. Right? So when the next developer install, runs Composer Install, you would get the module. From the log file? Yeah, from the json file as well, the log file contains the version information. But the json file contains the entries themselves. So both have to be in sync. That's why I said log file you probably won't ever touch it. You know, it's actually for Composer's internal use. So when you commit your json and log file and the next developer runs Composer Install, Composer knows, okay, there is a new module over here. Install it. And all the dependencies as well. I use Drush to install my modules. So do I use Save Dev? So it'll write to it? How do I do that? I'm sorry? Do I use, like, the Save Dev flag? So it'll write to the... Actually, you should not be using Drush to enable... Sorry, to download modules. To enable modules, you will still use Drush. And that doesn't make a difference. Okay. But to download modules, you would now use Composer Require. In the command line? Yeah, in the command line. Okay. All right, thank you. Thank you for the session here. This is Drupalcon. And you mentioned the concept of the update. Composer update, or I still use Drush. And that's the day of the month where... That's why I keep my hair short. Because I could be pulling out my hair on the contribute modules. You mentioned the advantage of Composer is you can express a range on the versioning. For myself, like, wow, I would never... I'm afraid to do that. Because my first step, whenever I want to update modules, I go to Drupal.org. I go to the issue queue for that module. And, you know, okay, there's a new module. I'm just checking to see are there any critical or urgent or major or minor issues on the update. Yeah. And the question probably isn't anything you can do about, but maybe there's someone here. Boy, it would be great if there was a connection between these tools, Drush or Composer, and Drupal.org, where the community itself could respond to a... There's a major regression bug in the modules so that Composer would somehow be able to read this information. Like, hey, we're not going to... That's actually a great idea. There's nothing like that right now. But that's actually a great idea. The closest I can think of, not in Drupal, but in WordPress has a user vote system that this plugin works with the latest WordPress version, something like that. So I think that's a great idea. I'm afraid we don't have anything like that. Yeah, well, I mean, as Drupal gets larger and larger and the community gets larger and larger, it seems something would need to be done because otherwise each individual programmer has to go through their own testing of the module and to learn now there's a problem here. I'm sorry to interrupt you. You may not want the whole community to have the ability to put a showstopper. Maybe there's a showstopper flag on the module and say, hey, there's a new module upgrade here, but this is a real breaker. So just a few thoughts. That's actually a nice idea. There is something similar in the other way. There is a PHP package which lets you define security constraints. I'm sorry. So if your composer rules allow an insecure package to be installed, this stops it. Something like this could be a great addition. I'll definitely look into this. It's a great idea. Hello. Can you talk a little bit about deployments? How this would go to staging? How this would go to production? Are you then running composer in all of your different environments? Yes. Well, the recommended answer is yes, that you would run all of them in different environments. But personally, I see two ways around this. If you don't have a lot of environments, sure, I would do that. I would run composer in the new environment and test it. But if you're using continuous deployment practices and so on, you're probably using something like Docker or probably hosting it using a host that does not let any such commands run on the production server. So in both of these cases, the way I would do this is have a separate repository. So this will be a part of your CI pipeline. When you push to your repository, it would look at the code, run composer install, and with whatever is generated now, the new file system including the web-slash-core and modules and content modules and everything, you could probably create a Docker image out of that. And that's the image you push to production. And if you're not using a Docker, the other approach I would use is have a separate git repository, not a clone, a separate git repository, and you push, including all this vendor and web-slash, this modules and everything, you push it over there. And run your automation tests over there. So these are the two ways I would go about this. Good question. It seems that what should be in .gitignore from the conventional D7 structure is different from what's on D8 with Composer. So when we do Composer install, does it know what should be and should not be in .gitignore? Yes. So the template actually handles this. Again, the screenshot we saw earlier, this defines where the modules are actually placed. So this screenshot is from a Drupal 8 version and we see that for again it's getting chopped off, but that's type Drupal-module. So for Drupal module, it's going to place it in web-modules-contrib $name. That's like a variable over there. But if it's Drupal 7, the entry would probably be web-slash-sites-slash-all-slash-modules-slash-contrib-slash-$name. So there is actually a Drupal 7 template as well, which I didn't mention earlier. It will exclude whatever should be excluded. I'm sorry? It will exclude what should be excluded. That's your .gitignore file. I mean, nothing stops you from committing all these files, actually. It's just a best practice not to commit it and you would add this in your .gitignore again, which is present in the Drupal Composer template. Okay. Hi there. Can you go back to that merge module you're showing? This one? So yeah, that one here. You said you have to do this to make the auto loader work properly, right? Otherwise, you've got multiple auto loaders. That's the problem, right? With the contributed modules. There can't be multiple auto loaders, so... What's the purpose of this merge? So if you're writing it, two things. Let me clarify this further. If this was actually a contrib module and the Composer or JSON was present, this was let's say committed to Drupal.ovg. Yeah. Now in that case, packages.drupal.ovg will take care of this. You don't need to do this in that case. So if you're using any contrib module, you don't have to worry about this. This is only for your custom modules. In this case, this is a project just like short background over here. This is a project I wrote for my organization to track all the contributions we make as a whole team. So this website actually runs every 20 minutes, queries Drupal.ovg and gets all the contributions like all comments, patches and all that we have uploaded. So we are querying Drupal.ovg API and I didn't want to do that in this module. Mainly because I had already written this library for another project earlier. So this library was available to me. I didn't need to write all the logic. My custom module just had a Composer or JSON which used that library. The problem here is when I can put my custom module over there, but of course that library won't get picked up the API client D.o API client won't get picked up because when you run Composer installing the root it does not scan each and every Composer or JSON in the path by design. There's no way it can do that. So this plugin, this definition says that okay you're reading this Composer or JSON but also read this one and get the libraries from there and it will kind of merge it. It will merge it so that it appears it's a part of this Composer or JSON itself. I mean internally and all the libraries are placed appropriately. So like in this case the library would go to vendor directory. That's the answer. It's pretty rare that you would have to do this I guess. I don't know. This is the only custom module in this website and I had to do this. So it's really a new use case. Okay. Okay. I'm sorry, can we take or I think the next do we take one more? All right. I think this has to be the last one. Yeah, sure. So like I have a couple of questions. So like is it installing the module or just downloading the module? It's just downloading the module. Okay. And when it's installing the module does it run the updates on that module also? For example like when I run the Composer again Okay. And it will get the latest module. So will it run the updates? No, it won't. Like I said, I think somebody asked this question. So yeah, you have to run RushDb again and that's something we are working on to simplify. Okay. And the way like in the Drupal we have the update manager module, right? That tells you, okay, there is a new patch is available for you. Do we have something for the Composer like Composer like itself says, okay, there is a new updates for you. You need to download something. So yeah, there is a command on Composer. It's slipping my mind right now which checks for all the updates to all your packages. It just tells them, okay, this is the version you have installed and this is available as per your rules. And I forget the command. I'll post it later. Okay. So the Drupal update manager module going to be like in align with the Composer because like it still is going to be the... Not necessarily because your Composer, Jason could have version constraints. Update manager will always show the latest version. You might have limited that, I don't want anything after 1.5. So it won't show in that case. But yeah, usually it will be aligned. Unless you're using these fancy version constraints, it will be aligned. Thank you. Thank you everyone for attending. Thanks. Hey. What's next to you?