 Okay, dope. Let's get started. It's nine o'clock here. So we'll go we only have half an hour So I'll try to take about 20 minutes for the talk and then we'll leave a little bit for questions So this is generating dynamic Drupal installers. I'm gonna be we'll be using composer and try to make Dynamic installers that are better than sort of standard distributions that work with Drupal These are some of the links for stuff will be going over. They'll also be at the end of this slide And then I'll post the slides of course. You don't need these links to follow along with the talk. So What is the point of all this? Why are we gonna do dynamic installers? So The company I work for Acromedia a typical project for us Doesn't just isn't just a standard Basic composer install out of the box. We usually have somewhere between three and six Repositories that have been added. It could be you know, we're adding npm Dependencies we're adding a couple of you know dependencies direct from version control Maybe, you know custom libraries or you know third-party stuff from a client that isn't on packages You know, there might even be a private repo We probably require about 40 different, you know modules libraries stuff We have dev requirements as well. We might even be adding an extra auto loader There's probably quite a few scripts for install paths and stuff that comes from Drupal And then we might be running quite a few patches I went over a number of our product projects and we might be running 20 up to even like 40 patches if we have a large suite of Modules and stuff hasn't made its way through to stable releases And so all that means that the composer setup is actually pretty complex and there's a lot going on You know more so than just a theme and a couple of prepackage modules and So these come from a number of different places too So if you take a project that we worked on and you look backwards, there's stuff that's specific to the project There's stuff that's specific to our internal development process There's stuff that comes from Drupal commerce because we usually work with Drupal commerce There's stuff that just comes from the composer scaffolding that's been made for Drupal if you're using like a Drupal create project And then there's a stuff that just comes from Drupal core to make Drupal work with composer at all So there's all these pieces that need to go together And if you want to change something in one of those middle layers It's not that easy And so there's a lot of you know you go. Oh well composer. I can just add you know My composer files from my modules or libraries and stuff and they can add lots of stuff. That's true And that's not true There are lots of options that are basically what they call root only options Which are things that can only be controlled by the root Composer file of the project, which is things like adding additional repo file structures Extras patches dev requirements scripts auto loaders, you know all those things that I listed as being part of the project And so you can't do any of those Just with a download you need to make those in your root And so if you want people to add those you need to like put those in your read me and say hey Go in your root composer, you know copy paste this put it in you know run this command again And it's pretty difficult people don't usually need want read the read me They just want it to install and work And so we're going to discuss some ways we can generate these composer files dynamically so that we can get around all these problems And it's like okay, so why can't I just make Distribution that could add some of this stuff I mean you can you can put custom settings in your composer file for your distribution, you know You can have all this stuff, you know, and if you have people download your specific composer file They could have all this it's just that it has to be exactly the way your distribution said it was and if they want to add something else Now we're back to those manual steps that they have to do it again. You've sort of made like a fork of Drupal You know your distribution isn't quite the same now as stock Drupal. It's your specific distribution You know, maybe things don't work quite the same You know if someone wants to merge two things together now They have to manually take all this stuff and they have to add them together. So, you know, that's okay as long as you don't want to You know Deviate from the distribution too much but kind of the whole point of Drupal is it's super modular and you do whatever you feel like and This kind of goes against that so Okay, what should we do instead then that was just listing off a bunch of problems We had this same sort of problem With doing stuff for Drupal commerce there we had commerce kickstart for Drupal 7 But We didn't want to do that for Drupal 8 because we knew all these problems that I just listed So we're like, well, what can we do instead? You know, how can we build this more dynamically and it's like, okay Well, maybe we can build these composer files out, you know, we can dynamically create them. They're pretty simple They're just JSON, you know, we could put one together and then we could provide it to you as a dynamically generated download Okay, well, I'll just get a library that makes composer files turns out no such library exists So I wrote a library to do that. It's not really that complex. It's like I said, just building JSON It sort of structs out everything That makes up a composer file. There's a pretty sound spec for what a composer file should entail and So we can use that and we can just make a library that handles all of that stuff So I'm just gonna go over some basics of how that works and then we'll talk about how to get more in-depth and do more advanced options and stuff So it's pretty simple. It's a composer library. You can just require it like anything else You load it up load its namespace up and then you can create a new We call it a companyist, which I thought was pretty clever And you can make that and that's basically a composer File, but you know, it's it's a dynamic object at this point. We can output it into a file later And so we can just make one from scratch and we can just start adding things programmatically Or we can make one and load up an existing composer file that we want to edit That's probably what you're gonna do most often is you're gonna take something that you already have as a base Use that and then add stuff or move stuff tweak it from there And so we'll get into those details here The basics so we can create like I said a new composer file and we can set any part of it We can set the description. We can set all the support links. We can You know set what our requires are gonna be we can you know set the name versions anything we want sets Setting stuff like this is it's not too bad But it's it's best for stuff that you want to overwrite the whole thing or it's something where only one of it exists like a composer File can only have one description one name, you know one set of support links, you know one support URL That kind of stuff, but it can have lots of modules repositories things like that. So this doesn't always work for That kind of a set So You can you know if you have a whole set of repositories from like a different one that you want to add in that's fine Or you know require modules, but if you want to add some and stuff Then this way doesn't work the best because it's a full override, but it is pretty powerful So what else we want to do farther? Oh, no, that's okay So we can do Merging and we can do adding it should have included an ad on this slide, but Where we can take two composer files We can put them together we can take an Existing composer file and we can add stuff to it. We can add three more requires we can add a repository We can do all that kind of stuff Dynamically so we can move on from what we you know our base that we had and we can add all these extra bits And that's stuff that we couldn't add normally like requires we could always add extra But if we wanted to add, you know a custom repository that things pull from or we want to a really good example Is if you want to pull stuff from MPM, so you want to like pull JavaScript libraries Making those as a require in Drupal is kind of hard You still have to manually add those somehow, but if you do this you can add that as a way You can use those as a composer dependency as well, and you can add them put them in the libraries Section like normal and you can have your module Or your site do that kind of stuff obviously you can just do that to your Drupal.org module Because it's not going to support this But if you're making your own distribution your own in-house setup or something then you can do that Where it becomes really good though is when you can do merging and so that's where you can take Like two different Distributions and you can put them together so where this came into use for us was say like if we have Commerce which lots of people just add to something else They might be building a different site, and then they want commerce function now They want to have a cart and check out that kind of thing so it's like okay What if I build my site with you know lightning, you know, I built it with thunder Built it with you know open social something like that, and I want all this commerce stuff But commerce has kind of some specific dependencies and the easiest way to do it is to create a commerce project You know, but then you don't have all this stuff from lightning So what we can do is we can even load up Both of those composer files and we can merge them together And so this is going to mostly merge stuff pretty well It's you know, you can decide which one you want to have if you want to overwrite or if you just want to be additive You know, you might want to customize a few things. This is a pretty short example If you look at the source for our commerce kickstart distribution, you'll see some more complex stuff if you might want to merge everything together and then You know say your own title your own version your own description because you've made a unique thing now So you maybe generate that automatically And so those things will Be a new thing but you can add, you know, now we've just put all your repositories together So you just whatever we have from one and whatever we have from the other goes together your scripts Everything like that goes together. You there are some caveats. You will have to deal. So what if two libraries Or two distributions require Two different versions of the same require It's not going to automatically Figured that out for you because that's not even something that composer will automatically figure out for you. It'll just say, hey, you have a conflict You have to deal with that. So unfortunately, we haven't fixed that yet. Maybe if someone wants to submit an awesome pr That would be amazing, but That one's pretty difficult. So You will have to deal with some of that stuff manually. You might have to check through if you have Dependencies or if you just know, hey, this is the one I want primarily And then I want to add in the sort of extras from this and work from that You can always once you generate this too, you can try to even automatically composer install Your file that you've generated and See if it worked Um, and then you can sort of do that as a bit of an automated testing way to be like, did we make a valid one? Or does there have to be some manual adjustment Um, it's like, okay So we've just sort of mashed everything together into a Big setup. Why is this good? Or why is that bad? What we had with commerce kickstart, which I touched on before is We didn't like it. It was trying to do sort of multiple things and it had these various problems with it It was also trying to do sort of a few pretty different things at once Um The initial there was actually two versions of commerce kickstart. The first version was Pretty much just a development framework. It was to help you have some standard modules About, you know, that went well with commerce because commerce had lots of add-ons That you were sort of needed, you know, you needed to add coupons and shipping and things like that But everyone wanted it as kind of like a demo and it gave you it didn't give you any content It didn't give you themes It didn't give you stuff like that set up and and so people were kind of lost because they would get Sort of what they saw as nothing when they downloaded it. So the second one Was made which was much more of a demo. It came with Products it came with a bit of a theme You know, it had all this stuff set up. So if you set it up, it looked like a fully working triple commerce site the problem is That made it kind of crappy to develop on because now you had to get rid of all this stuff And people would oftentimes build their sites on top of a bunch of this demo content Which wasn't really a good way to do things You you just added all this extra clutter and people would just kind of extend stuff where they should have sort of started from scratch And it wasn't good at being this Development framework, but it was trying to do that a little bit So it wasn't perfect at being a demo and it was trying to do these Multiple different things and you get it. We'd get a lot of Like support requests and stuff like that Which would say, you know We You're trying to do this on commerce kickstart and then suddenly that was like a whole new version of problems Or they would need hey, this is outdated on commerce kickstart or this doesn't work with commerce kickstart And it was like this whole separate thing You had to maintain drupal commerce and then you had to maintain commerce kickstart as well as with like a separate support channel Which was just seemed unnecessary and was taking us away from drupal adding more overhead And all that stuff the third thing is if you try to Make these any of these distributions and everything really slick and you want to add in, you know custom user interface stuff like that You can go away from it being sort of a standard Drupal install and so now it doesn't necessarily work with everything and You're losing a little bit of what is drupal to make your site specific And so we didn't want to get where it's like. Oh, what am I doing? I'm using lightning Well, no, you should just be using drupal that is also lightning Instead and so we wanted to have All that Flexibility and functionality still there So that kind of means that What you can do is you can have exactly what you want You can include everything that you need and you can get rid of everything that you don't want right at the start You can do this With you know doing it programmatically through the library and stuff that I saw but the better idea is that You can use this if you're building a distribution And then all your clients can just pick options or all your users can pick options when they go through this And build their own custom setup that's specific to their needs and doesn't have anything that they don't need But hopefully has everything that they do need So there's no this cruft that they have to get rid of or that they're dragging along And they're really just back to a drupal site, but it's just a drupal site that does just what they want So an example of that is we made for the commerce kickstart for drupal 8 It's not a distribution. It's just a site And so what it does is it goes through a whole bunch of options That you can pick and then it will build a composer file out accordingly So if you want shipping it will add shipping if you want to be based on lightning you can be based on lightning You know if you don't want certain options, we won't include those And so as we go along we can build exactly that and then we literally generate a composer file for you Um at the end that has exactly what you intended You just all you have to do is you take that download you go into the folder run composer install There you have a whole site. You didn't have to to fiddle around Managing, you know dependencies or looking at what lightning does and looking at what your other thing does and putting it together um, you know Checking out, you know library dependencies and adding script tags all that kind of stuff So it should be really really easy to click through and just get a distribution. This has worked pretty well from us We I think we have about 10,000 downloads from it There are like I said cases where it doesn't work, of course And you know we're working on that. I just released a new beta Yesterday actually of the underlying module For it because there are cases where you know, you'll use a distribution It won't work or something will change in that distribution and since we pull it in dynamically It might not generate one It's still a bit of a proof of concept, but we're trying to go that way where we can have these Dynamic custom ones instead of just here's our distribution. Here's what you get you're on our track now Where are we? Okay, we're doing not too bad I'll just go a little more and I'll just leave this slide up for a bit. Um, so That's kind of the goal we were trying to Achieve here and then I wanted to do a talk so that we could say is this something that anyone else Cares about that would be useful for other people. It's come up a little bit people have sort of tagged and mentioned the library But um, it hasn't got a lot of uptake. I don't you know, it's also is there a completely different way we should do this Um, how should we handle stuff like that with composers is something that composers should support? Natively instead is a stupid idea in general Um That sort of thing. Um, so can open it up to questions. Uh, we're you know, don't have a ton of time here because it's only a 30 minute session, but Where we go So yes, it it does fit some use cases that I see at UC Berkeley and Central IT. We one thing I've been wrestling with is how to Um, have a base distribution that can be extended and indicate and um My first question with this is If I you've got a whole bunch of sites With the base distribution and then some customization around them and you want to upgrade them all at the same time What does that workflow end up looking like? Is it as simple as? composer update on Every site or are there gotchas? It should be as simple as composer update on every site if you are pushing a change that it's like adding A script tag or one of those root only options Then it can be more difficult because there's no real way to Pull those unless you set up like a ci setup that would Pull an updated composer file like dynamically you could do that you could generate them all Again, basically and anytime you change a composer file do it that way bring it in and run a composer update that would work But you would have to have a ci step to push those along because obviously otherwise It's not attached at all. Um, and you could possibly run into Uh, I got you if You know, you had some sort of conflict or they had changed too significantly that you might not notice or whatever So you'd have to probably manage them all centrally that way versus like let people Customize them manually and dynamically at the same time Again, you could look for like you could manually flag those conflicts if you wanted to as well All right Just a quick follow-up on that and this is like probably a newbie question around Drupal 8 and stuff But like when the security release comes out You know for Drupal core, I suppose would you just be merging in the composer json from the right? The right composer json from Drupal core In order to get the updated stuff quickly into your mishmash of Yeah, I mean you're not necessarily going to even have to do that if it's just like a new Drupal core Especially that it won't normally change the composer file because it kind of can't because that's already been customized So just a regular composer update is going to pull all those new versions Yeah, I guess it's more in contrib where like you might have a lot of different things to update You know, yeah, all that should still go fine from a composer update because all that stuff is already meant to work That way without a custom one at all So you're just sort of adding on top of that so you shouldn't get any issues from there Unless they have a special manual step which would just be the same as normal So this is less a question for you and more of an answer for you With managing multiple sites. We manage about 70 sites at PNNL Pacific Northwest National Laboratory and Not all of them are composerized yet, but in the move to do composer I found that some of the root options you can actually do with a composer plug-in So like your script tags even adding repositories You can create a composer plug-in and require that Also, if you've got other packages that you want to use across all of your sites you can create a Package that requires those and just require that across all your sites. So then you can update that package And then whenever you can pose or update all of your sites, it'll push that out everywhere So yeah, that's actually a way that's done as well for like dev dependencies and stuff for Drupal core There are a couple of like Sort of meta libraries basically that handle all of that So they'll match the like the drush dependencies and things like that so that they play nicely with each other So that's the way the Drupal core does the same sort of thing. So you can do that if you have control over your own Distribution you have a bit more flexibility because you can do things like that where you can say hey Require this plug-in whereas like if you're a module or something or even you know a distribution that you people won't be using As a distribution you still have that sort of root problem that you have to provide somehow So there's different use cases there if you're a public one versus a private you have a bit more flexibility And I appreciate you doing this because I can see some use cases on that as well. So awesome. Thank you Any more questions Sorry, can you please speak into the mic? They just want us to do that for recording so So I can see that'd be useful if you're running multiple projects But like that wouldn't really be useful if you're running like a multi site, right? Like this is only one distribution one install Yeah, like if you're only using a single composer file like for a multi site setup It's not really going to provide any use you can just do all your customizations Manually there the only thing that might help you is You know if it's easy to generate it when you first start the project, you know Like if i'm pulling down a Drupal commerce test project to like um, you know just do Specific, you know regression testing or something I might pull from this because it's just a little bit quicker But that's all you're going to get so like if you're building it in-house just for that It's probably not helpful. Okay. Thank you Uh, we do still have a couple more minutes if anyone else wants to go in for a question