 Welcome to this call conversation about installation profiles slash distributions. Here are a couple of people which are involved in that. And we will just try to sum up the current state, give some people, like show the different users and use cases and pain points and how to solve that. And we hope you all have your own opinion and we should discuss stuff and help each other to improve the situation. So, hi, I'm Adam, Phenoproxima on D.do. I am currently maintaining the lightning distribution for Acquia as well as the migrate and media subsystems in core. Me next. Hi, my name is Christian. I'm one of the Thunder distribution maintainers. And I'm already involved in a lot of media stuff. Hi, I'm Ryan Aslet. I am on the Drupal Association, Drupal.org engineering team. I manage packages.drupal.org and the test bots and work with Neil Drum over there on distribution packaging and things like that. Yeah, I'm Daniel Wiener. I'm part of the contenter install profile and I'm involved in all kind of things. I have a question for people. I just want to ask a quick show of hands. How many people have used the distribution before to build a website completely? Oh, wow. This is going to be a tough crowd, man. OK, how many people out of curiosity know the difference between an install profile and a distribution? Less people. I guess we'll talk about that. Yeah. OK, well, yeah, do you want to cover this one? Yeah. OK, what an install profile is. It's basically a configuration with modules. So it's just like that. And everyone knows the standard profile. So every Drupal installation has an install profile. So there's just a small difference between installation profiles and distribution. But installation profile is just a list of modules and configuration to come. And the distribution is an installation profile plus the packaging aspect of so how do you fetch the modules you have in your distribution? Most of them are in 2-bit.org. Yeah, distribution is like I think of it as the install profile is like a little module that says, oh, here's my laundry list of modules that I want to install. And here are some changes I want to make to the installer if you're into that. The distribution is that plus all of the actual modules that it wants, and also all the JavaScript libraries that it wants, bundled into a single tar ball and distributed on Drupal.org. So that's the difference. I hope that helps. Yeah, here's just a list of distributions. That's the top seven distributions on 2-bit.org. I might ask a couple of questions because I know very little about distributions other than that we serve them. Would you say that a distribution install profile that a composer JSON file or a Drushmake file that has a manifest of what you need downloaded is an install profile? What you need installed in Drupal? Is that the difference? The install profile doesn't include a Drushmake file or a composer JSON necessarily? I mean, they're both manifests, but one of them is in Drupal. One of them is just on your file system. Yeah, yeah. So the install profile includes a list of modules to install. It's exactly right. Yeah. OK, cool. Yeah, that's what we also do. At our Thunder distribution, we are shipping a lot of modules and we're having a configuration page after the installation. OK, we installed a lot of modules for you, but here are some others you might like or you might need. So yeah. What Thunder is? I don't know if everybody have heard of it. It's a distribution from Hubbard Boda Media, and it's from a publisher of German for publishers. And it's focused on easy article creation. That's what we mainly did in the last two years. Now we are going a bit wider and want to include landing pages or creation of landing pages and stuff like that. But yeah, as I said, distribution could also chip a theme. So we introduced our own admin theme, but we don't have any front-end stuff. So just what is it? Bartik theme. Yeah, that's some introduction to some of those. Next slide. Yeah. So lightning is another example of a distribution. Its intention was it was built to improve the back-end authoring experience of Drupal. And that was done by improving media functionality, improving layout, and improving workflow. And we did this in various ways with certain modules, panels, IPE, workbench moderation, and media entity, which are all now in varying degrees going into core. So I kind of look at lightning as core plus plus. And it also, as with Thunder, is not opinionated about styling. It also, it's Bartik out of the box, seven in the back end. And that was its purpose, and that continues to be its purpose. It's just improved life for authors. And the final distribution we want to show is contentile, which is API-first distribution, which means that instead of using trick-and-plays and panels and all these kind of things to build your site, you have an API, and then you have some kind of consumer which pulls the data in and then renders stuff. And we are leveraging the out-of-the-box initiative idea of a recipe magazine to provide an example content. But it's a full distribution in order to get you started. So that's the main difference between lightning and thunder and contentile. Is that contentile is really just a starter kit? And in thunder and lightning, you get a continuous update path. So this brings us to the less organized part of the slides. From here, it gets a little bumpier, and we definitely want to open it up for a conversation. If anybody wants to ask a question or whatever, engage us in fearless debate. Just head on up to the mic over there. This is just a way of saying that all of us, we as distribution builders have encountered a bunch of pain points that are pretty consistent when building distributions. And so one of the things we want to do is just tell you what those pain points are as distro builders, which can also affect you as distro users, and hopefully come up with ideas about how to address these things, share some of the ideas we've had, and what Drupal.org can do to help as well. So this is where we start bitching about core for the rest of the day. So if you came here to do that, I hope you're ready. Let's start with not bitching about core, but rather about Drupal.org. So currently, right now, there was a problem on Drupal.org that composer Jason for installation profiles aren't respected. Instead, you need to use Drushmake, which is a node technology from, yeah, I don't know. And it's gone in Drush9. There is no more Drushmake. So you pretty much have to be like, if you're not using Composer now, you will eventually. And that's just a fact. So. Yeah. It's not just Composer. You need to, like, yeah, I don't know. Should we say anything? One of the problems right now in a distro, if you're packaging modules like address module is a really good example of this. Media entity Twitter also suffers from this problem. Simple OAuth as well jumps to mind. These are all modules that have Composer Jason files because they depend on libraries that you get with Composer. So these are modules with Composer dependencies. And if your distro includes modules like this, the tarball that D.do gives you is not going to have those Composer dependencies, which means that at some point your code's gonna blow up. And that is a pretty big problem for us. So the way we worked around that in Lightning was pretty unfortunate, honestly. We had to change our project page to just be like, please don't use the tarball. It's broken. Here, just run this command for installing the Composer dependencies if you absolutely must, but just use Composer create project to create a Lightning project and then everything will be happy. So this is really more of a Drupal.org problem, not so much of a Lightning problem or a distro problem specifically. So Ryan, what can we do about this? Well, if you switch the slide, sorry. We're rearranging slides as we speak. Did that not work? No, never mind. I'm on slide 19 now. As I said, it gets disorganized from here. So none of the bullet points are really relevant. Like you said, Drushmake is going away. So that means that we need to be able to support building tarballs out of Composer. And once we have that on Drupal.org, then distributions will be able to stop using Drushmake and stop using that as the way to provide a manifest and then it'll make it easier, of course, for the third party libraries to show up. And then another problem that'll help us address is there's always been an issue with licensing on the tarballs that get put out in that we've always had to have a white list of acceptable packages. Composer has a way built into it to allow us to check the licenses of the end result that we build with Composer. So we can check it against a licensed white list as opposed to an individual package white list, which will get us a lot farther down the road of making sure that whatever we wanna distribute is gonna follow the policies that we have on Drupal.org for licensing. We will still need to figure out how to handle JavaScript dependencies and assets. Yeah, that's been a persistent issue for a long time. The way it currently works on Drupal.org, if you're making a distribution, is there is an issue queue somewhere called the Drupal.org library packaging white list or something and there's a bunch of fixed issues in that issue queue which say please allow dropzone.js to be packaged in a distro in a tarball. And what you have to do is you have to submit like a link to their GitHub and a link to their license and then somebody takes a look and if everything checks out, they add it to some white list somewhere and that white list then allows that particular library from particular, cloned from particular URLs to be packaged in with a distribution in that tarball. Which mostly works, it's kind of a pain if your library's not listed because then you gotta file an issue and then you have to be on IRC and like poke people to get it approved. Or what? Or be an admin. If you're an admin, none of this applies to you. So if you build your own distribution, better get involved on Drupal.org. And good enough, all right. And it also be a problem when you don't have the correct license. We had this issue once when we wanted to integrate AMP module, for example, and AMP module needs some library from GitHub and this GitHub library is not compatible with GPL2 version. So, yeah, we got the issue back, so sorry we cannot do it and this broke our build, basically, so. Yeah, so it's like if the library that you want is not gonna be GPL2 compatible and you wanna distribute that in a tarball on d.o., you're screwed, pretty much. Like you're gonna have to basically put instructions on like, oh, by the way, download my library after you install my distro, which I guess you can do it but it sounds kind of pathetic and for non-technical people that can be pretty intimidating. The way I look at all this as a distro maintainer is that all of this goes away. I'm very in the camp of like, can we all just please use Composer now? And I recognize that's not the most realistic camp in the world but the way I look at it is like all of this goes away if Drupal.org stops shipping code but the truth is that to get people to use distros, I think that we have to have a way to get them that's at least as easy as just downloading and extracting a tarball. If you're using Composer, one thing that makes the pain of the JavaScript libraries a little bit easier from a technical standpoint is there's this thing called Asset Packages. I don't know how many people have heard of that but it's a packages repository that pulls in automatically every package on NPM and Bowers repositories and sort of massages them into being Composer packages so that you ultimately can get JavaScript libraries in via Composer, which I think is pretty cool. Lightning uses it. Some people think it's weird. Some people think that yarn, Bower, NPM might be a better idea and this is an open question so this is one thing we'd love to hear from people on if you have any ideas about ways that you would prefer to get JavaScript libraries into a distribution or into your site. How would you rather bring them in? And that's the thing. We can kind of do, as Ryan's told me before, we can kind of do anything on Drupal.org in terms of if people want to use yarn, I mean, Drupal.org could be made to run yarn while building the tar ball but there's sort of a matter of consensus here. Right, and consensus is important because we don't want to have a packaging build process that runs 15 different tools to try and pull in all the different things. I mean, I think Drupal Core is moving towards yarn as far as using that for testing and pulling in all the JavaScript dependencies for that so I mean, that's feeling the direction that things are going but with JavaScript turned, we may be using Anchovi next week so how many are familiar with Anchovi? I actually never heard of it until I, so there isn't. Oh, you really made it. The manager called Anchovi, but. You got me. I'll do that right now. I'm actually curious about this. Oh, I thought it was a domain. Thinking a little bit outside the box but what if you could have an external build pipeline and then you could use yarn and compile SAS and do whatever you want to do and then just upload your own tar ball and then maybe it scans the tar ball for licenses or something like that rather than, because we're kind of heading towards having a whole CI kind of build process on Drupal.org and that's really hard for the AI to maintain and that kind of thing too. I don't know as much of the, that's a really interesting idea that I think is really cool. I can see that potentially, I mean, I don't know how you would look at this. I mean, it sounds like a potential security issue but I don't know. I mean, yeah, we would be distributing code that we potentially hadn't processed or looked at but I think the main thing there is figuring out how we would resolve the licensing question of like, are you giving us stuff that we're legal to distribute? Yeah, he is like a, maybe naive question, could like, if you are able to like upload a tar ball, could you also able to like just point it to another, to like totally different URL or like GitHub or wherever you have your tar ball in case you have a licensing problem then like the licensing problem wouldn't appear but still there's like the central nice to discover place on Drupal.org. Well, I mean, there's two kinds of licensing problems. There's ones where it violates our policy of what we're willing to distribute and there's other ones where it's just not legal to distribute it that way. And so that's, we could run into issues where someone comes to Drupal.org and download something and they've got a proprietary SDK bundled with GPLv2 and they're like, well, what did I get here? Is this legal for me to be distributed to me? And then they come back on us because of like, but you know, that's one of the issues. I can see what that, but. So do we have any distro builders in here? Except us? Oh, cool, all right, a few. So out of curiosity, backtracking a little bit, would you guys prefer to be able to just build distros with Composer or, you know, do you, okay, I'm getting a vigorous nod over there or does Drush make sound good? Looks like kind of consensus is Composer. Okay, so I'm not crazy. Yeah, well, at least for me personally, like just using whatever is the best practice in like the entire ecosystem is the right way to go, especially also for JS dependencies in my point of view, like using Composer to download JS dependencies is one. Yeah, that's a little weird. Yeah, it's a work of, I used the Asset Composer thing and like we had huge performance issues with that in terms of like, it was just super slow to fetch all the dependencies from NPM. Oh, yeah. I don't know why maybe. We're fetching the dependencies from NPM or from Asset Package. Maybe we use a different package, but there was a way to like basically fetch the dependencies from NPM locally and then build Composer fake packages locally, but maybe there's a different way. I mean, speaking only from experience using Asset Packages, it's not taking any more than the usual 25 million minutes when doing a Composer update. So, yeah. All right, yeah, let's talk about the next pain point. So, yeah, do you wanna? Yeah, we're testing. Ah, yeah. Yeah. Yeah, for distributions, there is currently no testing on tuple.org. So what we as distribution maintainers have to do is we have to set up our complete own environment. So we are using GitHub for development and then Travis for testing our distribution. Same. And then there are two approaches how to test the distribution. We are using, for example, the normal Drupal test. So test beta testing the integration of everything together with mostly functional JavaScript tests. But that could be really slow. So because before a JavaScript test run, you have to install the entire distribution which could be take some time. And because of that, we search for our own solution. So before the installation, the testing starts, we will dump the distribution and then reuse these dump for every JavaScript test. And that makes it a little bit faster. The other approaches using Behead, I think Lightning is using Behead. Yeah, I kind of now, like since he told me about the database dumps the other day, I'm kind of like, man, we should do that. Yeah, see, that's smart. I mean, Yeah. For every single, oh. Yeah, I mean, I'm wonder, you know, why float the ideas of Behead as not, you know, the distro testing tool, but a pretty good one in many, for many cases. I think Lightning between the two of us, I think Lightning is the only one that's really testing with Behead. Most of our tests are Behead. Some of a couple of them are PHP unit, but those are like the tests. We don't really write web tests in PHP unit for precisely the reason you said, which is we don't want to wait. I, you know, talking with these guys downstairs before this, you know, what I come to think is like, I think BeHat's a really good tool for testing in a distro because a distro is really intentional. They're usually built for specific use cases and specific kind of problem spaces, industry verticals. And so for less technical people, the fact, just the fact that BeHat tests are written in, you know, whatever your native language is, is really, really fantastic because it's just native. It's a story, it makes sense. You can read it's like, and if this checks out, it's like this scenario that I as a non-technical person can understand, I know this works. I know my website will do that. So for distros, which are probably more, which may be more geared to people with more specialized needs that may not be, and they may not be quite as technical. It can be really, really helpful for that. The only problem is, of course, BeHat testing is really tricky. It's not contextual. You know, if I say, I click the save button, well, which frickin save button? If there's more than one on the page, I could be clicking the wrong button and in a headless browser environment, I am absolutely gonna have no idea and I'm gonna wonder why my tests are broken and my environment is polluted now, which is another problem because BeHat tests will are persistent. You know, normally, I mean, you're dumping and you're wiping and like, you know, restoring with every scenario. We don't do that. So it means if something fails and like screws the environment up. You cannot do that. So you have to, in the context, you have to clean up your stuff. Yeah, and I have done so much terrible hacking in order to do exactly that. So yeah, BeHat's like, so it's useful for testing a lot of things. It's useful for proving to non-technical people that your distribution works, that it fulfills their use case, can be trickier from a technical perspective. For me, BeHat is like this communication tool with your users or client or whatever. But I think like BeHat is really not well suited for like testing edge cases and like testing all the needy little bits of your distribution. Maybe also like testing the update path, which is like a huge pain point. Like how do you like have a, how do you even know which possible state could be the user's site in you want to update in to your next version of the distribution? I don't know. It's really great to hear experience from other people. Do you do testing for your update paths? We use BeHat a lot and we end up writing our own custom feature context. Test anything really. We don't like really crazy things which wouldn't be a hack if it didn't otherwise. But we can do this with custom feature context and we can write custom BeHat extension as well to do that. And I think it's a good idea to use BeHat and we're talking about using BeHat on Drupal.org and CI. Yes, that's definitely something that we're looking at offering. It was sort of a, it started with my first question is like, what do you need tested in a distribution? What kinds of tests you're writing in distribution? Because you don't want to replicate what cores are you testing or the module? Not really, because I think BeHat is a different type of test and it's not like, because there's the web test. There's HP unit tests like functional tests and BeHat's a little bit different but I think could try to use BeHat to do things that you cannot do with a simple test or unit test. I have a question. I think you already said earlier, but I just on double check because we have a situation when we running simple tests on Drupal CI which is the dependencies as we just spoke. Decomposer.json is not picking up dependencies. For example, I'm a maintainer of AMP and we are trying to test the AMP theme which is also added via Composer.json but that doesn't get picked up on Drupal CI. Are we trying to look into some solution for that? That should work. I mean, that's probably an open an issue in the Composer queue and we'll figure out, if there is one then I, sorry if I missed it. I haven't been looking for a while. We have this problem like three months ago. I'll revisit that and then I'll ping you. Okay, yes. It sounds like an issue with like including themes. Yeah, it could be because we're not testing themes currently. So it might be that. Just in general, could you like say which distribution you represent? I think that would be interesting for all of us. No, this is not a distribution. This is just an AMP module for Google AMP and it has an AMP theme. So we are trying to enable the theme but we couldn't, I don't know if it's possible now. We couldn't because it doesn't check out the theme and put in the themes for when it builds it. The dependency is there, Composer JSON. Right, I don't think, I don't think DrupalCI is actually set up to understand that it is a theme. Yeah, yeah, that's, it's not the actual theme problem site. Well, it could be, that doesn't contain the theme but it could be when it looks, I think it was like couldn't check out the dependencies at all. That was the thing. Yeah, I'm gonna revisit and I'll ping you directly. Since I'm here, I wanna ask you something which is being like always a problem on DrupalCI. And many people got tricked into this problem which is the clean URLs, they are disabled by default. And sometimes we write tests and they go like, oh, I'm going to this URL and I'm testing this and that and all works fine on my local machine and then they send the patch and it fails and it's really hard to debug fine. And I ended up doing like some crazy things to get the output on the screen and then I discovered that the clean URLs are not enabled by default. And I was just wondering if you could do that. I think if we enabled clean URLs by default then people might write code that expects them to be and that it's supposed to work with and without clean URL. But when you do install this standard profile on your local machine is enabled by default. Okay. So that would be like same thing like. Yeah, I mean DrupalCI just does what Runtest does. So that's actually probably something in Drupalcore that needs to be enabled by default. It's not a configuration on the testing. And no, probably there is something somewhere that is not configured. We can talk about this as well another time. Yeah, cool. For every issue here. So yeah, I maintain the egg of distribution and the version seven, like we tried to maintain the upgrade path and we just got so many, so many issues with people who had changed, you know, like content types or whatever they changed. And so like that became impossible to maintain. So we did it in Drupal8 and basically said, it's going to be a starter kit because we're not going to maintain an upgrade path for it. So let's just answer to that. And the other thing with BeHat, I mean, you know, I think you can, BeHat I think is probably not read by end users as much as you think it is. Like you kind of think that you're writing tests for people that, you know, the end user is going to read. I think you should, my view, I think you should be focusing on tests that developers can read and, you know, the issues of like, I think you can, I think Sam might know, but you can run like a browser test base on like an installed site. Yeah, there is a patch in the queue and which is 279-3445 and there's a thunder, thunder, yeah, you can talk about the thunder solution here. Yeah, I already mentioned. Yeah, before we are testing, we are dumping, so we do a clean install of the distribution and then we are dumping this installation and pulling this into the test class. So we extend the JavaScript test class, the functional JavaScript test class with the thunder-based class and then in that we are pulling these overriding the do install method or something like that and pulling the, yeah, dump into the test. I'm actually really happy you mentioned that thing about updating configuration because that's the next thing we wanted to go into and that's a big deal. Yeah, see, next slide, beautiful. So yeah, the issues with configuration are exactly what you said. You install the config, you install it once or from default config, you install it and then the site owns it. You can now no longer rely on the state of config. So what do you do with updating a distro which is, or keeping it up to date because they're heavily based on config. There's different solutions to this. I don't know if any one solution is right. What we just came up with in Lightning, normally we've just been documenting it. We've been like things that we can't, we try to update everything we can automatically but stuff that we just can't be sure that the user doesn't want to change it and even if it's like the smallest little Boolean thing, like in some, nestled in some view somewhere, we just, we document it. We're like, well, if you wanna do this thing, go to this admin screen, you know, this page, click on this and set it to that and then hit save. We introduced recently like just a command line thing to run those particular types of things automatically prompting the user for each one and then I think you have a different way in thunder. Yeah, a little bit. What we are doing, we are documenting everything as well but we are documenting it in Drupal. So in the UI, we are using the checklist API module to create a bullet point for every update we do and then we are creating some Jammel files and the Jammel files contains expected configuration so this is what we expect, what the configuration should be on the user side. So the state that we shipped more or less and another point for new configuration. So before updates will happen, we check, okay, is this expected configuration? If yes, then we apply our new configuration and if not, then we will not mark the bullet point in the UI so the user gets a notification, okay, there were some updates we couldn't apply, then follow these steps to apply it by yourself or just mark it and then you can skip the update. So that's the solution we had. Something people have discussed for a while now is to flip the thing around and like don't allow to edit anything. Unless you opt in into special things so like the distribution could explicitly say these are the things you can change like the side name and that's it and the logo is not allowed to be changed or something like that. It's like a really radical approach and would require a lot of communication because there are probably use cases to change much more than you actually expect but at least that way you could like probably provide an update path in the most safe way you can think of. It's like this, sorry, go ahead. So we are planning to be like distribution in Drupal 8 and we have the same problem and what you're thinking also is to use config filter plug-ins that would be like filter the configuration when it's exported and would lock only some part of configuration and this, what is locked is not configuration. It's an actual code so it's in the module so it's like a class, it's a config filter class that would just handle these kinds of things and this paired with some UI tricks would at least secure a part of the site, right? And then I like really your approach which is that you compare what the configuration should be for the update to work and then if you cannot apply the update then you mark it. That's nice. I think this two approach would kind of cover like 80% of the mess that we have to deal with probably. Sorry, go ahead. I'm just curious, like if you were gonna manipulate a view or add a field to something and from one version to the next, could you use the API to do that and have something that's using Drupal's REST API to do that as part of the upgrade path so that instead of it being like you ship with config that has changed, you ship with a script that manipulates the API to make the changes. Is that something that could be possible? Wait, how does the API come into this? Can you use Drupal's API to manipulate a view? That's how I usually do it. Like in the sense of like... Like I mean you're using the UI to manipulate a view or... Or like I load the view and I figure out where things are like and then I decode the gigantic fricking shoot me in the face array. Right, but I mean is there a REST API to modifying a view using the REST API to... Not that I know of. Kind of make a replay script to modify the config. I mean that's how I did that but without using REST. Okay. Yeah, just ran just normal PHP, so yeah. Right, okay so you're not shipping config with just a new, you know, you modify a view and you've added new fields, you're not just exporting that new... No, no, we're like, we're actually a writing code that just like goes and... Updates. Just low level changes, yeah. Yeah, but you also export the configuration again, right? For like new installations of the... Yes, yes, of course. So you need both. Yeah, this is something I'm very curious about. So the next problem is like semantic versioning. So if you have any kind of contract module dependencies, like it's really hard to like follow their updates because they might not follow semantic versioning which means that they might break something in a patch release and then like in another patch release they have a security update and then you need to follow this. So yeah, that's a huge problem. I think it's mostly a social issue that like module authors are not like, don't want to follow semantic versioning even if it's actually like the way it's great to do it. I think technical solutions don't really help here. No, modules don't have patch versions yet but in the way that we treat the module versions in Composer they end up being major, minor. So, but there isn't yet been a cultural shift to tell people that like in the major version this is where you do backwards compatible breaking changes and in the minor version this is where you do new features. And so by, I think if we add the patch version onto jubil.org, that's the time when we can also like communicate the cultural shift of like, here's how you should use it. There is a plan. I think people would be still scared to put out the new major release for breaking changes because they should. But I mean, but like people then still like try to sneak in like breaking changes in minor version. So it's, for me it's a cultural problem. And the introducing semantic versioning into contrib in on jubil.org, that was the issue that started the genesis of we need a Composer facade. So then, you know, we got sidetracked and built that. So now that we have that, now it's kind of a plausible thing to get accomplished. It's just there's a lot of code that touches versions and there's an issue in jubil.org that it doesn't handle semantic versioning properly. So we should fix that, too, before we can do it on jubil.org. Yeah, I tend to agree with Daniel. It's a cultural issue. I mean, like right now it's the jungle, you know, like will version 1.1 be the same module from 1.2? I mean, most likely, yes, but you know, I have made that mistake before. We're like this version and then the next minor version. We're pretty much completely different. And guess what, I got a lot of complaints about that. So yeah, I think like at the very, like Semver doesn't prevent people from doing that kind of thing, but it at least puts in guidelines. So it makes you look like a jerk if you break people's stuff, which kind of is a jerk thing to do anyway when you have rules against that, so. Yeah, don't break the rules. All right, another point. I'm not back to install for a while. Yeah, it's that like that's more low level on the Drupal core side of things is that installation profiles are both modules and they aren't modules at the same time, which is really weird, which causes all kind of edge cases you can't even imagine. Especially like if you like install a module and then like in your install hook, do you have the modules available or the install program? It's all kind of crap. Yeah, I don't know a solution for that. Personally, I think like an installation profile should not change anything on one time. And if you need to change anything on one time, you should put it into a module. But yeah, that's my opinion. My feeling is even more extremist in the sense. I'm wondering like install profiles, as we mentioned, I think in some slide before, are loaded, they persist after you install. They're loaded like regular modules with all regular modules by the module handler. They're treated as modules. You can say get module standard, and it'll work, which to me seems kind of crazy. I'm like, why does my Drupal site care which install profile I use? Because install profiles, I mean, to me, their purpose is mostly to deliver default config and possibly change the installer in some ways, add steps to it, and maybe modify certain steps. And that's it. As Daniel said, you're not supposed to put hooks into them. You're not supposed to put services into them. Lightning breaks this rule massively, but you're not supposed to put update hooks into them and stuff like that. But I'm just wondering why. I don't even know why they're persistent. I don't understand why the Drupal site cares what the install profile was. I got my default config installed, so what? Yeah, but then once it's done, and then the modules that are installed are stored in core.extension, I think. There are modules in the profiles folder. Oh, the modules in the profiles folder. Drupal needs to know where to look to find modules, so. Yeah, but I never understand why, because why can't composers solve the same problem? Yeah. Can you please go to the mic? Thank you. So now we find ourselves at the intersection of many problems. But that's just a packaging issue. I mean, once you have a composer, you don't need to put the modules inside the profile anymore. I mean, just the profile really becomes just something that you really need to adjust the installation, then it gets out of the way. But for our stuff, we are actually thinking we are doing a known profile. Like there is, because Drupal requires it, but it's an empty profile. And just proxies everything to a core module. And that one will do everything, because we will install it anyway by command line. So we don't need screens or UI things. And we find it very convenient, because this removes one layer of dependencies, right? So because the semantic versions can be only on the modules and not on the profile. So we get the profile really out of our way, which is very beneficial for dependencies management. Because the semantic version, we just care about the semantic versioning of the modules of your distribution, and not more than one of the profile. Yeah. Klausi, you mentioned that. Is there a use case where a composer wouldn't solve the problem? What do you mean? But I guess it's fine. OK, OK, cool. OK, great feedback. I know that the issue is it really install profile or profile, where by the use case, and I know others have the same use case, is not about installing a site and then going off and configuring it. Profiles are also a way of maintaining hundreds of sites that you want to have a standard install across all of those sites and keep them maintained together and have the updates. So we can have our install profile or distributionist and profile that says, these are the configurations and these are the standard configurations. And when we roll out version two, it's going to update all of these things across the profile, across 500 sites that are all running the same profile. So it's not just, and they won't have done any configuration. In fact, it might be exactly as described, all they've done is changed their title, their logo, and everything else will stay the same across all the sites. There is a use case for profiles to have updates and all of that part as well and treated like a module. Otherwise, we just end up with a module that replicates everything that would have otherwise just been in the profile. Yeah, that's kind of how I looked at it initially, too. And that's why Lightning's install profile is loaded with update hooks and all kinds of stuff. And not just update hooks. I think it, even as a service, is YAML. It has very small APIs in it. So it's everything I said not that I don't think is a good idea. I did. But yeah, it's just like, I guess for me, it's just like it's, I feel like there's kind of a scoping issue with profiles, with the install profiles in a lot of ways. Like, I'm not really sure what their, I'm not 100% sure what their job is, you know? Like, so if it's going to be that kind of thing that you just said, then like, OK. But if it's going to be like a smaller scope thing where it's just like, well, this is just my default config and the way I want to set it up at install time, then OK. But I think we have to choose, you know? There is a difference there. Yeah, I can totally see that. So one of the problems with this is that you can uninstall modules that the profile depends on. So if you have services that depend on the modules that it ships and expect them to be there, you can't. So, but if you build your distribution or your install profile, that it's just going to be installed everywhere the same way and you can have services and all sorts of stuff and you know that these are like always installed and the modules are there, then there's really not much of a big difference. And then they're the same, except they're not. But you treat them the same way, essentially. Like you have them as update books, but you could also just have no profile and have a profile module that does the same. Yeah. So it's, yeah. Which sounds like a scoping issue to me, so, yeah. Yeah, talking about the next. It's really related to the discussion about profile, insert profile inheritance. So, there's this patch. There are a lot of people building stuff upon like for example, lightning. So they build a distribution based upon lightning. And then the question is how do you like, we use everything which is coming from lightning. Maybe your agency like wants to have a starting kit based upon lightning or something like that. And yeah, that's a tough problem. There was a core issue for that, which, it's like at 400 comments. And we're at this point, the lightning team, we bring in this patch, but we know it's never going to get committed. And probably it's just as well too, because this is a compositing issue. This is about being able to do that. So what it's doing, it does basically introduce the idea of having multiple active installation profiles which change the behavior of your site. And you can install some of them, uninstall modules. So like it's the same problem what we already have, but just worse. So yeah, I'm not really convinced that this is the right solution. I think a better solution would be to have some kind of way of composing things. So the level of composability should be a module from my point of view. So if you have some kind of media thing, then you have your lightning media module. And then in your install profile, you pick the things you need. Either from the other installation profile or for your current installation profile. Which means that installation profiles like lightning potentially have to put out more modules instead of just the installation profile using a subtree split or something. Yeah, we'd have to split it apart, which I'm personally been in favor of, but. And then it just comes to picking the things you want. Which maybe even makes updates a little bit easier. Because you also have semantic versioning on the individual module level. So you can like not update the media stuff, but update the workflow stuff or something like that. I kind of think composer is the key to this honestly and having everything be composer based because part of the problem right now, I feel with this is consider lightning media, right? It's a part of lightning, it can operate independent of the other components of lightning. But it has like I don't know how many dependencies a bunch and there's no way we could put a module page up that was like so you can download this module, but I also need to download these other 10 modules and any dependencies they have as well. Plus any JavaScript libraries they might want and you would never do it and you would be right not to. So to me, if we had composer up there dealing with all this dependency stuff then that becomes composable. So it's like require lightning media and it's gonna get everything. So to me, I'm not like huge on the sub profile patch for precisely these reasons that it introduces a lot of complexity and that modules should be, I think the smallest unit of composition, but that depends on the resolution. How would you ship with some predefined config for something like that if you wanted to have just the media part? Predefined config and lightning? Just the media. Like say you wanna define a couple of content types that come with like a media bundle. So I have a site and I wanna add media functionality and I wanted to add just, it's almost like a distribution of just a small piece of it. Would that be possible? Like I think so, yeah. I mean, it would just be like a module, right? Right, but where does the config go? The default config, is that? Config install, like that's where I would put it. It's kind of easy to talk about like additional features like or like scope features like media or something, but it becomes hard when we talk about like you want to change something from an existing installation profile. Like the core extension.yaml is the perfect example. It's just one file to define all the modules you want to install, but like if you pick that module that might have other dependencies then you need to like merge them together or if you want to disable a feature that's really hard and I think it's a totally unsolved problem. Yeah, our problem actually, like our use of this thing came into existence because people were like, well how do I turn off parts of lightning and we're like at install time, we mean. And we were just like, we don't know. And so we ended up with this sub profile patch because the reason, the thing about it is that it lets you exclude dependencies of the parent profiles. So it's a way of blacklisting it. And then you know, an infosec guy I know told me that it's always better to whitelist than to blacklist. So it's better to composite things together and say what you do want rather than be like, well here comes the tidal wave of modules. Let me just black out the ones I don't want. So yeah, basically I guess I'm saying that's the longest thing I agree. Yeah. I just want to talk about the situation we had which is related to all this. We have a distribution and we use that for certain types of websites. And then we start to build some car websites series in that same distribution. So we had situation when the distribution enables loads of things by default and then we had to disable many of these things that came with it. So we're like, oh this is not all right, need to split this thing. So what we came up with is making stuff instead of like one distribution may like mini suites. So we have like SEO suites, we have advertising suites, car suites. So when you enable that, it comes with a bunch of modules just related to each other. Yeah, one solution may be to this update path. There's a module called config actions. Which allows you to define in YAML a way to update existing YAML files. So like, this is our pits I want to add. I haven't used it, but it's out there from the form of features people. It's not just about YAML, it's about the action. Checking out the module that you don't need. Car suites didn't need advertising. So we had all these, there being more on the site. So I want to agree on the white listing and black listing. But I'm working with this patch on how to use it. So talk about 500 sites, but actually they're split up into five types of sites each with a distribution. And I want to have a way of having those distributions inherit from a global company-wide distribution that sets certain things. Like, we've got our password policy and I need to know that that is on every site and I've got a way of ensuring it's there, installed in that config and I can report that every site has that set on it. But that's where this comes in useful in that they can all be based off that one distribution that's very small and lightweight. But adds a few things and then we can inherit from that to take our brand-based distribution so that it builds everything else around that site, the content types, the theme and the look and feel of it, but just pulls in and inherits a certain set of things. But it's controlled in a way that we can in a profile and make sure it's there. And the whole thing that I don't want them to be able to disable it, install them and blacklist and say, I don't want the password policy. It needs to be there. I want to make sure it's always installed on every site. But that is totally solvable by expressing it inside the module dependency tree. You could have a core module or something like that. Yeah. It is useful that we could, but it's like, yeah, we could have some using, doing module dependencies rather than profile dependencies. Yeah. Yeah, I mean profile dependencies as, and correct me if I'm wrong on this one, maybe I am. But I mean, when you list a dependency in a profile info, is it actually treated as a hard dependency? Because in my experience, you can uninstall them. All right, I don't know. Like opinion seems to vary on this because I've seen them be uninstallable and other people haven't and I don't even know, but. Yeah, it also depends whether it's uninstallable from the UI or the API. Oh, that's true. Like, yeah. Like the entire extension slash module subsystem isn't. Yeah, that's true. So you can use a. State. It's actually a good point. It's inconsistent. Yeah, it is inconsistent. You can do a lot in the API that you can't do in the UI. Yeah, that's why we really try to like, not increase the scope of the like install profile by using this patch even more because it's already a mess and it will just get worse. But you can't. No. Okay. It might make sense to try to logically split up what's currently considered the install profile into a module bit and an actual install profile which does add installation time and does nothing afterwards. Yeah. We ultimately did that in lightning. Like in the beginning, there was so much more code in the lightning profile itself that was doing all kinds of stuff. We had hook form altars in there. Like. And eventually we did exactly that. We split off a lot of that stuff into a module called lightning core and every core, every lightning component depends on that which effectively makes it a required module but exactly for that reason. We're just like, we did exactly the same thing. We're maintaining the open fed distribution which, so what we decided for our Drupal 8 version was that we have basically an upgrade module. Okay. And we handle anything for things like upgrades. We handle in code in that module rather than through the installation profile. Yeah. Just to avoid that, well, schizophrenia almost of having installation profile do stuff on an already installed site which really it shouldn't. Yeah, agree. And to keep things siloed nicely too, Nelson. Because we have this major rule that we don't want any custom code. We try to be very, very strict in that and the only exception that we allow, well, right now because the ecosystem wasn't quite there yet, we have some more exceptions but the only real exception that we allow is, well, the install, the upgrade module, there we can do some custom code but it can only be run at install or update time. We don't want anything running in there during normal operation of a site. No hooks, no alter hooks, only update hooks and things like that. It's not always easy. Nothing about distros when they become big enough is easy. Yeah, we're getting to that. So we had this kind of set up so that I was gonna talk about all the things group.org could do to help with distributions. Of course there's stuff in core but mainly from a high level there's what we've talked about before adding composer build steps to packaging and distribution and in addition to that clarifying and improving our licensing policy. Like there's been a kind of an ongoing conflict with what we have on Drupal Network for licensing and allowing for certain, because once you have a distribution and you have a whole bunch of things packaged together and a lot of third party and extra libraries so that you have a finished product that you want people to be able to use, sometimes that puts you into the realm of it's only GPL v3 and not v2 which means we can't host it according to our current policy and so those sorts of things are, if we can get that clarified and get that relaxed a little bit then we might be able to host more complete finished products that don't require end users to be forced to go download additional things to get a complete distribution. The testing that we talked about earlier, BDD testing, enable testing in general on distributions. Right now we don't have a configuration screen or anything. We would need to do that and figure out if there are BDD tests what does the output look like when you go to Drupal Network to look at your results? Like where do you see that? There's some things we've been doing to promote and enhance the discovery distributions. I'm not sure if you guys have looked at the industry pages that we've put on the front. There's the industry vertical pages. There's healthcare and there's education and government and publishing and on those pages we've been putting feature distributions. So like if there's a distribution that tackles a particular industry which is one of the things that we're hoping that distributions can become are you putting together a sports team? Well here use the sports team distribution or are you building a farm then use the farm distribution and here's a package solution that is more than just a bunch of legal bricks that can build anything and more of like here's some solved problems for you to put it on. So we're featuring those when they get polished enough to be featured then we're putting those on the front page. We could probably do more on just the UI and UX of browsing distributions and figuring out which ones people need. So along the same lines of we're improving our project pages and then of course semantic version for contrib is an issue that we should tackle. So I have a bunch more slides but I'm not gonna talk about it. All right. Does anyone else have some kind of ideas, problems, suggestions? Random questions. Free beers. Six o'clock. I was just thinking the distributions are already on GitHub and using Travis so it feels like we are duplicating stuff when we do the same infrastructure on Drupal.org, right? Also the same applies to assembling distributions. You can also do that in Travis CI where you can say run NPM, run Compose and then publish the table to location XYZ and you're basically done. Yeah. And just replicate and all of that to Drupal.org it feels like just a really, really lot of work and if distributions are elsewhere already then we should really take a hard look at that if you really need to do that and more improving the integration on Drupal.org to linking external tar balls and hiding releases on distribution pages really make that page nice. So more like investing in the color, the paint that we put on Drupal.org instead of doing the hard work in the background. Yeah. Yeah. Thank you Klausi for covering that because what I wanted to say kind of like really builds on top of this because the Composer stuff, I love Composer and everything but it was the first slide that you showed with address modules and so on. So if we have a tar ball for distributions it suggests to users that they can just use that. But if you add address to Lightning, I don't know if Lightning... It doesn't have address but it has two modules that one of... But if also address is not in Lightning so if I download Lightning and it's packaged and it has everything and so on and I add address to it which also as a download thing it will not work and as an end user that's really confusing and there's not really a good way that I know that we can solve this apart from like the Composer IO I think that... There's also some really interesting issue from Amatescu right now which is using a file file. So it's basically like as an end user you could add a new module on your Drupal page which would then like upload the Composer JSON file onto Drupal.org. Drupal.org would like package everything up and you are allowed to download a single file file which doesn't have like all the file system wide issues because you can for example add a signature to a file file. Yeah and the other thing, so I don't know if you want to replace this well sort of just kind of one thing at a time because yeah I had a nice long meeting yesterday with Boyan and Dries and Jess and some other folks about that problem specifically and the consensus was we really need to improve the update manager and actually improve the update manager to be able to allow people to add the address module to their site and not make it so that they just download a tarball from Drupal.org and just add it but that they have a path to be able to do that. Yeah so the thing is that the Drupal 7 way of doing things will not work in Drupal. No it will not. And the other thing I want to mention I was working on making sites reinstall from existing configuration and the profiles are like a huge pain point especially with the dependencies and the fact that they're special paced but not completely and only sometimes and so one of the solutions that exists is most likely to be added of course is what was it called? The install profile generator so you install your installation profile but then you create a new installation profile for the specific site and then you install that. So like your profile is out like the original profile is out of the way but all the modules, the original profile shipped are still there and are composed and everything so if all the update hooks and so on are in a module then the distribution as such is just the initial one and to reinstall and run the site is not being used. So I would be very happy if distributions like the installation profile of distributions would be as small as possible and do as little as possible and delegate everything to modules that behave like modules. Let behave like modules. So Dave's going to. Yeah, just a final slide. There is a code sprint on Friday would be nice if everyone would come. There are probably people working on installation profiles on their own. In case someone is really like motivated they could like update the installation profile inheritance issue and saying this entire room agreed that we don't need that. Yeah, feel free. I've got some really cool drama in the lightening. That actually instantly just comes down on emojis. Yeah, exactly. That's why we have emojis. We have emojis now. Yeah, exactly. I'm going to have a really fun discussion with the other guy on the lightening team actually when I get back to Boston about it. That's going to be an interesting one. Too bad for you. Well, I mean it's going to be quite the argument but we'll leave. Yeah, thank you. Thank you guys. But I really like the name personally. Yeah, I mean I mix a lot. Check this out. Yeah, you know I like that too actually. It's like great, great, great, great, great. I'm very pumped but without asking so we just like keep our things changed fine. Yeah. Right in there like careful, like upgrading all then you just like break something or like uninstall this first and bring it or whatever you know. Yeah. It's not great, yeah. I like that too. Oh my god. Yeah. I'm missing. Or like, you know, install configuration against like update configuration and I see like if there's a, or like not for install configuration it's against like they have this configuration to see if there's a difference and then like lock like that. Yeah. Or make it like. Yeah, I mean like honestly like one of the things. Did you open an issue in any? No, no. Personally, I've done one of the things like the lightning usually like if it's a mandatory, I mean it has happened where like the manual upgrades are going to be mandatory and that's always things like where we're moving.