 All right, thank you all for coming This is the session about advanced configuration management with configuration split and similar modules I'm Fabian Bircher or bircher on Drupal.org I work for Nuvole a hundred percent Drupal company well We also do other things with Sylex and simply since Drupal 8 adopted these kind of things We are a distributed team in Belgium Italy and the Czech Republic our clients are mostly international organizations and institutions and They have a need often for a fast delivery. So we have several developers working on the same projects and We need to frequently update sites and I'm pretty sure this matches almost all of your experience as well So we need a safe way to manage the configuration between different instances of the same site of the project Okay So the first chapter is about okay, wait a minute This Twitter notifications is not gonna happen. Yeah, sorry about that So the first chapter is about the configuration management in Drupal core as you may know Drupal 8 has this amazing new feature of configuration management And it is really great and we love it but the use case Drupal core covers is Kind of limited and so often we have more needs But nevertheless the the core scenario is Answers the questions of how I can develop and test configuration on a development copy and keep this site running at the same time and Then export the configuration changes and import them on the live site in production and that's it So I Will this is not exactly the same presentation as I gave before I will try to Explain the concepts so that then You can adapt them to your own needs and so the configuration management in core works essentially around a configuration storage. I will use the same kind of Pictures for for all the things so this square with the slightly rounded corners represents a configuration storage The blue here the Drupal blue is the active storage. That's the one When your Drupal site is functioning reads the configuration from by default it is in the database so That has a several reasons one of them is that you don't go in and just change stuff because when configuration changes Other fields or other database tables also have to be updated And it's also faster to read from you can read the configuration from it and you can Get configuration to read in a form and save it back to to this database There is also caching layer involved and and a lot more advanced things, but this this is a basic concept I put the little asterisks on the reading We will get to that later because there is a layer in between that allows for for overrides And that's also why the writing operation has a separate read arrow, right? So so all good. The next step is to synchronize the configuration so you have this active storage and You have another configuration storage, which is backed by files the sync storage and You can export the configuration which you can do either by downloading that tarbal or With drush or to a console and basically what it does is it removes all the configuration from the files It leaves all the files reads all the active configuration and writes all the configuration to the files again the import configuration is slightly more advanced Because it doesn't just wipe the active configuration It makes a diff and sees what is new and what is old or what what should be deleted and adds the ones that Are new and removes the ones that are not in the files anymore and Then also updates all the other database tables Like for the fields when when you add new fields and so on so when you Deploy configuration or you manage the configurations between different environments You have in one environment these two These two configuration storages the database and the files so you click around in the UA and you build your site this Track after all then you export the configuration to the files you add the files to get You on the other environment you pull the files and you import that configuration Of course, this also works the other way around so you can have several developers and you synchronize the configuration Over git as as you do with the other codes that you develop All great, right The configuration management in Drupal 8 core works perfectly for this use case But we need to cover a couple of more things when I when I showed you this Actually the database has to be the same it has to be copied from before you cannot just import into an empty database and This is what the next Chapter is about this in installing from existing configuration. It's a very important step and It's for bootstrapping another environment of the same Site that you install before There's a contra project. It's an installation profile called config installer and it creates a Site another instance of the same site usually when you Go through the Drupal installer and you install a profile like standard or minimal or your own it creates a new site With the configuration installer it's just used to install the existing site and Not and then after the site is installed the configuration installers is not used anymore We use it in all our projects and so you should you Or actually core it even has a nice UI so you can upload the tar ball that you downloaded from the other site so it You don't need the command line for that, but of course it also works with the command line There's two issues to add this functionality to core Profiles at the moment that the API of a profile Was never meant to be reinstalled so for example profiles They can have dependencies, but you can uninstall the dependencies and that really makes no sense I mean you you have a dependency, but then it's not a dependency and so this creates a lot of problems when you want to Bootstrap a site from from with a profile that is like has a dependency but not and so there's two issues that try to solve this and I hope we can get them rolling during this week again more Another thing that is already in core, but it's kind of outside this use case is for example if you want to differentiate the environment and Simple examples are error logging to verbose on your development environment or API keys on production that you don't want to add to git and You can do that With core and the there's different mechanisms for overriding configuration. The easiest one is using the config variable in settings of PHP So basically you just add this this override in settings of PHP The system logging and you set the error log a level to verbose and That's it and you could also do that with API keys To go back to our graphics this this is where this arrow Comes in so basically you read from the configuration from the database, but then you read the overrides and you You use the overridden configuration In like the site title or wherever you use the configuration That's all great, but it has a couple of shortcomings One of them is that you can only alter existing configuration Using this method and you cannot add new configuration to it or unset completely other configuration and You cannot for example Change which modules are enabled And and a couple of other things like you can't overwrite the color of Bartik this way um Yeah, so Can we do better? Can we do more and this is where the configuration filter module comes in? So this is the schema of configuration management and Magic this is the configuration management with the configuration filter You can see the only thing that changed is This sync storage so instead of writing directly to the file you write to this magic configuration filter storage Configuration filters or config filter the module provides configuration filters and Filters are like a glorified alter hook for the configuration Storage so you they can apply they can change the data for every operation that you can do on a on a configuration storage Filters are plugins or can be plugins and plugins are sorted by weight and then applied after each other so they they're daisy chained and The data that goes into one filter and then comes out goes into the next filter and and so on and at the end gets written to the Storage or and the other way around First gets written then passed to all the filters and then gets passed back to Drupal Plugins can be active or inactive the inactive plugins. They they're just defined, but they're not used So they're just skipped The modules right now has more than 6,000 sites using it or reporting to use it And so one of the top 100 modules right now and has zero bugs. Well Not exactly. There was one bug that someone reported after I refactored the code but it turned out that the APCU cache was turned on and this is because the The the API of this module that the interface that it interacts with Drupal is really small Like it doesn't do anything else. It doesn't affect the running of the site except for the synchronization storage it that's the only thing it does it replaces the storage and Other modules provide the filters for it So let's unveil the the matching it look look behind the this configuration filter storage and as you can see there is the the green file storage again from before the the original sync storage and Before and after the data goes in and out filters are applied High-level example of this is for example with configuration split where configuration split has a secondary file storage and splits the storage splits the configuration between the main one and the secondary one Another example is the config ignore Which basically when you read the configuration? You read it from the active storage and therefore It's the same as the one you already have and therefore there's no diff and therefore the configuration is ignored This presentation is also meant as documentation for configuration split a lot of people have said that It's a bit complicated sometimes and there's a lot of options and It's not really clear what they do and how they work and You have to find out, but then you don't know if you did it correctly or or what? So let we will go here now through the the different options that you have to configure it so The first part that there's three field sets the first part is the static settings and where you have the folder so you you specify the directory of the secondary configuration storage and The you can use an absolute path or a relative path to the Drupal root the the same way you configure the Sync storage in settings.php. It's the same kind of way If you leave it empty it will use a separate specialized database storage for this split You can have multiple splits, of course, and then they get Used one after the other depending on on the weight and the split can be active or not and All of these you can override in settings.php with the normal configuration override mechanism that I showed you before So for example, we will get to that in a second The next section is the complete split also known as the blacklist I still might refer to that when when we talk because that's what it was named for a long time and Yeah, I'm still very used to blacklist and graylist, but it's now named a complete split and Conditional split so the complete split has a an option to specify modules and What happens is when The configuration goes through it. So there's the the list of enabled modules is specified in corded extensions and When you export the configuration what what the split does is it removes the module that you specify here and Save them in the configuration of the split and so they're removed from from corded extension as it goes on and When when you read from it, it adds them again it also Automatically detects all the configuration that depends on these modules because the configuration that depends on modules that are not Enabled don't make sense. So it also moves them away and puts them in in the secondary storage and There there's basically two ways one is a Select where we've chosen the UI is very nice and the other one is a text area that you can use to Use wild cards for example, so you can you can select a whole array of configuration In in the end Both of them are the same if you write the configuration name in the text area. It's the same as the drop down the conditional split or formerly known as graylist has the same kind of way to Select the configuration Important to know here is that this configuration will not be deleted when you export it I mentioned in the very beginning the way the export works is you delete all the configuration and then you read it and write it again but the conditional split Does what it does is it checks if it's different than the one you already have Exported but to check whether it's different the thing still needs to exist. So It doesn't get deleted and this has an important application. We will see in a second Then there's a checkbox for the dependent configuration you can select to also find all the configuration that depends on the configuration that you listed and Condition is split that too and there's a checkbox for split only when different so then it checks and it It splits only When when the things are are actually different And this is useful when when using wildcard. So you have like block placements and then one of the block placements changed But you don't want to split off all the block placements, but only the ones that are not the same as you you want to deploy from there There's a CLI commands The config split import and the config split export If you use them without an argument they're the replacement for the drush commands prior to 8.1.10 and for Drupal console And if you use them with an argument use them with an argument of the machine name of the split and then it will import or export only that specific split and will leave the rest of the configuration alone So that was all theoretical. Let's let's look at an example. So we have three configuration names, let's call them a b and c a for not being listed in in the configuration at all B for the configuration listed as a complete split. So the first part that might be Module or something like that and see the conditional split Obviously, it's a very simplified But you I think you you'll understand. So let's write when let's let's export A what happens it Goes through the filter it it gets passed along as it is Let's try to write B It will go in the secondary storage and you will not get passed on to Do the next filter or the other storage because it gets split apart see Here we assume it already existed in the exported configuration If it doesn't it's the same as if it would be different So if it already exists it didn't get removed. So when you when we export it, it just Passed it on If you look at the reading it's the same it reads the a Nothing to it continue to a The B comes from the secondary config storage And the C you read the C and you return to see so now Let's look at if we have a changed C a C prime so We see the C was the conditional split and now this will hopefully make sense to you The conditional split gets split off now because it's different than the seed is already exported And so when we read the the C We we read it from the the secondary storage So it will be the the C prime and and you see the the config filter behaves the same way so if if you Give it a certain amount of a certain configuration you will get the same back when you read it, but what happens on the other side is Also consistent, so you you see like ABC prime and goes in and a C Comes out for right and the other way for read so far so good, right? So how do we use this for making environment specific modules or configuration? So for example, you have the develop module and you don't want to develop module on on your production site But the configuration management just is all or nothing So you would have to disable the develop module then export and you don't want that It's a hassle and it's air prone if you do it manually so You can guess You do it with configuration split You list the module that you want to split off And you add the environment specific configuration and then you override which split is active per environment and If the split is active The module will be active if the split is not active when you import it It's as if the module was not there and therefore we get disabled or Will not be enabled Another small thing is the if you want to have environment specific permissions as you know permissions are part of roles and so The easy solution is you you have like a developer role that you have only on one environment But then you you can't just use the The same users and then you have to give them in this additional role. So there's another module called config role split The UI for the moment is not great because they to make the proper UIs Like how you should be configuring this is not so easy. So right now. It's just a text area where you essentially post YAML. It's described on the product project page It's also a config filter plug-in. So it Operates in the same way as the config split and what it does. It's exactly the same as the Core extensions and you add and or remove modules. So basically you just add or remove permissions to a role And this is useful if you want to have in some environments additional permissions for Authenticated users or Anonymous users Of course this split can then also be active or inactive depending on settings of PHP or you can even split off the Role split in another split so you you can make this as crazy as you want One of the things that you should keep in mind is when you import the configuration it will take into account the active Plugins that Drupal already knows about so when you import A split like when a split becomes active while you import the configuration this split will not be taken in account Already or this role split, but when you import the configuration again like a second time Then the split might be active and so basically for for every new plug-in You you import a configuration again So what we usually do is just config import and then review and then everything can and then just run it again and to see The message that says nothing has changed and all is good So before we go to the next chapter Where config split is used? I also want to say a little bit more about configuration management in general because as talking with some of you During the events a couple of things happen come up again and And so we use git for configuration management. You don't have to but it's the easiest in my opinion And it's great because the configuration files or the configuration are exported to files Which are text files and git can handle text files But there's a little bit more to it and you have to be a bit careful sometimes With configuration management and git so this is important for the team of developers when You you share that the repository for both the code and the configuration and So how you do that when you bootstrap a configuration the first developer Initialize the repository and installs the site and exports the configuration and adds everything to git All the other developers or all the other environments I should say They clone the code and they install from existing configuration that that may be other developers That may be production and that may be continuous integration So like you you install the site instance only once and the first person that makes the first commit Is this the person that does that and everybody else? We'll just restart from from the Existing configuration on the existing site So you can have parallel development You work on your branch and you commit stuff and and you you code You you do site building your export configuration and so on And it's all file. It's it's all git and it all works right but You have to be a bit careful with that because git is just the files But Drupal needs to import and export so there's this extra step and you you need to follow a couple of rules and If you're not careful what can happen is that you lose the uncommit work You accidentally overwrite Work done by others Or that it looks fine at first, but it's actually not So here's the safe sequence that the holy grail that the rules that you have to follow Sometimes you get away with not following them But then you're in danger and so this is always safe this this this works so just to say it you Do your site building? Then you export the configuration you add it to git and you commit it At this point you have a safe reference You know this was your site that you just worked on you you have it in git you can always go back to it Then you get the code from your colleagues with get pull or fetch and merge Then you update the modules as it is specified So whether you install modules as tar balls or whatever version You run composer install for example Then you run the update hooks So drash up to be or you go to the UI update at which we then you import the configuration and Once that all worked you push your commit and which has the merge committed it of course So if you don't follow these rules what can happen? If you import the configuration before you export it All your work is gone like It's it's gone. I mean no backups. It's it sucks for you, but you're the one that didn't follow the rules, right? Yeah, and there's no help there If you merge before export And that means you you you have the Exported code from your colleague, but it's only in the files and not in your database when so when you export it it all goes away from your colleague and The files will reflect what you had on your development Everything works for you. Yeah, right, but you actually removed all of your colleagues works This is not a huge problem You can solve it with the advanced git knowledge and you can go back and make a proper merge but You have to discover it and you you have to then fix it and It's not just git pull. It's like checking out stuff and making new branches and properly merging them If you don't do the composure install You may have updated and not updated code so that Can lead to breaking things in several spectacular ways If you do not merge Or if you do merge before the commit Then you don't have a safe state that you can go back to and you might have Additional manual labor to to solve conflicts I don't mean necessarily merge conflicts because those git will tell you about but the conflicts will see in a second And if you forget to import at the end then you will continue with the state that you had before and not Have all the stuff your colleagues have been worked on so next time you export the configuration You're in the same places before except Some time has passed since and it will be more difficult to fix it so Say sequence remember, right? So but there's another sequence when you update the modules So you you get the new modules with Composure update for example then you run the update hooks The update hooks might change the configuration So you export the configuration again that with the updated configuration You add all of that to git and you commit and you push of course if your colleagues have worked in the meantime before the push you You go back to the other workflow I think that's all clear So I was talking about before you should always run the update hooks before the config import and and Like If you conceptually the update hooks they're there to fix the database so that it corresponds to your code that you run and When when you update code and the database are out of sync You live in dangerous territory. So the first thing is always update the database so if you want to abuse the update hooks to do stuff after the config import Then you're tempted to do configuring patient for first and then the update hooks but You live in this dangerous territory and there's an issue in Drupal.org that will not allow to import a configuration when you have outstanding updates And There's a proof of concept module It's it's on GitHub right now that adds Another hook like a hook update as the pre-config import hook and the post-config import hook And they get fired in a safe environment as in the code is aligned with the database and everything is fine And they get called before and after the configuration import There's an issue to do that in core because the other issue is blocked on that essentially Yeah, so Maybe that's also one of the things we can work on this week and hopefully move on with this Right now you can do both and Often it will work and and sometimes it doesn't and Maybe you will remember this presentation and Remember I told you so So let's see an example of how you can break the configuration with git You all know the standard profile, right? So you install the standard profile developer a so you both start from the same place developer a Makes a new branch and deletes the tags field from the article content type This results in two configuration changes Basically, it removes two files the field instance and the field storage because it was the only field instance There was on the site Great commit developer B adds the tags to the basic page That results in another configuration change, which is adding a new file Since these files have nothing in common with each other git is very happy to merge this it's there is no no conflict but now you have configuration that has field instance without a field storage and Yeah, Drupal will not allow you to import this configuration because the configuration is is not in a good state and so Always import a configuration and you will notice this and if you Share the configuration often the the differences will not be that big and you will probably find out what happened and The import will tell you what is wrong. It will tell you The field instance doesn't have a storage can't import and then you go back and so what happened Where does this storage go and you see in the other commit that it goes removed there? And then you just add it again and it's solved but Git is great for Managing files and Drupal needs a little bit more care for that So how do we handle configuration changes on production? I just told you Configuration changes is like development, right? So if you change configuration on production One could argue that you do development on production. I mean it's not exactly the same but Your configuration changes they go in your in your git repository and So if you change configuration production that should also somehow go into the git repository I mean the scenario is exactly the same So you adopt this great model and you have all your developers Properly syncing configuration, but then you have your geeky client that just overnight Changed configuration and so when you deploy All the clients changes go away and because they were not in the git repository and your clients not gonna be happy with you And he's gonna yell at you and the client so it's right. So you have to take care of that of course The solution that you might know You have someone that clones the database to your slot local laptop exports a configuration puts that in git and does that? It's yeah, maybe a bit painful to do that But it's a solution you have like some person that's dedicated to to be the the client's developer, so to say Um That that can work if there's nothing in between that goes wrong. So if like If if the client changes something after you exported to the database to your local stuff, it's still gonna be gone. So First option you don't allow the client to do that Problem so Often often though often though, this is not really a solution and the client has a legitimate use case for changing some configuration. So This is great. And if you can do it, it will solve you lots of headaches, but sometimes you can't so there's other solutions to Coming back to configuration split because it can also help you out there So essentially you you specify which Configuration the client can change with what what is okay for the client to do because the client will not go and I don't know change all the the view settings and cron and add fields to everything and I mean at that point the client is Really a developer so but you know, maybe the site name can change or things like that So you specify what is okay to change in in a split settings? for example the gray list then before you deploy you Export that specific split so that your clients configuration is safe Then you pull the configuration business usual you deploy the configuration with Configuration import and what happens is it will import the configuration from the files that you had and it will also import this second split All good. All configuration is in the files That the dogma that the configuration that is in the files represents the configuration that's on the site after config import is still okay But you have to do this extra step of exporting so the third option is with config ignore Which you can skip this this step, but like the first so it's still the same you decide which config is okay to ignore Then you just pull the configuration and Config import and the configuration is then imported from the files but also some of the ones that are in the active storage and it goes in in into the The magic config filter storage and nothing will have changed in this configuration. And so it will not get removed and everything is fine This is great, but it Departs a little bit from from the philosophy that all your configuration is in the files and therefore reproducible and You you can have like the files and you know when you import it will be the same But it can it can be a solution for for this case Finally for a shared configuration, so if you want to reuse configuration or Use configuration on a multi-site Before the last Drupal con in the US, I wrote a little blog post introducing configuration Dimensions so You have this vertical dimension of the same site in different environments and then you have another site like on a horizontal separation If you were to have the same that the graphics from before it would be like a third dimension between you so This this is something that core does not take care of at all and You you can use the same kind of things. I mean the exporting to files is great and so on so There's other solutions and For example features Is useful for for that? Where you can bundle some of the configuration in into their own features into their own modules and and reuse them on other sites? But once you install the feature The site owns the configuration and not the feature anymore So when you update the feature you have to make sure that the configuration that the updated feature then gets Re-imported into your site and then the site gets Deployed with the normal configuration management workflow as we have so it features is for development only It's it's like on the the lower level on on on this Dimension And but it's great for like starter kits when you have like optionally more features that you want to add But it it Can help you to solve this problem, but it's You have to find a good workaround For for multi sites the You can use configuration split. I've used it for on sites. There's two basic approaches that you can do one is you have a shared sync directory and then separate splits per site and the other approach is the other way around where you have Each site has their own sync directory, but you share some of the splits Which suits best? I don't know. It's depending on on your use case But there will be a buff on Thursday, so you can join us there to to talk about that a little more and Yeah, that's it join us for the contribution sprint on on Friday and Make sure that you leave some feedback that always helps to improve the presentations. Thank you very much If you have questions, please use the microphone because it will be recorded so one common problem is that you don't export before you Check out something with gate or merge from from upstream or something it would be feasible to either have something that a Mechanism that always exports everything then as soon as someone changes a conflict will be exported or maybe with a cron like every five seconds They import yes, there's two modules that do exactly that. There's config tools which Basically on each safe of a configuration entity exports it directly. It doesn't have like Disadvantages in some way well, yeah, it does Git or it doesn't like a file right on on every safe I mean, but it only writes like one conflict file. Yeah, it's not so bad. It's you can totally use it Yes, it's a good solution for this problem Okay, it would be maybe a good hook that prevents get from doing anything harmful unless the conflict is in sync or something That would also be an option. Yeah Yeah, but if you export it automatically then then you will know I mean if you just pull and then if Yeah Hi, what is your solution when you're installing site in database you have profile standard but when you export your profile in for Configuration you have profile config installer and when someone wants to import changes from configuration There is a bug then we can't change profile from standards No, you you don't change the profile to config installer So when when you use config installer, it will still be standard that is installed Okay, it's so the way config installer works is it installs another profile and it jumps through a lot of hoops to to do that and That's why there's this issue to to move this functionality in core because a lot of things like You know, you like to do for example, it can't handle with the the case when modules have or Profiles have modules in them Because then if you install config installer it it can't detect these modules and don't doesn't find them and and so on But so profile it when you when you install a site with Config installer the configuration you have at the end Will not say that the profile was installed with config installer, but with the profile that was installed initially Hello should notice about features and configurations you can use used features import or features drush features import or features import all when you pull new changes In your future just Zendron Drush import features all and it solves problem You are to delivery Yes, but so if you then change this configuration yourself, then you know, you know, you will have to review this process it's it's a and I have question about Station About in protein blocks you know blocks issue when you Create block instance in Yeah, production Conculation and cannot import this configuration because you haven't content Yes, exactly. So one of the things that I didn't mention now because I expanded the other parts of the presentation Is is the the issue that you have when configuration depends on content and the block is like the conan canonical example for that So when when you have configuration that depends on content, you you have to make sure that this content also gets imported one of the one of the solutions is this The pre-config import hook that I talked about another solution is you you implement an event listener that listens to the Content not found event when when you import the configuration. I forgot the exact name of it But it's something like that another option is use default content and then Have like an additional step to to import and export the content that you need You can also use deploy the deploy suite for more advanced Use cases for that in in in our case. We usually use default content and it works great Hi, how can I handle? Multilingual multi-side with the same feature set, but different base languages on each side so The configuration split works with multi-lingual configuration as well It it splits the configuration with its languages What you cannot do right now is override only specific Language so I mean you can with the Like No, no you can't really I Mean but it wouldn't be too difficult to to add another configuration Filter that only does languages so like The thing is the configuration split works with a separate config storage and so It works the same way and that's also how it deals with languages it natively just supports languages because configuration storage to support languages So if you want to only have special Languages, then then you need to treat the languages differently than than the rest of the configuration and It shouldn't be too difficult to add another configuration filter that Deals languages and saves them somehow in a in a separate way Yeah, but if you have questions about this if like if you're implementing a config filter And I know it can be pretty daunting because it's like a lot of things and and Don't don't hesitate to contact me and I'll be happy to Assist you. Yeah. Thank you very much lunchtime