 Hello, welcome again to the building track of Drupal South So we've got Adam here and probably one of my favorite contracts was working with Adam I have to say we had a very very interesting goal Of doing some cool stuff with Drupal 8 where we wanted to deploy from from git content And pretty much there was conceptually it was possible to do that and about basically Adam was like The hero piece don't want to like trump him up too much But uh, yeah, Adam doesn't pretty cool code on that project, and I'm really excited having been away for Six months or so from that project. I'm really excited to see the presentation today to see where they've got to so Can you please welcome Adam? Thanks, I Yeah, so today gonna talk about some really exciting stuff that I have been working on for the past almost a year now Config and content as code mainly content because we all know that Drupal 8 has got this great thing called CMI And around how we ditched that old canonical database So let's get straight into it Gonna go over the problem and the motivation behind this kind of thing. Why would we want to do this in the first place? a little bit of our infrastructure overview of how we've got AWS setup to handle this kind of thing Just so it's a bit easier to explain the principles behind it And we'll go over the project timeline what we started with some great groundwork from the main man here Simon and What we've got now and what our future is looking like Give you a rundown of our toolkit all of our Libraries and things we use to help out with this kind of thing And then I've got some amazing people to shout out and then I'll open the floor for questions so Problem and motivation so The main idea of this was everything has to be code content config The whole shebang we've got all of our infrastructure as code as well And we wanted to get away from this idea of a canonical database in Drupal Does anyone not understand the idea of a canonical database? Yeah, okay, so in Drupal you've always Well for me personally anyway, you've always got this idea of in production You have a database and that's your source of truth You nurture it you hold it and you back it up and you don't want it to go away because if it does it's good and That's great. I mean That's how we how we used to dealing with things in Drupal, but what if you didn't need that? What if you? Could spin up an entire website just from a get repo Not having to back that up that database up every hour or every day or however You have to do it we also so and Saying that we need to support a highly available and scalable environment We need to be able to scale out for load and scale back in and you know if a web server goes down We need to be able to spin one back up to the point in time And that at last was all the content needs to be there and that needs to happen without that database backup So how do we actually do that? we've also being government we I Think I'm a forgot to mention that I work for the Department of Justice. That is a crucial piece of information So we've got compliance requirements. We've got to meet We need to someone comes to us and says on the 23rd of September You guys had this piece of information on your website and it wasn't quite right With a canonical database, how would you? Go back to that exact point in time and be able to audit that and You know Verify what they've said is it's correct or Not so we are able to with our content as code spin up a site at any point in time using using get So yeah, that meets our audit trial and freedom of information requirements there We wanted to be able to manage our changes Be out of package content config and code all is one so We might have a large feature that contains You know a new landing page and that landing page has a cool map on it And that map is made of a view and the view is built from a module that we've added to the site And we want to be able to package all that stuff up The content the config and the code the modules and everything and to one release and release it all at once rather than Writing update hooks to add content Import the config and all that so You know, there's there may be a case where We have one of those large features and it's sitting in approval for three months We need to be able to Handle that and have that sitting there waiting for approval and once everything's all good merge it in and go live And also rollback so if something goes wrong, we need to be able to roll back That's all the kind of Why we want to be able to do this right so Now I'll give you a little glimpse at a very little glimpse at what we What we use in our infrastructure to support this kind of model so per site we have a single build server and He's the guy that runs the tests builds the packages for deployment Has a testing site on it if we want to have a look at those builds or see why the tests are failing or not We've also got UAT and production environments for each website and Her environment we have a separate authoring site So that is one of the crucial pieces of this whole infrastructure being able to have that separate authoring site Where people log in and author content and having completely separate web front-ends or view servers as we call them on an auto-scaled AWS structure and so Each environment has automated content config and code deployments just through Jenkins And those happen nightly to UAT or they can happen manually When we see fit the content gets deployed Every 10 minutes and I'll take you through a little bit more of that process later So the project timeline what we started with now For me when I heard about what they wanted to do in there and our text vision for this Honestly, I thought it was a bit crazy. I Simon and I I remember sitting in the room Multiple times thinking is this really going to be possible Are we actually going to be able to export everything on a Drupal site into our code and have that be able to be imported on another website And spoiler alert it is possible It's pretty hard so Challenge accepted. I'm gonna do this shit and We started the long long process of getting where we are today so Cmi Very dear to me. I Love it and thank God Drupal has it because I don't know about you but features Yeah, enough said, right So, I mean Cmi so that you know the smooth dependency resolution for the most part you've got you've still got the ability to override stuff and settings of PHP files, so You know, you have your settings. We have our settings sort of UAT PHP and that's where all our UAT stuff goes in terms of login credentials for external dependencies and whatnot and It really allows us to keep all of our config consistent across all of our sites so and all of our environments, so we actually use Yeah, one one group or folder or directory of con config for everything and Then just use those settings that PHP files to override Right, so config. Yeah, we got that for free. Awesome. We can export it. We can import it looks like that cool now this entity pilot Who has had any experience having a look at entity pilot apart from you guys So basically what entity pilot allows you to do is Package up content into flights so departures and arrivals So it automatically Calculates dependencies for you So you say I want a flight containing the about us page And if that about us page has a menu link and a taxonomy term and all the other bits and pieces It knows that it's going to export that all as well so entity pilot basically allowed us to Extend its transport mechanism, so it sends its flights over HTTP yes, and The way it's architected means we can just say hold on a minute write that to disk or read from disk So that was basically why we chose entity pilot to do this. We looked at things like default content or deploy But the way entity pilots Britain and architected really allowed us to extend that It's also really actively maintained the Lee Rollins legend and it made our requirements for those packages of content To play in symphony awesome we could We could use the symphony services and swap that piece out that transport mechanism really really simply so That was a that was a really good one for us as well, so Here's a little snippet of something you can do and you just say Here's a service entity pilot dot transport You're actually going to use my class now and Then all we do is override the methods for getting and Basically getting and setting those flights So symphony really allows you to take something awesome that someone's done and molded into what you really need So that was a really good one for us as well So into entity pilot get which is my pet project It's What I've been just describing so it's it's replacing that Sending over the web with sending to the file system And what you can see here is a what we call a manifest So when you export something It writes this manifest file it says we've got Some contents in that contents. We've got some block content and of type simple content And there's two pieces of content in there and just lists the UIDs So These manifest allow us to uniquely identify At any point in time what is going to be on the site and what can be exported or imported And it makes the content extremely transparent We can I can look at this and say there's two blocks and this manifest and if I were to import this I'm going to import those two pieces of content and What are those pieces of content look like? Well, they're just Jason files So On the right you can see our content directory nested by entity type and bundle and All the files are just named by the UID On the left you've got an example of one of those blocks earlier. It's got all its fields in there It's basically just a serialized version of that entity And it's yet to be open sourced. Sorry. It's coming though So with that all out of the way, we've got the config and we've got the content Now there's a whole bunch of other stuff that you've got to be able to do so make everything deployable Basically, everything has to be related by a UID so one of the first pieces of work Simon actually did on this was To write the UID condition module. So it allows us to place a block By UID instead of path that guarantees us that we're going to We know that block is going to show up in the same place on every single site or every environment Rather than relying on a path, right? So we've got blocks, we've got config, we've got content Cool. What about images Everyone loves uploading images to durable sites Or files, documents, especially in government so IntermediaDB was chosen as a digital asset management solution Basically gets us away from having a mount of file system Or sync files between environments kind of that kind of thing because remember we've got that separate authoring site separate View websites If we're uploading an image to authoring, how do we make sure that's represented on that live website? So I plotted the Enbridge module to Drupal 8 That's an example of searching for an image And basically just are able to choose images from that damn solution and directly use them in your Drupal site And this is using KuaiaJax modal windows by the way, which are really awesome So that's uploading an image in a field. What about images inside content? So Enbridge CK editor module allows you to embed content, images and assets inside that body content We're getting there. We're almost there. What about links on the pages? Node slash NID links aren't going to be deployable because you can't guarantee that node ID is going to be the same on the other website So we got in touch with the guys at previous next And worked out a solution for deployable links and menu links as well. So This is a node. I don't know if you can see that but it's You know, it's being displayed on its UID URL and that basically means we can have in page hyperlinks to content and Yeah, show that piece of content without having to rely on a NID path so Yeah, thanks previous next as well for continuing the work on getting these into core It's been a long process And menu links as well. So menu links by default here. This is an example of an export one on the node slash NID URL again, that's not deployable. So we need something to target that UID. So that's actually Normalized at export time and then denormalized at import time and finds its own It's a real NID in the project and that actually supports link fields as well So we're pretty much there now. We've We've got a config. We've got our content. We can play some blocks. We can upload an image We can link to a content and we can have we've got menu links. So that's pretty much a standard Drupal site But we've gone a little bit further since then so that was basically what our first site started with But then we started getting these more complex requirements like what about Collections of fields or as they're now known as paragraphs If you haven't used this module Awesome So we know export can export and import paragraph content through Export and import transport aliases so an alias in Drupal is actually considered a computed field. So normalize is actually ignore it. So By default, it's not exported So we've got that now and Wrote a case sub module for the Enbridge Module which basically means if our digital asset management Solution goes down or has any issues with connectivity our site can still serve those images locally from a Cached URL so it works very similar to image styles in Drupal Where It'll generate that image the first time right at the desk and then serve it from disk for the subsequent requests So that's pretty much That's pretty much what we've got now in terms of deployable content So how do we actually Do all this stuff and how do we what do we use to deploy? And manage that kind of content workflow quite a lot so We've got Jenkins Jenkins is Being coined as the heart of our project so it Keeps the blood flowing. It keeps all of our code and our content flying through our environment We've got Bitbucket, which is the brain. It's it knows everything. It's got all that content It's got all that config. It's got all about library dependencies composer files It's got our ansible tasks for infrastructure And then we've got our terminal helpers console and Drush to to alleviate the sort of the pain points of That kind of stuff over over Jenkins are we using PHP unit and be hat? Sorry Sam Just a side note I am starting to look at What Sam was talking about yesterday the browser test base and all of that awesome stuff now Be has really accessible for people that haven't had much experience with testing though, so that really worked for us and Yeah, but you know it has a lot has its limitations So ansible we use for all of our infrastructure So all of our web servers get provisioned by ansible We're using it so and we use it as well for development environments So I believe we borrowed heavily from a talk you might have seen yesterday beatbox and Drupal VM as well So we've got our own vagrant boxes using ansible and stuff like that and Thanks to those guys at beatbox with beatbox and Drupal VM for helping us get that done and Finally the biggest toolkit known to mankind AWS We Yeah, heavily use a lot of AWS services like obviously easy to got s3 in there just started using cloud watch and Elastic search and stuff like that So now I'm going to briefly explain how all this stuff fits together With some photo grams so How do we do a code deployment using this kind of methodology So we go hey Jenkins, would you mind merging these two branches together and he goes all right with a smug little face and goes off and There's some tests Can I merge that branch or those branches together? If there's conflicts or any problems with that it fails He then checks is there any content to export so we've got like a Jason callback that checks with site if there's any content That's been updated since the last export created If no, then we're all good to merge So we spin up an entire separate machine Next to our authoring website. We merge that Code together we run all about this and if everything's all good then we can merge and deploy So I actual authoring site. That's that's how we manage feature releases so What about content? So every every ten minutes Jenkins checks That same health check. Hey, is there any content to deploy yet? There is cool we Export it and Also the config as well because we allow our site owners to edit conflict some conflict Cool, we've got that in codebase. Let's create a release push that and get and then basically do like a blue green black white red blue swap with the view servers and One at a time just deploy those and then We've got we've got that new content so that's taken care of the last little piece of Kind of cool stuff we do with with all these tools is our pull request builds, so code review is a big part of our process and To to release a new feature to build or develop Jenkins sees that there's a pull request up You might be familiar with something this from something like github deploys that branch to a build server runs a test and Since all that content and all that config is there it Can just simply run a site install and a config a content import And then we've got a working website there that we can go and have a look at and Yeah, basically see That everything's working correctly And then Jenkins tells that bucket. Hey, that was a good one or that was a bad one And we don't let bad ones be merged That's really handy. So That's basically a We have six live websites right now and There is another 29 to go So one awesome thing about this platform is it really allows for rapid development We have what we call a vanilla build where all of our base build for the site or that our sites live and Getting that initial build going talk us about six or seven months Crime stats was Sort of guinea pig for that and then we've just been out of being our sites ever since we've got a couple more in development right now And yeah, as I said, there's a lot more to go the context we have a team of three developers to all those are front-enders and Three ops guys that work on the background background stuff. Cool. So what's in it for the future? I'm gonna get out of sorted We still haven't quite met our vision of absolutely no clinical database. We do have to back up for user passwords and And another thing which I'll talk about in a minute We're gonna be starting to abstract that content out of our Main repos into it's into their own dedicated repos for better release management As I said, we want to get to this node database backups point of view Right now, yeah, we've got the user passwords in there, but also revisions. So Another huge piece of work that previous needs is helping us out with is adding you ideas to revisions Currently when you normalize a piece of content, you only get the default Revision, so if it's published you get the published version If it's unpublished you just get the latest whatever's the latest version of it But we want to be able to export the entire history of a node or a portion of the history of the node That way our authoring environments can be completely on demand if You know out of business hours. We don't really need that authoring environment to run. We can shut that down Not even bother with a database backup Let's make sure everything's committed to get and then bring that back up and have all that Content in there just as they left it So what this is going to be able to do is yeah, they said on-demand authoring Cost-saving there, you know less servers running the better And some sites that we run only actually edit content like once or twice a year, so There's not really much point having that authoring website up and running all the time That's going to lend us to be able to have different charge models so we could You know have a different model for 24 seven authoring or on-demand authoring We get into things like dashboards for spinning those up and all sorts of cool stuff Yeah, there's a lot of performance improvements You can imagine going from exporting one one entity per node to exporting You know, maybe 10 revisions per node or hundreds even if Your content authors make a lot of mistakes So there's a lot of performance improvements to do there and we want to get to a point where we can You know have a new site coming onto the platform and completely script the whole base of our of Our build so they get repos the Config the Jenkins jobs and everything like that just so we're ready to go as soon as we need to cool So last but not least I need to give some shout outs to some awesome people in the community that have actually helped this project come to life So firstly Lee Rollins Author of entity pilot Has helped us out tremendously on IRC and previous next I've been doing amazing work with all that UID Kind of patches the core and helping us get that And kind of realize that Simon Hobbs he was the initial engineer on the project and actually is responsible I hear through the interview process of getting D8 into the department of justice Drew gaff he's our architect and he's that he's the one with that Grand grand plan that dream of having everything in code And Samantha presser is our superstar on our team does everything from front end to a BA work Works with our clients And obviously everyone else at the department that's that's kind of made this project happen So if you want to get in touch with me pretty much AC Bramley on everything apart from Twitter because someone took it and Those are the three issues that I've talked about today with the and involved with your UIDs and revision UIDs and stuff So if you want to get involved jump on those And that's it anyone got any questions Just one quick cool thing the whole things auto healing which I love saving the Jenkins is still the case like if If the Jenkins box go down they're all completely auto healing. So Yeah, it's pretty cool too. All right. Who is first? Just a very quick question and that's interesting But can you tell us about how many nodes are exported say in the crime stats database on any of the others if you know? Yeah, so I think crime stats last time I checked About a hundred and twenty nodes So, yeah, the sites that we're working on we're kind of working from Most simple to most complex crime stats while it's not the most simple. It's the most Feature rich in terms of what we need for the whole platform. So Yeah, we've got other sites. I think in the future. We've got a site coming up with which has something like 1500 assets So all that's you know, we're getting there in terms of the number of the sheer number of content, but yeah Good talk my originally saw so I do this the original part of this at Drupal gov And it's been one of these things that I've been really waiting to hear the conclusion for for some time Yeah, I'm gonna ask the question that I asked I at Drupal gov Which is when does it come to the community? Where's the release? Yeah, honestly, it's I could release it now and But I just want to make it a little bit better before You guys just put that in context. I really think the main part The main component is the entity pilot extension. Yeah I think with the entity pilot extension you could see effectively like a prototype give repository with just Just a composer file Another thing that I loved about this project is we got away from make files completely and we did the whole stuff with composer, which you know Difficult at times, but yeah, it's pretty cool. So you'd end up seeing a repository was effectively just a composer file Some basic structure But yeah, I'm looking forward to that too Looks like a fantastic idea Great vision. I think it's good that you're getting somewhere with it In the past doing things like doing migrations from Drupal 7 and porting a lot of content over I've seen things where content imports take a long long time with having all that content If you've got a big site with a lot of content, what kind of effect does that have on the performance of your install bringing a side up? Obviously the more content the longer it takes it's still still loading those entities and building them up and looping over everything but What that's one of the things I'm going to look at soon as how we improve that performance whether it be that batch or queuing or You know right right now. We actually export everything at once and import everything What I want to get to is on an existing site being able to import only the what's changed So then I'll speed it up But yeah, we have I mean I haven't seen any severe impacts of the number of content right now but I know when yeah, as I said when I switched on the revision new UID stuff going from importing 120 pieces of content to importing 1,500 Yeah, there was some performance issues there, but that wasn't that was around memory issues rather than The time it takes to import something Yeah, great work on this. I've been chatting decide about this as well But I was running have you looked at using you know the deploy module and multi-version and those sweeter modules in d8 to actually be able to Kind of syndicate this content out to other sites. So if you had D8 sites that were too small to really kind of do any work on or other d7 sites being able to actually use this model But push the content out to other sites that still use a database It's not something that we've actually we have looked at yet, but it is definitely possible. Yeah I mean, it's a pretty good idea something that I've taken mine Yeah, and just on that as well as a couple of architectural constraints that we we chose to adopt right at the start when one of those was we have one site One one site effectively in a repository on a branch that you know and so it wasn't like about syndication We're not a news. It wasn't a news corporation where we want to back send to multiple websites and spread that content out So there's things like that and around the size as well Like it was like well, there's going to be a point where it doesn't scale, you know And the existing models are still valid But for the majority of these sometimes brochure where sites at that department justice like the model makes a lot of sense Lee I Think this is the same question I asked Simon as well last time Obviously went through a selection process for the intermediate DB. Can you guys share the because that's an open source? No, that's an open source right Dm right Yeah, yeah, yeah, so Obviously you guys went through a process of evaluating DMS and just how you got to where you got to and yeah, that I'm looking at others and are you using like a hosted solution for that or are you guys right? We're hosting it. Yeah Yeah, it is interest and it is a interesting product. They've I've got some cool stuff Once when I came on board that that selection process has already been done I think There is a larger vision to what we want to do with the digital asset management solution, but we've kind of got a We've got a work out a lot of cans with that product right now If it was more of a If it was a bit simpler in terms of requirements, I probably would have done it with something like s3 I think it would have been a lot a lot smoother, but Yeah, for now intermediate is Doing pretty well On one of the slides you had earlier think was the deployment pipeline for production you had view one and view two down the bottom yeah, so If I'm reading that right that means Like one view stays in place while the content's being ported for the new one and then you just hot-swap the two of them because The databases cattle not pet type. Yeah. Yeah, so each server has an onboard database. Yeah So yes, so one's get one gets taken out of the load balancer That does a pull sorry an extract of the latest release Does it I think composers already installed doesn't import of the content then just hot-swaps that does it to the other one Then puts them back. So with the question that Bevan asked about how long it takes to import does that kind of make that mood because Absolutely. Yeah, so it doesn't really matter. Yeah In our case anyway, it's right anyway right now We would like to And we have had requests to have that content Deployment process a bit faster, but I think yeah for us right now That's yeah, as you say it doesn't really matter how long it takes because it's not actually serving Yeah, so the only reason it's too is because that's what our auto scaling group is said to If we were to bump that up to five today At the site would continue working. It would just build all those servers from the code base and away you go Nice talk nice idea. I think it's the reverse of migration like file to D8 migration kind of thing in my opinion so Is there a plan to you know convert that to a proper thing like in the migration we always do you know from file to P8 or something D7 D6 we have the modules for that in contraband now in core and This will actually give you The ability to import, you know as soon as D9 will come you can import your data to you know D9 instead of P8 Because you can always use serialized and stuff. So is there a plan for that in place? Um, I mean, it's not something I've thought of but Yeah, as you say, it's definitely possible. You could Just for example, if it was You could really do it with anything you could just implement your own normalizer because all it is is And that denormalization process as long as that returns an entity, whatever it may be at the beginning Yeah, it could be used for something like that Another question like from why? You know experience with entity pilot, it's a complete product of you know Importing data from one side to another and so on and so forth Are you just using it as a you know normalizer repository kind of thing which is yeah sort of so it It does a lot of the work for us with the Stuff like the dependency resolution Being out of package that content normalizing it entity pilot has a lot of So core has a lot of normalizes and so in itself the entity pilot takes it to their next level being able to You know normalize things that don't exist yet. So it's got like a unsaved UUID Plug-in where Say if I've got a node and that's referencing another node and No, this node is getting imported before this node. It knows how to how to figure that out. So Yeah, it's basically for all the normalization stuff like that. So are you still using Drupal UI to add all the content or yeah So what is still log into? the authoring website They do like a normal Drupal authoring experience there and then that gets exported and pushed to our Front-ends or view servers as we call them one last thing. It's just I'm and I know it's getting good, but I think in Drupal 8 it is very good for whole, you know Drupal 8 workflow because we are now able to use it for rest Application and stuff like that. So you are doing it, you know for your type of sites Where you having you are having the content in the repository repository, but you know, I think it's for the rest application And stuff like that. It's pretty amazing thing. So thank you for that Thank you. I was gonna introduce the applause, but you've got to in front of me. Thanks Adam