 In my session on managing Drupal 8 sites at scale, my name is Kevin Mall, I'm a Drupal architect at App Innovation Technologies. My handles are K-Mall at Drupal.org and GitHub, and I'm Kevin J. Mall at Twitter. So today some of the things I want to talk about are kind of what the needs of an enterprise organization are. I really should rename this because it's really the needs of any organization of any size. An enterprise just makes it a little bit more difficult with the scale that we have. So we need a few different, to modify a few different things in order to get things to work for us. I'm going to talk about a little bit about what Edison is and some of the tools that we use in Create. And then what I really want to do is leave some time at the end to talk about open sourcing. The company I work for is very keen on open sourcing almost everything that we're able to. So there's a few tools and things that we've created that we've already open sourced. So I'll kind of discuss those and show you where those are. Then there's a few tools and stuff that we're using that to be honest with you, we're not quite sure exactly how to open source it or how we want to. So I'm hoping that maybe some of you guys here work in organizations that manage a lot of sites or big sites and might have some ideas about the best way to open source those or even if some of the stuff we're doing would even be beneficial to you guys or to anybody else out there in the community. Because in the end, if no one's going to use it but us, then kind of makes it a little bit difficult. So let's talk a little bit about the needs of an enterprise. And as I said, these are really just needs of an organization for a website. If you want a website, you need to set up some things. You need some GitHub repos, you need hosting, you need some other services. So that all takes a little bit of administration, a little bit of setup and maintenance. You want consistency in your sites. If you have a hundred different Drupal sites, I can almost guarantee you that 80% of those sites are going to use the same functionality. So you don't want to use different things and different tools and different modules and different processes for something that in the end is ultimately the same functionality. You want to make sure that there's consistency along the way. You want to make sure you have some element of testing. This day and age is pretty dangerous to deploy code out there without having some sort of automated tests or some sort of functional tests to make sure that code that you're deploying actually works. Doesn't break anything that used to work. Things of that security as well. I mean, just about everything that you do on the web these days needs to be secure. There are people out there constantly trying to hack you. So you need to make sure that you have the things in place to make sure your site's secure. And I think to me, most importantly and of all, it is a good developer experience. You need to make it easy for your developers to build features and to get your code out into your website so people can use these features. So we've been working kind of within those constraints on this team that we've now called Edison. Edison is part of, is an engineering team at a large pharmaceutical company. And essentially we used to be called the platform team or platform engineering team, but we've kind of re-approach the way that we're looking at this with our D8 platform. And we've kind of made it more into a product, more into a branding thing where we're actually building a product that people are going to be using to deploy their sites and to maintain their sites and to kind of see what's going on within their sites. So essentially it's just a group of tools and processes that we use to manage our sites. And it consists of three main parts, really. Edison base, Edison CD, which is our continuous delivery, and Edison dashboard. So Edison base is really what we kind of consider our platform. We base this upon the Drupal Composer project. The building sites on Drupal 8, chances are you're kind of, if you're using a Composer workflow, you're usually basing it off of your Drupal Composer template. So this is kind of just our version. It contains all of our custom and contributed modules. It contains all our files and configurations that we want to deploy out that every single site is going to use. So whenever any site is built, it automatically pulls these modules, these files, these configurations. This is what really brings that consistency into these sites. Next is Edison CD, and that's just our continuous delivery system. So this is mainly a tool written in Python. It's essentially what we used to do in Drupal 7 was a lot of bash scripts with Jenkins, and that was what we used to do a lot of our deployment, a lot of automation. We've since moved that out of that, and we've moved to Python. We're currently using a tool, a CI tool called iron.io. So that's our continuous delivery system, and it's also our main automation tool. And the third part of it is the Edison dashboard. This is the main place for managing all of our sites. This is kind of the UI that we've built that kind of integrates all of these services and all of these tools. So this kind of gives the developer the ability to see what's happening, what sites they manage, what environments are in there. What's the status of these? Did my build that I just sent to our dev environment, did that pass? If not, why? Where can I get logs for why it didn't pass? Where can I see some of the testing results? I don't want to have to go to a million different services and log into a bunch of different places in order to see things. We kind of want everything in a consolidated place, easy to find, easy to see what's going on, laid out in a way that's kind of makes sense to developers. So that kind of goes hand in hand with the good developer experience. It's also good for administrators. This is where they can do a lot of things like create sites. They can see reports, consolidated metrics on how many sites we have, how many builds are passing. Do we have an issue with some of these because all of a sudden 90% of our builds are failing, things like that. We all, we want it all in kind of one consolidated place. So one of the big changes that we made when we went to Drupal 8 is we've kind of split up the way that our sites are made. Rather than having one GitHub repo with your whole site and then custom code, contributed code, Drupal core, all in one place. We kind of split that up and now we have what we call a site repository. And that's what our version anyways of the Drupal Composer Project makes. When you build a site, it pulls all of the stuff in. Custom code all goes in an installation profile. So it's a way to contain in one location all custom code configuration and things that are needed specifically for that site. So essentially the site repository is what contains the main composer JSON file and the composer lock file. We make that read only to our vendors. The way that we kind of structure this is we as the Edison team, we don't actually build websites. We build the base that people build websites on. And then we hire different vendors for different brands and different components of the business that need websites. So essentially we're not building any code, so we need to make sure that we have some sort of controls in place where an outside company can't come in and say, hey, we don't really want all these security modules here because it really hampers some of the stuff that we wanna do and just rip it out, right? So they don't have the ability to do that the way we structure this. So essentially we only give them access to the code that they need to use and alter, which is their custom code. So essentially the main site with that composer JSON file pulls in our Edison base, which is really just a PHP package. And it pulls in the installation profile, also a PHP package. So to build the site, you can run your single composer, create project, give the site repo, and then your whole site gets built, the installation profile gets pulled in. So just a few other things. So kind of to round out Edison, essentially it's the umbrella around all the things that we need to kind of create and maintain these Drupal sites. So obviously, we're using this on Drupal 8. Composer is a huge part of our process, and to me it's been a really beneficial tool that's been added to our tool set with the advent of Drupal 8. It allows us to do things like we were saying before where we could kind of create most of the site repository the way we want and then just kind of limit the vector of which the vendors have to work with with their custom code into a single place. We store all of our custom code on GitHub and we expose that to composer through Private Packagist. So if you guys aren't aware what Private Packagist is, it's essentially kind of a paid service from the guys who run Packagist. And that allows you to put some of your custom code that you're not actually putting out in the open space, your private GitHub repos, things like that. It allows you to expose those to composers so you can use them, pull them in just as you would with an open source Drupal module. So there's actually a conversation on Twitter where somebody was asking a colleague of mine kind of how we handle things like custom modules. And really there's kind of two ways that we handle that. There's the custom modules that we make that we want included in kind of the base platform in Edison Base. And then there's the custom modules that are going to be used by specifically just the vendors for like a one off project. So the way we do that is if it's going to be used by more than one project or all the projects, we create a private GitHub repo and then through the Private Packagist that gets exposed and we can pull that in. If it's just a custom module that a single vendor wants to use for a single site and there's not a lot of reusable potential there, then you can just put that module in the modules file in the installation profile. Then that gets pulled in when you pull in the installation profile into the site. So a lot of that just depends on kind of what it is and what it's used for. But Private Packages has really been huge for us because it allows us to kind of treat our own custom modules as if they were kind of open source like Drupal modules out there on regular packages. Or any other PHP library. So it's a great tool. We also use Travis, all of our testing runs through Travis. So anytime a developer makes a commit to the installation profile, we run tests to make sure that things are working. We run their PHP unit tests, which we require them to have. We run PHP code sniffer, make sure that they're adhering to Drupal code and standards and things like that. We also use Rundeck as a task manager. This was actually a new one to me. I don't know if you guys are familiar with Rundeck. I'm not gonna talk too much about it, but I think it's an interesting tool and wanted to mention it. It's essentially a task runner. And when I kind of describe some of the functionality and some of the things that we do with the dashboard, a lot of that is kind of run on Rundeck and I'll kind of mention it there, but it's a really cool tool. And our hosting is on Acquia, they host all of our sites. So all of our CI tools integrate with Acquia and with their cloud API. So I want to talk a little bit about kind of some of the tools that we've created to handle some of the needs that I spoke about that we needed before. So one of the main things is set up and maintenance of sites as well as consistency. So we have a plugin for Composer that we've called Hydration. Essentially what this is is it's a package that provides a Composer script to be used as a placeholder replacement mostly used by skeleton projects. So what does that mean? So everybody, for every site, you need a repository, right? Now if you have 100 sites or if you have 500 sites or 1,000 sites, you need to be creating a lot of repositories and to start doing that manually for every time you need a new site gets a little tedious. It's prone to error, you might miss some things and you really want to sit there and hit the add new repo button a thousand times and get hugged for every site that you need, I certainly don't. So we've created a way with a Composer plugin that you can set a template for the repositories that you want. So we have a few different templates, one being for the site repository that I spoke about and one being for the installation profile repository. So that essentially allows you to take a kind of base skeleton repository. I don't know if you guys could see this in the back. This is where this might get a little tough with the resolution here but essentially this is the outline of what would be a starter kit for a repository. You could put placeholders in the file names and within the files themselves. So and then when you run this script, it's essentially going to take the placeholder and replace it with a variable that you pass in, which would ultimately be the name of the site. So it's going to replace all of these placeholders file names and every place within that file with the name of the site. And then once that's done, you have the resulting repository with everything that you need in it starting off as a base. So that gives us a way to ensure that everyone's starting with the exact things that they need and the exact things that we want them to have. So we've actually integrated this with our automation tool, which has an integration with our Edison dashboard. So you can just go into the dashboard and say, I want this site. For this one, we're using PFTestal 1. I type that in, hit Enter. It calls out to our continuous delivery system. That would actually clone a repo, which is the template, runs our Composer plugin, populates that and then throws it to GitHub. So this is actually a plug-in that a colleague of mine made. And he's put it on the open and open sourced it on GitHub. So you can require it in your project just by requiring the package name. And I'll make these slides available. So if anyone's trying to write any of these down, I'll make sure that I post this so you can have all the info. So the way this works is, and I don't know if you guys can see this at all. All right, I'm not going to try to mess with this too much. But essentially, see, it's tough because it doesn't seem to. Yeah, hopefully it's clearer on the video. It's actually not even kind of clear here and present of you. But essentially, it's just a section that you're adding into your Composer file that makes available to the plugin certain things. And it passes in kind of some scripts here or a command. And this is actually the command. Let me go here. So this is exactly the command that you run. So it's just a Composer create project command. You're passing it the URL of your private packages. Or if you're just using all open source, you can pass a packages. And then the name and the path to where you want this to go, which includes the name. So it takes the last bit of piece, which is the name of your site and replaces all the placeholders in there. So essentially just loops through and replaces everything and then puts that code where it needs to be. So if you're creating hundreds and hundreds of sites, you can easily just loop through this and create all these GitHub repos. So the way we use it, as I mentioned before, is our continuous delivery system goes through, hydrates all of those repositories for any site that we pass to it. Calling Composer to kind of do that work and then we post it to GitHub. So it's a really quick and easy way to make sure that you have a consistent way to build out sites. Everyone starts with exactly what you need them to start off with. So that's one of the more beneficial tools, I think, that we've made. Since we started putting that into practice, it's made kind of set up of this a lot easier to do. So the next plugin that we use is a plugin called Paranoia. So this is mainly for security. And what this is, is it's a plugin that will move all your PHP files out of your doc route. So this is the one that's actually been pulled into the Drupal Composer project just a few months ago. Does anyone remember, I think it was a few months ago, maybe a year ago, where there was a security vulnerability with the coder module, I think it was, where you didn't actually need the module enabled on your site for it to be vulnerable. Just having the file on the server made your site vulnerable to this hack. I see that look and it's terrifying because you have a ton of modules that may not be used, especially when you're in a situation like this, where some of the modules that we include in our Edison-based platform, they're not all required. The good portion of them are, but some of them are just optional modules, but they get pulled into every single site. So if there is a vulnerability in one of these files that doesn't actually need to be enabled in order to execute, now we've got a security vulnerability on every single site. And yeah, it's frightening, especially in this day and age, with all the hacking attempts that are out there now. So this is a great and easy way to fix that. All this does is it runs on a Composer post command, and it just takes all your files out of the doc root. And it puts them in a folder outside the web root. Obviously, simlinking back and forth. So your web server thinks that they're all there. So your site still works. Everything's good. And what it does do is it leaves your assets that need to be there in the folder. So essentially everything's simlinked. Your assets are actually there. They kind of exist in both places. Again, to require it, you just require it like you would any other package. And this is kind of what you need to add in. I'm hoping you can see this a little bit more since it's a little bit less code up there. But essentially you're describing where your directories are and what you want them to be. So we actually put our app directory outside of the web root, and that's where when Composer runs, it puts all of the code there. Like you would hear, your Drupal module is going module. It can trip modules. Your custom code can go in certain places. And then below there you'll notice you have Drupal app root and Drupal web root. So you define your app root as app. You define your web root. We use doc root, which is the standard AQUIA directory for their web root. And essentially the plugin will go and look for all your source code and then put simlinks to your web root to make sure that there's actually no source code there. And all your assets will be moved over as well. So if you're developing this locally, you can actually run these commands too. So if you're adding a new file, you can just add it to your app root like you would and you could kind of treat that app as where you develop. And then say you add some theming assets, some logos, some images, what have you. You could just run this command and it will actually automatically put those to where they need to go. So it's a very handy tool that we use on all of our sites now. It's relatively simple and easy to use. It doesn't require much configuration and it drastically increases the security of your site in my opinion. So again that's also open source that's been pulled into the Drupal Composer project. So it is a separate package. I think you do need to require it specifically for that. So now I want to kind of talk about our dashboard. This is kind of where I've spent a lot of my time over the past two years kind of working and figuring out how we want this to work and making it a better developer experience. Make it easier to manage all of these sites and manage all the integrations with the different things that you have within the site. Unfortunately, I can't kind of go through a whole demo here but what I plan to do is kind of show you some screenshots and kind of explain to you how we design this and what the capabilities are. And then I'd like to talk a little bit about our thoughts and trying to open source this and seeing kind of what kind of needs you guys might have and working for other companies and some of the things that you've seen and to see if this is kind of something that you guys think would be worthwhile to be in the open space. I'm a little biased but I think it's an awesome tool. So essentially this dashboard is a place where developers can go and kind of see a snapshot of everything that's happening. So on the screen here, this is kind of the main site page. So I can see a lot of things from here. I can see what environments I have on the site and then I can see the status for a few different things for every environment just at a quick look. I can see when my last build ran. I can see who ran it. I can see if it passed or not. I can see when my tests ran. Did they pass? Who ran those tests? There's also, I don't know if you can see it's a little small, there's a log link there. So if your build failed, the next logical question is why did it fail? Well you can just click on that and we'll actually open the log that we pulled in from our CD system so you can actually go through the log and see what happened. You don't actually need access to our continuous delivery service to actually go in there and try to figure out what's going on. You can just click a link and know. So that is the site there. There's a few more things that are kind of, I've cut off in the screenshot but below here essentially would be, there's a few list of views anyways that are just like builds. So you can see all the builds in kind of a different view. So not just the latest build, but all the builds that happened. Also during all of this too, we have an event logging module where every time something happens, we fire events. We have this event module that will actually log a lot of the things that will happen. Your build was kicked off so we create an event. It's kind of like an audit trail that you can kind of go through. You can see what happened over time. You can actually search, go back, see kind of history of what happened. And so this is just a, we do have a kind of an information icon here. We're working on kind of making that a little bit more visible here but certain things that you may need to know about your site here where's our repository. So we have links here to the GitHub repository for the site and the profile. There's also links to Travis. If you needed to actually go into Travis and see the testing results on this, you can do that. We actually don't import the logs for Travis because there's not a lot of harm in giving people access. If you have access to the GitHub repo, you have access to it in Travis. So rather than importing that, we just kind of make links there. So those links for the Travis test actually will bring you to Travis so you can see the logs, see why your test broke, see what happened so you can kind of troubleshoot it from there. So this is kind of the environment view. So you can step down and get a little bit more specific into your environment. So say you're, you know, you're working on a brand new feature. It's mostly in the development environment. You can kind of pull this up and this gives you a little bit more information about that. You can kind of see the URL for this. So you can kind of easily click there. You can see what version of the platform, which is what we call our Edison base. Is it behind? Are you not on the latest version? You know, those are kind of all good things to know. It gives you kind of some versions of some of the other tools that we use like the, you know, this scale version. So as I mentioned, the hydration scale comes from a specific repository. If we want to update, say, some of the configurations that we want to store at a baseline level, we can update that scale repository and then, you know, repopulate the repository with these new configuration files or settings or whatever we want to add. So that's version just like anything else. So we can kind of see exactly where that is. So we can know as much information about our environment and the way it was built as we possibly can. We also have some information here. We have kind of a connector module that we use that gets installed on all of our clients, and that's essentially integration to an API within our dashboard. That gives us the ability to kind of go out to these sites and get information from them. We can see what kind of, what modules are installed. We can, you know, get system level information, PHP values. We can see what our memory limit is. We can see all kinds of things. Essentially anything you want to know about your site, you can put in the module and then it can send back to here and it records it on the site. So it's a really handy tool for really knowing everything about what you need to know about your sites and your environments all in one place. It also lets you build your site. We do have features where if you make a commit to a branch that has an environment on your repository, we'll build that automatically. But not everybody has that. Say there's a project manager who is managing kind of releases and they're not actually committing, but they're, you know, they know something's there. They want to rebuild the site for some reason. They can go into the dashboard here and, you know, fill in, the defaults are usually fine. And you click trigger build and it will go out to our CD system and it will build the site. One thing I do want to note is that we've kind of built this all in a way that it's all pluggable. We've really utilized the plugin system in Drupal 8 as much as we can because the way we saw it is, you know, a lot of these things, regardless of where you're hosting and what your CD system is, there's certain baseline functionality that you need in a tool like this. You need to know what sites are. You need to know, you need to be able to build your site, things like that. But, you know, who builds your site doesn't really matter. I mean, it matters to us and it matters to you, but like if you use somebody different, that's fine. You still need to build a site. So we've built that framework around that and everything here is a plugin. So we've created a plugin for our CD system or our Edison CD tool that requires certain things to send a build. So say you want to use a different tool, you know, CircleCI, you've got your own CD, you want to use any continuous integration system in here, then you don't have to kind of write everything from scratch. You can just, you know, write a plugin that exposes just the things that you need to build your site and it integrates all into this form and you can just kind of send builds from here. As I mentioned before, we have a connector module in all of our sites where we can go out and we can fetch data. Those are also all plugins. You need to fetch a different type of data from your site. You create a plugin for the dashboard, you create a plugin for the client module and then you have that communication. You know, you can pull that data from here. Where you store this data is actually pluggable. Right now, like certain things we store on entities in Drupal. Certain things we're going to store in S3, certain things we're going to send to other places. So where that data is actually stored can be defined as a plugin. So any information that you want to pull out from your environments and from your site to give you a better idea of what's happening and what the state of these sites are, you can get that information from here. We do have automated processes that fetch it on a schedule. But if you know that there's something changed and you want to get a refreshed look at how everything is, you can just go to the dashboard and click a button and pull out all the information you need. You can also run tasks from here and tasks. For the most part are kind of drush commands that we want to allow vendors to run. We don't necessarily want to give drush access to these sites. We really want to limit access as much as we can. So this is where that Rundeck implementation comes in. Rundeck is actually just a task runner so we can define a bunch of different jobs in Rundeck and pass it variables and then Rundeck will actually go and run all of our stuff. So if we want to say, alright, you can clear cache from the dashboard, you know they can do that. This is a development environment so I want them to be able to say, alright, we just completely screwed this whole site. Let's just start over, run a site, install, start from scratch. You can do that. We use Drupal-based permissions so obviously we don't allow them to do that in production. That would be a very bad idea if a developer accidentally reinstalled their site. So it's just ways that we can enhance the developer the tools and the ability to do the tasks that they need to do to make their job easier but we can kind of limit the amount of things that we give access to and we can limit that access that they have so they can't inadvertently do something that would be bad. So essentially, I wish I had a demo I could run through but hopefully you got kind of a good idea of what we built here and where we've run into and I think we've left a good amount of time and that was really on purpose because I kind of want to kind of open it up to a discussion and for sure, if you guys have any questions about any of the things I've talked about you can ask that and before I do that though, I just so a lot of those things that you saw, like some of them are very specific to us we, as I mentioned, we tried to build this in a way that's as generic as possible and build a framework around the things that you need to manage sites and then make everything beyond that a plugin so that you can write a plugin that is actually for your specific needs and then use it that way. You know, the problem with this in open sourcing this is and we've had a lot of discussions in our internal team about how we want to do this and we don't really have a great plan because this isn't just like a module where you install it and they're like, hey, install this cool thing and in five minutes you're going to be managing thousands of sites it requires some configuration it requires a plan to actually work it into how you're managing your sites so there are certainly little bits and pieces that we could pull out if that was deemed kind of the most beneficial thing to the community is to pull out bits and pieces of this but I mean, essentially what we want to do is we feel we built a lot of really cool tools they're working very well for us as we were kind of bringing more sites on board you know, as we were building these tools we kind of let a few sites in to kind of, what we call a pilot phase so we could test out a lot of these features we could see what works, what doesn't what brings value, what doesn't we think it's cool doesn't mean that it actually is going to benefit end users or the business who actually wants this site so as we build bring more and more of these sites on the system is getting stronger we're getting more confident in the way we've architected this and now we're kind of trying to figure out how we can give all this work and effort and things that we've built into this and put it out there for people to use so that's kind of what I want to start the discussion off with if people have questions feel free to ask them if people have suggestions on like you know, say hey, that sounds really cool I would actually like to use that then like, I'd love to know about that so Yeah, actually above the that looks really awesome I'm wondering whether for the site management component when you started this I think if you consider using AGR for that and if not, why not because it does a lot of it so it just looks like I'm just wondering why, yeah Yeah, and to be honest at least amongst me and the people I work with it wasn't necessarily a consideration mainly because I mean, to be, I know a little bit about AGR, I've never actually gone through and used it but essentially, I mean we have a lot of the hosting things all done, we host an AGR that's not going to change a lot of the things and maybe it was just our view of it but we kind of thought that AGR was really that other side of alright, well, we're going to host here this is how we're going to do this whereas we already had a lot of these tools and these build processes and this is kind of just like a UI that kind of fits on top of the tools that we need we also, at being a large organization there's a lot of stuff that I don't see and I don't know about in what they can use and how they have the structure for compliance in certain things so I'm not sure like a fully open source hosting system end to end is something they considered I'm not sure but essentially we kind of had certain criteria that we already had and that's what we had to deal with and then we kind of built beyond that what was already there basically like kind of like it's a different target we focus on the UI layer gives you more freedom because like I am an AGR developer but it is a monolith, you get it all or nothing and it's like they all depends on it to each other so by focusing on this layer I can see how you're talking everything together the things you already had and that's it like AGR personally when I think of AGR I have my own hosting here we host with AGR and that's not going to change not anytime soon anyways so it's not like we could rip that all out and be like oh well we have AGR we can just do all of this not going to happen unfortunately but we did need certain tools and this thing has kind of grown into what it is now like it started off with just a way to say hey here's a reporting tool we pointed our github webhooks and our Travis hooks at this so we can report and then over time it just kind of built as we needed certain functionality so I mean I was talking to John earlier and this is kind of one of the things that I want to bring up and discuss especially with you guys is this experience AGR is this monolithic thing as far as I know you can't really separate out some of these things you can't take components of and say I don't want the hosting part and correct me if I'm wrong if you're like oh we've already you could do that no problem that's just my look at it if there's components of this that would enhance your project that's what I want to talk about and say hey because the reason we haven't just gone out and say hey here's an installation profile for this management tool go for it is because we've actually tried that not me personally but other members of our team did that with Drupal 7 and they put a lot of time and effort into building out these tools and they put it out there and nobody wanted it it's like it's called workflow tools yeah and I work with Dave yeah and I talked to him about this and essentially like when we were talking about this that's kind of what the thing was we put a lot of time and effort thinking we did this everyone's going to want this it's cool and then like it kind of fell on deaf ears right so we didn't want to do that right I love contributing to open source I love whenever I can commit code back I really enjoy doing so I don't want to waste my time if I'm going to spend time doing something I want it to be impactful I wanted to bring value so that's why rather than just doing that we're taking this approach and this is kind of the first step I'm essentially here to kick tires and to kind of give you guys a glimpse to say hey we don't want that but we could use that how can we get that part of it and that's kind of what like I said no one's going to just take this and drop and no one's going to take the dashboard and drop that in and start using it right away it's just first of all it's integrated with our tools we made it pluggable but you at least need to write plugins for the things you need now when we started this we kind of thought there could be community plugins for this and that and like the community to build out the tools that the community uses more often than like we would use and maybe that will happen but I mean if there's certain things that you guys think could improve agar or dev shop or anything then like let's talk because we've already built it I'm more than happy to to toss code out there that would make your tool better you're approaching it from the right angle meaning like the plugins can really break me down to this core component so that's cool would you elaborate on the deploy form like what's happening would you split back to us? yeah sure sorry I'm not very power point proficient alright so essentially what's happening behind the scenes here there we go so everything like so we have a plugin for our rcd system and what we do is we have a form a few api endpoints that we open and kind of what I'll call the framework and then we implement in the plugin the form part the submit part so the form is actually created in the generic module but then there's a class in the interface for the plugin that says define deploy changes form so whatever parameters you need the system to deploy you can define them in your plugin and you don't have to define the whole form you're just defining this middle section here so those are all things that are specific that we have that we set parameters on the payload based upon that and then when you hit the button there's a submit function that calls the submit function should probably go back to where it's being recorded so that calls the submit function in the generic form that just calls the plugin form that you have so everything that happens is in the plugin but the core framework of that is so then in the submit function my plugin knows what I have to do it knows I have to send it to a certain service so it does that and then it gets the response and does whatever it needs to do so what is this one going to do this one is going to call out to our cd system send in the payload whatever I check or don't check here and that essentially our continuous delivery system knows now we're going to build our system so they're going to build the site and push it out to Acre and deploy it so the branch source is the branch destination is the environment? yes the destination is the environment so I mean essentially these are all like if I'm on dev those are going to be dev stages just a confirmation of those so this is specifically written to interact with your cd system? yes because that's the plugin which is our Edison cd which is based on iron.io so like if you wanted like whatever ci tool that you want you create a plugin and then just define what parameters you require to know what to build how to build it and then on the submit whatever action that needs to take you would just call it there so we also open the plugin as a field on the environment so like if you have different environment or you have different sites and you're using different tools like we're all one organization that we're building this for so typically we use one system but say for like an agency you have different clients out there say they want to use this tool for this site they want to use this tool for this site you can actually set the plugin because the site is actually just an entity so it's got a field on it where you can actually define what plugin that you use for your ci for say so I mean our goal and our thinking was deploying a site that's a pretty generic function right so we built all the wiring around what it takes to do that and then the specific things like it's up to you as for whatever tool that you want to build and for like some of the things that are used most by the community it was our idea that there would be a community plugin for that and then it could be customized and whatnot for specific use cases that's kind of a serious thing that's kind of hard to show the plugability in a bunch of screenshots just like it's one system really it's not your system I mean maybe and that's a good feat I mean maybe I should have gone through a lot more code on how it you know yeah and maybe I and if it's something you're interested in I'm happy we can open up a PHP storm and I can show you kind of how we've architected it so you mentioned Edison CD yeah that's kind of just what we call our continuous delivery system it's like our automation tool but that's not a part of I mean that's what integrates in this that's actually the plugin that defines this is the plugin that we wrote for Edison CD which is our continuous delivery so all of it is what you're helping to open? yeah so we want to open source that too as like a separate integration tool part of that is like the aquea cloud API integration which is actually my colleague that works on that part of the system actually has open source that I don't have a link in here but that's a plan too and not necessarily have them open source together as a package but to you know open source that and open source this and you know kind of put out documentation and content and talks like this and blogs about how we kind of put those two things together but it's just kind of roughed on anyways I just was curious about another aspect of something you talked about to switch gears you talked about having some software I'm curious what the format is on each of the sites that you're managing to expose the information about that site like the modules that are installed I'm curious what other information is exposed in there and if there's any limitations is that the module or something else? So essentially that's just a module that we re-include in our Edison base that gets pulled into every site and it's just an API and there's configuration in settings in there that points to the API that we have in our dashboard here and that our dashboard knows about the API and points on that site too there's obviously some authentication back and forth to ensure that you know it's just systems that talk to each other but essentially it's just a module with an API and they talk to each other and you can literally send it any information you want we pull out lots and loads of information we literally send back every single not just Drupal module but PHP package and version which is really easy to find through Composer and we send that back that way we know what's out there because a lot of times what we have we're a problem with so many sites you know there's a lot of sites some of our bigger sites they're active all the time they're constantly getting updates they're constantly getting attention but then there are some sites that there was a project and it's been sitting out there for two years well like right now we don't have any way of knowing like what version of things are like are there security updates if it's two years old I guarantee you there's security vulnerabilities in there how do you handle that one of the things I didn't really touch upon the main goal is with this is also push platform updates so for exactly that case like that site's two years old no one's touched it we need to update it we're going to update it with a brand new version of everything we thought it was going to be tough and we actually have a system that will do that I've successfully pushed platform updates to multiple sites it's drastically harder than we thought it would be and we thought it would be hard so there's a lot of things to take into consideration there so we actually kind of have to rethink and retool a little bit about how we're going to do that but we have kind of the base idea but having that module there that can communicate with our dashboard and can give us the information that we need to know what we need to know about that but I mean there's no limit to what you want to put like I said we made that so here each one of these is a plugin and that plugin has certain things and the plugin can also determine the storage so right now most of what we have is stored in Drupal entities but you can also store it you can store stuff on S3 you want to store things in a NoSQL data there are no limits because you can implement whatever you want in there so any information you want to send back you can do so I mean if you're looking for things that are useful I think that one is really useful it sounds like it's componentized enough to release that as open source but the problem with that is it requires the dashboard or else being available on each of those sites then you could take that from there yeah it could be is it the site itself like the module that's running on the site exposes an API that your dashboard calls to get the information and that spits out like JSON or something exactly and it's both ways so it also knows about the dashboard so it can say hey I'm going to send you data and then dashboard could be like well I haven't gotten data in a while I need data right now to see your fresh look I can pull it so it's kind of a two way API there so that could be handy on its own so I think we all need that right now it's a fine server already just two modules pull that out as two independent modules I mean I have if that's something that would be helpful for the people and this is kind of why I wanted to have this talk right because I don't know what's useful to us because I only know what I've been doing for three years which is stick my head into this so do you have client modules for all the Drupal versions no just for Drupal Drupal 8 I don't know if they have something for Drupal 7 or not but I've only worked on one for 8 so I'll leave it at that I mean in this system everything I've talked about here is exclusively D8 we kind of built this from the ground up for D8 so yes everything is composed, composer is absolutely fantastic you can use composer for D7 you can you can we do we use composer manager module and stuff but yeah exactly so yeah I would definitely consider pulling just those two out I mean that would be simple right because that's just you have some kind of UI for this seeing this data in the modules so right now so that's this most of most of this here is we haven't done a ton of work on like a way to do this we also have different views for the modules and the packages so you can sort and you can do lots of different things I guess we're almost out of time but you know essentially it's displayed here you can see that and where do we find out more it's not a website to see yeah you know it's I guess you know you can ping me I guess I don't really have a great answer for that question but I guess you know coming soon Twitter if I'm gonna announce anything I'm gonna do an open source it would be on Twitter I don't tweet much but if I'm gonna open source any of this I'll absolutely make sure I put that out there and I'll make sure to blog about you can check app innovations blog I don't blog there often but you never know I may want to do this now with all my free time so alright I guess that's the hour thank you guys for attending