 Beach room so again, we'll have safe and happy deployments using automation in this room probably getting started in about like three minutes So how you guys doing today? Good Everybody wake Everybody got enough food Steve's challenge how many of you have been to a word camp before? Okay, how many have you been have been to this word camp? All right, so we definitely have some new people I'll give everybody else a couple more minutes just to get in here a quick poll though. How many people know what deploy means Awesome, you're my audience. Okay. This is great. How many people don't know or want me to do a quick Prime around what deploying is okay? So deploying just means getting your files or your code changes out to a server. That's what I mean when I say deploy Cool probably start right Yeah, or shut the door Yeah Okay, so we're gonna go ahead and get started in here So first of all like let's give a warm welcome to Aaron he Aaron is the director of web engineering at Zeke He builds publishing solutions and architects robust deployment and delivery processes. So super smart But today he'll be speaking on a safe and happy deployment using automation, so let's give another warm welcome to Aaron Thank you, Tish. Has anybody ever deployed code? And and you and you've gone through it and it's been maybe a slow laborious process Maybe you've forgotten the steps to deploy the project if you work on multiple projects Maybe if you work on a team, you're wondering who deployed the code last Maybe you wonder what code was deployed last and You're wondering what exactly? What code is live? What what are the build steps? Maybe you have multiple build steps for each project? Maybe they're different for each project or maybe you actually Crash the site when you pushed up files. Does anybody done that or maybe that's just me. Maybe it's just me. It's just me So if you do that, how do you fix it in less than a minute because when that happens you're jittering you're like Gotta fix it, you know 5,000 people are looking at the site Well, definitely this is not the fastest way. This is this is slow transmit of FTP. That's that's gonna take you a while Also SSH Indian, you know trying to edit the files live also not a great key for Not a disaster This is this is what I'm talking about having a one-button system where you can deploy and you can revert Everything is automated So first so that's that's why we're doing this. That's why we're having this taught Have a system and it doesn't have to be my system. My system could be an inspiration for you You can I'm actually gonna show you some of the tools that we use But just have a system these are the things that I've identified about the system that we use and the things that I strive for In building the system that we use have a system. That's consistent You want a system, especially if you work on a team that everyone can deploy Everyone knows how it works because it works the same for every single project You want you want it to work the same regardless of who is using it So whether I'm using it or Steve's using it or Jacob one another one of our developers is using it Everybody's doing the same thing the same way because they're not actually doing much of the deploy. They're just pressing the button And you want the deployment process who who has worked had to work with multiple hosts You know what I'm saying you're not just working with one host You have multiple projects and they all live on different hosts because that ultimately is typically up to the client The client is the one paying the bills for the hosting most of the time in most setups so the client picks WP engine or liquid web or page Lee or Precibel So your deployment system can't Be tied to any one of those hosts. It has to be independent completely of the host You need a very transparent system. This is very very very helpful You need an audit trail. You need a way to see who pushed code. When did they push it? What did they push when it was deployed etc? And you can see what's currently live. You need a system. That's fast SFTP is very slow It's very slow because files go up one by one and if you're pushing up 2,000 files It's going to break the site it's going to break the site because the first file that gets pushed has code changes Or is calling a function that is in the last file? So you can't be sure that the site is not going to come down So you want a very fast system. You also want an intelligent system Something like our sync or get and we'll talk about that in a second because our sync and get are very intelligent They only send the files that have changed since the last deployment So you're not resending every single file every single time. I've done this don't laugh It's I've I've had a two-hour downtime for a site because we pushed all the dependencies every single time It was extremely extremely frustrating You want us you want a system like I've kind of alluded to you want a system? That's very simple deploy should mean hitting one button reverting should mean hitting one button So everybody knows what to do So if there is a problem You just go into your deployment process and you hit the button and everything is back to normal And then you can figure out what happened you can figure out why it broke if it breaks And lastly you want a system that is automated everything has kind of been alluding to this especially with the title of the talk But you want a system that's automated you want a system that has no user input or action required You want all the steps automated? Compiling building checking deploying testing notifying the team all of these things need to be automated So I will demonstrate or I'll show an overview of our system We'll go over this step-by-step. It looks a little bit complex It it is complex, but you you do the complexity now so that when you go to deploy it's much simpler so First of all What we're trying to do is we're trying to get code onto the server and in our case We're using get you don't have to use get but I do recommend that you use a version control system Very very important so get I will not go into this deeply Definitely check it out if you don't know what I'm talking about have your code in a version control system that way There is a process and there is a way to commit your code in different different commits different stages So to manage all of our deploys we use a system called buddy. We used to use the system called deploy bot We have since moved on to buddy and I'll I'll show you this in a second. I'll show you some of the interface here I really like buddy. I considered when we were rebuilding our deployment system I considered Jenkins. I looked at a few different options Buddy is what we we what we really were we're happy with I really like the interface I like how much configuration How many options there are available? There's a lot of different tools that they that they use the end of their feedback is very or their support is very good So if you have feedback about a feature you'd like to see They've been very very supportive So this is a little bit about our system Or this is this is our current build process as it stands right now And I actually just added the very top one so a PHP syntax checker because I thought hey It'd be kind of cool like as I'm going to deploy I don't accidentally goof up with something really simple like a frigate a semi colon and the site comes down So adding this in is very very simple I had the little plus sign and I can add in a PHP step So this is each one of these is essentially like a docker container that spins up and then it executes a command So and if it fails the the whole process stops and it lets us know So with the PHP syntax checker it loads up the code it scans the code and there's there's Tools that people have written to do this stuff So I'm just using a tool that somebody's built and a PHP docker container and I run that and then the next thing Is a security checker? So there is a tool out there by sensio labs that actually will check all your dependencies Versions for vulnerabilities, so it's really really helpful to see if any of the dependencies that we have Have have reported a vulnerabilities, and that's a good alert to let us know Hey, we should update this so we don't have Insecure code out there We also so this is just a zoomed-in view of that We also can set up and build environmental variables Anybody not know what it invent environmental variables are Okay environmental variables are Ways that we can alter the code base based on the environment. It's in so if I'm working on my local machine I want to set say the stripe key, right? So I've got a payment gateway. I don't actually want to accidentally charge a user My local machine that would that would not be good, right? That would be bad That would be bad, right? I Don't want to accidentally send a thousand emails to people via send grid because my my API key is hard-coded in the code Environmental keys are very very important So the way that we manage environmental variables is through buddy, and we use a file We use a package env.php is a package And We can build the environmental variables on the fly and then place them on the server every time So the live server has got the right stripe API key and I have the test API key So it's very very good So we don't accidentally get the wrong things happening in our local our test environments We install dependencies and I'll talk a little bit more about this in a little bit but we manage all of our dependencies through composer and NPM so all of our PHP PHP Dependencies and packages are all managed through composer So we don't have those in the repo the plugins the plugins directory Most of that is not in our repo. It's clutter. It's noise. It's stuff that we don't want just Clogging up our repository. We just want our core code because if we're bringing down a kismet for example We don't want to store a copy of that in our repo We just want the version that we want when we deploy we don't want to have to manage Each individual file so we can we install those dependencies. We compile assets so we compile JavaScript CSS or sass do anything else on the front end. So all this is happening Automatically every time you deploy so you don't have to do these things make sure they're done Increment the CSS and then commit that and then hope that that gets up there And then moving the files of the server. This is the whole point. This is the deploy part So moving the files to the server and you can see right there on the second to last line Zeke pusher So that is the system that I built which actually does a couple of these bullet points and does the moving of the files of the server and We also do visual regression testing which is really really really cool because if we're pushing files out there And I'm refreshing the page right like I'm refreshing the homepage to make sure it didn't break That's good, but I'd rather have it automated I'd rather have it so after it's done then buddy kicks off the visual regression testing system We use and it checks not just the home page, but it checks the about page. It checks the post page It checks things about all of those pages to make sure that I didn't just completely ruin the site And then after everything whether it's good or bad we notify the team via slack And this is what actually deploying looks like I click run now and it starts that whole process That's that one button Initiation that I'm talking about and then as it's as it goes through You can see each step so it's very transparent You can see each step you can you can click for more details in each individual area So you can see the top three security checker the ENV file And the the moving the ENV to the Zeke pusher those are all completed But I can go look at the logs to see what happened in that step And so eat and then in the Zeke pusher the Zeke pusher is an SSH command So buddy SSH is SSH is into our little VPS box and it starts running a command with a git Hash and then you can full screen it and you can look at what does that look like so on the second line I'm running the deploy script with this is the commit so this is the commit we want to do so whether or not That's the latest one or it's a reversion and we want to roll back doesn't matter It's just here's the code base commit and then it starts running through and you can see this is where it's installing composer And then it's pushing to WP engine via their git push system And then a little bit just an audit trail so I can go back and I can see for the Zeke BB demo I can see these are the these are the last deployments that happened And I can click into them and I can actually see all the logs about what happened in that step So it's very transparent And there's a lot of actions available like I said, so if you can think of it you can do it pretty much And if you can't do it with one of their pre-built Actions you can you can write your own docker container and you can put it up and you can actually pull it in so It's very helpful and Buddy's a newer service And I've been in contact with them because while I was while I was evaluating whether or not this was the right system I was asking some questions I was kind of pushing pretty hard to see you know like is this a system for us So I've been in contact with them and I actually reached out to them and I asked them I said I'm gonna be doing a talk at WordCamp Orange County I said do you want to offer some sort of discount and they said yes, absolutely So they're gonna do 25% off any plan If you sign up within the next 14 days just tell them Orange County WordCamp And there's a like a little support like chat icon So after you sign up just say hey Orange County WordCamp, and they'll they'll set your plan to 25% off so This talk is not sponsored or affiliated by buddy and anyway even though it kind of sounds like it I just want to put that out there I really like the service and things I'm passionate about I like to you know Tell people Bob because I like to share things that are awesome So I really really like it. It's a really it's a really fluid interface Which is not been my experience with like deployment managers and Jenkins and other ones now There are other ones out there like Jenkins deploy bot. So Figure out which one works for you But use something don't just use the command line and hope that you know, I'm pushing the same thing as Another person on my team. So that's the Zeke pusher And so this I'm sorry. That's buddy Next buddy kicks off saw that step with a Zeke pusher. So it initiates the Zeke pusher and we get to Zeke pusher Now what is this right? This is kind of a black box It's a VPS so it's so in a nutshell it builds and deploys files Okay, it runs on a VPS whether it be line node or digital ocean for us It's running on digital ocean, but it's just a small box you set up It can be whatever you want it to be. It's really just a collection of scripts It's it's a couple bash scripts that I've written, but it's an interface between the server and our code and it's It's an interface between like WP engine or liquid web and our code So it uses get an rsync and that is dependent on that's kind of where it levels out based on the host that you have Some hosts of support SSH access great You could totally use our sync some don't but have other options like get push like WP engine Okay, that's also very good And how it breaks down is to repose on this Zeke pusher box for each project So each each project has a core Repository and a deploy repository and why I had to do this Was because I needed two different things I had my code in GitHub and that didn't have my dependencies It didn't have plugins. It didn't have the vendor folder for PHP packages It didn't have the compiled assets because I wanted to keep those out I didn't want to clutter the code base with all of that stuff because that's not really what we're we're tracking So it doesn't have compiled assets. Like I said, it doesn't have dependencies. It mirrors the code base It's directly connected to GitHub versus the deploy repo. It's not connected to GitHub at all It's it doesn't have a remote to GitHub it mirrors the server So you've got these two repos this one mirrors the server this one mirrors code base the code base This one has the compiled assets. This one has the dependencies. This is where everything has already been built and The way that it works is there actually are synced across so they're on the same box they're sibling folders and Every time I push a deploy It runs the deploy script and the deploy script goes and it grabs whatever commit I want from from that core repo So the core repo is connected to GitHub it grabs that commit it sets that repo at that commit level Then it runs it actually deletes all of WP content in the deploy repo now It's all tracked and get but it it deletes the whole Directory because then it resets everything from the core repo with our sync so it brings everything over fresh and then it starts kicking off the build process and it kicks off the build process with Composer so composer is the dependency manager for PHP It's very very very very helpful if you haven't heard of this after this talk go check it out. It's it's how you can manage dependencies with PHP and there is a website that's Kind of a sister to composer which is packages So people take it's just like the wordpress.org plug-in repo really it's people that aren't in the wordpress space but they're building PHP packages and they put them out there open source and They're all connected to GitHub but this is like the the archive list of all of those packages where you can go to see them and then you can bring Them in just by typing in composer require a seven autoload which is a package that I built So you can bring that into your package without having or your project without having to copy files down or you know un install things Manually and you can set which version you want of it so you can lock your project into a specific version and You don't accidentally get an update if you haven't tested it which is dangerous because that can bring your site down And then so we talked about packages and that's like the list of all the Thousands and thousands of PHP open source packages you can use for free now If you don't if say you build your own PHP PHP packages for your own business or for your team But you don't want to open source it for whatever reason There is so what what runs packages essentially is status or the status is the private packages So this is where you can actually set up your own private packages Which is what we do for some of our our PHP packages that we just don't want to open source for whatever reason Maybe it's proprietary to the client whatever So we run this and that way we can put our packages up and they're available to just us and Then taking it a step further. So how you manage? plugin dependencies or plugins as dependencies is through WordPress packages and What this does is it mirrors the wordpress.org plug-in repo, but it attaches a composer file to each of those plugins So you can specify which plugins you want just in your composer files So everything's being basically could the composer file is a manifest then of which dependencies this project needs and Then taking it even one step further. I know we just keep getting more complex Sadis press so sadist press is just like sadist with a private, you know Listing of all the packages, but this is for plugins that are not on the wordpress repo So if you have purchased a license for a plug-in and you have it you can set up your own sadist press box And then you can receive Receive that plug-in through there. So every single dependency we can manage in some way and then We want to do some front-end building. So we did the composer. We did the back-end dependencies, but we also have some front-end building steps to do we have We have you know sass files. We have JavaScript files those need to be compiled down first You know speed optimization Maybe you're using more advanced JavaScript technologies where you can use Requires or imports and you want all that to be compiled this handles it so we Still use and we used to use a system called gulp. It's very handy. It's a task runner go check it out I'm not really gonna talk much about it other than to say this is one of the build tools that we use We are also using webpack, and I think we're moving more towards webpack. Is that right Jacob? Yes, I work with Jacob awesome guy Yeah, webpack is is a really cool build tool. It's a lot more complex though It takes a lot more to get up to speed with it, but it's very very very powerful So that composer actually has hooks in it where you can run scripts at a certain point. So Via the composer manifest file we can actually say okay when you get to this point run npn run build and that actually goes and it Executes whatever front-end build tool that we need So we've done all the building we're past that part again. I didn't do anything. I just hit a button So the Zeke pusher has done all this composer has done all this and we had moved over all those files And we went through that process. I just described Now we're about ready to push the files what we do is we commit all those changes now We have everything in there. We have all the dependencies. We have all the assets everything's in here It's not connected to github remember. It's just on this box So this is all committed and it's important because with WP engines get push or somebody else's get push You need to be able to do a diff and you need to be able to say here's what it was in the past version Yeah, we deleted everything, but we brought it back because Get is great about doing diffing. So even if you delete everything but bring most of it back It's only gonna say well only things that have changed are you know these four files or whatever so we commit it and Then we push it to the server and this push the server depends on the project So we're either gonna do our sync which I said like I said is it is very cool open source Shell utility tool. It's been around forever or we do a get push to server depending on the host and Those are very very very very fast. They are seconds. They take seconds not minutes Because of that intelligent file diffing you're only pushing the files that have changed which is Which is awesome when you're installing a bunch of PHP dependencies that can get pretty massive and take a really really long time So now we have the files out to the server So now we kick off the visual regression test We use ghost inspector. I really like this tool. Sorry guys. I wasn't able to get you a discount Not that they didn't offer it. I didn't have time to contact them But I do like ghost inspector. We tried a bunch of different tools This one is really neat because of how easy it is to create tests. There's no coding involved You install an extension in Chrome you you load up the site you want to test you hit start recording And then you just like make assertions you click around the page at different CSS elements and different divs and stuff and it records those things and then you can say okay I'm done and then it ships it off to your your suite and then you have a Visual regression test. It's it takes less than a minute. What's up? So the question was are there any tools that we can hook up into the Process that do it when we push is that okay? This this has a web hook. So all I have to do is just send I just grab the URL. They have a really good API You just grab the URL and I throw that into buddy and then it kicks it off in my build process Does that answer your question? Okay? So I really like this because it gives us confirmation now We have a dedicated QA person at Zeke which is awesome. I never worked at a place that had such a rigorous testing Setup it's really nice because as a developer you kind of get a little lazy and you're like you always follow The happy path does anybody know what I'm talking about with happy happy path not that happy path. I mean no You always you want your code to work So you're testing how everything is supposed to work, right? You're not throwing garbage at the functions to see what happens when they get stuff. They're not supposed to Our QA tester Katrina is great She definitely she is able to see ways that we have blinded ourselves to so having a QA person is fantastic It does this does not replace it, but it's a nice compliment It's a really nice compliment because maybe she's busy in a meeting or whatever and we just have to push something this just gives us immediate feedback within Minute or two minutes that all the things that we have asserted about the site are still true So the homepage is not a big giant white screen, you know things like that And then we notify the team so we just hook in with with slack or whatever hip chat Whatever you want to do we use slack and we just notify the team that hey something broke Get on it or it's all fine. So That is Zeke that is that is our process and the Zeke pusher is open-source as of yesterday I am not trying to claim that this is like you just you just hit a button and then it's all set up for you But it is open-source do with it what you will maybe it's helpful for you Feel free to open issues and Contribute as well. I would love some help on this project. I Definitely have plans to make this a little bit more robust But as it stands right now, it's more of an inspiration than then you can just you know take it off the shelf But you can see exactly what we've done. So My name is Aaron Holbrook. I'm the director of web engineering at Zeke and Yeah questions Yeah, sure sure, I'm sorry, can we get the microphone? Sorry, I don't think I could repeat everything you just said It's a beach ball for those of you watching on the video so my question was can you kind of elaborate on when Like something like a Zeke pusher would be necessary as compared to like Compiling everything locally then pushing the staging and then from staging to like a live server Yes, okay, so everybody got that question I would say the Zeke pusher was necessary for us because we were not finding The tools like buddy or deploy bot could fill that gap of that build step So you could absolutely build it locally But every single person on your team if you have teams Need needs to follow those same exact steps. So that's where I was talking about having a system that's consistent. It's simple It's it's it's there's no it's all automated, right? Like those build steps. You're talking about on your local machine That's not automated if you're having to manually do those so with with a system like buddy You can get most of the way there. The reason I built the Zeke pusher is because the options offered by deploy by by Jenkins by buddy didn't Didn't fulfill the git push to WP engine and the r-sync to the servers now, buddy has an r-sync Action, but it I needed to do certain things that they didn't really have available like so in the So I'm gonna pull up. I'm gonna pull up the Zeke pusher just to show you so the So this is the script and This is where I r-sync over the repo to the deploy repo And then the very last line so I'm doing some things here where I Exclude files and really really really important same with the git push you want to exclude uploads Oh my goodness because I'm doing a delete if that file doesn't exist on the server Which is you need if you're pushing a change in your repo and I'm removing a kismet, right? All of a sudden the kismet doesn't exist in our deploy repo It shouldn't exist on the server anymore if you don't pass the delete flag r-sync will just say it's it's more additive than it is Sinking at that point, so it's gonna add any new files, but it's not gonna delete files that don't exist So you need an exclusion To make sure that you don't accidentally delete all of your clients upload Which would be very bad. So These are the types of things that we exclude can everybody see that Okay, so we're excluding things like The ENV WP content in case somebody set the wrong directory. We don't want to delete all of WP content We don't really want to send tests Things like that And we don't want to necessarily delete so these are vendor specific So WP WP engine has a couple MU plugins liquid web has a couple WP or MU plugins And so we don't really want to remove those. That's fine. They can keep them there We don't want to add or remove object cache because maybe we don't want object cache in the our core repo So these are things that we would want to exclude this I know I kind of went down the tangent a little bit But that's kind of why I built the Zeke push your system because we needed to do more than what was available And And then it still ties in really nicely I really like having a system that is an interface for managing the ploys like buddy because you can start you can stop you Can see what the statuses of deploys and who pushed what code last if that makes sense? Yes in the back Can you can we pass the ball to her? Hello Did you consider using automation tools like Ansible probably and cistrano and what Would be the benefit or the Extra things that you might provide with this for for Ansible Ansible and an cistrano. Oh capistrano and cistrano. Okay. I'm not familiar with that Ansible I'm I'm I just investigated it wasn't a good fit for us, but I mean if it works for you great It's it's about having a system where I mean the key message here is just have a system That is really simple to use because when it comes time that you broke the site and I've done it so many times It's really you want the quickest step to get it back and it should be simple And it should be always the same process so And if you start having Requirements where you can do builds or you want to do more complex things to make your site more efficient or make your Lives easier as developers Having a tool to do that is is really the key Any other questions Yeah in the back there Yes, great talking Aaron by the way. Thank you Jacob. I have a question. I don't know him by the way He's not an audience plant On the WordPress plugins to add that you know as a dependency how you were talking about in your Project you did mention the WordPress package list and that's kind of how you find or get the plug-in that you need Can you quickly show like if I want to add like jetpack to my project? How can I find it in that directory that you showed and then what would that look like in that composer file? Very good question Jacob. I will definitely do that. I Did not tell him to say that I did tell him to give me a question if if he wanted all right So w packages so I Showed this in the slides, but say I wanted to add a kismet to my project I can just type for the I can search for the plug-in And you can see here's the results and you can see some of the versions the latest versions are up here And so if I click on this you get this little line of text Which is what you would put into a composer file and a good compliment to that question is what is a composer file? So I will look it up or I will show you guys an example. It's really hard to do I don't have a Exact mirror here. Actually, let's fix that. Is that still viewable for everybody? So this is the magical composer file So there's a couple things All right, so this is basically the manifest for the project there's a couple things here that I've Built out preferred install just means like we want distributions not the source The vendor directory this is really important. This is where all those packages those regular PHP packages will go So we put them we have a folder with most of our projects that are that is MU plugins app if you don't know what MU plugins is Definitely look it up. It's it's it's a plug-in that always runs On execution so you don't have to activate it So we put vendor files all the packages there And we can even do something special for WordPress plugins, which I haven't done here But it would look like this slash That name so any any Packages that are defined as a type of WordPress plug-in would go to the slash plug-ins area Which is what WordPress expects and that's what we want to do We don't want to place these in the vendor directory so These are a couple things that that we do and actually we'll put out an example template of like a composer file So you have an idea of like what we typically do and then you define some repositories here And this is where we get into like okay, so the bottom one is the W packages That's where we would look for the kismet. We also define our own status Press and status. That's the top one in second one that I kind of talked about in the talk and then if we wanted to add in a kismet We grab the string and we can just Put it in here and now when I run composer install or composer update it'll grab That that plug-in and we can actually do that. I know this side is really small. I'm sorry All right, so we are gonna delete a kismet and then bring it back in with a composer So this is the directory of plugins everybody got that Okay, so I'm using I term and I'm using Z shell. Yeah with Z shell. Yeah, there's a plug-in. I use antigen A-N-T-I-G-E-N Google it. They got a whole bunch of stuff that auto completion You saw was fish like auto completion fish is a shell. I know I'm getting really technical really cool though I love tools that really make my job easier the j that I typed is auto jump So you can just any directory you go to frequently You can just type part of it or misspell it and it gets you where you go. I think my Wi-Fi is a little bit slow Typically doesn't take this long But it is going and it's gonna figure out like what it is already installed and then it's gonna install kismet while that goes Yeah, there we go. So we're installing all our packages. I think I had some things in there that required a lot of other packages Does anybody have any other questions? Yes, and we only have a few minutes left. Can you get the beach ball too? I think you can throw it. Can you throw it? You broke the mic. No, I'm just kidding So when the deploy day comes so let's say you developed some future for a couple a couple of weeks And on the deploy day, how do you specify what to deploy is that a commit hash code or yep? So it's the latest on your master branch. How do you specify? Yeah, so it's on our master branch number one It's the latest commit number two So if we do here Alright, so Well, first of all if it's a serious project and there's it's already in production and whatnot We'll typically do a code review on github And then once that's passed and we've done testing and staging and whatnot then we move over to buddy and This one is set to auto trigger. So anytime a push happens to master branch. It automatically starts to deploy But When you're more on this is more of a demo site, so that's what we want every time I push to master I want it to do its thing I want the latest code out there, but when you are not doing that when you're on a production site So you saw I just hit run and then this is where you can both roll back and you can just deploy the latest commit So I can change the commit and I can go through any commit in this Codebase and I can select it and then I just hit run now and you can see It's it's going and I can click this to see the progress And I can I can look at each step You can see it's it's going through the the host So it pulls all the latest changes builds again. It's always up to date It's always up to date with with your version control system It's going to take a few minutes, right? What's that? It's it's going to take a few minutes then to deploy everything like how much does it? Yeah, I can take up a few minutes most of this process is happening in the build part and not in the actual pushing That's the part. We want to minimize the time, but yeah the push. I mean we're done. That's it And when you need to reward you do the same right you choose the yeah The revert would be essentially another commit and so I'm gonna run this pipeline But I'm gonna change the commit to this one and then I'm gonna now I'm not reverting back a commit So when you push the changes that does it go to the document route? You directly push files to document it depends on how your project is set up for Our sync stuff we just we just are synced to WP content, but for get push with WP engine WP engines get push is at the document route So that's part of why the script that I built had to move things to WP content and then commit them Are we out of time? Okay. Thank you. Thank you everybody