 Hi, so everyone going this is continuous integration Kind of a it's been marked in advance, but it's it's it is really an introduction to see I So I'll try to be clear about sort of what's in it in case you're expecting something different I'm I'm Boris. I'm a developer at previous next and and I've been a Drupal user for about six years about about four professionally and Might just get a quick gauge on who's in the room Who here is actually like no is pretty sure they're not see I is about or have built like Yeah, so you're not supposed to be here. Okay, that's fine, but it might be might be useful anyway So I'm gonna take a little bit of a different approach with this this talk. I'm not gonna Kind of do like a long demo where I just hit buttons on Jenkins or or anything like that It's not very tool-focused We're gonna try and get into some of the motivations for actually Doing continuous integration and try to get to some of the reasons Why we don't do it in Drupal projects for the most part Of those people who built see I set up. So are you doing this on Drupal projects? Can you put your hand up if you are? Okay So that's about probably about a third room so Think about half. You don't really know know what see eyes about so we'll cover that and Particularly how how it might apply to Drupal projects. This is a bit of a forward-looking topic to me because one of the one of the prerequisites for Continuous integration is a good testing and I think that's sort of a bit of a weak spot in Drupal and it's a Kind of a motivation for a lot of the Drupal 8 Core work So I think it's a good topic to start thinking about Some of these practices and I don't know if anyone is in the the agile project management talk this morning, but It was kind of mentioned that the Drupal Drupal projects are tend to be a bit behind the ball with a bit behind the Times with a lot of agile practices and all the agile talks I see it at Drupal cons for the most part tend to be project management focused and not not so much You know some of the classic extreme programming practices of which See eyes one or test driven development. I mean I'd be very Doubtful that many many Drupal projects are doing much proper test driven development and there's good technical reasons for that It's it's kind of difficult But that's sort of the the framing. So what we'll actually go through is just yeah, some of the motivations for see I wear What what problems we're trying to solve? It is not a tool focused Talk so I'm it's really about getting the idea that we're trying to improve quality and and Doing that by Not letting bugs get released basically so we'll go through some of the basics and definitions and and some some ways and some kind of first steps that I found kind of get started that that will make Later on tying things together with with a continuous integration server a lot smoother if you kind of start on the right foot so And a lot of it's about clean lines with your your practices so Deciding where your where your tests go and how they run and then What happens at the playtime and and and trying to get some real consistency around that so And of course we will cover continuous integration servers because that's where it all comes together so So I'm not going to demo Jenkins, I'm not going to Try to tell you which tools to use or give you sort of like one perfect one size fits all I'm really hoping that you'll come away with this understanding the goals and Knowing how to kind of build your own Your own toolkit like a framework to think about some of the things that Continuous integration I kind of brings up so I think it's a it's an interesting topic is like a Sort of a a catalyst for thinking about a lot of agile processes So even it's up down to the way you're dealing with user stories and then how how you're gonna actually Like I said implement things like test room and development or how you're reviewing your architecture It all sort of gets touched on so We won't be talking about writing tests. There's a good talk tomorrow particularly Drupal focused on writing tests you should go to that and and that's That's sort of the overview. Okay, so we're talking about Build build phase quality is the motivation here. So problems problems with defects So not pre-build but really these parts here where we Design development test locally then probably deploy to a test server so you should be familiar with this with this process and then there'll be review and then we fix bugs and That's when we start to get Sometimes some annoying regressions and we fix more bugs and it gets expensive and they can kind of loop and It all comes down to defects in the build Mostly well, that's that's what we can deal with with CI Other agile processors might deal with leaks in requirements or dealing handling that but This is our engineering focus. So right there Okay, so it's the quick quick review of our software engineering 101. We don't want to release defects because It's just cheaper to fix them now than later. Basically. So turns out that We want to this is some of the motivation behind agile engineering practices is we want feedback So that we can we can fix them early And you see the cost increases not only When it's the defect is early up, but also when it's detected later So earlier stage so required a defect in requirements detected in a production release is most Most costly so we're trying to try to improve this cost down at the front here so part of the problem is we do a lot a lot of parallel development and kind of developers go off on their own and and work on work on their own solutions and and You know that that's good, but it can lead to problems with Too much work in progress And obviously it has to all all get integrated at some point So maybe some of you have experienced sort of an integration hell moment of your own where You know you're working on team and and somebody has working on a feature and Then they go away for three weeks four weeks, and then when they try to actually bring it back in There's problems And So that's one that's one one motivation So we want to integrate small parts But you know obviously there's overhead with that integration can be sort of boring constantly sinking from from a common branch a Mainline and and then running tests again, and then and then going over and over So we'd like to do it, but we want to kind of Make it make it a good process something that we want to do all the time. So remove remove that friction. So that's sort of some of the some of the things we're dealing with here because And that's where where some of the processes and tools come in so we're going to try and automate Automate the integration task and try to make it not so wasteful in terms of time But but do it often so It feels good do it Integrating it can be painful and Integrating small changes reveals problems earlier that solves the problem. We're trying to try to solve and regularly produces feedback Quicker than integrating Larger granularity so That that's why these people came up with the idea to integrate all the time Which can sound crazy You've never seen it before But it can be done We with tools and with a bit of discipline And a bit of practice and and and a bit of a bit of careful thought about what what you're actually doing so So when we say Continuous integration for those that are unaware of what we're actually talking about Any change to the project and that can be some configuration or generally source code and application would Whenever it's done in a common a common mainline that would trigger Some sort of automated integration tasks. That's what we're sort of aiming for so the goal is again To not release defects into production So we do that by keeping a careful eye on code quality Project what the quality of the project at every moment so so We're also Going back to those those ladybugs Want to want to keep them in check and not let them sort of go off on their own and Build incompatible solutions You see in a dribble example Maybe use a module stack that that is against what the team is chosen. So maybe you're building a building a project with a You know Context module stack with the ties in with a may go and does a lot of stuff like that and they go and install Panels to do some layouts when you've been building a bunch of stuff with Delta module or something so that's it's one sort of The one sort of motivation is to try to keep that under wraps. So that's one that's one type of quality is Sort of site building architecture But it's mainly transparency. I mean we just we want to know what state the the code base is in everything from code styles to of course test running because Yeah, we want to fix defects early. So that's that's the main the main goal So it's a bit of a different mindset You might be used to You know having a development server that you're kind of happy Happy to be relatively unstable states at all times and you kind of I'll fix it later I'll fix it close to the launch or I'll fix it at an end the end of an iteration so this kind of makes you take a step back and and And makes you agree as a team, you know, no, we're gonna we're gonna actually maintain stability In in this branch so so wherever code gets integrated it has to it has to be clean So starting point any developer can in theory come on download the source code Trigger a build Go over that insect and and get going and not worry about you know Some broken state they have to deal with and whatnot. They should always have a clean point. They can start from so and there's some other workflow processes that this touches on around How you handle defects after the site as a team are we are we gonna you know, how many defects are we gonna Deal with how long we're gonna let the build that the state of the code go bad and Do we stop everything that might be a good option? you know just all everyone everyone stop and and fix it if it's if it's like gone a day and and and the codes the red lights gone and You might want to fix it But that's it gives you sort of a framework to make those decisions as a team but obviously the goal is to keep it fixed so It's not about just sort of installing a robot on the server that automatically deploys necessarily it's big topic and I Think it's something that we'll think about more as As triple eight starts to reveal itself and we're writing more unit testable code in particular and I think it's It's something that's very applicable to distribution development and using triple as a product platform which at trees mentioned this morning so if you're you know thinking of a startup or something like that then This is the sort of process you probably want to keep and keep in your mind when you're setting up your workflows Okay, so part of it is thinking very clearly about When when you're performing your various activities and putting a bit of structure in place So it can be a bit prescriptive workflow wise, but But there's flexibility in terms of actual implementation So you should you should be built building up these processes from the bottom up and But but being careful about about we do and we'll go through some examples, but So some clear definitions of what what I mean by build we kind of throw these words around in CI and Don't really think too much about about what it means. I think it helps to be a bit more concrete. So build is The confusion is because we use PHP a lot of the a lot of times we'll sort of you know Deploy with version control or something like that and the source code will be sort of considered the deployable the deployable item, but I think you'll notice that you know if you're using sass or Again, this is something that's probably gonna come up more with triple 8 There are things that you always have to do sometimes you do it in the deploy but Something you have to do to the code base to actually get it executable or runnable and so It's a bit it's a bit abstract and a bit waffly, but it's it's some it's some result of So you start with your source code and then you do a build and then you'll have something at the end and that can be You know some some process code with sass run on it. Maybe you run tests. You've got test output and and And then that the executable part is ready to deploy to any environment preferably so So test testing you know kind of enough said but we will Test as part of a build process and then have some test results and that will be considered You know, that's an artifact of the build and it's about trackability So when we do automate this stuff with the robots and get Jenkins to to actually start or whatever you're using to start Start running these processes these builds then you'll be able to trace use this data in much more useful ways the test test outputs So and then the employees the easy part You know what it will change from environment to environment sometimes but You know, you have some little little extra extra steps, but for the most part I mean you just that once the build is complete you just copying the file somewhere so So they can be executed Maybe with some configuration Okay, so There's sort of some core like Rules that you're supposed to post to go by a sort of textbook continuous integration But I think it's useful to sort of quickly go with them When we say integrate we mean We mean check into mainline, so that might be a little bit confusing if you're coming Like you've only known git or I mean, I imagine pretty much everyone here would be using it to manage their projects So what does that mean mainline? Well, it's a it's it's not as obvious, but it's it's it's Any nominal branch that we've picked as our stable branch, so we don't have a Separate develop branch you are you develop in a branch in a topic branch or a feature branch experimental branch and then when you want to integrate You you do it would be the push the integration would be the push but You can integrate locally before you push so and remember the goal is to keep that mainline stable. So we'll be Testing as a real human element of this that's kind of forgotten. It's not about offloading The the tests to the test bot, you know and kind of not really testing it properly and waiting for it to fail and come back It's really about Extra extra gates to quality. So we're really serious about quality. We're gonna fix We're gonna test against any new code now before we before we push it off to to github or the central repository or whatever and and and And we're gonna we're gonna catch up before so that the continuous integration server later on that will be another gate that Where the goal is to never have that go go bad? So Now it should be fast. That's another another important thing and test everything that can break so from a Drupal perspective Just to to tie that into the first point We want to run all our tests locally because we don't want the build to break But that doesn't mean that we're sitting there running the whole simple test suite for core every time We do a change local or every time we want to merge into mainline So we're only concerned with you know things that can break We may not even run all those tests on we probably won't even run them on the on a continuous integration server I mean You know that could be a real later stage and we're getting to testing and how you can kind of divide up the different tests into different stages, but So building a testing should be very fast and again, it's because We don't want to make more work. I mean, this is a this is a hardcore You know the view of quality But you have to recognize that you know, we're lazy and as soon as testing becomes hard And I think you know, this is obvious for Drupal testing Drupal that is hard and So we don't do it as much so If you're gonna gonna start down this route You don't want to do sort of install, you know install a CI server or or make some make some process That makes it difficult. I think you know developers have to be able to Run tests locally fast do builds locally fast. So it's it's this is tricky tricky stuff So build build speed and test speed all all have to all have to go go quickly Okay, so I think it's important to To kind of highlight this is again probably looking a bit more forward to Drupal 8 when we will have more unit testable code You can write Unit testable code now for Drupal 7 for your custom modules I'm not sure it's done much. Is any there's anyone here write proper unit tests or as opposed to just simple tests You know a couple. Yeah, so it's the domain is quite limited at the moment If you're you know writing some Anything that maybe interacts with an API you might might want to might want to look into it And use in that case you would not use simple test You would use something like PHP unit mainly because You probably want something like mock objects or some way to run Just to be clear when we're trying to use testing we're talking about essentially testing a function a method a class So some block of specific code how the code actually behaves in terms of what functions it gets called What what what it returns the kind of internal API's not the end-to-end functionality if you've seen simple test tests they're more about creating nodes and Users and how how functionality interacts. That's functional tests So there's a big difference between the time the time that these things can take is testing one functions return or one classes methods How they behave versus how a bunch of modules behave that needs a Drupal install to get run This is a huge difference. So You don't Don't necessarily want your developers to have to run Every unit test every time they make a change. They may want to run unit tests Only when they make it every change and then when they go to merge run Custom simple tests. These are the sort of strategies you'll want to come up with and it's all about removing the friction and making testing not fun, but Not, you know, not something you try to avoid smoke tests Probably pretty useful if you're Wanting to test configuration going back to we want to test things that can break So I think it's not really useful to sort of write simple tests that are only going to test Whether or not your wiring is working your solar service talking to your application but you still want to test that stuff because You know things to go wrong configuration can Can kind of get it right sometimes? You want to detect bugs in your your build and deploy process. So you can just do simple things like have a Shell script that you you add a bunch of commands They can do curl commands and check output and just check pages up and just you know Real simple stuff and you could you could run them on every you know every integration build externally Which will make sense or pick the right right point for that And then or it goes all the way down to automated acceptance tests Which I've actually noticed a lot more activity in Drupal around that sort of stuff a lot more excitement but they generally would take a really long time to run because you're test the testing end-to-end user functionality either through a browser or a headless browser and There's a lot you can read up on on behavior-driven development from an acceptance test user perspective if you're interested in that but it's not really in in the scope of Of CI so much, but you would if you ran them you'd run them again not not Will not make your development team run the whole the whole suite every time they want to merge so Yeah, we want to split split them up and be careful about which tests run when so Some yeah might not be that obvious that this is related to continuous integration, but it actually is because You want to have this few differences from environment to environment as you can so Anything that you're doing any process so Say the Sat the sass example or If you're if you're compiling sass manually, and you're not checking the CSS in You want to you want to have all those tools all those things that you're automating it as kind of It's consistent as possible, and it's it's just easy to maintain that you can actually write builds that Have very little difference when you when you have to run them in different places And you know where those differences are and they're there in very specific spots So all tests should be runnable like locally Even the even the functional tasks, it's just that you don't doesn't mean you have to run them every time So these are these are the sort of policies that you kind of are coming up with and again the goal the goal is to to have the sort of a Local development environment where developers don't get caught If they they come up to a come on to a new project They don't have to necessarily sink from a sink from a particular database or If they do then it's it's well-defined they don't have to Think about eyes at using sass or not do I have to drive the means that myself and kind of dig in it's all Automated and consistent and that's where build tools come in So It sounds like a bit of work or a bit of a change, but I think that if your application your if your site can't Can't exist without like a particular database. I mean that's a bad smell, right? So this is just sort of taking that a bit further you know it should be Installable and and someone should be able to pick up the source code and just start working They shouldn't be have all these external dependencies So that's that's sort of one of the reasons for dividing being careful about You know automating a build and and and keeping every every step that's involved in every requirement In check so these are just some Okay, I'll skip through these a little bit quicker This is one once you've got an automated build and you kind of you kind of get going These are the sort of things that we're aiming to do so so you know sort of best practices to maintain that quality So not taking a broken build running test locally before merging we cover that keep the main line build stable Know how to manage instability. So it's all about having these policies and then Having probably a system around who's responsible for fixing things that break. So this is An interesting one is Dealing with architecture reviews because these things are hard to test and hard to get a red light to go off I've seen some some tools, you know some PHP static analysis tools that might help But it's a tricky one. So you'll need to consider that as well because that that's technical debt Which is you know another form of defect If it's if your system is difficult to extend and maintain but it may be difficult to test. So and Slow tests. Yeah, a Big no-no because it will it will make people Stop writing tests Code style is sort of one of the simplest simplest things you can automate and again, it's about sort of just Picking up a sense being a bit more hardcore about quality pick up a sense of when You know, maybe someone was rushing in rushing some work or something like that It can be a good if your team's really buzzing along and sometimes you'll you'll I've seen a test where they not so concerned with strict code quality fails getting everything to fail all the time but being a bit tricky with Analyzing the changes in the defect numbers so that So that they'll see like it, you know, it's a variation a statistical variation and there'll be a sign that something needs looking at but these are easy to do for Drupal obviously because It's a static analysis, so you don't have to write the test But you know it only gets you so far so Automated build is sort of What you need we want to start with I think it's a better place to start even then then, you know Some auto deploy which I think is where some people start when they're trying to implement CI they'll they'll they'll you know do automated deploys or code cell checks, but before you get to that point I think thinking about about Your your build process going from source code to something that can can be deployed or run locally and having that consistent is kind of overlooked So Let's just quickly Switch over Give you some idea of what if you haven't seen a build tool used before There's a bunch of them out there you can use shell scripts Preferably you want something that's going to make it easy to get Status codes when things go wrong so that again when you automate it later You'll get you'll get something going so So we've got a got this the simple a Simple project repository So it's not You know the actually for this one. It's the use cases in store profile Development so we want to develop in the store profile maybe push it to Drupal.org So it's gonna have very specific repository requirements has to use It has to use these kind of specially named make files for the Drupal org so that when you download it It gives you the the zip of the whole distro So that's that's the source, but the the rest of the project management stuff the build build scripts We check everything in a version control so Configuration and build scripts to get a strong source to something that could be deployed or run for run locally So this is thing it just lets you Define tasks that you're gonna perform and generally you want to run Have the default be something that could be it would be the one that gets automated and Kind of a pre-deploy step, but locally it can be configurable. So I'll give you a quick example I'll go to I'll go to another folder and clone in So copy of the same repo so this repo has so this repo has The install profiles of submodule in this case You know, that's just an option because for this case it's thinking about a Drupal org submission So that's a decision that's been made Some some stuff to help set up a database user and then the build tasks so If I just rent thing it would run the default the default build task, but Why am I doing that? I have another one though that will do the default build Install Drupal and then actually kick up a PSP server so you can actually sort of just run it So rather than do it instead So this is this is sort of the The long the long build this this is the sort of thing you still configure your builds. You have the same process and try to share Remove duplication in your in your build scripts. So that that ran And what did that give me? this gave me a pub a Doc roof that that I can actually in build and you can see there there's Drupal and I can point a web server to that and Developing in it But that default build is actually the production build. So I've got this Got this configuration and Probably my steps for developers would be download the source either copy this or run it and then set set development to true Set development to true and that will give me some slightly Different I want to do different things for local builds. So this is sort of where you can share that build code and have it sort of be reusable and and Probably have a template or two actually there is a build XML file for For thing That tries to cover all the bases in terms of you know, whole whole directory make files and make files That sort of stuff. I think it's probably best to start from scratch But use the same process of trying to try to reuse the code So if I rerun thing with the development mode instead of doing that whole Download of everything from scratch and starting from nothing which you might do on a production System to make sure that's all running it. It's development focus. So it just rebuilds install profile so that would be you know adding a module to a make file and quickly kicking that off so the sort of sort of sort of things you can do with automated builds and Obviously now I would I would do Sassafal compilation things like that. So tests tests are part of the build They're the other artifact other than the deployable object that you're generally gonna have and you know, so for continuous integration There is a standard this J unit standard that you should be aware of Once you once you start digging into this That will be well supported by different continuous integration servers So I think This build had some tests already done. There's there's the test folder And you can see there's a bunch of XML files. So that's that's also in In the build so we've got our you know, rend this on a production machine. We would have code coverage reports test output and then of course the deployable artifact being the application production ready change log removed everything static analysis, yeah, same code coverage code style checks These should all be all be checked and and considered a Part of every single build every time we're gonna merge in we want to develop generate all this stuff and make sure it's up to scratch so deploying I think it's you know There's some things you have to do From environment to environment on where you still want to keep the same project repository But maybe there'll be production some sort of production difference. I mean that There might be a particular Particular requirements for a hosting provider about sort of where Where code has to go? So again, we're really getting away from sort of a common model of just running get pool to deploy I mean, it's very very far from that but The advantage is that we can do things like Use a tool like Capistrano just for deployment. So very very it does this one job very well It moves it moves files from one spot to another Maintains versions can trigger backups and then it does Basically atomic switching by changing a sim link swinging a sim link around so that's a lot more preferable than just doing it doing a get pool and having Database even if it's only for a moment, you'll have database and code out of sync I mean, there is there is a possibility I mean, it may seem like really nitpicking but there is a possibility that your schema and your and your execution And maybe non-destructive it may just be someone, you know clicks on the page and it doesn't work And then when they go back and the update functions have run that's complete but But we have to we have good tools for this stuff. You set it up once and and they're very reliable so So it's kind of a different a different view going Defining your build first and your build tasks and try to keep that stuff a bit more a bit drier and and cleaner and The event is being that once you go to actually Automate this stuff with a with a continuous integration tool Then it really sort of falls out quite easily So The CI server doesn't do anything really magical or special. It's just glue sits between some version control positive tree with That you know that sort of code that I showed and and the build tool that or build tools or scripts that get run to Create your artifacts so The advantage being that you can also collect information about the build state so all those test outputs and graph them build times performance testing So you kind of off on the right foot if you kind of kind of start from the start from the bottom up and Rather than the other way around Jenkins is a CI tool that Most likely use or I know something we've probably already already installed and are using it It has a lot of advantages. It's very powerful. It's free But I think it's it is probably worth looking at some other tools I think sometimes we get a bit lazy with the stuff and kind of just go to the common one so there's some some interesting Interesting ones Thoughtworks have one called go which has a lot more a lot more of a focus on on the gates and visualization of the build because again, we're sort of tracking tracking build quality and only promoting builds that The pass out our gate quality test first a local gate and then the test that run on the CI server and so on so we know that it when we pick that build and we move it up, it's exactly exactly the quality that we need So Travis see I sort of an interesting one Haven't had much internet here today. In fact, no internet I'll quickly This is like probably the easiest one to set up if you just want to sort of have a play you've been playing with with the automated build and You want to you want to you know start doing it for real? you can You can put a single YAML file into your your project repository You do need to use GitHub. It's not going to be usable at the moment for client work It's at the moment's open source tool. I think they have a commercial tool in the works using the same same stuff, but it's It's a mate fight. You can get up and going in about five minutes If you if you have an automated build already working So you just I'll show you the ML file You can declare a couple of different environments PHP 5.3 or 5.4 and some other ones. I think And set some database stuff do some some environments it up So the main main one for me is installing some Some pair dependencies thing of course and and and related and then Then I just run my same thing builds. This is exactly the same build script that I've developed for the local set up my local environment Which even that I have a different a different database config for this one But actually that could be that could be easily refactored. This was just the first attempt But the the install script is exactly the same and And it all goes pretty smooth Anytime now the way Travis works. It's tight into GitHub So you put any GitHub account you can see all your repositories and then it just picks it up So anytime I do a commit it will actually There's the build output. That's just the thing output and then if it you know if it errors and it comes up read You get an email just like you would with it your own Jenkins And then you could you know do some other stuff obviously What are deploying from here would probably not not work, but if you've got If you've got a good build process where there's very little difference and you're using the same the same build scripts then you could use this as a check and then I think it's an API and you could you could kind of wire something together. It is for open-source This is of course the classic CI demo as you change the documentation So of course I would have I think see I probably I would have ran the build locally and make sure all my tests pass Have a meeting So that's sort of the you know the the long-broad How to get started on on continuous integration? These are these are some books that I found sort of useful particularly actually the continuous delivery one Which isn't about continuous integration, but Touches on a lot of the same sort of quality philosophies This Dripple thing build file is probably not I wouldn't use it like just download is it for a Project, but it's good if you wanted to see some examples you can show you pick some stuff out of there I think it needs it some rewriting in places But it's definitely good start I'm Putting up a sort of a real simple like starter starter template that will cover Different different layouts but with some a bit more factoring in there and I think that's about it check the build see if it went green Up there we go. It's run yellow means. It's running This is a full full clean slate Production build, so it doesn't even have drush on there That's it. I think I'm ready for questions if anyone has any Or if you want to talk about something that touches. I know this is a lot of a lot of different topics Asking questions about specific tools or or pain points that you've had understanding or implementing Yeah Yes There's a there's a dot-com page. I think that they've got up So they've obviously got a commercial service running at the moment, but it's at the moment. It's free for open source projects so Definitely if you had other PHP projects that were open source I would highly recommend looking into it Because it's just so quick to get started but again you can I Think things pretty trans transmittable Skill tool like it's reusable in different environments different different situations works great with Jenkins And the output kind of comes in sort of automatically that's a plug-in, of course Right. Yeah. Well, that's I mean that's I would consider the question was Would I think you mean database sync? Yeah, like say production data to staging I would consider that a deploy step to staging probably Or you know or it could be a special case for your for your This is why do I deploy is a kind of funny It's so specific to sort of what you got and and how much how much data you have if you got a really big database in Production you might not be doing that so often you have some other process in place Maybe a daily a dump and then you're thinking from that So there are there are some environment specifics I'd like to keep that more probably in the deploy the deploy configuration because it's specific to an environment Simple test if it's functionality There's a there's a talk tomorrow on simple test Okay, so you're like you're doing some site building and you want to have some automated testing around that I Would I would look in the simple test as a first step? and Maybe later you could look into some acceptance testing stuff with the hat and mink, but I think I think Simple test is probably first go to and you can include the test in the feature module And and have it run So I didn't really kind of get to that that there's that output that XML output was just simple test output so Generally for your build But you'd have all those simple tests running on on every build like for just your modules Not for not for core and contribute Okay, thank you