 Drupal come Amsterdam and for some reason I'm again up here on stage. Can anybody explain that? So my name is Chris buttered if you put Dries and me together on stage you get these buttered We actually pulled off that trick a couple of years ago Seven so Works better when I switch it on Brief background. I used to be a software developer 15 years ago 20 years ago, and then I ended up doing operations work Today my role is I'm the chief trolling officer at inwards.eu which it's one of the larger open source consultancy Companies in this part of the world. We actually have offices right here in Rotterdam Antwerp, which is our main office We have one in Kiev and we just opened one in Prague Which has absolutely nothing to do with developer with the Drupal come product last year Um Well kind of not really What I'm here to talk today about is DevOps what DevOps is about and the current state of DevOps and Drupal What I've been doing for the past couple of years is pretty much Evangelizing DevOps in in all kinds of different ways and when I'm doing that in the Netherlands We have a special term for that then it's called actually on wiggling soap so I actually wasn't planning on being here at least not giving this talk by myself Kevin Bridges submitted this talk With exactly the same title and then afterwards he asked me if I could actually join him on stage to give the presentation and Then somewhere in the August he had to cancel his trip to Amsterdam. So I'm standing up here all alone So Who knows what DevOps is? Are you guys sure? I'm gonna take you back into a small trip about five years ago Paltry gruba and a couple of other people we started doing this really small conference 60 something people probably Around the size which is in this room right now and We wanted to bridge the gap between Development and operations and figure out how those two groups could work better together and Well, yeah, everything else is pretty much history Now we have DevOps days worldwide Every larger city in the world is organizing its own DevOps days. There's local meet-ups or as local communities Actually at Drupal con this is already the third year. We have a DevOps track. We started doing that in Munich With I see quite a lot of speakers from Munich Barry and Ricardo and who else was there who spoke there and then Doesn't feel that way but in two weeks. We're actually going to have the fifth anniversary DevOps party in again It's already five years Since we've been doing that it has been a growing movement and a lot of people actually are really interested in making this thing become a success well kind of almost because there's also Because of its success a bunch of vendors slapping DevOps onto every single product they tried to ship Which has usually nothing to do with DevOps We have new job titles like DevOps engineers and DevOps Teams and there there's people actually selling DevOps certification. By the way, if you give me 10,000 euros you get certified But that's pretty much stuff. We really didn't want to see it happen. So let's try to define DevOps for a while so That was doing ops work that DevOps You'd wonder how many organizations actually think that we should just fire the ops people and let the developers do the actual operations work You're laughing, but it's reality in the back. Yeah, isn't that exactly what's called as no ops? It's the question. No, because with no ups There's actually nobody doing operations because nobody has to do operations, right? If there's a batch exploit nobody needs to do anything, right? If there isn't a self vulnerability Nobody needs to take care of it, right? That's what no ops is about and Fundamentally what no means is that you pretty much have outsourced everything to a software as a service party and you don't care anymore That's kind of what lots of people do ops call no ops So on the other side of the fence there's Operations people writing actual codes that's DevOps, right? No, that's infrastructure as code. That's actually Experienced system administrators doing their job right and then there are some vendors we claim that continues integration and continues delivery That's what DevOps is about Well for some of them it is but it's only a small part So yeah, no ops basically using the cloud. That's all broken definition of DevOps to me There are so many broken definitions that We don't know where to start. Is there anybody here was a DevOps engineer? Nobody this is a pretty good crowd. Usually there's a bunch of DevOps engineers in the audience I always ask them are you doing development or are you doing operations or are you an agile and then a look at me like What do you mean? There's also the new adoption of what what people call DevOps teams Is anybody working in a DevOps team? Did you guys have coffee? No, I figured so the DevOps team is They're in different flavors and the most awkward one is the one where they actually put a team in between the developers and the operations people To create more confusion so Let's figure out what what DevOps really is about and in order to explain DevOps in a better way I always have to come back to The definition from Damon Edwards and John Willis So Damon Edwards and John Willis were the two guys who actually took DevOps from Europe and made it happen in the States they started organizing the first DevOps days in Mountain View and They also had what still have but it's pretty silent these days. They have a podcast called DevOps Cafe and In DevOps Cafe, they were interviewing a lot of people. They were talking to a lot of people who they Assumed that we're doing DevOps the right way and After a couple of those episodes, they finally figured out like there's four key components which actually make DevOps DevOps and Those four key components are Measurement and sharing and For a lot of people that actually resonates with what DevOps is about It's about how an organization treats his team how they work together how they deal with failure how they deal with Learning how they deal with adopting new people and new principles It's about automating everything you have from built to test to deployment to pretty much everything you can imagine It's about adding a lot more measurement than you are used to do and both on the human parts How do people perform but also on the operating part and how do you use your platforms and how is your platform behaving towards your customers? And it's also about sharing not only the source code, but also the experiences. This helps if you take a little So We had camps and then the naysayers came along and they said well you can actually Rewrite camps as a scam Which wasn't really what we meant so Jin Kim stood in Jin Kim, which you might know who you might know from the Phoenix project these days And tripwire before of that And he actually noticed that the large influence of lean into that so Now when we speak about Cams we actually put the L in there so it's clams and we can actually not make an anagram out of it so it doesn't become a scam anymore and To a lot of people that actually is a large good definition of what DevOps is about and This one might also be a really good one DevOps actually is a culture and professional movement and that's what Adam Jacob currently I know what its current low at chef is but Used to be CTO then he got some other tiles But that is also a fundamental part. It's not a Recipe on if you do a Bcd, then you're gonna have a billion duck Billion dollars in the bank in a couple of weeks, but it really is a group of people trying to become better at what we do So what is the real problem here? The real problem is pretty much this and I keep using this slide or a variant of this and each year I hope it's the last year I'm gonna use it and each year I have to run into the fact that it's still today's reality There are still organizations out there who go to the operations people on a Friday evening around 430 like hey guys We have a national marketing campaign. Can you put this stuff live? Who who has this happened this year? Oh Fuck that's way too many hands I was about to increment and say we had it last year or the past five years, but doesn't matter so This is the fundamental problem people put all their budget on building a piece of software and Don't think about how to deploy this how to manage this how to keep this secure and and typically operations people come up with hey Does this need to scale how many users do you expect? What about security was gonna do the upgrades in six months was gonna actually do backups of the to he even need backups because sometimes people just don't bother and The result of this discussion is that this is the relationship between developers and operations so What do you gain from adopting DevOps practices? For a lot of people it Comes down to getting more features out faster. There's a lot of competition on the market There's a lot of customer demand on getting new features out there And if you don't get the new feature out there to marry then your competition will have so you need to be able to be much More agile in delivering features and software faster You need to be able to get your features out in hours as opposed to years if you look at banking Six to nine months to get a feature out of the door That's the normal bank Banks were adopting devils. They go much faster You need to be able to do that much more secure Security is one of the most undervalued parts of our business But given the issues we have this year with hard lead and and last week with bash More and more people are getting the wrong idea about security But it is something we need to pay attention to and we want to do this deployment and upgrade stuff in a reliable fashionable Hopefully we're not too many humans are involved So if on Wednesday evening I get a mail from Drupal that there's a security issue I don't want to spend all night fixing it. I just want the machines to do that for me and The end result of that should be that we are as a trade as software developers and operations people We are becoming oh, that's the buzzer going off He's on call Or he needs to write his slides for tonight. Sorry for that So the end result should be that we have more happy customers developers Ops people managers and all that kind of stuff So DevOps and Drupal Is it really that bad? Some of you might remember that about two years ago I set out a survey to figure out what the actual state of Drupal and DevOps was how Drupal people dealt with delivery how they use version control and all that kind of stuff and I presented the results of that. I think in other at a meet-up at our office or partly also in Amsterdam. I don't Munich I guess So we had a bunch of results there and then Kevin Bridges took that as an actual Drupal project And it's actually now on Drupal.org So you can actually rerun that survey if you want to do that internally or whatever and he rerun it for Austin and We've seen some changes So I'm here to discuss a bit those changes and while I'm doing that teach you once again about what DevOps really is So what did we notice? There's more people who actually started adopting DevOps or They think because half of them has a broken definition. So it might actually be that they're adopting. No ops. We don't really know That's that's a problem with Surveys there's lies stamps lies and statistics. So I'm not really sure if you can base yourself on this because we haven't spoken to those people There's more people who starting out to do this and there's more people being pulled into DevOps groups And then there's less people who don't know about it So that's already a good sign, right? We it seems like Like a lot of our communities the Drupal community is adopting more of these best practices to take away their pains So that boils down to the question. How do they do this? How do you get there and The first way for me to start adopting DevOps is to think about software development and deployment in a completely new way, it's If you are part of the HL world, you know that there's a lot of discussion about having a definition of done at the end of each print and as your team it tours it Changes from being able to build code to having tests in the code to actually having Code which is deployable to actually being able to monitor the code which is in production and your your definition of done starts growing and To me there there's even a further step to me. This is your definition of done as long as there's anybody using your software You are not done. You still have work to do and that's the change You need to get aware of you need to have a team which is building a platform Which is building a product and take up a shared responsibility for that so First steps to get there is to enable communication between development and operations between the product owners the business owners The people who actually support the platforms the end users and to get that communication and discussion going on See what is supposed to be delivered do that as soon as possible in the process of your project and then Brainstorm see what you need Don't go up as a developer to your ops people and say hey we need MongoDB by tonight Because they'll go like are you really sure do you really want to use your data and Talk about why you want to do this why you want to work together It's about really building a much more reliable platform being able to deploy faster and Not to have work late evenings and nights so Technically that means the first thing you need to adopt is basically version control Who's not using version control in this room? Okay Who's versioning all the artifacts he needs to deploy his infrastructure? All the software who's still download stuff from the internet What if the internet goes down? Can you rebuild your platform? one of the large discussions I have with people doing software development both in the Java the php and other worlds is What do you need to version to me? You need to version all the things everything needs to be and get Don't pull out modules from the internet Don't pull out libraries from the internet because when you need to rebuild that and the upstream vendor has decided to well Shut down a site for maintenance You cannot rebuild your stuff. You really need to have everything locally. Otherwise, you cannot rebuild So you need to have the source code of your application the source code of your infrastructure All the builds scenarios you have you need to have all the tests in version control all the Recommendation all the monitoring script everything needs to be in there So how do you think the Drupal community does? They're doing fine There's 10% more people who actually use version control according to the survey. There's about Everybody moved to get But there's still something wrong with those 4% who don't use version control So once you start doing version control, you want to be able to move forward and build a platform, right? So you want to move to continuous integration because you want to have tests that verify that the code you just wrote It's not gonna break your platform. You want to have tests that are proving that what you wrote is actually what you wanted to wrote, right? so You need tests that actually take the different components of your Infrastructure and your source code and the applications and some third parties and keep testing that and see that you don't break stuff and in the ideal world We go to what just humble calls continuous delivery where we do this through automation We do build deployment. We do all of these things and there's a lot of collaboration between Developers and operations people which allows you to do continuous delivery And continuous delivery has some requirements First of all if you look in the inner side, you cannot do continuous delivery without doing continuous integration If you do continuous delivery, you don't do any tests Then what are you delivering shit? Do you even care what you deliver if you don't test anything? So build up on tests and move from there and If you want to be really good at it You can go to the point where you have sufficient tests in place where you are Material enough as a team that you can actually say we can actually constantly deploy Everything that goes through the different stages in our pipeline. I'll come back to pipeline in a while so This is really something you need to build up. You cannot start with continuous deployment on day one You really need to start being testing adding more tests building up tests to a point where you are satisfied with the tests And then you can move over and say like now we're going to actually deploy this stuff automatically when we're satisfied And to a lot of people like I said before this is what DevOps is about But it doesn't take into account everything else. It doesn't take you into account monitoring metrics It doesn't take into account the change of culture you need to do in your organization to actually achieve this To a lot of people it's purely the technical part where we build a pipeline and we just do this And this is what a pipeline should look like You check in your code There are some tests and builds it fails you go back to stage one you start over I've worked with a couple of organizations where if there's a failure down here somewhere They fix it and then they just roll over they just move on they forget about all the tests They've done before and they just fix that one thing Potentially breaking everything else and just move over to production Which is not what you need to do You really need to go through the whole pipeline again, and then here is an example of a pipeline which is Building Jenkins which have a lots of tests in there, which are all green so it's really a good pipeline So the exercise you need to do is figure out how to build a pipeline You need to start usually with an empty pipeline Try to figure out how to deploy hello world to your production platform Because in most environments you don't start out from scratch you already have existing code Which you need to get to production If you're in the lucky setup where you can start from scratch and just build something then it's your first Commit of the application which you need to be able to push but in most real setups You need to be able to take a part of your application and start Incrementing and building tests on top of that and move further from there And then you need to grow those artifacts they need to go to a pipeline and The definition of an artifact here is it's a pretty important thing an artifact is something that needs to be Unmodified as soon as within the process of your development track you modify the artifact You basically invalidate all the tests you've done before If you make changes in that artifact, it's not the same anymore It might be a small subtle change, but it's still it's something different from what you tested before So you need to go over your tests again. So how about the Drupal community you wonder? So according to the survey Jenkins usage has doubled. There is not really that much other continues integration tools out there that are being used But they're still according to the survey about 40% of you guys that are not using any kind of continuous integration tools You're not using any kind of automated tests So that's the map with what you guys think was using some kind of continuous integration tool Yeah, that's about 40% who doesn't I think actually it's more than 40 who doesn't and I tried to figure out why that is and I Was chatting with Joshua's in the next room just before this talk and What we will conclude is actually it's not really a Drupal dilemma. It's it's more of a PHP dilemma Typically a lot of PHP and Drupal projects are short on a really small budget and people Don't have the budget to actually invest to set up a test framework to actually write a test It's not because the developers don't want to do it But because it's just not part of the budget the customer does not want to pay for it but on the other hand The real question you have to ask is can you afford not to? Can you afford to not automate what you are building so that when a customer comes up three weeks down the delivery and says hey? Can you fix this in this and you go in there when you break it then you actually waste more time you have as a budget or Six months down the road where there's a security issue and you really need to go updating the site But you don't know if you're gonna break it. So you're once again spending much more time in Manual testing and fixing code and figuring out that you broke it after the customer called you just cannot afford that so it's always a difficult balance between Building it up front or realize that you're gonna have to spend the effort afterwards anyhow does anybody here is Familiar with the concept of technical depth a Couple of people so Adept what happens if you get adept with a bank depth You pay interest well, you don't only pay interest, but you need to pay the money back with interest so each time you actually Ignore parts where you know that well, we should do this, but we don't really have time Chances are really good that eventually you'll need to pay back with interest So when you talk to your customers and your manager next time about no, you don't want tests Might be a good time to explain him that they actually need it because I cannot afford not to have them so We were talking about testing and that testing really needs to be in there. So whoa Typically when you start doing continuous delivery exercise continues integration exercise with people you ask them What's in your pipeline? So guys what what's in your pipeline? What's in your built pipeline? Does anybody have a built pipeline I See hands slowly moving up Do you need more coffee? Yeah, okay more coffee. I see a lot of nothing more hands on the coffee So okay, if you if you're not awake yet, I'm sorry But typically the start of the pipeline is that you check out the code you do a syntax check on that to make sure that It isn't broken like hell if you have a team sometimes you have imposed a style and You check if that style is actually being lived up to You do some code coverage on see if there are sufficient documentation a sophisticated sufficient testing in the actual code base And then you move on with doing actual tests and when you are doing actual tests you end up with a bunch of testing requirements and To me that means if I want to be able to test something I need to be able to build up the whole stack on which I'm going to test and I need to be able to do that over and over again because Whenever I want to run test a I want to see that the result of test a is B and If I cannot start each time at exactly the same point It means that the second time I run test a it's gonna start at point C and the third time It's gonna start at point D. So it's each time gonna have different end results F and G so I cannot compare so to me having a reproducible infrastructure where I can actually deploy and redeploy and Over and over deploy the code. I want to test is a testing requirement That means that for for those applications You also probably need some kind of bulk provisioning where you take actual production data or anonymized production data and put that onto the stack So you're testing on the relevant workload It kind of means that you need in order to automate this infrastructure as code and infrastructure as code comes down to I Want to be able to build up the infrastructure as code Without human interaction without somebody pointing and clicking on a GUI I'll like Luke you need said in FOSDM 2007 if if my computer cannot install it your installation is broken and That's kind of where you need to be yet So infrastructure as code brings us to you need to look at your infrastructure as code You need to look at the configuration you built on how your search systems look like and We're doing infrastructure as code. So we also need to take into account all those best practices We got from the development work. We need to do Continuous integration on that. We need to do version control. We need to have different test development and acceptance platforms and we can if we do that come down to a Setup or the combination of the infrastructure code the application code together that actually builds the service you want to run One small remark when we talk about infrastructure as code we don't talk about Scripting everything we really talk about defining a state. It's not about install Apache. No, it's about make sure Apache is installed The difference there is that we move from an old Scripting module which a lot of the proprietary vendors are now taking and putting a GUI in front of that to actually modeling your infrastructure and defining the state it needs to be in and a lot of commercial vendors still have not adopted at all though These are IDs which have been around for seven to ten years So With that in mind with with what Luke said about I need to be able to Deploy my infrastructure and I need to be able to deploy my application. I kind of made this bold statement in Munich. I Think if my computer cannot deploy your sites, it's not worth my time actually deploying it and That obviously brings us down to a lot of the new well not really new Evolutions in Drupal, but it comes down to how do you deploy a site there? There's a lot of ways on how to I'm at the key automatically deploy a site and and what we've seen from the survey is that People manually configuring and enabling modules from the GUI. That's down by 27 percent. Yay And using features. There's a lot of more people actually using those so Looks like all the hard work and evangelizing is finally paying off Um On the same Parts site profile installs. There's from less than 1% people who could actually do a site install from the profiles It actually up to 6% Luckily people using database dumps Is really down and there's Some increase in using features to actually do deployment, but it's it's still Not really big figures. It's not like the majority of you guys are actually doing automated deployments of your sites so Our pipeline wasn't finished yet We built something we do some more tests I like to package stuff to make sure that the artifact which I'm pushing through the pipeline is unmodified I Uploaded to a repository and I actually deployed to a test platform and when I deploy it to a test platform That's where monitoring kicks in We have the sub-movement in the DevOps community, which is called monitoring sucks because monitoring well up to like three four years ago really sucked and It's improving now But part of that is that we need to put monitoring into our continuous delivery pipeline if you deploy something We need to test something we need to figure out that there's a new service Which is pop-up to which we automatically need to start monitoring and we don't want to manually change those things in our monitoring setup And why is it important it is important that we also need to monitor the actual functionality of the site Because part of the function of monitoring functional tests if you're writing them for your Pipeline already you can reuse them for monitoring so when you do deploy to test you can actually figure out Hey, I didn't break my infrastructure My site is still running on the right ports. The right domains are still there and it's actually more working so testing testing testing you need to Pretty much do all of this who does all of these tests unit test integration tests system tests performance test security tests Who does all of these? Yeah, but you worked for me. That's normal How do you think that you pro community does Well given the fact that there are still not a really good adoption of Automated tools a lot of this stuff is manual and there's a large diversity on what you guys are testing on But it seems that the focus is on performance usability and GUI Which kind of makes sense from a designer and an end user point of view If you look at automating those things, I don't know how you guys test But if you're still not using tools like selenium or a cucumber or be hats, you're doing it wrong If you're really taking the testing is about hey, let's go to the site manually log in and click on some stuff That's not something you can reproduce over and over again That's something which is gonna bore the hell out of you and get you a burnout in like three to nine months from now Who wants to burn out? Yeah, I guess that nobody so There was improvements But not as much as I would have liked to see Maybe one of the things to mention this could also be a cultural thing The survey changes from 2012 to 2014 might have been done in a completely different time zone because I was making noise on the European side and Cyber Swat was actually making noise on the US side, so it's not really 100% the same audience, but still But anyhow, if you have sufficient tests in there if you have a pipeline which has a Number of tests which have been performed and people who have been doing checks on that then there's the next part There's the automatic or manual build promotion and this is actually a screenshot from a Jenkins server sending a mail like hey I've done all these tests. They were pretty successful. Do you want to push this through production now? and this is if you can reach this step where you can Have Jenkins decide or any other tool circle CI or Travis or whatever you prefer Have them say We did everything so far now It's up to you to decide if you want to go to production You can have a bill to promote it and you can have that bill promoted to the next phase or to production or and if you can really read the small print this is an actual Drupal pipeline promotion Or if you are so sure about your test you can even skip this test and Skip this step and do this automatically But that really needs you to have a mature team that has sufficient testing in place And it also has some business requirements because you cannot do this with every type of business Sometimes you can only launch a marketing campaign at six o'clock in the evening and you can only promote to production then stuff like that So we promoted to a test platform. We have some on some more tests We have promoted into acceptance the actual customer has looked at it and said it's okay He also got a similar mail he pushed to production and then you actually push stuff to production Are we done yet? See at least one person nothing The rest needs more coffee So no you you really want to close the feedback loop there One of the ways we do that is by sending a metric to graphite with a timestamp and we say hey We got a new deployment here and then you can put these things On the dashboards and you can actually Give your developers insight in what's happening on the platform or How many times they have deployed or what the impact on the database is or what the impact on a patches or the load on the system Did that change in your code actually trigger an increase in load or an actual decrease in load? You just pushed engine X in between. How does your platform react to that? But more importantly you can actually now also Visualized business metrics from because from all those logs all those metrics you collect you can most probably figure out What the revenue of your platform is or how many people actually sign up how many? Conversions there are from free to paying users or if you're hosting an API you can measure the number of times an actual API call is being made and that kind of stuff can be interesting in figuring out where you need to focus on if there is if you are exposing a couple of APIs and There is only limited use of one you might not want to fake Focus the next print on improving that because of those three users But if you don't have those metrics you might actually spend your next two tree sprints on a feature that nobody's using Which then is pretty much giving you back that feedback cycle which allows you to do continuous improvement so bunch of tools there a bunch of best practices, but It really it's not about the tools. It really is not about Writing code and pushing into production. It's really about changing the way we operate Platforms today. It's really about people talking to each other and Working together into building a much more and a much better platform Actually of homework for you guys if you haven't read them yet Really go read just humbles continuous delivery or at least if you're not really a fan of technical works Which would really scare me in this room go read the Phoenix project Give the Phoenix project to all your managers They'll love it and they'll realize How wrong they've been so far And that's pretty much what I wanted to say There has been To my opinion some improvement, maybe not as much as I liked into adopting DevOps into the Drupal community But if you want to discuss that later, there's tomorrow. There's above Which is gonna be right during lunchtime For those people who want to listen more about DevOps topics We have a meetup the Amsterdam DevOps meetup group is actually meeting up across the street tonight Ricardo would just flat working on his slides is going to speak there Bastien is going to speak there and there's also going to be one talk about get lap Because the get lap guys are kind of from the back if I'm not mistaken And for those who care about really advanced stuff like mesos There is going to be a talk from Ben Hintman tomorrow at the docker meetup group Which is going to take place But in order to get into ing you really need to register on their site on the meetup site and With that I'm open for questions DevOps adoption and Drupal. Yes Okay, so somebody's actually giving to way just humble book. Is this a scientific copy? No It's not a signed copy. Okay, so somebody's actually sorry. I don't know your name. It's going Okay, so tomorrow there's going to be a giveaway in the continuous delivery session Three no what time are you five at five? So if you want a book go get it here other shameless plugs or questions It's actually good to hit managers with When you end up in their office and it's an open shrink wrap I Have that one. Yes Okay, so the question is The deaf people and the ops people are kind of cool with adopting more practices But any kind of advice on how to get the business people more involved? I've seen a couple of things happen there business people and project managers a lot of times They don't really feel the pain. That's how they're They don't see that developers are up till late night to fix a buck which the operations people have found Because one of the things is they're not on call They don't focus on putting the right priorities according to the deaf and the ops people into the next print if they're doing a child Work so so one of the things I've seen happening in a pretty large organization is that they actually put business owners on call so that they learned which issues the other people had to deal with and To them as a result that meant that in the next couple of sprints the priorities got completely rewritten They started actually focusing on stuff which was hurting them and Now was actually hurting the business owners The second part was kind of already in my slides Because once you can visualize business metrics once you can actually see look We have an issue on the storage platform and that means that the revenue currently is down like hell Or we have found this bug and we see that there is no sales and there's no new sign-ups For the business people that means money They realize that they now have visibility in their platform and they can actually tie incidents much more closer to Where they're putting their money or where money is not coming from and that's also helping Putting in metrics is usually one of the things I try to do as soon as possible in the process There was another question there. Yeah So pretty much the question is we have a lot of small sites Which are which have a short lifetime? How do I see testing and continuous delivery for sites which are short-lived? If you do a lot of those What I would try to do is make a really good standard template from where you start building them so you can at least deploy that standard template and With that standard template have a lot of testing in there and then try to the word as little as possible from that framework and And and really try to push it into a product because that's the only way you can afford to sell it to your customer because even if it's a Site which is going to be only online for like three to six months What I've seen is that that kind of sites are typically high-profile sites which are announcing an event or putting up something for sale and If there's a security issue there, you don't want to get your users to get everything for free So you really want to be able to do the upgrades and specifically in projects where you only have those short time frames You just cannot wait to upgrade till it's over. You really need to fix things right now They're usually also really agile like oh well users want to use it the different way You only have like three months to get it fixed to get the users on your side It really comes down to Hoping that this customer is not the very first time he's actually building an application because if that's the fact you're pretty much