 I'm Jake Scruggs and I'm going to give a talk today about there is such a thing as too much testing Which is a sort of reflection of some hard fought and won lessons. I've had over the course of my Lifespan as a consultant and software development or developer Integrating some sort of automated testing on a lot of different projects If you notice that everything has kind of been shifted up I took Yahuda's advice and moved all my slides up a bit so that you can read the stuff So first we have to go through the little who am I stuff I Used to be a high school physics teacher, which was a lot of fun until I got tired of writing bathroom passes and I Taught myself Java. I apprenticed an object mentor, which was a really interesting thing to do right after you had Just learned what an object is to hang out with a bunch of people who like spend their whole lives thinking about objects I Worked at ThoughtWorks. I did about four years there I wrote metric foo, which maybe some of you use and now I'm a consultant at abtiva and The genesis of this talk was I was starting yet another Project from scratch and we were spending a lot of time arguing about Shoulda versus our spec and some of the team really wanted to use shoulda and some of the team really wanted to use our spec But what we weren't doing is having a conversation about like what our testing strategy should be and This is kind of a pretty common thing is that people sort of Nobody has a conversation about testing right anymore a lot of times It's just sort of assumed what we're gonna test and then there's just a strategy sort of adopted And then that strategy just gets implemented and maybe it's the wrong strategy for your project You know because what you tend to do is just sort of use the same strategy used on your last project Which is you know, you know yesterday's weather is a pretty good thing unless you happen to be doing a You know moving to Antarctica between days so This is gonna start off with like a tour where I'm gonna go through all the projects I've been on and talk about what mistakes I've made and what we did and Don't worry. It's not just stories We're gonna be kind of getting some patterns and then we're gonna like extract those patterns and discuss them So to start off the tour So After kind of talking my way into becoming a paid consultant with little to no education I got put on the dealer portal, which was my first ThoughtWorks project and it was a pretty cool project This was a large manufacturer of tractors that you may know and What they had is they had a whole bunch of websites where what they want they could Dealers could go there and they could do really cool things like submit a sample of the engine oil of a tractor And then go to the website and find out like diagnosis stuff on the tractor And there was a million of those like sites that did a million different things And they wanted a portal that would integrate all these things and so since it was a portal We were gonna use portal software and Java and web sphere and of course struts because this is about five years ago and struts was pretty popular so There was about eighty to eighty five percent coverage, which if you've been in the Java world is a pretty good amount of coverage It's hard to get in all the nooks and crannies of Java Web unit was something that was big on this project and web unit I don't know if you've ever used it is a way of simulating clicking through the Browser without actually having a browser. It's kind of like a rails integration test and Because it was really easy to write these tests and you got a lot of coverage out of it They were everywhere, but that led to sort of a crazy amount of build time This was a relatively small project with six developers and it had a 20 minute build We were about four or five months into the project Right so extrapolating that out, you know, like how long is it gonna be after a year or two? You know, like it's was already kind of a daunting build The other big problem was that web sphere took five minutes just to start up, right? So you want to kick off your test suite? It's just kind of waiting around for five minutes while web sphere sort of does whatever it does And these are on pretty powerful machines so it was an interesting project in that there was sort of this heavy burden of testing and Definitely if you broke the build near the end of the day and we had this red flashing light that would sort of go and it would drive Everybody nuts when you broke the build you had a minimum of 20 minutes before you could fix that right because even if you just went Oh, I left off a semi-colon and you just check it right back in right there You had to wait a little bit But the team was definitely into it the team was a supportive team the team was into testing and the Management was definitely down with the idea of testing software So we didn't really have a bunch of pushback and everybody sort of chipped in and made it a more bearable So that was a long build, but it was it was supported So moving on I got put on from a very small team to a very Large large project. I think it was something like 800,000 lines of JavaScript when I joined and it had been running for about seven years Maybe eight years and it was the huge financing app and this is you know, basically if you want to buy a large tractor yet again you basically take out a mortgage and That mortgage has to be managed through something and so this application managed all that for people who would sell you the mortgages Right, you would go to the dealer and say I want to buy this thing But you didn't have six hundred thousand dollars in your pocket. So you would essentially like take out a large loan So this was like at one point it was a proudly proclaim that it was the largest Java app You know in existence like which was kind of a dumb thing to be proud of but They it was and I used EJB 1.0 So that's entity beans and all that use struts 1.0 on 10% of the app because they wrote their own way to get around pages Before struts and then they thought wow struts is cool. So we'll integrate that But that didn't work so well. So they stopped and never backed it out So you'd be in this app like working on a bug and you'd be like this page is crazy It doesn't work like anything else in the app because that was the 10% that was a struts app and then you'd go Oh, let's go dig through some XML and then you'd figure it out So when I got on I was like the first person I think ever to run a coverage on it And it was like 21% test coverage Four years later. They hit 25% and the problem with this thing is that It's a large application that once had like, you know, 20 developers on it and was very active And now it's kind of like five six developers and it's sort of in maintenance mode, right? They're not really doing anything major. So they're just never going to get to all that code They're sort of fixing and testing the code that they can get there But what I joined it was actually kind of a test hostile world where you couldn't write tests because the EJB is demanded So much just to instantiate an object. She's like, I just want a freaking object, right? And it would be like no, no, you have to be connected to some server or something and you'd be like I have no idea. So it actually kind of beat the testing out of me. It was it's kind of brutal We had a silk suite which is like mercurial or selenium or whatever to click through the app thing and that that run like eight computers 36 hours straight and 80% of the tests would fail but Never the same 80% Because I kept saying why don't we just cut out the tests that fail all the time and they were like, oh, no not the same tests and So we didn't really know What we were running it for and I kept asking a lot of like really uncomfortable questions And I basically figured out at some point like it just made the company feel better to run these tests So they had like a guy who had to sit in a room with like eight computers And it was noisy and he had to like run those tests and then he would come to the stand-ups and say things like Well, I don't I don't know 20% passed All right, so It was kind of the worst of both worlds like we had not really much coverage and we it was money Right, like this is like millions of dollars possibly hundreds of millions of dollars passed through this thing If we screwed it up in theory, this is you know bad, right? And it also did a 20-minute build so there's good times So after that Or actually sort of during that I kind of got frustrated with Java and taught myself Ruby And one of the great things about ThoughtWorks is I just sort of said hey guys I'm a Ruby developer now and they went well all right cool And so I had sort of been learning on my own And I've been running the Ruby users group and they put me on a small little Ruby project of the large managed hosting company and the really cool thing about this is that When you manage stuff You manage hosting you have to manage all this IP space and it's all binary weird Calculation so it was a lot of fun to be in this domain right because you get to do binary math And there's all these problems you have to solve like hey Do we have any space left because it's not obvious if you have any space left in IP space? And also somebody wants a big chunk of numbers that are right next to each other Can we do that and where is that chunk and if you're running out of space? You have to be able to prove to the governance bodies that you are indeed running out of space So this was the whole idea of this app and it was a cool amp in a many many ways First of all it was a it was a Ruby on Rails app But also we had near a hundred percent coverage which was like sort of a new thing and We sat maybe like where those doors were was where the like one of the founders of the Company who sat and at any point we could walk over there and say like hey man Like you know you want to see something and you go sure and we'd show him something and then he would go I was really thinking about this or that or the other thing so We had a really fast suite it was a cool little nine-week project and it sort of had the proper level of testing We got it out the door and but the testing enabled us to occasionally just go in and rip out a bunch of code and like change things when obviously requirements change So that was kind of a success and that was fun and then I moved on to the small social startup Which was a Ruby on Rails our spec project So this is back very early in our specs career. I was an early adopter because I know this Chalinsky guy and He kept talking about BDD and I didn't really know what that was but our specs seemed kind of cool. So We had a hundred percent coverage We had a really cool Selenium build that would kick off when we checked in and it would run all these tests and make sure everything worked It was one pair. So it was me and this other guy a friend of mine and luckily we got along because it was just the same guy every day for three months and We had all come and then there was like a project manager And we had all come from like super heavy projects where it just had to run So we had used yesterday's weather when deciding how to test this app, right? We just said like okay Well, you know on my you know previous apps that everything needed to run There was no way you could afford to fail. So we're gonna do Selenium. We're gonna do a hundred percent coverage We did really extensive view testing in our spec which seemed like a really cool idea at the time but was actually very brittle so One of the weird things about it though is it was just this tiny little startup That they wanted to get out the door and see if it made any money at all So the whole point of this thing was like this guy had like kind of a crazy idea And he was just thinking well I want to write this site, but I need some help so they brought in the consultants And we were gonna build this thing and they just wanted to get it out there and see if anybody cared at all And if so let's make it something really cool and if not then maybe it just goes away But that was not what we did right we wrote this sort of bulletproof app that like was Crazy well tested which is you know good if you were like, you know trying to do money transactions Not so great if you're trying to do a quick startup and and see if like there's any Use to this thing at all or interest so this is like you know the first sort of like big failure That happened and it will be just made fun of me for it when right before you there is I don't know if you're here right before it so Yeah, a lot of lessons learned on that project So yeah after this and we'll summarize all this later on so just hang on We got a couple more projects to go So after this I move on to sort of the opposite project right it's like in theory They're both social networking sort of sites But one was like tiny little startup and let's just get it out the door And then the other one was like huge company that wants to get into sort of the where to go What to do website business across the nation by the way coming soon to San Francisco anyway And so we built this app for them it was a rewrite of a Java app that had listed that had lived in one city for like ten years and They wanted to make it live in like 37 cities So we took the ten-year-old Java app and rewrote it in Six months with ten developers in Ruby on rails There was a couple of mistakes really early on before anybody ever showed up and one of the biggest ones and Please learn this lesson is if you write a spike to figure out if you can use a new thing right you're going Oh, maybe I'm gonna use Django We'll just hack on Django for a month and see if we can get anything to work And then you're really happy with the output do not under any circumstances just go okay cool Let's just check that in and then start building on top of that because it's a bunch of junk right you wrote it as fast as possible with zero testing and Like they even let the manager write a bunch of code, which was just just terrible code and so basically we started off we had like a Couple of thousand lines of code and zero coverage and no real knowledge actually of what it even did right So we would play a story you'd start working on a story And then you'd find out that it actually already did that but in a very weird way somewhere over here So you'd go over there and try and figure out what's going on They wouldn't let us pair so that was fun So like 50% of the team that the consultants part I was working for ThoughtWorks at the time We wrote tests and then the other 50% either wrote no tests or really kind of bad Like I'm just new to testing sort of tests and that was kind of intense because It developed this very us versus them sort of philosophy You know you probably experienced this before I would guess like maybe we've been on a team where you're the testing Zealot or you've had a testing zealot and he kind of drags everybody into the testing sphere, right? But then everybody else is kind of like oh man like I wrote all those code and it works and now I got to like write some tests Which is of course wrong you should wrote the test first, but They would see this the test is sort of an impediment or they would do this major Refactoring and it would break a bunch of tests because they were brittle tests And so they would learn this sort of like lesson that like tests really slow you down and it was this You know sort of a very strange thing because it was sort of a dictated level of testing So there was a lot of backlash But it was all very subversive and nobody would ever say anything so you'd go like hey you like testing right and they go Yeah, sure, but then you'd see like his next check-in would have like 20 change source files But no change tests and you go like he doesn't care about testing No offense so It was a weird thing because we had sort of too much testing from the team's point of view But also it really wasn't a very good amount of coverage right like and we didn't really have Confidence to go in there and just throw things around we had some very big problems with the project But we didn't have the confidence to change them After that I moved on to the VoIP sign-up app so large voiceover IP product provider Needs to have a sign-up app and what they do is they hire us to come in and we're gonna write this thing for them And it's like you think oh, it's just a wizard it collects your name and your phone number and it collects a whole bunch of things And then it just hooks you you know sends out you off a box in the mail But I didn't realize like how crazy important this thing is because in the VoIP world. There's a crazy amount of churn So like maybe they'll get 30,000 customers a week, but they'll also lose 20,000 So the 30,000 has to keep coming in the sign-up site must always work If it doesn't work then they're still gonna keep losing customers, right? And then they're gonna like be in a very very bad position So this was like a really high level of assurance was needed to keep this thing up and running But we had just just sort of crazy team one of those crazy teams you get on and you look around at everybody And you go like that guy is smarter than me that guy is smarter than me that wait a second You're like oh my god everybody here is just like kind of like crazy good level So we had Ruby on rails and dust with dust is sort of an our spec like Thing that was popular at the time and also the guy who wrote it was on the project And we had near a hundred percent coverage. We touched no databases no No outside things so we ran all our tests in something like 10 seconds, which was pretty awesome We also had this cool integration thing that would run as rails integration tests Locally, but when you checked it in the cruise control server would then run them as real selenium tests, right? so they kind of did double-duty it was kind of a neat sort of DSLE type approach and Then we could actually run the the selenium tests in Firefox and Internet Explorer Which would drive out a lot of weird bugs that of course since we're developers and we all use Firefox or Safari We would not see and we caught those things really early, which was pretty awesome We had like ten different builds like we would literally have the Firefox build the selenium build the quick unit test build We had a build that migrated up the database and migrated back down and then did some checks to make sure it really did migrate the Right way, and then there's like one of that's four. There's six other ones I couldn't remember I think there was like tests of the old version tests of the new version and just a lot of Really cool things It was a very advanced testing suite, but it's a very advanced team So a lot of times like this selenium suite would go down and I have some weird problems But we would just like throw some developers at it and beat it until it like came back and we figured it out so In looking back on this sort of app. I think this is like Any other team I've ever been on this would have been way too much testing this just wouldn't have worked out the team would have rebelled management would have rebelled and You know the code would have suffered because we would have spent all this time sort of babysitting this code This testing suite that nobody cared about but this team and this project just sort of demanded this and could support it So we were able to actually kind of have like ridiculously well tested code We're at some point They actually introduced in like hey You know like there's this one path through the app and now there's going to be two paths because there's gonna be a mobile offering and The other one and we were able to like rip out like just huge chunks of code and put it back in and have Almost zero bugs because we had it really well tested and we knew what was going on So social behemoth part two so I quit ThoughtWorks and Then I get hired back on to this social behemoth, which is kind of weird So I'm back as a different contractor But it's still working on the project and things have sort of changed like there's sort of a bunch of new People on the project. They're up to around 60 or 70 percent coverage They've had three separate attempts to add Selenium which have all failed Yeah, we'll talk more about that in a bit But yeah, so like they had hired like a bunch of contractors come in and Selenium They had hired like a one person whose job it was They'd also at some point hired Jason Huggins to put in Selenium, but then he quit and went to Google So that didn't work so well He now actually works for Sauce Labs, which I think I'm gonna mention a little bit later but so It was more the team was more test friendly, but we just had all this technical debt Right, we had this controllers that were like 600 lines long Like not just the controller, but like one method was like really long and it was just a big like hey Are you something well then do this is a something well then do this It was like this switching on type thing which is like textbook refactoring right like you're switching on type What are you doing? so it was like kind of Intense for the developers right because they realized they had all this sort of gook to overcome that Sort of happened because the the original stuff had been written really fast and hadn't you know been sort of well designed So it was kind of coming along it was kind of getting there when I left this project you know Then I got to work on like a this a tiny little startup It was this just this little inkling in someone's mind You know this guy he had a crazy idea and he had a little bit of his life savings and he was gonna hand it to some consultants and this probably should sign kind of familiar to you at this point and Having learned the lessons from before I got to sort of apply them here I walked into this thing as the tech lead and went okay We're gonna do this link building app which the idea behind link building is say you sell shoes online Right and you want people when they type in shoes into Google to come to your site But how do you get that right? Well, there's a lot of really evil ways to get Google search rankings But there's also a very good way and the way is you figure out What sites link to your competitors, but not you and Then you write a really nice email to the site that links to your competitors not you saying hey I noticed you have a roundup of all the people who sell shoes I sell shoes Would you like to check out my site and link to me and that works? I mean it's really polite and nice, but it will build up your links and people generally who compile these Lists are very nice people a lot of times they're librarians and they take their work very seriously and Google ranks them very high So you can just sort of slowly step by step build your sort of search engine You know your Google juice But you got to manage all that and the way they were managing it that this guy was managing it right now It's with these huge Excel sheets right that would have all these like URLs like crazy URLs and they had this pearl script that would generate the people that link to your competitors But not you and then they would have little notes like contacted on 1213 said maybe and So we wrote this app that would manage all that to you for you And it would also go out and find the people who link to your competitors But not you and then later it would check back periodically to see if they had linked back to you But never sent you an email So it was a cool site it had to take credit cards It had to get out in three months and it was only two people so we made some decisions That we're just not gonna really have a super advanced testing suite We're gonna basically can test controllers models and helpers. We're not gonna test views. We're not gonna do selenium Now this being said I still probably had like a 2.7 to 1 ratio of code to test Or wait, did I say that wrong? Test to code I still try to test paths through the methods Right, so if I had a method that had four passed through it I would write four pretty quick tests for it, right and that sort of blots your average of your ratio of test to code But it was able since they were all unit tests and I didn't hit the database They all ran in under like 10 seconds, which was awesome, right? So it was a very low burdensome suite even though it was actually a pretty comprehensive suite and the cool thing about this Project is that this sort of let us get out the door with enough assurance that it was gonna work But also quickly so that they could figure out if this thing was gonna make money or do well And it still exists and it's doing all right, you know like most startups It didn't set the world on fire like the guy thought he was gonna do but it's pretty cool So check it out by the way. That's the URL So verdict right for the project All right, last project project. I'm on right now social behemoth part three So I left the social behemoth. I went to work on squid and did some other things and then they hired me back So they're now actually writing a new offering which they're gonna launch in like 27 more cities By the way coming to San Francisco sometime soon so The idea behind this new app was to kind of have a much more like the old app We're acquired an army of dudes to generate content and the new app is going to be much more generated by users So we actually started writing a new rails app, but it talks to the old app through active resource in various ways But that's a little bit of detail behind the scenes So new product we had to sort of develop a testing structure for this But now we sort of had the advantage of like having a lot of lessons learned from before So we have above 90% coverage We have a metrics build because I'm you know, mr. Metrix guy and that's kind of really helped out a lot When I gave this talk about a couple months ago at Lone Star, I was talking about yeah We have like a low-level Selenium suite that just sort of hits the important stuff and it's okay And we've since ditched it I gotta say I have integrated Selenium on a many a site and it's rough Because I feel like web rat and all these other things out there make it really easy to get started with it Right within maybe a couple hours You can be up and running popping up a Firefox clicking through the app and you think well, this is cool This isn't so bad But when you sort of like get it into your cruise control and you start failing the build on it You can get into a really bad situation where this build fails pretty randomly, right? Because your box isn't powerful enough and it times out or just weird things happen when you try to run this You know on a Linux server somewhere that is underpowered so The JavaScript testing has actually been a kind of a boon to us It's a very web 2.0. We type app and it has a lot of JavaScript So we use a Blue Ridge to do a lot of JavaScript testing so we can test first our JavaScript, which is nice So this is sort of a medium level of assurance, right? It needs to work But if it doesn't work, it's not like the world ends So this is kind of a good enough level of testing Yes, moving on so let's start thinking about lessons learned here So there's three axes to my lessons learned so we'll start with the first one What's the right level from the codes point of view and this is a question that really doesn't get asked Often enough right like people sort of develop a testing strategy And they don't really focus on the fact that testing supposed to be about design, right? So I mean does do you feel like the tests that you have right now are encouraging good design Does it feel like when you're writing the tests and you have to you know do something? That's ugly that means that there's ugliness in the code because that's kind of the idea right like the idea Is that when you're sort of feel like oh man, I have to put like 10 mocks in here just to call this controller Maybe this controller is doing too much right and then you start pushing stuff down in the model That's sort of like a thing that you need to kind of periodically go back and circle back and say like as our testing strategy Really helping our code, but also like you know, there's also the bug idea right like how often was our selenium test catching bugs? And at some point we realized Well, it isn't like we just it fails and then we spend like a half a day Like fixing it which was a shame right because I mean in some ways It was a really cool integration level test, but it really wasn't catching that many bugs We also already had a qa team who would hammer on it So it was basically just sucking up a lot of time and not really giving us a lot back What's the right level from the team's point of view? Oh, I'm sorry. I forgot this part How easy was it to write new features up? Probably one of the most important ideas behind this talk is like if you have well-factored code The time you know it is when you jump into that code to add a new feature and you say oh wow I can just take this thing and plug it in here and this thing and plug it in here because it's all modular Right, but if you feel like oh, I want to grab this thing, but it's doing this other thing that I don't want You know it's not modular right and hopefully the tests are sort of encouraging you to sort of break things apart into nice easy pieces All right, what's the right level from the teams point of view also sometimes ignored right you get somebody on the team a Couple of people on a team or a couple of consultants on the team who are a big testing zealots It's really easy to kind of jump in because nobody really wants to raise their hand in this day to age and say that I think test Suck, you know like there's just not a lot of that going on So they'll just kind of quietly kind of go along and be like alright, you know, but if it's a really intense You know kind of high maintenance thing like for instance JavaScript testing is kind of the Wild West right now Like I like Blue Ridge But there will be times when like m.js does something really weird when trying to simulate the DOM and you have to spend some time messing around with it Right. Okay. Well, we've got a team that can handle that and we've got some JavaScript guys who could figure that out But you might not be on that team. So maybe that's not a level of testing that you want to take on the unit test is king right if you start with models and controllers and Helpers that's sort of like a really well trodden path, right? It's really pretty straightforward to sort of get those things to be super reliable and super helpful So I would say sort of starting easy is a good important thing There's also this idea of the top-down thing like what's the level of buy-in for managers? Sometimes you can have managers sort of dictate like I want a certain test coverage or I want, you know This level of testing or I need selenium and that can be kind of bad Right because if the testing isn't sort of ground up you can get a lot of backlash and you know just sort of You know some bad behaviors from your developers Talk about in a few slides So what's the right level from the applications point of view? So this is of course What is the level of assurance that you need? But if you ask anybody and you take somebody who says like I'm gonna take $70,000 of your money now. How is it important is it for this thing to work? They will say like well really important, right? But they might be proposing something like Twitter which you know Twitter was sort of genius in its way to say We're gonna just get it out there and if somebody loses a tweet They're not gonna die nothing terrible happens and then we'll figure out how to make it reliable later Right, it just they got it out there. It turns out. There's an audience. Holy crap. Hey, let's work on it some more Because it's easy to lie to yourself about this level of assurance So make sure you sort of ask why five times like does it really need to work really are you sure what bad thing happens? Did people lose money does somebody lose their life? Does somebody just have to like type in something into a web form again, like you know different levels So this is the matrix Apologies for people in back. It's it's hard to fit this on here so this is me summing up all my projects and Figuring out what level of testing I had and if you'll notice there are nine projects on here and the word appropriate shows up four times so This is sort of me saying to you that like I've spent Like you know five years thinking about the level of testing and learning things and I've sort of failed more often than I've succeeded and That's because it's a hard thing to assess, right? There's there's a lot of ways to lie to yourself I sometimes get really worried when I talk up to talk to startup people because startup people are all like well testing's fine But you know like hey man like we want to just get it out there And it slows us down and and I feel like they could probably be doing more at the model and controller level than they think They can right because those tests are not particularly hard to write or maintain But they're in this mentality of like I got to get out there man Like let's not do anything keep in mind at the lowest level of testing that I did and this was for a startup I was still doing something like 2.5 tests to code ratio that was models and controllers and views, right? I just wasn't doing anything else, and they're not hard to maintain so Try not to you know like lying to yourself is pretty easy to do We need sort of a brutal honesty when assessing like what the level assurance What the cold quality and what the team buy in our for these things? All right lessons learned, okay, so Team is new to testing easier way in right. This is start slow Selenium water tests It's there's I mean I feel like there's this embarrassment of riches riches right like with these modern things It's so easy to get into these and it's so it seems so wonderful that you feel like well We're stupid if we don't do it But if you're gonna do this you really need to commit to doing it You got to have a serious machine to run Selenium tests You need me maybe talking to sauce labs about running your tests in the cloud or whatever paralyzing it because Unit and controller level tests do not take a lot of time and effort to run But these Selenium and click through and water and click through the app sort of tests They need a serious machine and if you have the type of company who's like hey Can we get another machine and it's like yeah fill out this form in triple kit? Maybe you shouldn't be doing this right like you know how how easy is it to get more resources? Yeah, and the other thing is that if you're gonna do integration tests back to the web unit lessons learned Sometimes you can do bad things and replace what should be a unit test with an integration level test right, so you have a test out there that The typical thing I've seen is like people will test the controller, but they're really testing the model Right, so it'll be like oh if I save this model without a name Then it should like blow up because names are required That is a model test that is not a controller test right so write the model test And then if you need an integration test that covers that too, okay, but don't ignore the model the model is much more important to test So this is a lesson that everybody says the fact the test suite has to be fast and reliable and yet How many people out there have been on a project where you have a slow suite or it's an unreliable suite It's kind of common. It's an embarrassingly common And you start noticing these bad things right nobody wants to check in at the end of the day like after three o'clock everybody's like oh Guess we'll just check in tomorrow Because you don't want to stay late because the build fails or you're spending a lot of time on various Websites right because you go to check in you got to wait 20 minutes for a build And the worst part is that the tests are not seen as this awesome Putting a wall up that you can put your back against you know everything behind that wall works So it's awesome right because you can move on to the next thing. That's what tests give you they also help you do design If you're just seeing it is like oh there's this impediment here Then that means that you're you know like the team is sort of you know has to think about what his testing strategy is And I love Unreliable test suites we we just had that one with our selenium thing people just stopped caring about the failing selenium They'll just it wasn't important anyone for anybody and then people would go in and just like oh Like it's supposed to assert that five things come back, but now seven are so I guess I'll just change it to seven It's supposed to be seven it works now So slow test suite fix it right there's ways to fix this you can paralyze it is thing called Parallel specs. It's pretty cool Running on a fast machine or in the cloud. This is a fight to have early and often with management is to say We have a problem. This thing does not run fast enough. Give us a bigger machine External services are the devil, please don't hit them in unit tests Profile your suite and examine slow tests a lot of times There's somebody's just doing something dumb like I saw something where somebody created 35 objects because something happened When you've made more than 34 so they would like put in a before filter like they created these 35 objects But really only one test needed it and there was like 50 tests there But because it was in a before thing it got happen it happened every time right so that's a lot of objects to create Especially if they're active record objects Yeah, you know cuz a factory girl is a great thing right but factory girl makes it really easy to spool up a ton of active record objects Remove redundant tests important So, yeah, we're about wrapping up here. I think this is the last slide. Yeah, so I can take questions now Questions for me. Yes How do you give me recommendations on Implementing sort of the TV culture on a project where there are no current tests And it's very important that the project be well tested. I need to help IT So how do you implement a TDD culture? When it's very important to test It's it's kind of a thing of like striking at the low-hanging fruit will get people into it So like something that's purely algorithmic like one of the best some of the best fun I've had testing is when you're testing like I had to figure out if a point was inside a polygon not so long ago And that's just a joy to test because there's nothing outside, right? You have a point and you have a polygon and you just like Start like, you know testing to see if it thinks it's inside or you think it's outside And then the great thing about that is that just you know is something that is very self-contained and people can say Oh cool. Now. I know that this works. I've tried all possible weird ball cases So now I can move on and do something with the point inside a polygon so like The things where you have to integrate with other things and there's all sorts of variables and you have to like do mocking That's you know kind of intense Maybe you're gonna hold off with mocking and stubbing for a little while and sort of focus on Some core stuff and get people enthused about that because if you throw in mocking and stubbing right away That's it can be kind of a burden right because I know when I first started doing mocking and stubbing I totally thought I got it and then like I look back at it a year later and went oh my gosh I like just did should receive an expectation on it everywhere when I really wanted stubs, right? But when the stuff and when to mock is kind of a mature testing level decision that you have to kind of you know Learn over the course of a long time Yes, I'm back Yeah, it's a good idea the question the question was am I gonna put metric food on something else besides github so I've been really busy for a couple of months and apparently there's this thing called gem cutter now and I Actually ran into Nick Corino. Was he somewhere here? Yeah, so we we shared a shuttle here And I was like talking to him and then I realized like oh my gosh He's a contributor to metric food, which is awesome and then he was talking about gem cutter I was a kind of sad and embarrassed like oh man. I probably should do that So yes, that's on the list of things to do. I'm trying to get this project out the door. So Yes So the question is are we happy with our JavaScript tests and the answer is Not as much as the unit tests It's I like it versus no tests because I've been on projects where you have like really complex Interactions on a page and it's all just like like either in the page in the HTML itself, which is horrible Or it's just this, you know, super long JS file that you have no idea what any of this stuff does So I like the sort of self documentation of the tests I like That you know, I can actually like have a reasonable expectation of what's going to happen But it's it's still kind of a work in progress. I still feel like it's coming along They're not as reliable. Sometimes they'll just fail randomly and like you just run it again. They pass It's it's not as mature So it's getting there like a couple of years like two years ago I tried to do JavaScript unit testing and it just wasn't there and I had to give up so Better than before getting better every day Yes There's stuff on me So the comment was you'd been frustrated with Blue Ridge, so you created jazz RB J A Z R B Based on gas and set of screw units kind of this Like the stack trace is all screw unit and it can be hard to figure out like where was your problem So, yeah, I mean those are all all these things I mean I made it so there's a jazz RB stuff Runs against Jasmine, which is like screw unit, but it has So I work very heavily on the mjs stuff, too J A Z I didn't want to have Jasmine every single time RB and it's not just in specific I actually added two unit to it. It's just I picked the name very quickly Yes What we found is that 90% of the failures in our apps having to do the dump strip jQuery that the Dom structure of the new change and open the jQuery behavior is no longer being attached Inclusively with class or IDs Modified so if you add these types of integration level assertions to say that a particular You know class president that moves that's where you're a jQuery behavior comes in I think that's an appropriate trade-off where you're not So good comment the idea is that When you're testing JavaScript a lot of times you're testing against like a stub something that has an ID in it But maybe your designer later comes along and changes the ID or the class and now your ju unit can't attach to that right so you you could do a very you know a view unit a view test or integration test that Asserts that the things that you think are in your DOM are actually in your DOM comment on that dr. Nick's fork of Blue Ridge as something where you can generate the your your fixture From from code So you would avoid that for them. So I was writing something like that then dr. Nick said he'd already done it So there apparently is a folk of Blue Ridge that Creates the the stub from not not really a stub. It's actually from your actual From your actual view now. Does that work with Hamel or is that? Yeah, it uses it uses our aspect view aspect view test to generate it Yes Your application So the question is why should I test models? Instead of testing models through the controllers if the controller is more of a public API And it's going to be more stable where the models could change more often And one of the short the short answer is math that like if I'm testing a controller There's a certain amount of paths through a controller, right? So so I have an index action. Maybe there's five paths through it, right? I got to test five paths But now if that index is interacting with three or four objects, how many paths are there through that object, right? So now I have to do Okay, five times I don't know something times something and I have now like I should write like a hundred tests Maybe for this one index and most people just don't do that, right? Like it's very hard to exhaustively test the path through something when you're doing what? You know it's sort of a you know it to not purely a unit, right? Where you're combining a couple of things sort of you're testing a locus of things that interact together? It could be very robust, right? If you're willing to do it because now all you're doing is basically saying that like I want this behavior out of the Controller and then you're free to do whatever you want to your models as long as they behave the same way But it can be very rough to make sure that you have extensive testing through it Yes With your selenium test had you tried using or was it available like cucumber stories to make a DSL above your So didn't you try cucumber? No, we did not we started out with Polonium because we had a pivotal guy on it and he liked a polonium and then later we moved to web rat We have not used cucumber The problem was never really writing them writing them wasn't too hard. It was the problem was getting them to run consistently Yes, over here Yes The question is how how do you write a test suite that allows you to refactor code that's the hard problem, right? I mean like really testing is supposed to be about design and you just keep iterating on it You keep going like are the tests helping me or are they hurting me is this thing like brittle and getting in my way Or is this Forcing me to start thinking harder about my models and like wait Do I have too much going on in here? Do I have too little going on here should I put two models together and that's object oriented, right? You have to know objects in order to figure that out The best thing the test can do is help you start asking those questions I have to wrap up because we're we're after two o'clock here But if you want to track me down in the hallway feel free Thank you very much for coming to the talk