 Okay. Hey everyone. This is my talk. My name is Francis Wong. My talk is testing heresies. This is a Byzantine mosaic of St. Augustine fighting the heretic. Apparently I wasn't wearing any pants. And now I want to talk about testing. I'll let you know up front if you don't do a lot of testing and don't see the value of it, this is not a talk that will try to convince you of the value of it. There's plenty of talks you could see or things you could read about that. This talk is about how testing has been adopted and evangelized inside of our community. Largely with Rails but you know with Ruby as well there's a lot of sort of overlap. It's going to assume a good amount of Rails knowledge for you. But it's kind of about testing in general. And yeah but you know in case there's any doubt I love testing. It's a great thing that we're doing it. I have some qualms with the way we've been talking about it. Let me tell you where I'm coming from so you can sort of get a sense of the context of how I see the world and the experiences I've had. I've been programming mostly the web for more than ten years. I have never written life critical code which is to say I've never had to worry about someone getting killed by a bug that I pushed on a production. That's probably most of you. But if you are writing life critical code I don't know how applicable all this is to you. I've written my code mostly in agile context and people have told me many times I guess this is bragging but that when I'm writing in simple and stable domains my code is very clean. I've been complimenting that a couple of times which means I get thrown at the stuff that's harder. I get thrown at domains that are complex and I got thrown at domains that change over time which is going to inform a lot of what I'm talking about now. So you know heresies are counterpoints to dogmas. Dogmas are sort of beliefs that communities hold on to sometimes against logic or sometimes they don't brook discussion. Sometimes they don't want to hear them challenged so that's sort of the point. This is the first one I'm going to start with testing in Rails is easy because we have fixtures. This one is a bit of a straw man so I didn't really bother writing a heresy for it but you know if you've probably a lot of you have already been through this but basically this is your basic fixtures file. It's a YAML configuration file that's set up stuff like in your user's table whatever. It works fine in your like one page blog post telling your Rails newbie how to like test but as soon as shit gets hard then all of a sudden you're using YAML files a goddamn mess and you're like why is Tom who had to ban Susan for making comment too in this file when there's nothing to do. I don't know who Tom is in my test right. Now Rails deserves a tremendous amount of credit with getting a lot of people to be better in their habits. You know I don't want to take anything away from DHH's marketing efforts. I don't want to take anything away from Rails scores or the whole movement but I do think there's a difference between sort of essential complexity and accidental complexity right and I do think you do a disservice sometimes to people when you tell them oh this stuff's easy and they believe you then they jump in and so I see this in the New York City group people show up to our hack fuss all the time and they're like I don't know what's wrong I must be using fixtures wrong and then you have to tell them again and again no fixtures suck and you were lied to. So this is a dogma this is a dogma testing is easy it's not easy it's hard but that's okay easy jobs don't pay that well right that's that's the thing it's hard but it's worth doing anyway right we we have serious jobs there's money involved right I mean for most of us I would imagine these days right so so you know let's be a little professional not professional in terms of us having to wear suits and ties you know but professional in terms of actually taking this off if this is an essentially hard problem let's treat it like that let's be adults so here's a dogma with tests I can write code with confidence and that is true but it is it's worth complicating some extent keep your eyes on the testing perimeter this is my like funny little made-up term and try let's let's say this is a some kind of fanciful map for your application and the things you've tested off in one corner is your domains maybe user classes and over somewhere else some controllers and stuff and you've got a couple of inner classes and so on and so forth you're zipping around in that territory feeling pretty good about yourself your bug rate isn't very is very low every time you refactor it's a pretty easy thing to do right and then you kind of have to stretch out into some new territory and this is what happens right I think actually it is worth noting that when you've been testing for a while you may not notice it but what ends up happening is that there's two kinds of code in your application in your system there's a stuff that's tested really well and the stuff that's a giant pain in the asset test they're totally different to work on right and it's you will probably always have it but it's worth it to think about it right it's worth it to be conscious of it when you're in the stuff when you're inside the parameter you can tear around really quickly and you can just tear the guts out of stuff and refactor all the time but like you know if you're running a Rails app you have a ton of Ajax logic and you know testing JavaScript is just harder than testing Ruby code you know like when when someone comes along it's like oh yeah I want to totally redo how this JavaScript logic works you have to be more cautious right that's just the way it is so you know what's your testing perimeter right the application you have and the team that you're part of right what layers are there that are harder for you to test for technical reasons right what domain areas have you tested less for whatever reason and what did you write in a hurry right because code lives code is not without context right code gets written by human beings who are imperfect and sometimes decisions don't get made in the right way and then you know dogma more tests equals less bugs always again this is worth complicating less bugs per week or less bugs for feature this is vaguely related to the thing I just said right because when you are inside your well tested area you can just tear around really quickly and the odds of you adding new bugs are very slim right but the faster you move and if you're not paying attention you can slip across the perimeter so to speak right and so I think it's there's there's different ways to test there's different things to prioritize on right so this is my fanciful formula testing is velocity times quality right so on the left hand side you have one way to do testing which is as you have more testing your features per week increases but your bugs per week more or less stay the same this actually happens I feel I've definitely seen this in in in projects and teams your code gets more and more solid it does more and more things but the number of times that your your clients have to email you and say hey this one page when you click on this link it just breaks stays more or less the same now with your clients notice that you actually have less bugs per feature or are they more sensitive to bugs right so these are these funny kinds of things about how the relationship works on the other on the right hand side you're talking about actually having the same features for we but actually driving bugs down so it's not about moving faster it's not about putting things off the checkpoint it's about being rock solid so if you were for NASA you probably think this way right if you make pacemaker software you probably think in that way right but if you're writing like some kind of social network tool that like plugs into Facebook and uses flicker to send things to Twitter or whatever like right like bugs per week is probably okay marking is the right day way to deal with test data this is a big one I've left about after I'm hoping to leave about 10 minutes at the end of this talk for mocking proponents to yell at me this is why I don't mark mocking focuses on services but state is what matters so let's kind of go through some terms so so mocking as as most of us understand in the testing community is this idea that you will use surface calls right and and this is pretty prevalent in our spec you can use it in a test unit and so on and so forth this is a right from the aspect documentation right so you're testing a per people controller and then you're you're saying oh the person class let's not actually go to the person classes normal implementation right let's intercept that call Ruby's very dynamic we can do this and say oh when the person gets new it should return this variable person right when person receives new and then you have the should receive and should receive create with name points to as lap right this is right out of the aspect doc this is probably what a lot of people see when they first see mocking now there's a there's an article that's actually a couple years old that's by martin full up father that's a really great introduction to sort of the breadth of this subject it's called mocks aren't stubs and in it he quotes Gerard Nazarus who sort of has breaks down kind of test doubles that's anything that pretends to be anything else in a test context into four categories so you've got dummies and those are passed around but never used so a lot of times you're dealing with fake objects inside of a parameter list this doesn't really happen much in ruby but if you were writing some win 32 API maybe I don't know right fakes which actually work but are unsuitable in production so there you talk sequelite for example is a pretty good example of a fake it actually works but you don't you know you don't deploy your rails app with sequelite or a lot of us don't stubs provide can answers to calls made during the test may also record safe for information so that's a little more safer and then mocks which is showed these are these the top three are kind of of one kind in the bottom and mocks is sort of its own right the top three kind of care more about state for example there are ways that you could make two different kinds of call into the same stub or fake or whatever you could do find or find by sequel on an active record thing right and if they actually return the same results you don't care what was called right it's sort of the underlying results it's not whereas mocks is very much about the you cut away individual layers right and this is exactly what it has to talk to on the layers blow it so Martin father says breaks it down into TDD classicists and TDD moccasists right so classicists use real objects whenever possible and then a double if it's awkward to use the real thing any kind of double really and then a moccas will try to use mocks for any object with interesting behavior so what you've got really is moccas to the idea I think and you know hope I hope you forgive me if I'm accidentally mischaracterizing the position but the idea with mocks really is that you're going to slice away your your system into a bunch of discrete layers you're going to really every little part of it you're going to wrap it in sort of oh this is the stuff that happens right above it and this is the stuff that happens right below it and you're trying to think of the controllers and isolation the models everything else right classicist is much more sort of feature by feature I'm going to write this one feature I'm going to push it all the way down in you know up and down each layer if I can now I have a couple problems with mocking number one is I'm too dumb to think of the problem and the solution at the same time and I feel like mocking really makes me do that so let me let me give you sort of an example this is from some code that hopefully is seeing public release on Monday my company diversion is working with sling.com to make a site that aggregates professional video it's very similar to Hulu in a lot of respects so you have TV shows right the office so on and so forth and then you have a show list page right show me all the TV shows that are on this website now it only makes sense to show a TV show if it has one video you can watch right because really videos are the center of the web so this is not IMDB right so we have a thing that says okay let's speck out the list page let's create a show so this is the this is the classicist approach to it this is how I do it right create a show which is actually save it in a MySQL database right then create a video now it's dead it was it's dead as time now minus one hour right so you could you could watch it in the past you can't watch it out expired and then show update public videos kind of that's a little cron thing or whatever and then you get this and you say response body should imagine so the name of this show some new sitcom should not show up in the response body right so that is a classicist approach in the sense of let's try to set up the entire universe in which this case might be interesting and then we'll try hit it and see what happens this is a mockest approach and this because this is actually what's happening inside this controller method show should receive fine with all select blah blah blah conditions shows public videos count and return empty array that would be sort of be the mockest approach now the the logically interesting thing is what's happening is conditions is shows public video count is greater than zero if you want to break that out but the other problem is that the select is in here the reason the select is in here is because we've optimized this method right of course we've optimized this now we're optimizing a ton of stuff we're like totally paranoid about performance at diversion and what will happen is you'll write it first without that and then at some point someone else will come along on optimize it and then this spec will break right because the surface call has changed even though rationally the exact same data the exact same html got out so that's why I have a problem with this kind of approach because you end up kind of binding on these fine-grained layers and here's the other thing if you have a really tough bug that actually got on to got on to production it's going to be extremely difficult for you to write a test a spec with mocks that captures the right layer call we have video ingestion so we actually take XML posts that say oh here's a new video from some syndication partners CBS or how cast or whatever right I introduced a bug into production early on that was not properly deescaping right so you can see this is about a test that's like we'll get an XML right and we'll put its title single apostrophe double quote ampersand right less than a greater than that's every single those of those are your core XML things that get escaped and then I'm going to execute this ingestion post and I should have a video as a result of it and that video full title needs to be exactly that right no ampersand APO a semicolon right at the time I wrote this book first I see the ticket or you know someone reports the bug to me and then I start digging around at the time I'm writing the spec I have no idea what I did wrong why would I right if I knew what I did wrong I wouldn't have done it right so and somewhere in the guts of video ingestion post there's a rec some L parser right that's and I did some wrong call I like I called the wrong thing in expath or I used one method instead of another method what the hell like why am I supposed to have to think about this when I'm actually it's hard enough to actually capture the bug so that's I you know there's a difference between sort of black box testing and white box testing I like black box testing primarily because it's you know I can sort of say okay right this minute all I'm thinking about is the problem and then I can sort of switch to the other side of the fence and say okay now I solve the problem I just wrote down the other thing is that both can sneak in between layers this is buggy code but you won't know it if video has a make live method and you're saying show should receive make video live then notify subscribers and then below that you're actually testing make live and notify subscribers you've introduced a bug but these specs won't catch it right because the layer of interaction is actually the fact that show should receive will not actually do do what supposed to do you know and and so this is a question right like is that an interesting bug I think it's interesting if it's going to get out to production so I'm a TD class assist you know like Martin Fowler actually what does that look like concretely that means every single time I'm writing tests in a Rails app or whatever if it's in the database I'm actually running a my SQL database in a test context right you don't do SQLite because if you end up having to do any optimization or any kind of specific stuff then then SQLite isn't really gonna work out and the fact of the matter is I guess I'm a believer these days that like this idea of like complete database abstraction is like a bit of a is a bit of a pipe dream unless you're actually writing something that never has to do anything hard in a database then you should just go ahead and have that be bound up with your tests and then when you do use test doubles it's always a big deal and you do from time to time but it's it's it's never a light decision so here's one ensuring that an optimization is being used videos come in from syndication partners when we get this ID from partner right now now this is not globally unique right this is a but but and and the thing is also that it could be arbitrarily long you'd like it if everyone was using numbers but they're not sometimes you end up having to grab a URL or something like that so we this is not a varchar field this is actually a text right because one day in like six months we could sign up some syndication partner it turns out that all their IDs are actually 600 characters long and the first 400 characters the same if you truncate a 255 you are like fucked beyond belief right because you have 200 videos that all seem to be the same thing so make a text now here's the problem is that you have a hundred thousand videos and if you have to look them up buy this ID from partner field it's very very slow so there's a bunch of problems ways you can do it you could try to do different kinds of indexing inside of my sequel but we decided to actually add a hash code field right there's just an integer right so you'll take the partner ID in the ID from partner you'll hash them together into an integer you can look up by the integer and that integer field is very indexable it's very fast look up maybe you get one record if you get three then you iterate through and figure out which one you want right this is this is kind of a you know this is what hash codes are really good for this is kind of what it would look like on the video model right so you're saying oh this is the path this is the defined hash code function right partner ID ID from partner put in a string call dot hash on the end you can hash anything right and then we're actually going to write find by partner ID and ID from partner and then we use the hash code in there I left that logic out but okay so this is this will work this will work just like find all find comma first blah blah blah with conditions bloody blah and so it's a different if it's a difficult challenge if you actually want to test this because now you have two ways into the same data they do the same thing but one of them is just a hundred times slower so it's actually a good use for mocking because you want to actually make sure don't use this surface it's slow as shit use this service which is faster right what's the way to do that this is kind of how I ended up writing it it's it's sort of like mocking it doesn't use our spec mocking but I actually overwrote the find method and say if you're calling find first with these various conditions partner ID and ID from partner in there I'm just gonna like freak out on you right so any test in the video ingestion post is going to have a serious problems you go through that one route you basically like put a roadblock in there right so that's actually blocking the surface interaction which is exactly how you want to do it at the end you have to do after all and you have to remove that method this is this is not easy to do and I actually got it wrong the first time and broke everyone's test week so that's an example of why you would do that and you know pretty hard right yes I mean the semantics are tricky but I think it would be a mock because you're primarily concerned with service interaction and I'm not like mocking isn't dependent on a specific library or syntax for actually doing the mock it's really just it's really just you can't get these results this way because I like you can get them another way right the other thing is you know you're calling super so it is actually just blocking sorry yes sorry so he asked would that be a fake instead of a mock so I think this still qualifies as a mock yeah either way it's it's a little bit of work to do you don't do it all the time you don't do it lightly you know yes can you say video I don't think there's a bunch of different kind of mocking libraries there's the one that's basically in our spec now and I think there's a couple of people who are trying other approaches I don't believe the current aspect mocking library will give you a good way to do this because in fact this conditions and the tricky thing is the conditions could be a lot of different things and the fact I'm just regexing it right so I don't think and and yeah so I don't think the current aspect mocking would give you a way to do that but I wouldn't be if yes people are talking about that if it were possible to do it with our specs mocking as it is now I wouldn't be opposed to it at all I it's yeah the second example is faking an external web service and this is like insanely involved but I'll walk you I'll try to walk if you don't anyway sling.com has a restful member service that is not written by us it's not written in Ruby I've never seen the code for it I have no control over how it's deployed right it's just out there right and it's the kind of a single sign on any sort of thing which makes sense when you're dealing with like big organizations that might have a base of members that are shared across multiple applications if you were working at yahoo you'd have the same problem so we've rails and it can talk to this restful member service it's it ends up being a pretty complicated member service so we end up writing a method a class in between it the member service class which has all sorts of methods authenticate you know member with email member with pat pseudonym blah blah blah that talk directly to the rest service on behalf of the rest of the application right so they kind of you're wrapping all the restfulness in this one class it's pretty complex so you have a test on the side you have member service test this is a test unit example it doesn't really change the change in much okay so you have your rake test right are you really going to run your member service test inside of your rake test you're going to have a test suite that like hits this restful thing every time you're probably not it's insanely slow member service test by itself maybe takes like four or five minutes to run but the other problem is that since it's a database that we don't control it could be down for a bunch of reasons they could be in the process of deploying some experimental thing they incidentally when we run on tests or in development mode we're running against obviously not a production instance where they've got a staging kind of instance right so but you know you don't want a situation where you everyone else is working on totally unrelated stuff and like 20 of their tests break and then I have to go I am some guy and he has to go I am some guy and look into whatever everyone stops working for half hour right so you take it so so our process is basically that you won't you can run the member service test it's standalone it's not part of rake and you don't run it all the time you run it whenever you're touching that layer right whenever you're adding another function or trying to refactor a thing in there because it's a big deal and it's a sort of a pain right and then of course you shouldn't be actually hitting the member service in the rest of your tests there are actually a lot of other tests that as a consequence of other things go through log in forgot password blah blah blah so we ended up writing a member service stub right which is intended to basically duplicate all of the functionality of member service but obviously it only lives within the Ruby process and it it it has lots of little backdoors which is what you end up writing in here right so you can say oh the next time you call anything on the member service stub raise a connection error because I want to see how my application handles it right raise all these various errors and so on and so forth you can sort of tune it and all these you set it up in all these different ways so it's underneath all the other stuff you're testing but it can sort of like bubble problems up and you can try to catch them the right way problem is that the member service stub actually gets complicated to so you write a member service stub test which you need because it's a complicated thing and and that's you can run that in your rake test because that's really fast and that's all inside of Ruby it's not testing touching any kind of rest anything right and then the last step ends of being that you can sort of see that we've got on both the member service stub test and the member service test you actually have common test methods you have test can't update email through modify member which actually makes sense right because you have the member service and the member service stub the member service stub is trying to look exactly like the member service so in fact a lot of the tests should be the same so you break it out into a module and now this is kind of what you left with you have these member service test cases methods you have a couple of things that are unique off on the side but a lot of it ends of being in this member service test cases file which is defining most of it I'm glad we did this work I think in this case there was a tremendous amount of complexity that we we're going to struggle with if we didn't sort of figure out how to wrap and abstract it but you know there's no question it was a tremendous amount of work you know and this is this is the right way to do it if you if you try to not have a member service stub test then you're exposed to actually having bugs in your member service stub that give you false positives right you think that your code works when it actually won't when it runs against the real thing incidentally did anybody here see me speak in RubyConf three years ago top-to-bottom testing one person to okay a couple of people all right I spoke about top-to-bottom testing and I talked and I was talking about Lafcadio which in which is an ORM I had written Lafcadio's approach to testing was different and I believed differently than I believe then you should have actually had Lafcadio should have had a sort of a I think it was like an object store stub or a mock object store or whatever which actually when you wrote a application that included Lafcadio when you ran in a test mode it would sort of switch out and do this totally Ruby in-memory thing that would never touch my sequel and that after wrestling with that for a while I basically changed my mind I think that was wrong because because fundamentally if you're going to have any kind of you're never going to really be able to actually abstract away every little thing that a database does and you should actually just let yourself drop into sequel and then at that point there's no you can't run your tests against something that's non-sequel basically or non my sequel or specific to the environment you're in so just want to say for the record it's wrong active records approach was definitely better and I'm you know I go in that direction now and I think it would be good if you know people on a regular basis would just get up and speak in public and say I was wrong I think we'd all be better off better here's a dogma and this is related to mocking right because people say well yeah but I mean if I run through the database for every single test I've got then everything's going to take a long time and this is the dogma everyone's always talking about if your tests don't run quickly there's something wrong with you you personally are at fault you are not being virtuous enough right you haven't stuck to the religious line closely enough your faith isn't out I'd also like a pony you know I yes if I could I would have my test run in less than 30 seconds that would be awesome that would just be great but given the traders I've got I'm going to just stick with tests that take a while but help me out in other ways so let me let me take a poll the room who here works on the main application where the test suite takes more than one minute to run all right more than two minutes okay more than five minutes so even more than five minutes you got maybe what a third of the room a quarter of the room okay I'm here to tell you today that you are okay that you're still good at what you do for a living I'm here to I'm here to spread a message of love and acceptance and tolerance and I want to thank you all for coming out of the closet with me as I let you know that my tests take a little longer than I would like how often do you run the test? sorry the test takes so long how often do you run them? uh I try to run them basically what ends up happening at our company is that people sometimes kind of use their it's not ideal so I'm not saying we're I'm not saying we're at the promised land yet but what often happens for us is people say well if there's a bunch of stuff going that I've done or merges coming in that make me suspect I'll run the whole thing and wait but if it's some little tiny thing I'm just going to try to by hand run just a couple of files and then check in and hope probably we should be doing some continuous integration thing with that but that that helps to some example that doesn't really fix the problem you know but but yeah so that's what we're doing now it's not perfect yes one thing I've noticed that was helpful is if the tests are taking five minutes and you use auto tests it wouldn't matter anyways auto test will rerun any test related to code that changed and then if once in the background and you don't feel it right there happens just pop in your face so this young man is advocating hey Ryan are you here I'm here okay now well okay Eric you can beat me up for Ryan I don't use auto tests you jerk do you want to know and it's and I'll tell you why I'll tell you why it's not because I don't like Ryan or Eric I like you guys look it's because I find that if I'm writing a whole system-wide thing auto test no computer program is going to be smart enough to figure out oh you change this model but actually you're doing it to support this thing on a controller so on and so forth so my experience I've tried it a couple of times and my experience is that auto test doesn't really work with the way I want to write tests you can change it okay maybe we can talk about this later maybe I just don't want to probably get it actually after it passes if reruns all tests but it doesn't pass that's when it just reruns the last test that was failing so it does make sure that there's no way to look for whatever reason I've it's it's been extremely hard for me to get auto test to work with a lot of my workflow actually ends up being this which is which is I'll write some very top level sort of integration test or whatever that explains kind of this very top level story of like well if I I forgot my password I get this email then I click on the link and then I forget whatever and stuff I'll run that that fails on the first step that's fine that's like some 40 line test hopefully you know hopefully less than that then the next thing is oh I need to write a thing on a model to maybe I need to tweak auto test but then I need to write a thing on a model to support this thing that I haven't even hooked into it yet so I write the thing on the model it seizes the change and runs the integration test again of course it broke so it yeah it depends on your testing style for whatever reason many people I know have just kind of support your point that you can buy a test that runs slower and with auto test it wouldn't be a big deal like that's the other part I was trying to say I'm glad it's working out for you and I'm sure it's working for other people because a lot of people use auto test so yeah it's only natural I think the tests are going to run long and at a certain point right like if your application is actually big enough which is often a sign that it was vaguely successful in its early stages it's just going to take a while like right Rubinius's tests don't run in less than 30 seconds that's because it right yeah it does a lot of stuff yeah there's nothing wrong with it I think I think it's also worth noting I'm going to be go out on the limb and be optimistic about sort of new efforts to use you know homebrew parallelism to fix this DEVER is this very early stage are any of the DEVER guys here actually DEVER is an early stage Rails E startup that is trying and they're very alpha and their stuff could be totally vaporware actually but what they say they're going to do is they're going to set you up with basically having your code repo hooked up into an EC2 cluster so as soon as you want to run a test suite or something it'll get all the code changes out on maybe 10 machine 10 instances or whatever and run them in parallel which makes perfect sense because tests are supposed to be isolated anyway right they should be parallelizable right and they have written on their blog they have the first alpha customer they got someone whose tests took what is it yeah 40 minutes down to two minutes right so if DEVER DEVER or another company may bring this to us but this makes perfect sense to me cloud computing is getting easy and blah blah blah so this is a this is a parallelizable problem so we will attack this in the next couple of years also it's worth noting Brian hey Brian are you here Brian Helmkamp sorry I think it's probably more than one maybe Brian's not here Brian Helmkamp's working on a thing called testier which does the same thing sort of locally that's more like you buy a couple of Mac minis maybe five or 10 and put them in the closet they listen over bondure and try to grab stuff and so it's a it's the same sort of thing I think there's no question a lot of wrinkles will have to be worked out but in theory I think the idea is very compelling it's a lot of future there so here's another dogma which has which is vaguely related to the mocking thing which is your code should always be well structured and and I hope this doesn't get me fired saying this in public but I don't believe that's to the case under structuring can be an agile practice and good tests can help you get there so what do I mean when I said that this is a diagram I kind of cribbed from lean software development by Mary and Tom Poppindia great great book by the way you know and the core one of the things that started allowing agile to be a actually a legitimate you know an easy any possibly even one day mainstream practice in our business is the fact that cost of change is going down for so many things there are many many decisions that you could change later on they won't be that much more expensive right so and then and then there's changes in high streaks constraints right so you have to isolate which decisions are expensive to change later and make them early and make them right so using a database is actually you know if you want to try to switch from MySQL to Oracle in the middle of the project that's going to be really painful so you know early on you're making a commitment to that but you're not making a commitment to like oh what do we name this model do we do this with a Boolean field or an enumerated no no no right you don't care about that stuff so the the great thing about agile is that you most things are on that lower curve right the cost of change does not go up much over time but the other thing is that it is an agile practice to try to push as many things as possible from that higher curve down to that lower curve right if you can say more and more things are okay for us to defer then that's a good thing and I think actually being a TDD classicist helps me do that if you sort of imagine a fanciful blog post controller that's creating a blog post and then maybe it's doing something incrementing some counter doing whatever on a user and also on tags this might be this is sort of a very quick and dirty sort of visualization of what that would look like in a in a mocked way which is you're testing all the layers in isolation right so you've got the test on the blog post controller you've got should receive so you're not actually touching the blog post you have tests on the blog post and you got should receive so you're not touching the user in the tag but if you have to refactor this really heavily you have to drag your tests along with you right because you have encoded you've taken time to encode the the layer interactions into every single sort of part of your stack right now if it looks like this maybe you don't have that problem and this is kind of how I like to think of things which is let's think of the surface of things which is probably HTML and web requests in a Rails app I mean if you if you want to go even further you can do sort of a web testing Selenium whatever kind of thing but you know you do this and you just test the hell out of the top thing right and then below it maybe you have a lot more freedom to move things around as you discover it a lot of this I think comes down to what kind of work do you get and what kind of domains are you working in right so this is why I said I get put a lot on very complex domains which is to say I get them wrong the first time right and I get put on domains where where they shift which is to say that sometimes the customer doesn't really know what they want but you still have to grope your way forward anyway right so in in those times when when naming things is even really hard I like to to have my tests feel more like this right I'm gonna do the very top level thing I'm gonna specify the living shit out of it and everything else I'm gonna be able to slide it around I like to make an analogy which is having to do with your apartment this is a a flicker photo from a complete stranger I just looked it up someone who lives in Manhattan in New York City those of you who've been to New York City know that real estate is insanely expensive and people will live in very small places so they can live near all these beautiful restaurants and bars and stores and so on and so forth this is someone's home office I think in I think in Greenwich Village and it's it looks very nice I gotta say but it's it's very precise right everything kind of has a place for it because when you don't have a lot of space you really need to be precise about it I lived briefly with two friends of mine in a very very very small apartment in the East Village it's probably was big enough for one person there were three of us in there this is not this is common in Manhattan and my one friend said God I feel like all I do here is put things away right because it was true you didn't have you didn't have the space to fuck around right you leave your coat out and then your roommates will go oh my god right so here's where I live now this is my home office now it's in Brooklyn it's four stops out it takes me a half hour or 20 minutes to get to Union Square maybe 45 minutes to get the lower east side but I have a ton of space which means if I want to leave a corner of the room messy for a week or even a month I can just do it it's awesome so obviously I've made my trade-offs now now this you know I do kind of wish I could live in Manhattan but so it goes I think of that as analogy with coding I like this is the trade-off I want to make I want to test the hell out of the top layer and go to go to more work to do it so that I have room to make other just I don't have to constantly clean up after myself when it's not that important here's another dogma fine grain focus leads to high quality code and I think this is something that we we see a lot sort of in testing literature because it's all about let's get the smallest thing possible right so maybe this is veering outside of the subject of testing but I feel like it's worth talking about anyway never forget the unknown unknowns and for this we're gonna try to see if you can know can you hear that unknown unknowns there are things we do not know if we don't know okay did you hear that okay now Donald Rensseld is a maniac and they should never put him in charge of a war but but he's onto something right which is which is sort of the nature of how you know things basically some and actually the thing he's talking about comes from this philosopher Carl Popper who I don't know does anyone he actually all right all right so Carl Popper was really into the philosophy of science and epistemology which epistemology is literally the philosophy of how you know that you really know a thing right and although although Rumsfeld's sort of echoing Popper Popper also comes up a lot among some hardcore hedge fund guys George Soros talks about them all the time and so does Nassim Nicholas Tullib now this was a really big pop science book that came out I think two years ago the Black Swan it's actually awesome and I don't read a lot of pop science actually the Black Swan is the idea of highly improbable unimaginable things coming in and sort of how you know they're not going to happen Carl Popper did he did he said you can observe the existence of a thousand white swans but that does not disprove that does not prove that Black Swans don't exist and after he said it years after he said it they discovered Black Swans when they discovered Australia so you know Tullib is actually the funny thing is Tullib's book is all about sort of the arrogance of thinking you know things but he's an astoundingly arrogant writer and he's like really into telling everyone else they're an idiot but it's actually a great book one of the things he mostly talks about finance and hedge funds and so on and so forth but he's talking generally about statistics one of the things he talks about is going to a risk management conference that's hosted at a casino in Las Vegas right and thinking about what major risks does a casino face right there's luck everywhere there's Rules and Blackjack and poker and God knows what right and how much money could you lose that so someone who was like a major risk management person at a casino told them well these are the four events here are four events that either a cost casino is a huge amount of money or came close to costing them right number one Roy Horn was mauled by a tiger they cost the casino 100 million dollars in medical fees refunds lost reservations so on and so forth they weren't even insured for it they were because Sigfried and Roy had lived with these tigers for years and everybody thought it was completely absurd they had insured against the audience members being attacked by a tiger but they didn't insure against their performers because because how could that possibly happen it would never happen and it happened to a disgruntled contractor planned to dynamite basement pillars he was injured on the job he was extremely unhappy with his compensation settlement so he actually got his hands on dynamite and blueprints of the building and was planning to you know basically bring the whole thing down now they caught him so nobody was hurt so the cost is really zero but even so it's a sort of now he writes taxes for years an employee neglected to mail certain casino specific tax forms to the IRS and this person wasn't even trying to commit an act of fraud this person was just flaky and it was this employee's job to actually just put it in the mail and make sure the IRS got it and instead they just put it in a drawer for years and so when they discovered it actually they had to pay this massive massive fine undisclosed fine to the IRS so they could keep their license to allow gambling at all and the fourth one was that an executive's daughter was kidnapped and he tried to pay the ransom out of company funds what's what's interesting to consider about these things is that none of them involve a guy getting really lucky at craps right none of them get involved a guy getting really lucky at roulette and we you know we're all like hardcore math nerds so we love this idea that oh yeah if you roll a seven blah blah blah blah you're out of right you uh I mean we all learn that we learned it in our first year of CS we learned it playing Dungeons and Dragons whatever right and but but there is a certain amount of of validity that is going to fall outside of kind of what is stated rules right so I think I think it's it's just important as we're testing unit testing or what not to not lull lull ourselves into a false sense of security somebody has to think about weird failure states somebody has to think about unknown unknowns which is not not a science the furthest thing from the science so these are kind of like intro systems design issues right how likely is failure and what will it look like how can I find about it quickly and how can I limit its damage right if you have a cron that runs once a day and then all of a sudden it breaks for some completely random mess reasons are you going to figure it out right away or is it going to sit there for two weeks totally unnoticed when you do notice it will you've lost data permanently or can you recover easily from it right and and it's funny because to think about these kinds of issues requires a sort of a perverse imagination right and and you have to plan for failure but nobody actually plans to fail so but but it's just you know it's important for us to not lull ourselves into a false sense of security testing automated testing is awesome for a very specific subset of our jobs but it's not the only thing we do they can't solve all of our problems it's because from my last point which is testing makes my job easier which is probably true testing makes it easier for others to do my job unless my job keeps growing we we work in a competitive field and it changes all the time most of us we're not writing Ruby five years ago some of us were you know and so it's it's definitely you know and continue to think about where we're going with our careers and what different kinds of problems how those fit into our careers right you sort of imagine a 1998 .com or whatever maybe this is how much effort they're technically this is obviously a very fanciful graph it's not much effort there you know the technical staff spends on various aspects of putting up a website in the simplest business logic was a pain in the ass those of you who were around you know those of you who've been programming for 10 years will know like and especially Rails has really like just nailed this right with with active record and getting all the stuff and it's great but it's not like the pie itself necessarily shrinks it just the pieces shift around right who here knows more about systems administration than they did five years ago yeah you know why you know more about systems administration because the domain logic part of your job got so small that it got it's like less than a full-time job right so there's nothing wrong with that I think that's actually awesome it's really really great but we we can't lull ourselves into a sense of security and thinking well the domain logic part is just like really really easy right there's actually you need to be stretching out into the parts that are harder even if they're hard to test so if that systems design sort of on the left complex business logic which is testable but harder you know I feel like you got to go there right because you have to stay big you know even if the business logic goes small that's my talk thanks for listening there must be someone with a question or a comment yes please yeah when you're talking about the fine-grained approach in the unknown does that a fine-grained approach to a certain extent help you eliminate unknowns that you may not know about for example if you have 10 lines of code and you have a test that tests that tests supposedly all 10 lines of those codes there's 10 lines that were something that you never anticipated could go wrong but if you test every individual line on its own with its own tests and take a look at it and delve into what could possibly go wrong with each line then you have a much better chance of covering an unknown that you may be able to know because there was given then did I know here when you said that do I should have a clue okay so it's about sort of whether it would be the unknown unknown thing you can kind of capture it with lots of little fine-grained tests even on every little line I would say a lot of times you can't because you end up I mean maybe I'm doing it wrong but I feel like a lot of times when you write tests you end up saying here's these preconditions the website the databases in this state or the model is in this state now I'm gonna call this method on it right any non-trivial application is gonna have lots of different states and you're automatically always thinking about oh it could be in this state or this state or this state I'm gonna cover all of them so on and so forth if it gets into a state that you didn't even imagine then it's sort of worth asking you know to what extent to what extent does it hold up you know I think I unfortunately I'm not able to come come here with a grand recommendations as you sort of how you handle that because I don't think I'm an expert at all but I sort of feel like like the fine-grained there's definitely a lot of times where the sort of the fine-grained approach to unit testing and so on and so forth is very good for nailing down there's very specific oh I'm gonna treat the string this way I'm gonna parse it or whatever I'm gonna extract this port from this host on this URL string these kinds of things but when you're trying to think of the whole you can't actually keep all the combinatorial possibilities of the whole application in your head and in fact most of them you dismiss outright because they're invalid and you can't imagine how they got that way I think but I do think there's just sort of like a process of training yourself instinctively to know like well this is probably rock solid but this thing I don't know do I need to write these sort of like other layers of cascading failure handling oh yeah what if the queue backs up what if right there's there's some things where it's like oh I'm just parsing a string it's gonna be awesome you know and there's some things and knowing how those things can change and break over time is very tricky error you can't go and can help with the things you want to do that does mutation to testing right right so yeah so have a yeah yes you mentioned uh referring um testing in a classist way I think that's the equivalent of unit testing and functional testing you're testing straight through to the database on a black box level then white testing the injuries units but yeah could be it's tricky because the term functional test is going to be used today no okay but all right but yeah and some people do functional tests that way yes one of the problems I've actually had in just testing um functionality it's like when I'm talking to a payment gateway for example a lot of times the payment gateways the test gateway doesn't actually match up with the test gateway with the production payment gateway system right and so I actually have to use more because it's the only way I can actually replicate the production data that we're actually getting because the production data you know the production system actually has bugs for the test data is the test gateway is actually not a it's not the thing replicate all of the states so what do you do in situations where you don't really have full control of your environment so you don't don't don't don't don't don't so to repeat this point he was saying that he's he's had difficulties for example where you have a payment gateway and the actual production payment gateway actually acts differently from the testing gateway in an ideal world you'd be able to email them and be like test your test gateway fix your test gateway dude but of course no um uh yeah I I think that stuff is really hard with the sorts of things you know depending kind of who's the person on the other side of the phone right like if they really know the thing and they can describe all these these interactions to you very concretely and correctly then I think it's easier to write out a stub or whatever and certainly a payment gateway is the sort of thing that you do really want to stub out all the time but yeah it's not an ideal situation certainly yes hey you uh do you use firing key constraints in your database when you're testing and does that slow things down because you have a lot of dependencies to create before making each of your admins so that was asking if we use foreign key constraints if I use foreign key constraints am I testing it if it slows things down I don't but I do actually try to sort of you end up creating creating this cluster of obviously this is sort of like the object mother pattern that people talk about so not it's not a database thing it ends up being your active record validations it's kind of the same thing to some extent it does slow you down I don't know I I I for whatever reason I don't think it's such a big deal I think it's actually you know maybe this is a stupid way to put it I think it's actually virtuous to be forced to write an extra six lines and say this is how the system would get into the state by which this case is interesting you know I think that's actually a useful mental exercise like it's a little it's a little bit of busy work certainly and then what ends up always happening whenever you do this is you always have these sorts of like you know quick convenience things where like make me a user it'll be the same as the other user but the name will be different these sorts of things and certainly there's a lot of like libraries out there where people are trying to like get exactly the right way to you know a really good way to do that that's that's still a frothy area of inquiry I think I think there's a lot of stuff happening and things are changing really quickly I haven't seen anything that I'm totally in love with yet I kind of do my own thing but I had a question about the slide where you were showing us how you like to write most of the test against a controller yeah you like to play with it bringing it back there so I usually like that approach without having any of the models I write the test as a custom I don't want to think about the design yet I can put all of the implementation in the controller or all I care at first and then I gradually refactor and create block codes and then user and then tag one roadblock that I run across a few times is sometimes as a user I decide I need to reuse block codes with completely different contacts as a different controller yet I don't want to repeat all the tests that I have for this controller that are covering the block codes washed out in the second controller so what's one way to do it? do I sort of repeat your point you're saying that you kind of do like this sort of approach but then sometimes you end up finding that you came in somewhere else and you have the blog post and you have a bunch of repeated functionality there is definitely I think sort of a a bit of a delicate dance for you like actually you know what the blog post thing is becoming real like really solid like I know what this is everyone on the team knows what this is let's let's just cram a bunch of unit tests on it and I don't think there's anything wrong with that you know if the domain is extremely well understood and you feel like it's not going anywhere then yeah just like bang just like yeah write a kajillion test on the blog post I think it depends on sort of the problem you're trying to solve so in that case I would say what ends up happening is you're going to have two things you're going to move some of those tests down to unit tests and there's nothing wrong with that and I also would say that in the process of like if you write this first and in fact you never created a blog post model which is sometimes how I end up doing it you create a blog post model that lives in the code and people look at it for a little while it's completely legit to say okay you know what like one of these controller tests tests how this method works on blog posts implicitly I've got a couple little ed cases I want to really get solid you just add them to the model or test them directly on the model there's nothing wrong with that if you know it's all on like it's a case by case basis right if you feel like your classes and your methods and your interfaces are really solid if you really like how they sound to you and everyone knows what they mean then yeah just like wrap them up right they're done which is great but but just as often there's code that you get out there and they're like oh I don't really this isn't this isn't this is not the best way to express it I can't think of a better way right now but I'm just going to push it out some more Trotter so I think though that what you guys hear is slightly misleading because I think that what ends up happening is yes an analyze has some blog post controller but you should really have a lot of test some blog posts but I think the key here is that you removed all the model in between so it's fine to have a lot of test of blog posts it's good to have a lot of test on blog posts or a lot of test on user a lot of test on tag that makes sense but when you go when you get tested with your model and then you have this if you really now you're making this entire test suite with a whole box you should actually this test is you should probably have a couple more tests on the on the models below it yeah I might agree I mean obviously it's a fanciful it's just a bunch of floating boxes and then an Omnigraph will say yeah yeah and when you do a black box test and something explodes you don't know where it is sort of whereas if you're slicing it by layers you can't usually you can't do you have any practices that go long so that it's problem so the point he's making is that when you do this sort of black box testing without mox and one thing sort of in a low layer breaks for whatever you change something at a broke you might actually create 20 errors all at once which is pretty true um do I have any practices I guess I don't have anything more sophisticated than just saying I like digging through the code you know I mean maybe that sounds ludicrous but but I I still think that's okay I kind of like I mean you know I like the idea also that you kind of there's certain smells of oh if five controllers broke the lines it's probably in some common code right but that's not a huge help I just think that I don't know in my experience that's not that much of a cost but maybe I'm a freak that way I don't know yes please um in terms of the whole modeling thing and the problems that a lot of people experience like what you were describing it's it's helpful to understand that that the way we use models today evolve out of extreme programming so you start a customer acceptance test which are and then tests that so you've got that high level of coverage that you're talking about right now that you're getting at a slightly lower a lot so you end up not really running into the same kinds of problems where you might be mocking a certain method and you change the method elsewhere and now those things don't align because they'll show up in your end-to-end tests and one other thing that I might add it's also the original intent of it was as a design to the point is that I'm working on this class right here and it's got to use this other class that doesn't exist yet so I'm just going to put in this thing and tell it how to interact with that class that doesn't exist yet and now I can focus on the thing I'm doing now and B I've designed in part EPI or the other thing that doesn't exist yet so yeah and I think did people hear the second point there kind of about how you can you can think in layers in isolation and you can if I'm characterizing or you can sort of test one layer without actually having to implement the next thing and sort of know that that's correct yeah and I think there is sort of a you know I don't I don't have any doubts that people who practice mocking are good at what they do and you know sort of I think there is actually maybe a very core philosophical difference which is I don't and Martin Fowler talks about it in his essay which is that you know people who are mockest people who do mockest TDD tend to like to think of their he said they you know they think of their codes sort of in layers right and they want to focus on one layer and then fix the layer below and stuff whereas sort of the classicist kind of actually thinks in terms of one narrow feature that goes all the way down I think that and you know it's going to depend on a lot of things sort of how your head likes to work probably and probably how the team communicates about domain design and stuff I actually like like moving like stumbling forward in this weird soup of classes that may not have quite the right name yet or may meet another sort of inner inner class or some intermediate class to like solve some complicated calculation I actually think like I feel I personally feel okay about that now I want to convince you to change your practice I just wanted to put some context around why my thing is the way it is and then that can actually be used very effectively but there is some pieces that have to be in place and a mindset that has to do with this part I think um may work way over it's like it's 615 so I'm just going to cut it I think we have to go eat but thanks again everyone