 We're not quite starting yet. We have a minute before the the timer officially kicks in but Before I do start I just want to say thank you to everyone for Coming to this talk instead of going and watching Aaron Patterson talk about garbage collection and his cats That's absolutely a hundred percent where I would be right now As an aspect maintainer knowing how the garbage collector relocates pages It's actually like super important to my day-to-day work But I suspect most people in this room are actually application developers and Being able to actually test your code is maybe more important than knowing what shiny new Under-the-hood features the Ruby VM it has I hope or maybe you just like aspect a bunch or you care about open source Anyway, so Yeah, I guess I guess they've closed the doors, which means I should probably Start so this is aspect no longer works with active record My name is Sam Fippin just to be absolutely a hundred percent clear. This is not a feature announcement Sometimes when you wake up in the morning and you're working through your email drinking that first cup of coffee a Line comes in a subject of the email that fills your heart with such dread and Terror that everything else stops and you must immediately focus on it for me that was this aspect mocks issue number 972 Allow no longer works with active record classes And so while the title of this talk is slightly hyperbole in that aspect was not Completely broken with active record This basically is one of the most core and important functional pieces of the aspect gem And if aspect doesn't work with active record classes, then you know, there's much gnashing of teeth maintainers get sad every kind of thing kind of breaks and explodes and you know, no one is happy Another way of saying this is that when aspect doesn't work with rails My email inbox suddenly explodes and I think nowhere in my head Is there a more poignant example of this than a different aspect issue? Undefined method cash in tests for rails 4.2 point 5 point 1 which if you zoom in had 37 comments come in Overnight while I was sleeping you see this version of rails got released late Pacific time Well, I was still living in the UK and that just means I was in bed Mining my own business and I wake up to discover that all Of your tests aren't working every single person in this room who did the security upgrade had non Functional tests and I had an inbox full of very sad email people sending pull request issues debugging trying to work out What's going on? I'm just like everyone please calm down It will be fine and it may sound like I'm complaining, but I'm really really not I'm glad people care the aspect works I think it's the sign of an incredibly mature open-source project that people care enough about how the aspect gem works that the moment there's even the slightest Tremor of an issue and a very specific component that applies to them be it mocking stubbing expectations or rails That they will come in and file an issue but this is a gif of me at the end of the day after work most days and You have to remember that open-source projects are mostly worked on in evenings and weekends and Maintainers have to divide their attention very very carefully Amongst their projects or all will be laid to waste and so support a maintainer hug a maintainer all of that Good stuff So anyway, let's get back to our issue and actually focus in on what we're talking about here The rails issue that I've mentioned affected expressions like this where you say allow user to receive and then some kind of scope Methods in this case. We're using confirmed, but really this could be anything and return something Executing this simple test would raise an exception and I don't know about you But when I look at that I face screaming in fear that Has to work that is a very very simple expression in our spec an active record objects are one of the core things that people Want to mock on this is really bad like when I saw this issue. I was like, oh, this is this is not going to be fun Before we get too far into debugging I wanted to actually do a little bit of an examination of the bug report and think about whether it's good bad how it might be improved and what sort of things we could do as Contributors to actually make this issue better If you look at the full text of the issue as submitted It says what dating a rails 4.2.1 project from aspect 3.2.0 to 3.3.0 I got a lot of failing specs I noticed the issue can be produced simply and then the user provides this test that we just looked at There are a few really really great things about this bug report The user provided a rails version and an aspect version and as a maintainer That means that I can hone in very quickly on which exact things I'm going to need to begin debugging What's going on here? And so if you're filing issues on projects if you're having a bug report with an actual problem in an open source project These are really really great things to provide In terms of things that aren't really great. We don't have much in the way of steps to reproduce We only have a failing test We don't have any of the production code. We don't have any of the rails configuration We don't have any of the other surrounding stuff that that application is Going to be providing along with that test to make sure that it's not some other gem some other piece of tooling something The user has done that is outside of our spec and we also don't have a Ruby version In the modern times basically if you're on Ruby, I don't know say 2.2 plus variances between Ruby versions don't Matter that much anymore. They're mostly stable and work all the time at this point But providing that information can be really really helpful Especially if you're on an alternative interpreter like jruby or just if you think something really weird is going on And so because we're missing a Ruby version here, we might have a slightly harder time debugging now for this particular issue It's not the case But I do think that is a really helpful thing to provide if you do have the time and ability to do so In terms of things that could absolutely be improved with this issue. I don't have a backtrace I don't know what the exception the user is seeing is I don't know where in our spec the exception comes comes from I really don't know what's going on as a maintainer I'm gonna have to go into this like thing and do a bunch of an investigation before I'm able to actually Work out where my bug is coming from and Whilst I applauded this issue for providing rails and our spec versions We don't have extended dependency information. I don't know all the other gems. I don't know what was in the lock file I can't really begin to know whether or not this is an aspect bug a Rails bug or something in a gem, which is monkey patching one of those things or monkey patching user objects These days I think a really really good standard to aim for is providing a full application a full reproduction case that the Maintainer can clone and immediately execute to be able to begin debugging this issue So basically what that would mean is instead of filing an issue and providing a single spec You file an issue and provide a link to a git repository that I can clone Run bundle exec R spec and see exactly is what going is going on because that will provide me a gem file Gem file dot lock all of your tests all of your code and everything I need to see in order to begin to understand What is going on? One of the things that's really great about aspect is it's a multi-maintainer project And so what happened immediately after this user filed this issue is another aspect maintainer Myron comes in and is like Thanks for filing this issue. Please. Could you provide us a complete backtrace in order so that we can begin Bugging this and then the thing that happens next is kind of like one of those amazing beautiful platonic ideal Open source things where a different user to the one who initially files the issue comes in and Provides all of that information and like this is this is how open source is literally supposed to work Multiple people are experiencing the same bug And they're like literally collaborating on the issue and this is like an amazing thing that we can only do because our code is Open and our tools are Collaborative and so like if you are using an open source framework And you're encountering an issue in it and you go to the issue tracker And you see that the issue is immediately at the top of the list because it's a new bug Usually then you You can collaborate right if you can provide more Information than the person who initially did and that information is helpful That's really really great and no maintainer is ever going to tell you off for providing extra data points It's a really helpful thing and I super want to encourage you all to do that, but We're still missing something from this issue We still don't quite have enough information to be able to begin debugging it and really that's the ability to reproduce this issue in an Automated repeatable easy to use fashion, which is basically what you need to debug any real issue in open source these days So a third aspect maintainer comes in and provides us with exactly that Here we have complete steps to reproduce rails new what to put in the gem file Which models to generate and how to write a test and so let's actually work out what this issue is Let's actually debug this issue together live on stage and work out what's going on So I'm gonna go ahead and new upper rails application at version 4.2.0 The reason for that is that this issue was filed against that exact version of rails and like We're now in the future and rails 5.1 exists And so I need to be able to go back to that older version of rails in order to do so once our application is Created we can then begin getting ready to debug this the first thing that we're going to need to do is put our Spec actually into our rails applications so that we can begin Testing our bug and here what you'll notice I've done is actually specified path dependencies to the aspect version Instead of like the actual official gem versions The reason that I've done this is that I have all of the aspect repositories locally checked out on my computer So that I can do debugging insert binding dot pry insert print statements Do everything I'm gonna need to do to actually debug this this also makes switching versions really really easy I can check out a specific git tag and get the exact version that I need in order to do so You might also notice I've specified rake 10 here That is because the current version of rake is rake 12 and the older versions of aspect aren't compatible with rake 12 But that's just sort of a side note. So now that we've updated our gem file Let's go ahead and bundle update and bundle install so that we have everything ready to go and Then we'll actually begin Debugging this issue and making sure everything's good and to do that we'll basically do this very standard rails generate our spec install and That will take a while because bundler is slow Sorry is Andre in the room I'm sorry Andre. I love you and then we'll go ahead and generate a user model Which is beginning to get us towards the ability to reproduce our bug And once we've generated that user model. Oh, we're having some video troubles I had to rebuild all these videos like half an hour ago because they were a bit blurry. So please bear with me So that's gonna go ahead and generate our user model and Then we'll migrate our database because you have to do that. Otherwise, you won't be able to run your tests So because we didn't ask back install on our rails application It's automatically going to have generated a spec models use a spec for us Which we can then pop open and actually like replace the default generated test With the one that's going to reproduce our bug now because we didn't actually create a scope method on this user object I'm gonna go ahead and stub new instead of stubbing a specific scope method Which should actually allow us to reproduce this bug exactly as we expect it If I do that and run my tests, you should be able to see that this test Passes and the reason for that is that we've checked out aspect version 3.2.0 To confirm that this did actually ever work The user has reported that this is failing and we want to make sure that it did work on the previous version as they are claiming So what I'm gonna do here is go ahead and change into my aspect Development tooling repository, which is a sort of meta repository used by aspect maintainers to make working with various versions of aspect more easy What I did there is run a command rake get check out version 3.3.0 Moving us from 3.2.0 to 3.3.0 Then I changed back into my rails application and ran my tests and you can see that they are now failing So we know for a fact somewhere between 3.2.0 and 3.3.0 our tests broke reproduction case Check and We have a good commit range So at this point we have a commit range we have a passing and failing test and that is sufficient I think for most people to start Doing debugging because this issue is small and like probably result is resulting from a small change It wouldn't be that difficult to just go ahead and like stick a debugger in and work out what's going on But we can do a little bit better the year is 2017 and we have great tools for debugging Systems now and for when a regression gets introduced like this something used to work and now it doesn't There's one tool in particular that is better than all the others for working out what's going on and that's git by sect For those of you who aren't familiar git by sect is a tool where you tell git Where the thing was working where the thing was not working and git will move you in exactly hard of the commit range Helping you find where the problem occurs The important commands that you need to know a git by psych start which basically tells git you're going to do this Get by psyched good which labels a particular commit as functional and get by psyched bad Which labels a particular commit as non-functional and by using these commands get will interactively walk you through a series of commits and help You debug and find the correct commit in order to identify where the issue came from Unfortunately, we are slightly complicated in our ability to find the specific commit because there are potentially two repositories at play here aspect rails and aspect mocks For those of you who aren't used to working with the internals of aspect aspect is broken down into multiple different repositories to allow us to Ship features independently from each other aspect rails manages all of aspects integration with the rails gems And aspect mocks is the gem that provides these mocking methods allow receive and so on and at the moment We don't know where in either of these two gems it could be now I sort of take the lead on maintaining aspect rails So I tend to assume that bugs exist in that repository first So let's actually bisect across aspect rails and take a look at what's going on here So here I've moved into the aspect rails repository get bisect starts label version 3.3.0 is bad Check out 3.2.0 label it as good And now we can begin debugging and you can see here gets actually moved us to a specific Revision and then we go back to our rails application. We type bundle exact aspect and we explode The reason for this explosion is that bundler can't actually resolve aspect core equal to 3.3.0 with aspect rails equal to 3.3.0 pre So far so cryptic. Let me explain what's going on here All of the aspect gems are released at the same version numbers so that we can maintain compatibility while aspect provides a completely stable and semantically versioned public API our internal APIs don't have to follow that same compatibility guarantee and aspect individual aspect gems are allowed to call each other's private API is where it makes sense because after all these are sort of one Combined project. So if I have aspect rails checked out to 3.2.0 I must also have all the other aspect gems checked out to a 3.2 x version and in this case They're all checked out to 3.2.0 The same is true for version 3.3.0. They must all also be checked out to that same version But what's happened when we started the bisection is that the version tag for aspect rails has changed to 3.3.0 pre This is this version tag doesn't represent a single commit of aspect But it's instead the version number that we use in the master branch of the repository While we're developing the next minor version of aspect but earlier we checked out all of the aspect gems to version 3.3.0 and 3.3.0 pre is not equal to 3.3.0 because trust me and so This won't bundle There is a very very simple solution to this which is this command git check out head carrot What the carrot character says is git? Please give me the revision that's immediately before the one I'm specifying and head is the revision We're currently checked out to so what I'm going to go ahead and do is Change to every single one of my aspect repositories and type git check out head carrot in what is a furious bunch of manual typing In order to get them all on to the like 3.3.0 pre versions of the aspect gem so that we can bundle our application Once I've done all of this I can go back into my rails application And now if I run my tests they will actually run because the bundle is now Compatible on those aspect versions that are checked out in this case my tests are failing so I go back into the aspect rails get repository Get by site bad Go back to my rails application run the tests they fail go back to rails give by site bad And this happens basically in a loop marking revisions is bad every single time because the tests are going to fail for a while Coffee This is this is what open source is like this. This is my life So anyway, I think we're nearly done Zero revisions left to test after this one. So we run a test one more time. They fail. We go back to aspect rails We try to run the test, but we can't because we moved and I'm bad at typing and we markers bad. We run a test Really, this is you you too could be a maintainer who gets to do this Okay, and get has given us the revision where we fit where we started failing and this says something really interesting It says bunk to three point three point. Oh pre Well, that's kind of curious because that would be a commit immediately after we released a version of our spag So here we look at the git log and we see indeed the thing immediately before is releasing three point two And we can see here that our patch has absolutely nothing to do with active record scope objects and mocking and any of that stuff Which means our by section failed. Oh No You know what this means. Oh, yeah. Oh Yeah Time to do open source people. So what we're gonna do is Run our tests in our rails app make sure they're failing here I've reset everything to the three point three point O versions of our spec and then go back into my aspect mocks repository this time and Reset by section because I was doing some earlier check out the three point two point. Oh release market is Bad sorry three point three point. Oh release market is bad check out three point two point. Oh market is good And now we're going to do the same thing Changing back and forth between our rails application and running our tests This will be accelerated because I like my time. So this time our test passed We type get by sight good run our tests and this time we get a blow up We don't actually get output from the test suite. This is interesting, right? It says cannot load such file our spec support object inspector Here's what's going on Bissecting old code across multiple repositories is a dangerous and scary game Because we've been actively developing in all of these repositories classes have been appearing and disappearing as that development has happened and so if we go into the repository that has that file Objecting specter and take a look at what's going on. We can quickly identify the issue here We're looking at aspect support Which is an internal aspect library to help us develop code and you can see here that's a commit Which just literally deletes this object? Which my tests depend on during bisection because again, this is active development So what I'm going to do go ahead and do here is grab the commit before the object was deleted Check it out and now go back into my rails application And here we can begin by section again and here where the test fail I'm going to type git by sec bad where the test pass I'm going to type git by site good and so on and so on like through All of this process and basically what will happen is the same thing git will identify a specific revision where the Project started breaking and once it's done that we can actually go ahead and understand what's going on So I'm going to go ahead and skip Like a bunch of bisection here and sort of see good bad good bad And eventually the commit gets popped out at the end So what we go ahead and do is grab that commit number and head on over to github and see what's going on and That can't be seen at all because the contrast on these projectors isn't the best See can anyone read that? No, all right Basically what's happened here is a patch has been introduced that causes our spec to call do a callback every time you Try and verify methods on an expectation on all of the subclasses of this object As we sort of debug through various lines of github we find These methods which do the things we search for them. We discover. They don't exist. We search for different ones We discover they do and eventually we're led back to this piece of code the important line of which is this one if Your object responds to a method called define methods Which is defined on active record base and everything that inherits from active record base Implements we we call that before attempting to do any aspect work on it The reason we do that is that when you have an off an active record object for the first time before it's ever spoken to a database All of the methods for all of the attributes don't Exist right if you have a database column called name and you try and call the method name That method isn't defined yet And it's the method called define methods which actually define attribute methods which actually does that I Can show you this by going back into my terminal and firing up a rails console and trying To call it if I call active record base define attribute methods It's going to explode and the reason for that is that active record base is abstract It doesn't represent any specific table and so there are no attribute methods to be defined It doesn't have any attributes and this is identical to the blow-up That the user who provide us it provided us with a backtrace reported in their issue So we've done it. We found the cause of the problem It's that when we invoke this method our spec explodes and this method is being called because our spec is calling this method on all subclasses that respond to it The problem here is that active record base is a subclass, but it's not valid to call this method on active record base We only want to call it for the subclasses and not active record base itself We solve this by implementing a check that we're not going to invoke this on the thing that explodes And also because you know our spec is a testing framework We add a bunch of tests that actually verify this behavior exists and works There's a little bit of aftermath to this issue whilst this solution works for active record The generic idea of being able to do this kind of crazy Callback chain isn't the best and so a different patch gets applied to our spec mocks Which is more generic and doesn't only work with rails Well, we like to fix the immediate issue. It's also useful to come up with generic solutions if you have the time So let's do a quick retrospective on this issue. How how did we do how how was this well? It wasn't great that we shipped a version of our spec rails that was critically broken I think a few people need our spec to work with rails applications just sort of a hunch But we had a really really great debugging strategy, right? We had a backtrace We had version information. We had steps to reproduce and then we did Very very carefully calibrated by section by section is a really great method for finding specific bugs Exactly where they were introduced in the system It's also worth noting that initially we bisected the wrong repository But that was just based on intuition if we'd spent a few more minutes looking at the backtrace that the user provided It would have been obvious which repository had the issue in it But it's great because we actually eventually ended up finding the commit Before I go too far. I think it's also worth noting that for this bug report 99% of the time that was spent is actually in trying to find out where the bug came in The actual fix was trivial. It took about five minutes once it was obvious what the bug was and that's so common and so true I think debugging is most of the time and then just implementing the fix. It's not hard so Let me give you a few pro tips. Let me summarize all of the wisdom that I've been trying to distill into this talk There are ways that you can make your bug reports infinitely better with just a few small tweaks so for like One star the most basic level of filing a bug report is just telling a maintainer. There is a bug this thing broke Here is some code It's fine But it doesn't really provide us with enough information to be able to begin to assess and triage the issue And that's where we started with this issue and then our maintainers got to work actually Working with the community to begin to have enough information to solve this problem a Second star I can sort of award to you for providing a backtrace because backtraces are rich with information They don't just tell us where the bug is But if you look carefully you can also see that there's like version information baked in to this lines these lines here They actually tell us more or less exactly what stuff you had checked out and what it was doing when the problem occurred The next most useful thing is dependency information if I know every piece of code that was running in the system at the time a bug occurred I'm much more likely to be able to quickly and suddenly identify your issue if there's a gem Which is monkey patching what aspect does or monkey patching what rails does? I'm able to tell you it's probably an issue with that gem and probably not an issue with aspect or rails The big one the thing that I wish Everybody did is providing a reproduction case that I can just clone just type bundle install bundle exact aspect and see exactly What the bug is this kind of automated reproducible bug reporting is kind of like the golden standard These days many maintainers won't even look at your bug report unless you do that now because it takes so much time The Ruby ecosystem is really really complicated these days and if we don't have this It can be almost impossible to work out what's going on and then like for five stars But really this should be more like infinity stars is doing the bisection for us If you can come not just with an application that I can clone run a thing and see it fail But also then spend the time to see if this is a regression to actually bisect across many many revisions of rails or aspect to determine what's going on That's that's gold because then you've done most of the work and that five-minute fix that I talked about earlier That can happen straight away. It saves me all the time in the world to not have to bisect your repo on Your behalf and in fact like I have a sort of pithy quote about this from Sean Griffin Not Sean's baby. Just to be clear a few years ago. Actually, I think last year I had a I had a bug in rails that was badly affecting us back and I had actually bisected down to a single commit in rails and I was like Sean the bug came in here and It was just like bam six hours later. It's the issues gone He said to me if everyone who filed an issue on rails gave me an exact commit where the bug was introduced There would be no more open bugs in rails Now there is a whole army of you and There aren't that many people with a rails commit bit. So if you all want there to be no more bugs in rails This is a thing you can do right. It's not that specialized So that's approximately where I finish talking about bug reports It behooves me to just say one more thing, which is you've probably noticed digital ocean has been all over this conference I work there who enjoyed our party See see that's that's just easy pandering to the audience Here's not pandering to the audience. Um very specifically My team at digital ocean is hiring not just digital ocean is hiring my team at digital ocean is hiring I have I think one open headcount for this year and for open headcount in 2018 And I want to build an amazing team filled with Ruby s's who can ship great software We work on internal tools, but also we write a lot of aspect tests And if you want to see how aspect maintainer writes aspect tests, this is a open opportunity. Please come grab me afterwards We have socks I Am I am wearing them? I will I will give socks out until there are no socks left first come first serve Please don't tackle each other So that's it. I'm done. I'm Sam Phippin everywhere on the internet. That's my email address