 Quick show of hands how many people here have heard of code.org use code.org. Okay. Nice mattering And show of hands how many of you have ever used or Like investigated a test framework like selenium or phantom. J. S. Oh, okay. Nice. All right. We're in good company today Yeah, so I'll be talking about our journey starting to visually test stuff at code.org but before that for a couple reasons I wanted to just Quickly tell a little story about how I got started with stuff This is me and my three brothers four boys a poor mom. She was always hoping for a girl If you can guess which one I am Hard to tell left Right. I'm the one on the way right. I'm the youngest. I was the last straw for my mom It's when she gave up. She's very happy now that now we have now she has daughters-in-law So I think that made up for it. So this was me we were fortunate enough for for two reasons when we were younger to Be exposed to computer programming on the one hand we had access to computers when we were younger my Dad started out his career doing programming software work. So we had computers around some super old computer looking there I think I only used to play games on it some DOS games. Does anyone recognize this? this browser fortunately or unfortunately Fortunately for me when I was probably I don't know ten or eleven or twelve I Noticed there was this button on the browser that said edit like I don't I don't know if any other browsers I've seen since then had an edit button. I'd never done any programming. I didn't know you can make a web page I saw the edit button. I clicked it Well, actually I was afraid to click it because I was on some big website like New York Times comm or wired or something And I was like if I edit it is it gonna get in trouble I was pretty sure that if I edited the page and click save it would change the website Which was both exciting and terrifying. I clicked it anyway made the change and then I refresh it and it was gone So I was upset So I got more and more into Making web pages my brothers having older brothers who had like gotten into some tech stuff Before was very helpful and they would show me how to style stuff up better than the Netscape editor Let's you do and started learning HTML and it's a little bit of CSS. This was the extent of the CSS So this is a web page I made for a sport fishing Charter company that I met on an AWOL chat room who would pay me twenty dollars to make a A web page which was probably a good deal for them You'll notice I was getting pretty into the Photoshop the the layer filter effects And the oh and also of course the nice table-based layout I kind of missed that but it sounds like we're getting back to the ease of that now with the With the box model and everything so yeah, this was my first CSS And it was probably a snip that I copied I copied and then changed some stuff out You'll notice the mismatched capitalizations of the style tags. I was never sure whether it's supposed to be all caps or not Some nice some nice weird semi-colon usage So anyway the point of the point of telling you all this Then when I was sitting down to to make an AIM as you know, this is very popular back in the day You had to have an AIM Username I was trying different names. I tried Brian Jordan. It's taken I tried this thing is taken I Love turtles. I was like obsessed with turtles. I think I wrote some report about turtles So I tried like turtle taken turtle man taken and then I thought what if I add that cool magic language I don't know how to use yet CSS. So that was my AIM username was turtle CSS I'm just telling that story to say that I'm really excited to be here at CSS con It's kind of cool Turtle CSS finally made it. I'm really it's just super cool knowing that like if I'd known back then that there would be an Entire room of people who knew about CSS and that I would even know a little more CSS. I'd be like what? So anyway back to back to the talk thing This is me my brothers ultimately we all ended up taking different paths and ended up all now doing programming And it was it just makes me think I think now and then like geez what a weird like series of Coincidences or just like luck or privilege to have to be able to have access to the computers and to be able to Learn to program like none of our schools had any computer science classes the entire time You know people who also had computers just never and never ended up having the opportunity to learn to program That brings me to oh, yes, and I wanted to plug a talk Alicia's talk tomorrow at the end of the day I think we'll be a deeper dive into The those sorts of efforts and what you can do to help that So what I do and actually those two other brothers of mine also now work with me at code org Which is funny. I wouldn't recommend working with your siblings. No, it's okay. Sometimes It's a good word We're non-profit It's based in Seattle The goal of code org is to expand Participation in computer science. So we actually think that the best way to expand the the participation in computer science is to Bring computer science classes into schools and be taught next to algebra geometry in K through 12 So the wit we there's a few ways we do that there are a lot of missing pieces of the puzzle when it started in 2013 Computer science wasn't seen as something anyone could do computer science wasn't Wasn't there weren't great curricula available for K through 12 yet teachers hadn't didn't have the expertise To teach computer science or feel confident with that yet So we've kind and and also stated there's a lot of problems states also didn't count computer science as a credit So we've kind of been attacking it from every angle One of the things that we did was we ran the hour of code Which is a yearly event which which says hey teachers and students Why don't you run one hour of code with your students? It's you know a little bite-sized trunk and it's a puzzle based puzzle based games that students can play and they learn Stuff like sequencing and loops and Just yeah event handling and some basic concepts using drag-and-drop blocks to start for the for the younger ages There's a little video that kind of shows it'll be it's a quick one shows kind of like the other pieces of what we do There's so many pieces are So many different pieces going on with it. So this kind of gives a nice overview of the teacher trainings All that sort of stuff So yeah, the funny thing is a lot of people haven't a lot of people haven't heard of code.org But it's it's actually getting into a lot of schools like if you ask if you have Mises nephews sons daughters you can ask actually them and they might actually already be doing computer science in their school Or their district might be planning on doing it Most people have just heard of the hour code though. Yeah, that's the main So this will give you an idea of what we're gonna be testing So students used our tools to make these apps They can make little games Fun little stuff even fruit keyboards It's pretty sweet 40,000 teachers have been trained over the teachers having completed the training and We have a bunch of school districts a lot of the largest school districts inner-city school districts have signed on to basically implement computer science across their whole Their whole district K through 12 Just pretty sweet so 75 percent of schools still don't cover computer science All right, so let's talk about how to test this and I'll talk about if you're interested in Volunteering we're all open source and that sort of stuff at the end So yeah, we're working on full curriculum district partnerships professional development policy change Fortunately, I don't have to do all of that. I only work on our K through 12 CS curriculum And that's what we'll be focusing on testing today So anyway teachers are coming trained that must be old because I think it was 40k a lot of people using this stuff The cool thing is that because we're in schools and during the school day We have a student population that reflects real population It's not just people who you know have computers at home or can make make it to an after-school program It's actually in the school day. So that's very cool so How to code org start automated testing, how might you start automated testing? So in 2013 to 2014 2013 was the first hour of code. It was a very You know last-minute sort of thing like oh, let's put together a tutorial. I was not there at that point I joined in early 2014 But you know as you imagine when you create a new thing very quickly and push it out to production To a lot of people who there the there's bugs that come up And as we'll see we use a lot of kind of bleeding well not bleeding edge But bleeding edge for school computers technologies in our tutorials So yeah, there wasn't really so much testing So why would we test? Why would we invest the time to and Yeah, the time in infrastructure to actually set up automated tests I think the automated tests aren't necessarily for every For every team or you know the the full like cross browser automated tests I think you want to think about what what are the what are the Use cases for it. What are the things that break and how bad is it when things break in our case? It's pretty bad when things break. We're not doing fine. We're not doing financial stuff, but Think about it. We're we're in schools. This stuff's being used day to day in schools you got teachers who I Yeah, I don't know if you've ever like been in a classroom. It's kind of pandemonium We'll go in the teachers will teacher will come in get the kids all set up on the computers It takes like five ten minutes, you know, I gotta log in press the login button teacher They raise their hand then you got to go over to them. It's kind of it's pretty stressful When on top of that they say teacher the level is not going forward or this page isn't loading It's it makes everything way worse, and it's super stressful It's I've seen that in classrooms before very rare, of course now and of course schools if you remember schools schools have quite a wide array of situations I remember Yeah, I nine we got Chromebooks iPads and a mix of everything. I remember this one It's one issue. We were doing the Minecraft hour of code last year. It was like a little tutorial where It's kind of like a 2d like grid-based Minecraft and you like program Steve or Alex to move around I Got this bug report. It was like two days before the hour of code like millions and millions of Students and teachers are about to use it in their classroom, and we get a support request saying whenever Whenever we go to the Minecraft hour of code page the browser closes. I was like the browser closes. Oh My god, that's the nightmare. I was I was on the hook for that in a day millions and millions of people are gonna Maybe I don't know have their browser closed. So I needed to figure this out So I was like I was testing it on every browser. They weren't very specific in their first request I think they said may have said Chrome and they just said like and I was like Is it is it just Minecraft? Is it the other tutorials because we're using a new JavaScript game engine phase are very cool game engine I didn't know what might have what might have been wrong. So I emailed some diagnostic stuff Can you try a different browser? Can you try these other pages different tutorial and I was just so confused The the Minecraft hour of code page would close in every browser multiple browsers and not the other tutorial pages so Finally I it dawned on me I think it was like the night before the hour of code and I'm like I'm having nightmares about this I email him and I say can you try going to Minecraft net the main Minecraft web page? And he's like oh, yeah that closed too So I guess the school had a filter set up that sits there and closes the web page if the page says Minecraft on it anywhere on the page so I Said yeah, there wasn't much we could do to excise every mention of the word Minecraft in our source Good luck. Maybe try a different Ask the IT department about that one. So yeah weird stuff happens Also, we support a lot of languages like 40 plus languages including Right-to-left mode which we take to kind of an extreme where we actually flip The entire layout of the page over so the run buttons on the right still towards the visualization area And then the blocks the drag-and-drop blocks can expand out that way. Oh And yes as I alluded to earlier Bleeding ish edge technologies such as canvas and SVG which are now not so bleeding edge But turns out there are a lot of bugs you can find in a bunch of different browsers in those that you'll discover when a lot Of kids are clicking around and dragging stuff using it. So this is this is our so yeah that See right here. Yeah, you can see like the drag-and-drop blocks students drag-and-drop blocks out to Program that and think now like how would you test that? How would you make a make a robot sweet testing robot drag those things around because we'll we'll be getting to that This is this is called app lab. It's our IDE. I love it for creative programming actually Where you can drag out interface elements write that stuff and Then create JavaScript events. So this is this is for our late middle school to early high school So they start out just your drag-and-drop blocks in the earlier grades and then later on move on to This and that actually you can try you can transition from blocks to text there So you can you kind of get the goodness of self documenting blocks and then you can transfer to text How would we test that so many things to test? Let's get started so Yeah, it's kind of daunting sometimes thinking about How do we actually how would we actually get started testing at our you know on my project or At our company because it's like what do I like invest like a month and you know say like okay? We're gonna do this thing. We started very small The founder of code org He had he he works with a bunch of companies advises companies and they had said that they would They had said like hey you guys are having all these cross-browser issues You should check out this cross-browser testing. I knew nothing about it My teammates knew nothing about it one of our engineers Brendan set up just our first little test and it was Just hitting a few browsers go to a page and then and then we started running that So how our tests actually organized and run and when Brendan first implemented it We had it use a Selenium web driver, which is Selenium is the kind of lingua franca of Browser telling browsers what to do and we use cucumber as kind of a high level higher level language like DSL language Where you define your own steps? to just kind of make little reusable snippets of steps as I learned there were other patterns for doing this like the page object pattern, which is like I've gotten called out for not using that before but this works and it's it's kind of cool So yeah, we have features. It's our big list of all the tests that we have Step definitions where it's like okay if they say click on this. What does that mean in the code? So like on use the browser object do that thing? So this is what one of our tests looks like So you're you basically say given. I'm on this web page. So go to this web page And if you're running on a different host, it'll point it to a different subdomain and a rotate the landscape So we're also testing on iPad simulators and some of the browser providers actually have real device testing Which we have not explored but that is interesting for native stuff You wait to see a dialogue and then you close it and then that element should be there And then eventually we actually drag blocks to blocks press the run button, etc And a bad habit of ours is we often make cucumber steps that are That let you just say selectors and stuff and apparently you're not supposed to put selectors in your cucumber text either But it works So some takeaways from doing that stuff Yeah, so building up a library of steps is actually pretty useful Because you're building up very reusable pieces, especially if you have some JavaScript You got a trigger on the page or if you have some assertion you want to make about You know some some thing you can build up reusable steps with cucumber So in this case we have something that lets you drag block a to block b And then so now in your step all you got to write is that you don't have to reference any JavaScript or know too much about the stuff Underlying it the other lesson we learned was hey don't test on real production data because Then that's going to change because we have our education team is going through and editing levels adding levels changing the order of things So set up test stuff separate test data if you can Even better to have it programmatically set up in our case We just we use our own level editing pipeline to just set up a test only levels Which is kind of a smattering of levels of different stuff on our entire site Also a handy page to use if you're if you do want to do some manual testing It's like oh here's everything on our page So actually something that would be useful if you have like a style guide a you know a set of patterns that your team uses For visual stuff Well, I guess we'll get to visual testing in a second But I think that's a good target maybe for something to point your tests at Are the cool thing annotations cucumber lets you kind of flag tests on and off You can say don't run this on IE because we know it's gonna not work or sometimes the web drivers have their own issues So you want to flag that So on our team who writes tests? So on our team we Because we just kind of got it started when the immediate need came up. We all kind of Did our part to set up the system and write our own tests So everyone out on our team writes writes these tests alongside their features Or oftentimes after the features done because there's a bit of thrashing sometimes when you're writing a feature like that And that's actually worked out pretty well, although I haven't worked somewhere previously where there is there are dedicated Test test automation to engineers. I've heard that's very nice and save some of the pain that we've also gone through So mad respect to our friends in test automation. They they do some awesome work So who are what actually runs the tests? Well, we run the tests when you're developing a test But then when you actually run it so in 2015 We had one pipeline of tests. So there was actually a server sitting there and saying oh to get update cool Let's run a build Spit out a bunch of output on hip chat. So we'd be sitting in the hip chat room And watching the output come through now we're on slack But no longer do we only run it through that one machine sitting there when get updates There's also the thing is our tests if your tests take a while. You have a lot of people pushing code in then It's gonna get backed up in your tests You won't find out about what broke until later now what we got going on is Tests UI tests actual cross browser tests for every single commit. So now this is using a circle CI You basically just set up a little a little file that says here's how to build my project and then the last step there You say run my run our tests against this version of the project and then that gets running It's I think I'll yeah I'll talk a little bit more about how awesome that is But I can't understate how much better it is to have your tests run for every single commit every single feature that someone writes Versus building up these buckets of failures where it's like oh like eight failures like okay And then you got to figure out like you look at the big list of commits who did what? It's very awesome having Flourcus tests. So what challenges did we end up facing? Yes, and how did we address those challenges that I alluded to earlier? Multiple browsers we support a lot of browsers What we use is that selenium web driver because it's kind of that got that lingua franca Sort of factor of being able to point to different browsers We can point it at any given browser And we don't actually have to set up machines to do that because we use sauce labs Which is a basic cloud provider of these of these VMs with those browsers in them So you literally say just point your selenium test in your configuration of the URL you just say point to sauce labs Give me this browser and then your tests start running and they they deal with all the stuff behind the hood There are ways of setting up your own cloud, but it's nice to not have to maintain that and they're always adding new browser support So this is our browsers.json file It has a list of the cut basically the different browsers we want to run against So some of these are the the sauce labs identifiers You can say what what operating system you want you can say what? browser you want and Then when your tests are running you can even see they record a video of the test running So if there's a failure you can say okay, well what actually happened during that run It adds adds a little nice ability to debug that Now how about testing new changes so you're writing a new test or you locally change something You know a new feature that you know is gonna break a test. How do you test that so? If you're doing it locally what you can do is use Chrome driver in our case where Where you point the test at a local browser and it'll actually pop the browser up in front of you And you'll see the robot like doing stuff. It's kind of fun I'll show you a video of ours in a second and then we use sauce connect So if you want a remote browser to be hitting your local machine It's just like an SSH tunneling tool that lets it hit your machine It also has some fanciness like it caches stuff on their end so that the requests are a little faster It's not that back and forth business going on and pull requests. I love pull request This was like my holy grail. I'm so glad we now have it. I just I think I we just got it in the last like month or so So now our UI tests also run on circle CI and will Tell you when thing when your stuff broke stuff and so you don't have to worry about those buckets of failures anymore So how about drag-and-drop? The standard Selenium drag-and-drop didn't work for us. So we ended up using jQuery simulate Happened to work with our SVG based drag dragging and dropping editor Visual responsive changes. That's the cool thing so in Was it last year or the year before I was doing a hackathon actually with my the our oldest brother We got into a hackathon and I was trying to think of a project idea Based on like what is what's a project that? That would be really useful. What's a pain point at work and how can I fix that? And I was thinking we had just had some bugs where it's like a button went away or something And it's like I don't even know how you're gonna or a button show a second button copy showed up And I'm like, how do you even test for that? You have to like say, okay, there's gonna be one button It's gonna look like like that's too many steps when you're writing those individual manual steps And often looking looking into who broke what when and finding out finding that out later from you know users or You know the CEO is not very fun So I figured hey, what if we just I thought this was like a super unique idea What if we just took a screenshot of the thing and then compared it with another screenshot of the new version? Turns out a lot of people have done that before but I thought it was cool at the time In fact, there's even a tool. We ended up finding during the hackathon. I think it was called pdiff that let basically lets you Over it'll kind of show you It'll detect what percentage of the pixels changed and it'll kind of provide this cool view of like which what changed in Between the the two things so we set that up. It was still a pretty cool tool You just literally put the URLs in and press run and it was cool, but it turns out It was really hard to do that, right? Apparently when you take screenshots Things things change that you wouldn't expect change like sub pixel Shifting of stuff and it depends on like the browser's Operating system renderer. It's it's yeah, you see a lot of weird stuff and also, you know If you have a something randomly changing on your page you have to deal with that Turns out It's an actual thing though. So if there's something called Apple tools that we found after and I was like, oh Why don't you just tell me about that? so Apple tools they're they're basically they're doing just that so they provide an API that lets you Compare the images and adding more regions and everything like that So actually in our case it was because we already had the selenium test set up and they basically plug on top of the Selenium web driver We just have these new steps that say eyes dot open And then you can have a we had a new cucumber Thing here step here that says I see no difference for this and that's basically a tagged baseline a tagged image So say I see no different difference for the homepage And then it says cool check the last version of the homepage if if it did break then it'll throw up it'll be upset so Here's an example of one of our features that was converted to an eyes feature So we mark it as eyes to say basically make the stuff available to run that and then I say when I open my eyes to test this thing Go to the thing. So click around. We're still using selenium commands And then at some point I say and I see no difference for Blank game screen and so it takes a screenshot at that point and gets it ready for our review The other thing is this is cool cucumber has this this thing that is Scenario outlines where you can basically write a table of URLs and stuff and it'll run the same tips for steps for all of them So yes ignoring regions as I alluded to earlier if you have random elements on your page things often change on web pages Pretty often rotating screens or in this case a random assortment of angry birds Tiles on the screen we can do is you you basically draw you click this button and draw add an ignore region within Which you want to ignore changes So yeah, it when there is a failure when something changes. Here's an example of a failure Where this button is showing up when it shouldn't be showing up. It was due to there's a version of this that has parameters and a version of it that doesn't have parameters and There was some Inherence mix up there that caused that issue, but we caught it before it broke the tool This is this is kind of the interface and here's another example of a failure where The step button showed up when it shouldn't show up And that was present on a lot of different steps in that same test. So you can kind of tab between the different steps there and yeah The other thing is they also when you use the phone Version of it. It'll show the top bar of the kind of the whole simulator and screenshot based on that So what do these tests look like when they are running? We'll do a little bit of this speed run Do it on time So yeah, I just find this so cool. So there's a computer. This this Selenium robot is sitting there Learning how to code so it's code. That's writing. That's learning how to write code. I just think that's kind of cool That's silly Yeah, it's kind of fun watching your site get used very quickly Cool So wasting time or not wasting time. We don't want to waste time time is valuable We can do things with time so one thing that sometimes will happen is Tests if you write, you know, you write a couple tests and it's like, oh two minutes That's fine Like I can do a minute long test I can wait for that and then you know your whole team's writing tests and then more People are writing tests suddenly your test suite is 60 to 90 minutes, which means you're not gonna run it locally You're not gonna run it very often and then those buckets of of changes that you're gonna have to figure out What broke what are gonna be larger? So we started doing was Parallelizing our test runs at the test in the test runner so you can basically say run this at parallelization 50 And it'll spin up 50 browsers all at once and be you know telling them all what to do at the same time All hitting your same server actually oftentimes the bottleneck is the server your local server because it's getting hammered by like a hundred different VMs So in 2015 we got back down from 60 to 90 minutes back down to 20 to 30 minutes and we're still around there Oftentimes when we're having like team retrospectives and it's like what sucks about being you know our on-call person or looking through the failures oftentimes You know people will question the premise. Why do we have tests? What if we just didn't have the tests and sometimes it's hard to answer that or like or what if we just cut out Firefox or Safari like I'm out. It's just around like one browser. There aren't those issues all gonna be the same reason jQuery It's gonna it'll it'll sort itself out And I'm always like wracking my brain. I was like I could have sworn there was some Firefox bug or something So we started doing was keeping a bugs tests caught wiki page that are kind of like it's kind of like the trophy haul for The bugs that you caught I'll share some of the items from here. The nice thing about being an open source project is we can show you everything Here's here's a instance where a button went away Here's an instance where the button got taller randomly Here's an instance where this weird margin White margin got added to the right of the tool which is like that stuff used to happen all the time And it's like it's nice to get notified about that early. This was a funny one So Over there was some kind of like re-jiggering of the whole of like some of our top level styles And then overflow hidden ended up applying to This text up here. So if you can see the words are kind of getting cut off I don't even know if I was just clicking through I might not even notice that but we caught that one Here there's the duplicate button button got duplicated. That's always a tough one to catch Without the visual stuff. And here's one where there's one Tiny gray pixel I Don't know how that happens. Well, we figured it out at some point Yeah, it catches all that kind of little stuff. Here was a big one that I caught So this was yeah speaking of bugs that bugs still do happen, especially when you're using SVG Yeah, get the box when it's hidden blows up And here's one where chrome 50 deprecated SVG element offset with and it broke our little layout of that text caught that one And here's my favorite bug ever Let's look at this look at these two slides and imagine you're you're doing some QA manual QA on your site What's the difference between these two webpages? There is a scroll bar, but why but why see that see that text down there Yeah, and see actually even above there stuff's getting shifted down to the line height changed everywhere The entire websites line height was changed during a routine refactoring of top-level styles So we caught that and yeah, I'm pretty certain if we didn't have these tests in place We would still have that that our website would just be slightly longer forever Who investigate failures? We have a dev of the day when failures do come up to kind of triage the stuff It's not pleasant to be the dev of the day. It wasn't didn't used to be pleasant to be dev of the day It's it's slightly better now. So right when you're running in a big bucket and you have a bunch of failures You have to figure out what caused them So you're looking you're looking at any random failure from anyone's code from that entire day and then you have to figure out who broke what and then Make sure that they get their fix in before you do a production push or revert it Now that we have those pull request tests You don't have to do that because it just doesn't break because the yeah So now now that we're actually running in every single pull request before you do the pull request You'll check that you have to merge it. What's next for us? We're gonna get our Test results like add a little github comment with the screenshot diff. I think that'll be cool in your branch when it breaks If you guys want to learn more about cross browser responsive tests Dave Hefner has a sweet Introduction to that and you can see there you just define different resolutions and browsers to do that Where's some takeaways start small? Everybody tests Don't have to do that if you have someone to test then maybe not everyone has to test but it worked out pretty well for us and Periodically invest in speed-ups so your tests don't get too long I just want to thank all my team who worked on many pieces of this because we all kind of shared the burden on that and Of course the folks at the various providers that we worked with Thank you. Ah, yes, and if you're interested in getting involved code.org slash help Let's have some Q&A with turtle CSS here I'm not gonna reveal my first All right, so we did have one question Some folks are saying they've used selenium for testing before and they find that as applications change the tests become really fragile and Difficult to maintain and they frequently have to be fixed. So how did you avoid this at code org? Yeah, so you so that that that one that one tip I think that was having a separate like a separate portions of your site that don't thrash as much is helpful and then I think also Making it okay to break tests is is making the workflow for breaking tests a little less painful I think that that made a big difference for us because yeah It's it's when you find out about the test a day or a week later that it's extremely painful to like figure out who broke What and what you did to break it? So I think I think that stuff mitigates the pain of Having tests that are so fragile and I imagine when you decided to write test One of the tenets is that now that you're adopting this everyone has to be writing it and everyone has to submit tests So how did that become part of your culture and day-to-day? Yeah, that's a good question So I think the way we kind of started out is you know a couple people Everyone's kind of like different their toe in the water It's like there's a new project a new file like the new steps to try out to run test locally And they'll be like, okay, maybe I'll like try writing, you know for this moment my feature Maybe this thing broke often like it oftentimes if something breaks Team member will say hey, like don't we have a test that covered that or what if we had a test that covered that? Okay, I'm ready to test that covers that Yeah, so it kind of like grew from there And then it's like you don't want your stuff to break and then especially if you could have prevented it by writing tests Like all right probably should have written a test for that So it's kind of ends up being on the owner of that which works for our team because I think we have kind of separate areas of ownership like features Wonderful, and then it's just kind of adopted its way through and your bug list keeps everyone that's right Yep, the fun list of all the bugs we caught I like that. All right. Well, thank you so much, Brian