 I just had lunch, and I'm not feeling as awake as it sounds like all of you are. But just in case, it is second day of a conference, lots of emotion so far, we just had lunch. I know I'm feeling a little lethargic, so right before we get started, and also because we might wanna pad out the time a little bit, let's all stand up real quick. Everybody get your hands up, stretch, just way up to the side. All right, let's take it left, my left. Let's take it right. All right, now bring it back up, and everybody clap your hands. Perfect, thank you, you can go and sit down. I actually, unfortunately, maybe this isn't a good idea to get started out on deception. But unfortunately, I don't actually care that much about your health, really, I just wanted a picture so that it looks like everyone here is giving me a standing ovation. So when anybody asked me afterwards how this talk went, I can just show in the picture, it's awesome. It went great. Do you see, look at how they're reacting. Thanks for playing along. All right, cool, so we're here today to talk about concurrent feature tests. So I wanna start at the very beginning, I don't wanna assume any knowledge that you may or may not have. So let's define what a feature test is, because unfortunately there's a lot of different connotations for these kinds of things. So feature tests, when I say feature tests, I mean a driving a test from the UI, a test that goes end to end that ensures that a feature of your application is still working correctly. So how that might look in a real kind of web application is something like this. It might have a name field, an email field, a password field, and a sign up button. What do we need to do with this? Well, we need to fill it in and then we need to click the button. And then once that's all done, we need to go and check that something happened. We need to make an assertion. Just familiar with those three As with testing, right? That third one there is assertion. So we'll say something like this. I'm gonna look for some sort of flash message or modal that says, hey, thanks for signing up. And we need to assert that that actually happened. Feature tests are great for this kind of stuff because it actually ensures that our applications are working the way that we intended them to and that we haven't made any sort of regressions when we make changes to our application, when we do refactoring, and all those sorts of things. So ideally, a feature test, one, represents real users and two, they provide confidence in our system as a whole. But if you've written a lot of feature tests and just with real quick show of hands, who in here has written feature tests before for the application? Oh, a lot of people, turns out. You all know there's kind of a dark side to these things because while we ideally like them to represent real users, they only represent real users sometimes. I don't know a lot of users who can click around in a browser as fast as Selenium will click around in a browser. And oftentimes, we can accidentally start interacting with things in a test that a real user could never actually interact with. For instance, if that DOM node is actually hidden, Selenium doesn't always tell you that. And so you have to actually know and be able to correct for those things. And they provide confidence in our system when they aren't just sort of randomly failing all the time. And judging by the laughter, you've probably experienced this, right? And why does that happen? Well, I mean, JS happens. You just don't know, especially in this day and age of really rich client-side applications that rely heavily on JavaScript, we don't know what the state of our DOM might be. DOM nodes can go stale in between checking if a DOM that exists and then checking if it has the right value in it. DOM nodes can just disappear or they might appear asynchronously. You might try to wait on Ajax to fire and then in between Ajax firing and actually rendering anything into the DOM, you do a query and then nothing's there. All sorts of problems, all these sort of timing issues can start to crop up. And this is where you get the test suite that's like just rerun it, it's probably fine. It's not a big deal. And that degrades your confidence level. How can you actually be confident in your tests if you have to just rerun them and hope that they pass? Beyond any of this stuff, though, there's the big problem, the problem that we all talk about with feature tests, they are slow. Orders of magnitude slower than any unit tests that you might write for your application. Because what are you doing? You're starting up a browser and you're running through a browser and it's hitting your application and it's going down into a database and it's executing SQL queries or whatever the case may be. And that just takes a lot of time. The problem, though, is they're also useful so we want them, but these are the trade-offs. So identifying all these trade-offs and trying to think about ways that we might be able to solve that stuff. Utilizing some of the power of Elixir is why we decided to build Wallaby. I think this is the cutest gif of a dancing Wallaby that there is. But if you have a better gif of a dancing Wallaby, I'd love to see it. So what does Wallaby do for you? Wallaby manages multiple browsers for you at a time. It's concurrent by default and it assumes asynchronous interfaces everywhere. We assume you're using JavaScript because let's face it, you probably are. So that's enough chat. Let's look at the TLDR. I realize that that's why you've all come here. You just wanna see the code. So let's look at it. So remember this example. We have a name and email field and a password and a signup button. Let's build a test. So this is what a test might look like. By the way, I'm gonna go really fast through this and then we're gonna break it down so don't stress out. We've read a test. We're gonna say the test that users can register. We'll get a session. Once we have that session, we need to go visit a page and then once we do that, we need to find the form. After that, we'll fill in some stuff. We'll click a button and then we'll go and we'll assert that something actually showed up by getting the text out of a flash message, asserting the message and saying, hello, you actually logged in. Boom, there's our test. So we'll go through this real slow and we'll try to break down each piece and ideally here, I'm gonna give you an idea of how you might stretch these things. Really what I wanna do is not give you a textbook like here's how, well, all of me does. If you wanna do that, you can go read the read me. But ideally, you'll actually start to understand some, a little bit more about Ecto, a little bit more about Phoenix, maybe a little bit about plugs and you'll also start to learn about how we kind of manage these things. So let's break it down. What we're gonna talk about here are sessions. We're gonna talk about queries and we're gonna talk about actions. So we'll start with sessions. So a test session typically looks something like this. We start with a test, that test talks to Wallaby, Wallaby talks to Phoenix, Phoenix talks to Ecto and then you get data back eventually. That's the idealized version. What's actually happening under the hood is something a little bit like this. Wallaby's actually managing a pool of browsers for you. Right now it's a pool of phantoms, if you use phantom.js. Coming soon will be support for Selenium and Chromium we'll talk about that later. And we use poolboy to actually manage this pool for you. So if you use poolboy, you're familiar with that. If not, you should go check it out, it's a great library. When you actually wanna start running a test, the first thing you do is you tell Wallaby to give you one of those browsers. That browser then talks to Phoenix and then when you're done with it, you just put it back in the pool. Pretty straightforward stuff. The way that it looks like in code is this. We simply tell Wallaby to give us a session and then on exit, this would be part of your test case or whatever, you'll end the session. And I'm showing you this code. We actually give you this code to drop into your application. You don't have to write this yourself but it's worth going through it just so you understand how it all works. Of course, if you have multiple browsers talking to a single Phoenix, the Phoenix is talking to Ecto. Ecto's then responding, you have to have some way to sanitize that data. This is why concurrent feature tests are sort of an interesting problem, right? It's because the browsers and Phoenix doesn't know, it's just, they're just calling into the database, right? The data is whatever is in the database. And so you could end up with unsanitized data. But luckily, Ecto's awesome and actually provides support for this kind of thing out of the box thanks to all the work that James Fish did. And we can actually take advantage of sort of nested transactions and all the transaction ownership model stuff that he implemented for Ecto too. So we can avoid this problem. The way that looks is something like this. We take a single browser and when we talk to Phoenix from that browser, we actually pass up a whole bunch of ownership metadata. That ownership is like the ownership of the transaction that we're currently in. And then all of that gets pulled out of the request in Phoenix via a plug and then passed into Ecto so that we actually can understand who owns this transaction and if it's safe to run more transactions or to rollback or whatever. In code, it looks something like this. We'll break down this down line by line. So the first line here is Ecto adapters SQL sandbox checkout. So we're just getting, we're checking out ownership of our, in our connection pooling. The next thing we do is we craft some metadata and that is gonna be stuff like you own this transaction or what the case may be. And then finally, we just pass that into Wallaby. Wallaby knows how to inject that correctly and then have that get passed up in the browser. Obviously you need to be able to rip this stuff out, like I said, with a plug in your Phoenix application. And so the configuration in your Phoenix application looks like this. Inside of your endpoint, you just add a plug. So that plug Phoenix Ecto SQL sandbox. You can add that into an if condition that way it doesn't get added like in production, let's say. And that plug will do all that work for you. And you can just start taking advantage of this now. If you have Phoenix Ecto in your depths and your mixed steps, you already have this. You can just use it right now. So it's ready to go. Cool, so that's Sessions. Let's jump into queries. Sure, seeding? Oh, sure, that's a great question. Yeah, so the question is how do you seed a session like if you need data like in the database already? Because you've already checked out the ownership, you can run transactions inside of your test setup and it'll handle that, mm-hmm, yeah. All right, cool, so let's talk about queries a little bit. So going back to our example again, we have this alert message thing, right? So thanks for signing up. And the HTML for that might look something like this. Div, you get an alert div. You have a span with a class of message and with some, you know, text. We can write queries against this DOM like this. We can take our session and we simply pipe it into a find. And we can use CSS selectors. And we kind of assume everybody knows how to use CSS selectors. They're pretty ubiquitous at this point. And this will allow you to go and actually get that DOM node. If you want to use XPath or you wanna use something else, you actually can specify that there, but you have to pass in a tuple. By default, we assume you wanna use CSS. If you have something more complicated, like for instance, a list of users, right? You have a list of users. You have, this isn't technically a list, right? It's dibs, but who, I don't know that any of us, I haven't used the real list because it's styling and whatever else in a long time and rip the web, unfortunately. So we have a list of users. Those users have names. They have emails. And we wanna write a test against this. Let's write a test that's like, let's get all the users' names. So the first thing we'll do is we'll find all the users. And we'll try to do this. We'll say find user. But as you can see, there's actually multiple users here. There's two different user dibs. And so unfortunately, this is an ambiguous query. And we'll throw you an error that says, this query is ambiguous. We'll tell you exactly why. We tried to find it this many times. We expected it to find it once. We found it twice. And if you wanna fix it, drop this snippet into your code. And that snippet would look something like this. It would tell you to specify an explicit count. So if you wanna find both of these, you can do that. And again, our error messages will just tell you how to do it. Once you have that, you get an array back. Or excuse me, a list back. Too much Ruby. You get a list back. And you can map over that list and then call more functions. So let's say we wanna get the names of both of these users. You could do that. You could just use enum.map, as you would for anything else. And then go find their names. And then finally, if you wanted to get the text out of those names, you could again pipe it into another map and just call text. And then you would get those names out. And you could make assertions on this or where the case may be. If you want to do more scoping, we can look at different ways to do that. So let's say that we want to just get Grace's email here. We can actually write a query that looks something like this. We can say, go find the user with text that matches Grace Hopper. And that would scope our query down to a single user node because there's only one node that has that text in it. And then from there, we could go get the email and get the text from that email. And again, we get Grace Hopper. As we talked about in the beginning with some of our caveats, when we go to actually query this stuff, we don't always know what's happening because timing issues because of our favorite friend. And we might be rendering stuff. We might be in the middle of doing an asynchronous call, whatever the case may be. And so we assume that everything, when you try to talk to a browser, is not gonna be successful. So we actually will try to block for a determined amount of time until this thing actually becomes true, until you can actually verify that query and it returns you something. And if it times out, eventually we throw an error that says, hey, we couldn't ever find this. And you can configure that timing. So queries. Let's talk about actions a little bit. So we have a simplified version of our example. We have a name field and a signup button. And the HTML for that probably looks something like this. You've got an input field with a label and a button. If we wanna actually write tests against this, we can do so like this. You can say session, pipe that to a fill in. And you can see here what we're saying is fill in name. It's an interesting API decision, but we sort of assume that you don't actually really wanna couple yourself that much to your underlying CSS. And that's brittle. And your CSS probably changes a lot because styles and flat design is in vogue and then all of a sudden material design's a thing and then your CSS classes are just gonna change. And so tying it to that is a place where you will start to have issues long-term with your tests. And so we're trying to create an API that allows you to attach things to less style presentation and something maybe a little more descriptive. So you can actually read this as a full test and say fill in the name with whatever the text is. That also goes for buttons. So if we wanna click on the button, we can just say click on and then the name of the button. And that'll go out and find the button. Under the hood, we do all this with XPath. And so it allows us to be really flexible with our querying for these sorts of actions. And we can also, we don't have to scope it to just name like that, which is the label text, as you can see. We can also use the name of the field itself. So we could say something like fill in username, which happens to correlate to the name of the input field. If we wanna extend this and add a checkbox, it's like, hey, save my login information or keep me caching cookies or whatever it is. We can do that. And then the way we would extend the test here is we just say check. You wanna check that box. And again, you can pass it the label text and it'll find it and go and check those things. There's a whole bunch of other actions that you can use. You can use fill in, as we've seen, choose. For radio buttons and that sort of stuff, you can select from select boxes. You can attach files. And so you can do file uploads. We're working on some others, like being able to drag and drop out of the box. That's a sort of all upcoming work. But yeah, but by out of the box, you get a whole bunch of these different helpers to be able to use to fill in forms. And they're all chainable. They're all pipe-able, as you'd expect. So that's a bit about Wallaby, right? We have sessions, queries and interactions. Sessions allow you to create multiple concurrent browsers, all running against a single Phoenix instance. Queries allow you to interact with the DOM, or excuse me, to query the DOM. Interactions allow you to interact with the DOM. And interactions also follow the same blocking procedures that we use for queries and everything else. Wallabar runs on the same engine. And there's still a lot of other stuff that we want to do with it. We want to add more support for selenium and for chromium, as I talked about. If you've ever done development against phantom, like let's go have a beer together, and we'll just cry, and it'll be awesome, and we'll share the pain together. So there's still a lot more work to do there. We're talking a lot about trying to help even eliminate even more risk conditions. We're trying to work on errors and provide better error messages for people. So if you want to get started with this stuff, you can go to GitHub where it is. If you want to look at these slides, they're up here on speaker deck. And finally, I want to talk about one other thing, tangentially related to all this stuff. But I want to tell you just a little quick story, and I want to stop and talk a little bit about the Elixir community and how awesome it is. So about a year, not a year ago, but at the beginning of this year, I set up some rules for myself, and the third rule was that I was going to contribute a lot more stuff, and I was going to talk more, and I was going to try to get involved in the community more. And about a month later, I actually went and made the initial commit for this repo, and I've worked and working on it since then. And I just want to stop and say, like, the Elixir community is awesome. The Elixir community is amazing, and I don't know if you guys are as excited as I am about being here and seeing all the awesome stuff that's being built, but we're at an awesome time right now. We're just at a fantastic point in time in this community where we can really get involved and contribute, and we're all learning together, we're all growing and sharing. So if you remember nothing else from this talk, I hope what you remember is, like, let's just build awesome stuff together. Let's go out and continue growing and learning and sharing that knowledge. So thanks. I have a lot of time for questions. That's hopefully, I kind of went quick because I wanted to get some more questions than anything because I think that'll be more interesting. So yeah, so let's do questions. Hey, great, thanks. That was awesome talk. So just one question, really. So I'm one of the unfortunate ones that still work with Ruby. Have you ever tried running this against the Rails app because that might be a way for me to sneak, like, an Elixir into my work? Yes, I love that suggestion. It's totally possible. All you need to do is dial the pool size down to one. Yeah, so literally if you just say pool size one, you can run it against whatever application that you care about. Yeah, I should write a blog post about that. Hey, Chris, thank you for this. So I've had a lot more experience with Hound. Okay. And the thing I noticed about Hound and I've noticed about your examples is they're still really coupled to the HTML structure of the page. And a lot of the stuff I've done, Ruby is more text-based using I18N on both, for the page and for the test. So is the text node that you have a visible text? I was looking through the docs, I didn't see the extension there. Is that what that's for? Yeah, so we try to emulate real users whenever possible. So if you try to, for instance, interact with a DOM node that a user couldn't see but is in the page, we actually throw an error and say a real user could not interact with this. It needs to be visible because of these reasons. Along with that, so for any of the text stuff that we're getting, it'll provide you whatever we believe is visible text to a real user. So you could do the same thing with figuring out this text should be I18N, whatever, and then just induce that instead. So, and it should use the same stuff. Does that answer your question a bit? Yeah, totally. Yeah, yeah, totally, yeah. And you can nest this. The question, the follow-on question there was like being able to scope instead of getting like the full page. Yeah, if you continue to provide scoping, we'll just give you the text wherever you're scoped at. So, other questions? I think there's some over here. Hi, I'm giving a talk in Windows New Rails about using page objects to kind of abstract the coupling of text and CSS to your tests. Like Tesla's Brittle, have you experienced any, have you tried using page objects in Elixir? Yeah, absolutely. I just called them page modules. It was really the only difference. That is actually, so this is actually a bigger question that we have and it's something I hope that as more people maybe try this stuff out, we'll be able to have a bigger conversation about, but I'm of the opinion that we should always be using some sort of page module type thing. I think that is a good way to encapsulate some pretty messy logic about, you know, coupling to DOM nodes, coupling to CSS, that sort of stuff. I don't have a good answer for it yet other than to use these best practices, which I feel like is not a holistic enough answer. I would actually really like to eventually figure out a way to almost encourage people with the framework to put that stuff in variables like that you can change or like in a function. I don't know how to do that yet other than just telling people, but if you have ideas about how that API could grow and change, I would love to talk to you about it. So, other questions? How is the synchronization between Wallaby and Phoenix done through the browser? So like say that we were using Phoenix as a backing for a front-end JavaScript app that lived in a separate project, what kind of plumbing would we need to add to that intermediate app to enable parallel tests? Okay, so as long as Wallaby can run against your page and if it's being backed by Phoenix, and if you don't screw around with the user agent string, which is where we store all the metadata, then it should actually be fine. In terms of getting that running in your environment, that's, as you know with trying to get stuff working with big single-page applications, it's pretty ad hoc, it's pretty bespoke. It's like whatever you have figured out for your app. Yeah, but as long as you're talking to Phoenix and we can hit your page and that page is talking to Phoenix, it'll just work, assuming you can figure everything. Hi, I think the killer feature here is being able to parallelize on a clean instance of the database, the sandbox as well. That's just awesome. I've done a lot of work with Protractor. Mm-hmm. I've done a lot of work with Protractor and I've done a lot of work with Protractor and I've done a lot of work with Protractor and it has blocking functionality as well. Could you maybe expand a little bit on how technically you implemented blocking and does blocking apply at each line? I imagine you're just pulling on the DOM with some kind of frequency. Yeah. But perhaps you could elaborate first. Sure. So we pull the DOM. Essentially we take your query, the thing that we believe you want to do and every one of those actions will block until the DOM ends up in the state that you've specified in your query. And we do just retries until we either time out or until it becomes true. And so every action blocks until one of those two conditions happens. It definitely adds overhead to your tests doing it that way. The trade-off there is like, it works with JavaScript. And if someone has a better idea of how to do it, like, if there's some magic JavaScript thing, like I would love to know how to do it. Essentially all the other test frameworks do more or less the same thing. Even in the JavaScript ones, it's all done via setTimeout and they try again. And then eventually either throw you an error or they return you a result. So yeah, so every operation and wallaby blocks up until your query becomes true. Each of the tests is running in a separate process, I can't imagine. So I guess there's overhead, but it's all concurrent, right? Yeah, so every test case, every test case obviously is running in its own process or whatnot. And we disable, we don't do timeouts internally in a wallaby. There is the global timeout that we will break your test eventually. Like, only retry this for five seconds. We'll break at that point. But for all the gen server calls that we do internally, we have some agents that run that store state for like, we capture JavaScript logs and JavaScript errors and then rethrow those for you in the test run. And so that stuff gets stored in an agent. And in all those calls that happen, we set the timeouts to infinity because they get caught by a larger exception. Or the test timeout times out eventually. Does that answer your question a bit? Cool. Hi, Chris. So I'm one of those people who've written literally thousands of tests in Capybara against Rails apps and Angular apps and have lost months of my life waiting for them to run. There are always two really big problems and I'm totally blown away that you've solved the first one, which is the sharing of database connection between the test and the browser. That's always caused enormous problems. So we've described that. So I just want to say how impressed I am with that. The other big problem that almost made me give up on that kind of testing is something that happened with pretty much every project is we would build out the app in the first version, we'd write all these tests that automated the forms and then design would come along and of course modern design requires us to basically replace all of our form elements with JavaScript elements. So suddenly our selects aren't selects their divs and our checkboxes aren't checkboxes their divs and all of those rigid operating functions start breaking. Do you have any thoughts about how to mitigate that kind of problem? Like beyond training your designers not to do that. So like my less obnoxious answer is by default all of our actions are assume you have real HTML. A select looks for a select. And in fact this gets into limitations with WebDriver. Like if you try to go click on thing or if you try to click a link in WebDriver and the link doesn't have an href tag it's not a link. So it can't click it. So we actually find those sorts of problems for you. Like if your test fails because of that we actually find that problem and we throw that in the air. So we're like I can't click this link because it's missing this href tag. That's not really the answer to your question. So having DOM just like kind of building JavaScript widgets we do give you all the underlying stuff that we use as part of the DSL. You'd have to look at the docs. You need to scope things like node.click after you can find it. There's nothing stopping you from just like finding the elements that you need to click on and then clicking on them. So we give you all the tools to build that stuff. All our actions are just built with the same stuff that you can use in your test. There's going to be more work unfortunately. As is as tends to happen when you askew good semantic web stuff as we all do unfortunately. Just a follow on to that if you can't select your element in your test you're also probably don't have a very accessible website for people who need assistive technology. So that might be a way to push back on designers who want to do things that make it hard for you to find things in your test probably means that someone's using a screen reader they can't probably find it either. Yeah, I can't echo that enough. You know the best way I found to solve this problem? I did this once to some degree of success. I sat down with a designer who was building these widgets and we actually used a screen reader and that kind of drove the point home like oh, well that sucks because you just if you don't use a screen reader on a regular basis you don't know and you don't know how broken the web is in a lot of ways. So it's worth everybody's time. You have one installed on your machine already if you're using a Mac. So just go turn it on and try it one time. It's very enlightening. Anyway, other questions? About a year ago I was working on an application that had a solar back end and so we had to play some tricks with a mutex and blocking so that the solar tests wouldn't all step on each other with parallel tests and rails and all that. Have you bumped into any edge cases that come up often enough that you just say we can't run these tests in parallel? What are some examples of things that are just too hard right now to parallelize? Well, I mean, right now we live in one of the great things about living in Elixir is that you can just have GIN servers. We can't roll those back. So if you go manipulate as part of your controller action whatever this feature is doing manipulates external state outside of Ecto we have no control over that. That's your application. We wouldn't sort of presume to know how to roll any of that stuff back or to correct for that. So stuff like that, like if you're calling into external if you're calling into other GIN servers and mutating states somewhere else you're going to have to either make that test not async or you're going to have to roll it back manually yourself after every test run. So stuff like that currently the other big the other big gotcha for this is channels. Channels obviously if you're doing a lot of intercommunication between browsers via channels they don't filter via the same mechanism that Ecto does. So while it's not super complicated to pull that same metadata out in your channel and then use it to filter out like broadcasts it's not something that we again presume to do for you like I kind of tend to feel like we should just be giving people snippets for that kind of stuff so that you can use it in your application. So the two big gotchas right now blog posts coming soon on how to kind of get around both those things it's we thought about it for about two seconds on how to like how would we solve that for people and then realize like we'd probably get it wrong and we'd just make your life harder if we tried to solve it for you. So it's more about it's kind of have to know that sort of stuff. Anything else? Yeah. So this is just a potential solution for the question about brittleness in the tests when designers change the classes and attributes. We use this at Thoughtbot and it works really well. We actually don't match on the classes, IDs or elements and we use data attributes. So we'll create basically a data role and name it something specific to how we're using it in the test. So we might do like data role users and so we match on that in the tests and the designers just know if they're going to change the class or the element then they just keep the data attribute and basically the tests never break. Oh cool. I would if you have time after this we should chat more about it. I'd love to see some of the implementation of that. Like that sounds very interesting to me. Not that thing I've thought about before. So that's cool. Anyone else? Anyone else? Thank you. Thanks so much for coming out.