 Hi, my name is Dima Kovalenko, I'll be talking on the topic of managing Selenium Grid and slowly growing your grid and most of us talk is going to be on the topic of managing the nodes themselves from sysadmin point of view instead of just the test themselves. This is my Twitter handle, it is also my handle in the IRC room for the Selenium IRC room so everybody can just come in and pop in there and ask a lot of questions. So crap, I messed it up already. So before starting to talk about the grid itself, I would like to talk about the Selenium web drive or community to begin with. A lot of people don't realize it but a lot of the people who are giving the presentations today are actually petrified of being in front of a crowd. When you're standing up here, this is what it looks like crying out loud and there's a bunch of people and every year I've been come to this conference, the crowd just doubles and doubles and it's not very good for social anxiety if you're just, okay, this is getting worse and worse. But the interesting thing happens, every year people who are scared of getting up here and talking still do that. Why is that? And part of it is that they really, really love what they're doing and they really love the product and they are willing to put themselves in danger and embarrass themselves horribly just because they've discovered something great and they want to share their discovery with everybody and that's kind of the exciting part about this whole project. You see any of the main developers walking around in a hall, go talk to them. They're great people. They will answer any question you have, jump into the IRC room any day. They will drop everything to help you out as long as you're not trying to do something illegal but that's a completely different topic. Quick shameless plug. I just finished writing this book. If you wish to please go buy it, it will be shipping by end of September. In it we discuss topics such as managing test data in your continuous integration environments, how to deal with testing production data versus local data with fixtures, how to test behavior of the application through page objects or without them or with QComber or RSpec or any other behavior-driven software, spend a lot of time on making the test more stable especially with Ajax and all these funny animations and jQuery and all these other nonsense. So that's the shameless plug. Another shameless plug is for my company who keeps sending me to these fun conferences where I end up learning more things every year instead of just sitting at home, they keep kicking me out. Every time I come back I learn so much more that I did the year before that I kind of feel like I need to up the ante the next year and just like, yeah, let's find this something even more exciting to talk about. All right, after all of this is done. Let's talk about the grid a little bit. So as I mentioned, this is going to be more on the sysadmin side and less on the code side. And one of the most common misconceptions about the grid is that the grid is actually this Orvalian Microsoft access thing that knows everything. The truth is the grid nodes are really dumb. They do not do much except for register themselves and then pass all of your test requests down to the web driver. That's pretty dumb. The hub is even dumber still. All it does is it passes your request down to the node and it just keeps track of the nodes. Which one is there and which one is not. It does not know much else. So when we're thinking about dealing with the Selenium grid, it is almost as if there is no grid. The smallest unit of the grid is one node. And that happens any time you run a test locally. That is one grid, one node. What happens when you add external nodes? All it does is repeats the same process but just kind of sends it out externally instead of locally. And keep that in mind so the grid itself is not very scary. That's basically what it is. All it is is basically a HTTP proxy that passes all the click commands and all the things down to the browser that lives somewhere far away. It's not more than that. All right. So let's talk about priorities when dealing with the grid. One of the most difficult times that I had with the grid was trying to achieve speed and trying to achieve more browsers being added than achieving coverage. And very quickly I've learned that this is actually more or less the right order. You have to achieve the stability first and then you can maybe work on speed and then you can start to add more browsers to add more coverage. If you have the choice between stability or speed, always go for stability because if your test fail for whatever reason, because they're running too fast, you're gonna have to run them again. So slow is fast in the long run. Let's talk about stability. So to become, to have stable nodes, you need to first become a sys admin, sort of, kind of, no. A lot of the stuff that come with stability come from the fact that, especially Windows has a lot of bugs that make the everything crash. And this is not just about the grid. This is not just about Internet Explorer. Just Windows operating system tends to be very flaky to begin with. And so a lot of the work in stabilizing and scaling the grid is working around Windows operating system. So if possible, move as many of your grid nodes to a Linux instance. If you can have Firefox and Chrome running on Linux, on some Linux distribution, you're not gonna have any problems whatsoever. And just reserve your Windows nodes just for Internet Explorer. The more things you throw onto the Windows node, the less stable you become in the long run. The next tip is try to dedicate only one test instance per node. Especially if you're running Internet Explorer and Safari. Those two guys, they have this funny way of not being able to keep secrets from other instances of the browser. So with Firefox and Chrome, you can create a different profile for every single test you add and new test instance. So hey, you can have your own cookies for each test. With Safari and Internet Explorer, if you have one test already running and the second one starts and you're testing, for example, the login. Well, the login session from the first test will be already there in a second test. So your test become horribly unstable. On top of that, you can run multiple Internet Explorer things, test at the same time. But every browser you add, you're gonna reduce your stability several times fold. If you run just one Internet Explorer, much more stable than two. That's the general rule of the thumb, and I just covered that point. And to improve, the next point to improve stability, we need to start thinking like a sysadmin where configuration of the node is treated as code. It's not this afterthought of, hey, we're gonna configure this node to run with only one instance of Internet Explorer and call it a day. Well, what happens when somebody comes in and logs into your virtual machine or your box somewhere and just flips one bit or changes your resolution or does something funny? All of a sudden, your grid nodes start to fail tests and you cannot explain why. And worst case scenario is when one out of the ten nodes has some funny configuration on it and now you have one in ten to one in a hundred failure that nobody can explain. So using something like simple as git or SVN to store node configurations, we're talking about timeouts. Those things are very important. We're talking about instances of the different things. We're talking about all these node and hub configurations. Storm in some sort of version control system. That's gonna help you out a lot. When it comes to managing the nodes themselves, use a great free tool like Chef or Pop It and just have that Pop It manage the node. When you need to push out a new version of Firefox, just disable the automatic updates for Firefox and Chrome. Stay in one version when you're ready to upgrade to the next one. Have the configuration management utility push out the new version of Firefox to all your boxes or have it push it out to half of your test boxes. That way you can test it on both Firefox 30 and 31. And when 31 is stable, push it out to the rest of the boxes. If you don't do that, you will be logging in to the nodes every single time. And I promise you, your wrists will hurt by the end of the day because of all the clicking and changing and all the things you will do, it's gonna make you miserable. So having a good backbone for pushing things out, all the configuration to the nodes is a great way to save time. Let's see, what else? When it comes to managing configurations on the Windows or on the Linux box, crons or scheduled tasks are your friend. Have your box restart every night. Take a time, 3 AM, 4 AM, when nobody's in the office, nobody's running tests. Restart your computers, especially with Windows. You don't have to do that so much with Linux boxes. They can go on for months at a time, but typically what happens with Windows box is that after a week of running several tests a day, your node will become so unstable that it will just become unusable. But as far as the grid is concerned, and remember, grid is not the smartest thing in the world, the box is still connected. Even though it is at 99% CPU usage and 99% RAM usage at idle, the grid says, hey, we can still send a test to that. And that test will fail every single time. So when you finally figured out a way to stabilize the operating system in itself, and figured out a way to keep track of all the JSON configurations for the nodes, you are in much better shape. So once you got stability, we can now move on to speed. We can, in general, it is better to have a bunch of very, very small nodes that will run only one browser at a time than to have three very, very powerful machines that will run ten browsers at a time. We are starting to run into things like input output on the hard drive. The writing speed becomes slowed down, and so every time you add a new browser, you just reduce the overall speed. If you have a choice to go to your IT graveyard and dig up 20 very old desktop boxes or laptop, use those. Hey, they're gonna be thrown away. And they can run one Internet Explorer and some Java, that's more than enough. So use low end machines. Again, it's okay to sacrifice a little bit of speed for higher stability and to speed and stability when you distribute it as far as horizontally as you can. All of a sudden, you get both. Finally, after you have got your grid to run fast and you got it to be stable, we can start to add new browsers. So this is a common question. Hey, can we please add Internet Explorer 6 or 7 to the grid? That way, we can test that 0.5% of the statistics that we get from the users all the time, and the typical answer is no. Starting with IE 9 and up, things start to get a little bit more stable. But something about Internet Explorer 8, I'm not sure what it is. But I'd say 50 to 80% of my free time at work is dedicated trying to figure out why Internet Explorer 8 crashed. And I don't wanna say it is the fault of the web driver or anything, I will flat out say it's the fault of the Internet Explorer. Because anytime you have a website that is simpler than Hello World and has three lines of JavaScript, Internet Explorer is already going at 99% of CPU. So if you add a little bit Ajax to that, you have Crash City waiting to happen. One of the problems that I've been tracking down for the last three weeks is that one of my coworkers, he was writing tests for Internet Explorer 8, and it just keeps crashing randomly. And we tracked it down that he was injecting a little bit of JavaScript code to simulate a hover event over an object to get a tool tab down. And just adding a little bit of JavaScript to simulate hover, it's like 50-50 crash rate. And nobody knows why, and all of a sudden you get this cryptic error. Timeout, nothing else, Internet Explorer crashed. The timeout actually happens a step after when the step tries to click on something new or tries to navigate to something else, but the browser is gone. So the action times out because there's nobody fulfilling that action. So yeah, what I want to do is propose a very, very special day for one of the people, actually the person. Mr. Jim Evans over there for writing like the amazing job he's doing it. So I am proposing buy Mr. Evans a beer day. You see him, you buy him a beer. For those of you who cannot see him, this is Mr. Jim Evans, okay? I actually did an internet search for Mr. Jim Evans. And for some reason there was some very similar images, I don't even know. So basically if you see one of these people buy him a beer, all right? It doesn't matter if it's Jim Evans or not. Also go to iTunes and buy his music album. He's pretty good rocker, so hey, let's get back to the topic of stability. As you know, as Simon mentioned, RC is being deprecated. So you need to avoid this situation. When you start the grid and you look at the node, you see remote control at the top. This is bad. The reason it is bad is because it gives people an option to still use RC. And when they have the option, they will always copy paste code from somewhere else that will use RC. So by simply changing the role from node to work WD for WebDriver, it will make a lot of your tests fail right away. But you will fix them, and you'll be happier in the long run. That's the best advice I can give right now. Get rid of RC tests, the other problem is, yeah, sure, this looks great. Most people will use WebDriver interface. The RC is there as a backup. Well, think of it this way. Right now, this grid has one Internet Explorer and one for remote control and one Internet Explorer for WebDriver. Well, this means that your test might be running in WebDriver. But the grid picked up somebody else's test that's coming in from RC. And all of a sudden, the second Internet Explorer window opened up. And now you're getting into the whole sharing of the cash state between the two of them. And these type of things will not be easy for you to debug in the long run. Because all you'll see is a time out error or some sort of a crash error. And you will spend, like me, a couple of weeks watching every single build for that occurrence when kids, my God, two windows opened up. One of them crashed and they were like, how could I be this so stupid, stupid. So just get rid of the remote control, you will be much happier. Okay, let's see. Couple more points on stability. I've heard somebody give a very good tip about, replace the Internet Explorer path to the Internet Explorer or to iedriver.exe that you specify in your grid with a batch file. And have the batch file clean up, delete cookies, delete any cache. All these things, especially with Internet Explorer 8 and below, Internet Explorer has this awesome notion of deleting cookies. It doesn't. It just says I did it, but it doesn't. If you go back into the file, you will find them sitting there. And if you replace the executable itself with a batch file that goes and pre-cleans everything up for you before it even launches the browser, all of a sudden you're going to find yourself in a much happier place. As I mentioned before, create a schedule test, create a cron job. Restart your grid nodes and restart the computers that run the grid nodes. And when I say the grid node in this context, it's the jar file process. It is sadly written on a very old version of Jetty. And it is full of memory leaks, like no other. If you leave it going for more than a week, the thing is out of whack. So just have it restart. It's a great way to save yourself a lot of debugging and a lot of headaches. One of the things I've set up is auto login on the Windows boxes and on Mac OS 10 boxes. Basically, if you're going to have a cron job, make sure that after the computer reboots it goes straight into the user that is going to be running your tests. And make sure that you add some sort of a task or some sort of a startup item to restart the node right away. That way, even if you restart the node, it is back online and back in rotation within, I don't know, 30 seconds, however long it takes to reboot your virtual machine or Windows box. The other option you need to remember about logging in is if you do not auto log in and set it up as a system process, the grid node, and the machine will stay on the lock screen. What you're doing is you're putting your machine into googly-less mode. And so Windows will try to emulate how Internet Explorer runs. And that is not the same way as how Internet Explorer actually runs. So you will be getting very, very weird errors coming from, especially JavaScript, especially from the DOM. It's things that are supposed to be there and are there. When you're looking at it from the browser, when Windows goes into googly-less mode, those things end up missing. So auto log in and stay logged in with that user. Make sure probably some sort of administrative users who can bypass some of the Windows Vista plus access control things. We can talk a lot about stability. There's hundreds and hundreds of these little tiny things that you will nitpick and nitpick. I keep using this as an example, moving the mouse out of the window. If the mouse is within the window, you're going to get some weird results, and you will not know why. Moving the mouse outside of the window makes Internet Explorer render the webpage much better. There's dozens of these little things. So to remember them all is actually quite horrible. So part of the problem I ran into that is just I was using a lot of auto-wit scripts and a lot of batch scripts and things like that just to deal with every single nitpick that makes the test fail once every 50 times that you cannot track down. And about a year ago, right before the conference in Boston, I decided, hey, maybe it's time to actually bring it all together. So I've announced a project called Selenium Grid Extras that year, and so it's been one year now. And what it does is you can find it here. What it does is it tries to document all of these little nitpicky, silly, silly things that we have to do to keep the nodes stable in code. And I did not actually try to be famous or anything to write this thing. All it is is just me writing code to help let me do my work better and not be busy so much trying to rebuild and figure out where are the things, where are they and things like that. So document grid stability with code, but it's going to be in poorly written Java code because I'm a poor developer. I will say that upfront. But the idea behind the project is to make the setup of the grid and management of the grid and making the grid stability as simple as possible. It's just a, if you download the release and you run it the first time it will run, it will just ask you a bunch of questions and then it'll say, okay, the grid is set up. You don't need to worry about anything anymore. So that's the basic idea. You push it out to 100 nodes and the 100 nodes just automatically do things for you so you don't have to go and manually push configs out to every single one of them. A couple of these are just like a small subset of the different features. But for example, on Internet Explorer, every time the node reboots, it changes the Internet Explorer's protected security zones to be all the same. And it's one of those very pesky bugs where somebody changes the protected mode on one of the four items in your browser, all of a sudden the test will not run on you. So things like that, as I mentioned, move the mouse out of the way. Before every single test session begins, it throws the mouse to zero, zero that way it's out of the way. And after the sessions are done, it gives the ability to, hey, if the Internet Explorer browser crashed, but it has that little pop-up saying Internet Explorer browser crashed. That means that IE Explorer.exe is still running. So some of the old data that was from the previous test run is still somewhere in memory. So it goes and tries to kill all these processes, including IE driver.exe and Chrome driver.exe. Those things sometimes are left behind also. It's a controversial feature, but it goes and every time it reboots, it goes and tries to update WebDriver. It downloads the new JAR file, it downloads the new Chrome driver, and IE driver executable as they're pushed out. You can disable it and just lock it into one version and just push it up one at a time when you're ready. And relatively new feature that I've added was to every time a node registers to the hub, what it does is it configures itself in a JSON file and then it hands that JSON file off to the hub for storage. And so what you will have on the hub is a directory with a couple of hundred configs for every node. And every time the node reboots, it goes and redownloads the config from the hub again. That way if you want to change any settings such as you want to bump up or bump down the amount of browsers that any given node can run, you change it in one place, better still if you store it in Git or something. Change it in one place, keep track of those changes and your nodes will automatically pull down the new updates. So the three latest features is to restart node every 10 builds right now by default. We wait for the node to be free and available. After it's executed 10 tests, it just restarts, you can bump it up or down. Again, this is very useful for Windows, more than that in a minute. There's a Jenkins plugin which is a bit of a work in progress and automatic video recording. So let's talk about the rebooting. So just by rebooting the Windows box, every couple of builds or just once a day, I've seen drop in test flakiness by 50%. And just that one simple thing, all it does. If you don't do it, something like this will happen. I've seen this many times and it's just a horrible sign. And the virtual machine becomes so bad where it's, you cannot even log into it. And then when you click on the start button to restart, it takes about a minute or two to just open up the start menu. So reboot before you run out of memory. Next thing is video recording. I just finished part of it last week. And the idea there is that every time a new test begins on a node, we start recording a video of the test. And then when the test is done, we compress the video and then we send it off to Jenkins as an artifact. So that if you have a test failure, you know exactly why it failed. Let me show you a quick demo. This slide is good. Everybody knows that doing demos, it's just the worst idea ever. So where is it gonna pop up? Come on, quick time, don't do this to me. So the current feature that is done, is that gonna resize? Okay, it's kind of hard to see right now, but the current thing when the test is starting to record is, we throw in a little title screen where it says the current session that was used at the time stamp and the name of the machine that executed that test. And this is a very big pain point that is trying to figure out out of the 20 Windows boxes that you have, which one failed and why. And so if you have a little time stamp at the bottom saying, hey, this is the machine that we were running the test on. When it fails, you can go and see maybe that guy has a corrupted older version of Java or it has a corrupted executable somewhere or something like that. As the test starts to run, we grab full screenshot of the operating system, including the browser itself. And the reason for that is when you take a screenshot with Selenium right now, the only thing you're gonna get is the rendered view of the webpage itself. If there is some sort of a pop-up like basic authentication login pop-up that came up or Internet Explorer window crashed pop-up came on or Internet Explorer wants to update, that will prevent your test from running. And so when you have a full view of the operating system, that helps out a lot. And one of the things we're doing is capturing some stuff at the bottom. Just working on capturing the last action. So it's kind of hard to see, I'm still I need to find a UI person who can tell me which colors to use that doesn't burn your eyes. But again, same information, computer name, what the test is doing. And at the bottom, what is the last action that was performed, whether it was a click, whether it was trying to fill out a form. What are you trying to accomplish? And so when the test fails, you know exactly what happened. As right now, we are recording about five frames a second. And it's somewhat single threaded at the moment. But I've recorded up to three tests at the same time. And it was performing pretty well. This will work especially well. I don't know if anybody noticed there, but the command changed. The last command changed because I was faking it. But I'm hoping to add this automatic recording feature by the end of next week and push it up. If we record the video on, say, OS 10 box or in a Linux box, this will be very easy because we will send the video of the browser to a different display. And so you can see it, the downfall with it on running it on Windows is that Windows has only one display. So if you have two tests running on top of each other, one of them will be covering things up. So after the video is complete, the next project will be to redo how grid assigns which node gets the new test. Right now it sends it to the same box as many times as possible, and it doesn't try to spread out the test load as sideways. It just tries to scale it in one box. So that's the video demo. And the other feature that I'm working on in parallel is the Jenkins plugin, and the idea there is to see as Jenkins is executing one of the Selenium builds or multiple builds is to see all of the nodes and how they are as they are right now. And some people pointed out, well, this sounds like a very, very fancy glittery feature that, hey, it's nice to have. But in reality, yesterday as I was showing this during a workshop, I noticed that one of the boxes had that. And that box was failing tests, and nobody could figure out why. There you go, there's a little Internet Explorer crashed window that came up. So just by being able to see the whole overview at a glance, you can say, no, let's go fix that box, there's something wrong on it. So that's roughly it. I don't know if I'm finishing it early. Hey, Gemma, do you know if I'm, do I have time? Anybody to give me a track? Anyways, I'm willing to take any questions right now. Yes, anybody? Yes, sir? Okay, I'm sorry, it's difficult to see. So to record 10 tests at the same time? Yes, it can record 10 tests at the same time. If it's, yes. Yes, because the last mode what it does is it starts a video recorder on per test session. And so each test session will have the little timestamp of what is going on in their own video. That way, even if you have three tests running, each one of them will say slightly different than one of the other things I want to do is add CPU and RAM usage. So you kind of see it going up and down on the per test basis. But yeah, trying to avoid running tests on three tests or one test more than one test on the machine is probably the best thing if you're using Windows. So, yes, absolutely. Well, yeah, in the problem, your Windows computer can be extremely powerful. I was able to run up to 10 tests in parallel on a Linux computer. And I did not have a single problem because for every single test, every single browser went into a different virtual display. And so all the Windows would be in their own display. What happens with Windows is that if you're running 50 at a time, they're gonna be stacking up. And there's a bug in Firefox where certain JavaScript events will not be triggered if the browser window is not at the very top. And so you can have a failure in tests where you fill out a form and it sends an Ajax request to validate your form. Well, if the Firefox window that is running it is on the very bottom, that request is never triggered. And so the workaround for that is before you do any actions such as click or type, you force the window that you're currently using to come to the top. And so if you were to watch the test run, all the Windows would be just jumping one on top of it because they're all fighting for the focus. You can do that, and that's part of the reason I was saying that it is better to get a bunch of very, very old machines that can run one single test at a time than to have one that runs five tests at a time, but all of them are stacked on top of each other. Maybe one day Windows will figure out how to do multiple displays properly, and that issue will be gone. So, okay, I'll, and then I'll say it in the back in a minute. So I'm thinking of doing it as a configuration right now. Again, it's something that I'll be working in the next couple of weeks. But the idea is, hand it off to Jenkins and let Jenkins figure it out whether you want it to archive only the failing test or do you want to archive every one. At the moment, haven't figured out how to do good data encryption for the video. So right now it's just a MPEG for video. And so it's roughly 100 to 200K for a couple of second video run. But I've had luck before with H.264 codec where I could record a test run for one minute for one megabyte. So it's not that big of a deal. And then you just throw away anything that is older than two days or something like that. I believe it's, I haven't seen it. But again, I've only used it for about a week now. So I guess whatever is the hard drive you have and how much RAM you have to, I do believe it writes as it goes. The video codec writes to the hard drive as it goes and then it finalizes it. So it doesn't keep the whole video in memory. But it's really at this point resource constrained like anything else. Yes, sir? Yes, absolutely. Right now it's defaulted to 10 and I'm also working on a feature where it does not restart the whole computer by just restarts the Java jar. And one of the, I spend about a week figuring out how to do, there's with Selenium Grid, there's three different timeouts that you need to know. And all of them need to match up and if they don't, you can get a, you can put your grid in such a weird state where the grid thinks that the node is connected but it's not. Or vice versa where the node is connected but the grid doesn't know about it. So after spending a week just parsing the code and setting up all the timeouts, right now within a second the node that, when the node goes offline within a second grid throws it out. And since we're using virtual machines, it takes about seven seconds to restart. So from the point that the last, the 10th test finished and we started to restart, the machine actually has enough time to restart and join back the grid before the next test starts. So if you're gonna be using older laptops, for example, they're probably not gonna be as good but virtual machines are kind of great for that, so I've been looking too much on this side, let me, yes sir? Yes, so what you can do is write your own custom prioritizer class that will do that. And with that, with a prioritizer class, you can say, hey, I want all the Internet Explorer builds to go above, even if there's five Firefox builds waiting to happen, I want the Internet Explorer to go on the top, and you would have to dig a little bit through the code. And there's a couple of methods to figure out how busy the current node is. And another method to compare is this node busier than this one. But again, you would have to write some Java code to do that. That's why I wanna do, after the video and Jenkins thing is done, rewrite how the prioritization works and just kind of spread it out. And if I have ten nodes and I have ten tests running, it will be spread to all ten machines, and the eleventh test will go to one of the, to double up one of the machines that's probably gonna be free the soonest. So that is definitely a big pain point that a lot of people have been pointing out, yeah, absolutely, no, everything is an option. That's the idea is that you just configure it on per node. I wanna record videos from this node but not from this node, things like that. So that's just something I wanna, yeah. Yes, sure. And you do separate tests on each one of them. The question is that, would it be better to create virtual machines on that one-part machine as in running? Yes, absolutely, and that's what I mean. It's better to have little tiny small nodes. And then you run. What version of Windows are you running? Okay, last time I checked, Windows 7 requires 12 gigabytes of hard drive space and at least four gigabytes of RAM. So yeah, it's not the greatest thing in the world. It's gonna, I mean, you can, what we're doing is actually, we have purchased a lot of blade servers that are running Linux. They're running CentOS or Ubuntu. And then we're using a Zen, the virtualization called Zen, XEN. And that allows you to spin up a lot of the virtual machines relatively easy. It's not as sophisticated as VMware, but it's free. And what I ended up doing is once I set up a base image for, say, IE9, copy and paste it a bunch of times and each node up individually. But yeah, you would really have to think it's like four gigabytes of RAM for Windows 7 or more, so it's quite painful. Well, I have not, my problem is I'm relatively new to Java, it's been only about a year, so I can only keep up with things and let me throw up the project URL again on the screen. So that I'm kind of hoping that somebody who is experiencing a problem can just send a pull request and we had several people who already added some features that had been pretty cool and they are experiencing this one problem and so they go home, they add the feature and everybody gets it for free, it's kind of great. So I'm not sure if I don't have right now any plans for capability tests, but I did have somebody at my work ask for ability to distinguish between Windows 7, Windows 8, and Windows 8.1. And being able to choose that, what I would also want to do at some point is add another capability to choose a specific machine. That way, when I'm debugging, I wanted to run on PC1 and not on a random one. And I just force it to run on PC1. But that's again, it's a wishful thinking right now, it's not done yet. Yes, sir. So you could probably, I've heard David and a couple of guys from Salesforce talk about having multiple grid hubs available. And what you do. Sorry, the question for the video is increasing availability of the hub, of the grid itself. And basically, one solution that I've heard of, but I have not implemented myself yet, is having multiple hubs running. And having the DHCP server point to the one that is currently live. And if it goes down to automatically switch to one of the other ones. And since the nodes are not very smart, what they will do is every couple of seconds they go and they check in with whoever they think the hub is. And so if you have a dedicated network name for the hub, it doesn't really matter which server it is running in. As soon as that couple of second timeout hits, the node will go and say hey, I'm here. And so the new hub will automatically get all the nodes switched over to there. And you can take the old hub down. So that's one solution that I've heard of. But right now, the short of rewriting grids to run would say Tomcat or something like that, there's not much else you can do that I know of. I'm just trying to figure out what time I was supposed to finish. I think them, is there any more questions? If not, I think we're done. Sorry. Can you, yes. I'm just gonna be coding the video causes a significant hit to your TPU to draw your run in the test. I haven't done too much extensive testing yet, but under relatively old IBM ThinkPad that's about five years old right now. I've only seen a bump of say 25% on one of the cores. So it hasn't been too significant. But the encoding part, I'm thinking that it needs to be done after the test is done and whatever is send it off in a new thread and just encoded afterwards. And whenever it's done encoding, only then upload it to Jenkins. So it's not gonna be an instant reply. But part of the other reason is, I haven't seen too much of a hit, is I've only chosen to do five frames a second instead of 24, which is your typical video. And so that kind of reduces the amount of time it can. And the last thing was to add, I added a timeout. So if a node has not received a new command in so many seconds, whichever you configure, it will stop recording the video and start encoding it this way. You don't have a situation where the browser crashed and the test already forgot long about it, that node. But the video just keeps going, filling up the hard drive space, so. So I know that there's a couple of Firefox plugins that will capture JavaScript exceptions. I'm not sure how to do it in Internet Explorer. Yeah, so right now, I'm not sure. I mean, if anybody else, if anybody knows how to capture exceptions in a node or from the browser if it happens, please come talk to me. I'll come talk to you and it'll be great to share that knowledge with everybody else. Okay, so if there's no more questions, I think we're out of time, all right? Thank you.