 Welcome to the session from Rigidity to Resiliency and Evolutionary Tale of Selenium-Based Frameworks by Tariq King. And without further delay, over to you, Tariq. Thank you, Ankit. Let me just share my screen. All right, hello, everyone. My name is Tariq King, and I'm pretty thrilled to be here presenting this session at Selenium-Conf from Rigidity to Resiliency and Evolutionary Tale of Selenium-Based Frameworks. So, who am I really to be talking about this stuff? I'm just a guy that fell in love with software quality and testing, which is a hard problem while I was in college. When it comes to professional experience, I'd started off my career at IBM as a research fellow while I was pursuing my PhD. And after that, I decided that industry actually wasn't for me and I joined North Dakota State University as an assistant professor. So, I was there for about three years and as life would have it, I would realize that it wasn't that industry wasn't for me, it was that the company culture was more important than working for a big name. Of course, if you can work for a big name that has a great culture, that's awesome. But sometimes that doesn't always align. And so, I left academia and joined Ultimate Software as a test architect and would spend most of my career there as a test architect. And actually, the bulk of this talk is around my experiences there as a test architect, but eventually it would grow through up the ranks into eventually the head of their quality program. Most recently, I'm known as being the chief scientist at TAS AI, where a bunch of hacks there, including platform leader, head of R&D and that sort of stuff. All right, so let's get into it. So Ultimate Software, more recently, you guys may know it as UKG, the Ultimate Chronoscrew, but back then it was Ultimate Software and their tagline was People First. And this is true because they put their employees first, but also the software that we were building was all about people. It was all about the employee everywhere from recruiting to retirement, right? And so what this meant was that there was a main product, Ultipro, which was all about human capital management. And it covered all aspects of the employee, all sorts of complex domains from human resources to payroll to compliance and so on and so forth. But there weren't separate products. It was all one unified solution. So all of those domains had to come together under one umbrella, which meant that testing within each domain and across domains was of utmost importance. So in terms of the software process, there was an agile process in place and so you had cross-functional teams with stakeholders like product owners, business analysts, software engineers, quality engineers and later on ops folks. And then you had validation and verification happening throughout the lifecycle, right? At the requirements, design, unit integration, and of course the system levels. So, I was coming into a pretty established software process and a quality program that was already running. And as a test architect, it was really about enabling the teams to succeed. I wasn't assigned to any one particular team. So I worked across all the engineering teams in the company and helped them, I don't like the term best practices, but with good software engineering and testing practices and tools and frameworks. And however, I would find out really quickly after joining that I would be forming a team to work on a very specific mission. And that mission was really to replace their UI testing framework, a framework called SWOT. That was a homegrown framework. We'll talk about SWOT quite a bit and I didn't make up that logo. That's the actual logo that they came up with back then. But I needed to replace SWOT with a new framework that was functional, maintainable and sustainable. And as part of that, also go through the process of rolling out the framework and migrating what was about 200,000 test scripts and have all the teams run those continuously across all browsers, right? And so, you get a mission and a task like that. And I always have one thing that I practice whenever I'm given something new. I'm new to the organization at this point and it's a new project and a very critical one. And I always know that the task is never just the task. The task always is a little bit more than what is described because what is usually described is the now. And you always must consider the past as well as the now as well as any future considerations, right? And so, this is important not just from a technical perspective, but also from a people and change management point of view. And so, what was in the past? Well, it wasn't really in the past because it was still being used but there was the legacy product, the back office. I think we've all been there and it was tested using Mercury WinRunner. So there's all these WinRunner automation scripts and that back office was actually quite loaded in terms of functionality and features. And the features that were not any web product and so this was heavily used by customers and the very critical and important customers. And so it was not something that should just be ignored, right? So what about the current last gate, right? And if you didn't know about ultimate software from this picture, you probably then you once the big three came to Miami and we sponsored the Heat Jersey because the logo was all over there. But this was literally my face. I think when I saw what was going on at ultimate and how complex everything was, there were 19 teams, over 200 people, 100 engineers, 60 testers, talking about six million lines of code at that time and all of these tests, right? And quite a high percentage of them automated which was remarkable that you had such a good ratio but not everything was all cookies and cream, right? The framework, like I mentioned SWOT also integrated with fitness and the windrunner stuff, like there was a lot going on there that needed to be unpacked. In terms of platforms, it was a .NET shop. So we're running on Windows primarily and on Firefox and IE where the primary browsers back then. And so let's talk about SWOT. For me SWOT was the now of what everyone was just trying to kind of get rid of and get off of, right? And, you know, looking at it, there was a lot of different aspects. SWOT stands for Simple Web Automation Toolkit and really was meant to automate web application testing in multiple browsers, right? We had the .NET product on the web. SWOT had an editor with UI recording and it integrates with fitness to allow more non-technical QA folks to be able to write tests. So what do I mean by that? I mean, if you consider the C-sharp syntax for SWOT, which was really just embedded in the programming language versus the fitness kind of wiki style, it was easier in terms of technical know-how, I guess, to write. And, you know, as with any framework that's meant to do something, if you just look at a quick example of what was going on in SWOT, so these are some high level test steps like navigating to Google, searching for Agile Alliance, clicking on the manifesto and then making sure that the manifesto actually is displayed. And if we did this in SWOT, this is what the script would look like, actually a part of the script. And you start to notice some things right away. Like you start to notice that there's expressions embedded in here that really map to what we would call selectors, but they're very well ingrained using regular expressions. There's some interesting non-determinism, there's a sleep in there, there's a lot of stuff. And imagine these were very large files and so there was a lot going on. And so when you thought about SWOT, this is the picture that would come to mind. It was supposed to be simple, but it actually was not so simple at all. From the perspective of the design itself, it was a very stovepipe mode of building the framework. There was not a lot of reuse and very limited abstractions. Of course you had the fitness on top of everything, but everything was pretty limited, right? There was kind of just focus on the company's text. Which is to be expected.net, SQL Server, i.e Firefox. And then this was the hard part. The hard part was that there was really an average of like three months, right? To get anything updated in SWOT, right? And again, sometimes these were most usually over the three month mark, but anywhere between one and six months cadence for releases to the framework. And the team that was building this was really just primarily doing bug fixes. There weren't really any enhancements coming. It was just trying to kind of keep pace with things. So that was the rigidity of the kind of stovepipe nature. On the other side, then these scripts, as you saw, there was a lot of implementation coupling where the test script itself just referred to elements of the page. It was not very readable. There was the non-determinism in there. And it was very difficult. Another part of the fragility was the browsers, right? Like this was a framework that had to handle each and every browser on its own, homegrown, keeping pace with those browser releases. Very hard problem, no vendor backing from any of these things. It was all just on those engineers building the framework. And then the users as well, right? You had your technical internal, there was no kind of external contributors. So there was just a lot of kind of churn on the framework itself and not many people to kind of help, right? So the first thing that you're probably thinking at the Selenium conference is like, well, why didn't they just use Selenium, right? Well, there was no Selenium at the time, right? There was no spoon. Selenium was still in the making and I shouldn't say that there was no Selenium. There was pieces of Selenium all over the place, kind of popping up, I guess. And every company I think was working on their own version of trying to solve this problem. But in terms of what we know Selenium to be now and where the project kind of went, that wasn't there, right? And as I said, it was not just ultimate software. It was engineering teams across the globe that were dealing with this problem of testing web applications from the user interface and dealing with the pace of browser turnaround and so on and so forth. So that was the current landscape. And obviously you don't know everything in the future but one thing that we didn't know in the future and this is actually a glimpse into the future there, that mobile was coming, right? That we had a web product and then we would go mobile and if there's a mobile product then there needs to be mobile testing, right? This is our actual LT lab a few years later. So what is the kind of scenario here? Well, you start to get the team together and it wasn't a big team. It was literally two test architects and two interns but we also had a lot of support from other engineers, full-time engineers in the organization. And we actually called it the virtual team because we considered those engineers that were embedded on other teams that had a full-time day-to-day that were also working on the framework as part of the team, right? And we set out to do mission possible, right? Where we thought that, hey, you know what? This is not an impossible task. We just have to consider all these different variables but it's something that can be done. And it's interesting because the first thing that you do in a situation like this is do your research and see what is out there and what was happening. And thankfully, after I first joined the company the first conference that they sent me to I joined in January and then like a few months later they were sending me to the Selenium Conference in London. And Selenium had really picked up since SWAT had been incepted. And it was starting to get the support of a lot of the major browser companies and there was a lot of community around it. And it was one of those things where I said like, you know what? This is something that we really ought to pay attention to and this is probably where we need to go with this so that the maintenance isn't on the development team that is building the framework and that we have a community of support. Obviously like a lot of ex-unit frameworks to support different languages. And at the time BDD was also a huge kind of push towards this ubiquitous language. And so you started to see patterns arising like the page object pattern and a lot of application modeling going on and being able to separate that model and reference it so that the test didn't have to hard code those things. You had all of your support from a test driver standpoint was set up and teared down. And this was nothing new. This was all there from the unit tests but it was now really heavily being used for all the UI testing and stuff that was going on as well. And then of course your BDD and kind of Gherkin give them and then support. So, these were kind of some of the trends that were happening in the space. And it was important to understand that like each of these things, the modeling, right? Abstracting the user interfaces and applying that in the context of a web application was something that you could look at and say like, yes, you know what? It makes sense to kind of model it like this and separate it. You talk about good design principles and then have the behaviors also embedded there with that abstraction of the user interface. It was quite natural. And then the scaffolding that comes along with it, right? That is now easy to write those tests very clean. You can document them like using a kind of fluent interface by chaining these things together. And then on top of all that support and infrastructure for driving test execution there was also things like the Selenium grid and all these other pieces that were really kind of taking off and helping to transform that landscape. And then of course, there's BDD, many different tools. This is StoryQ, but you know, I wanted to see that there was always a push even from the SWAT days to kind of bring the business user closer to the testing because they're the ones that have to accept the stories and so on and so forth. And so again, you were seeing fluent interfaces with Gherkin style and being able to execute and look at the results of tests using this kind of domain specific language at this level. All right, so what was our solution? Our solution was to build Echo. And Echo was really a middleware architecture that sat on top of Selenium and other frameworks, right? And the key thing here is that you wanted a level of abstraction above these frameworks. So that if you had to move off of any of these frameworks, you wouldn't need to necessarily get rid of the entire Echo framework. You would just need to reimplement those interfaces with the other framework. In addition, you're talking about moving towards the future and being extensible, integrating things like Opium for mobile, also thinking about the back office and being able to pull in coded UI to still test the back office and the runner scripts could go away or be converted and then other things, right? Database interactions. And so Echo was built on those principles that were good from the community, right? The application modeling and then having layers that represented, you know, levels, multiple levels of abstraction to help with this problem. First being the test abstraction layer, which is really your internal DSL. And then taking those commands from the internal DSL and translating them into something executable. But I was on top of an abstraction for the framework and then an interaction for the framework, right? So you have these multiple chains and of course there were some common and shared resources amongst these things. But now you have this kind of framework in place for integrating and translating tests from a relatively high level all the way down to the execution level and then getting the results, right? And so what does this really look like when I talk about it? How was it even different from just straight up Selenium? Well, there was a lot more modeling that we wanted to make sure we did up front. You know, the product, the app was one level. You had the whole page, but you also had different controls like the menu bar, the button bar. You had other controls that you could model. The grids were a big thing because you wanted to be sure that you could manipulate the grids, both the kind of header and footer of the grid of the grid control itself, as well as verify the roles of different things. And there was a lot of different implementations for these. And so the kind of language was this fluid interface where you could do like, okay, here's the application context, here's the product, here's the page. I can do an action like a click or a blur or an assertion, some sort of verification. And then that actually changed down into the different elements on the pages, right? And you get IntelliSense and all these good things that come along with it. So some examples of some standard commands, right? Including reusable functions like, you know, login as that are, you know, this is how you actually log into this page and you just pass that on. We call them macros at the time. And then also the database interaction that we needed for not having to, if you weren't doing a end-to-end journey and you just wanted to do some direct linking and do some setup that you could actually manipulate the data stores and so on and so forth. And so this internal DSL again, nothing super special, except the key parts of the abstractions that were in place to make it very extensible. And, you know, we built all sorts of stuff for the teams, test templates. There was a lot of configuration and then, you know, documentation of course. And, you know, we had, what I would call a lot of good hits, right? Through this mission, right? The framework itself was very unified. It was quite flexible. You didn't have the rigid stove pipe nature of development. It was easy to add other things in, like we didn't put Sikoli and some of those things in there from the beginning. And it was quite scalable, heavily due to Selenium and those aspects that were happening with the Selenium grid. And then, you know, the rollout was successful. We had releases that were more of on a continuous basis. And then we became contributors and committers to the Selenium project, right? So now we were part of this community that was not only benefiting from this framework, but now contributing and giving back to the community in that respect. I would say somebody, the misses here was that even though this was better than the SWOT syntax, it was still very technical and because it was embedded in the programming language, it was very hard to kind of, you know, just think that you could get away from the technical aspect of it and have a non-technical user write test. It was still inherently fragile because of the selector problem. Even though you had it in a separate abstraction, if those selectors weren't written properly or if there was a change, you still had to go update it. It's just, it was easier to update in one place. And then there was some accidental fragility because we were an internal DSL embedded in the programming language. You found a lot of folks doing all sorts of weird and trying all sorts of weird things who weren't really programmers, but were just kind of playing around with learning to program at that point and putting those things in automation, which kind of introduced fragility. Governance and standards then became a huge thing that we didn't have immediately that we knew we needed to get to. And while we solved the problem and got it working for the company, there was kind of like this bigger or biggest picture that we missed, right? A lot of the integration with the other aspects of the business were areas of opportunity. And so, one of the first things that was easy to kind of put in place was guidance, right? Like if you don't walk on this path, you could really harm yourself. And so we started to put standards and playbooks into place. And another school that we had a lot of that guidance there for UI testing, which is an important thing to have. But then we thought, well, what if we could go further, right? Like the Japanese art of pokoyok, which is mistake-proofing, like can we build it in such a way that people actually can't go and make those mistakes but it only fits in a particular way. And that's where we started thinking about internal versus external domain specific languages, right? And, you know, internal DSL, which is embedded in the host language just opens up a lot for people to do. Whereas an external DSL, which is kind of a separate standalone language with an independent syntax is harder to build but gives you full control over what people can and cannot do because you just throw compiler errors if they're doing something that's not allowed. And so since we had like a good foundation with echo, what we did was that we extended echo with legend, right? Which was a Visual Studio plugin and a full-fledged external DSL that had its own compiler, own syntax and quite a lot of integration. So there was this DSL editor that you could write your tests in still within Visual Studio as a DS plugin. And you had like a sweet information section that could have the information about this particular test we'll dive in and see how some of these things work. You could declare your products or whatever contacts you were dealing with. You had your setup structures and then you had your test scenarios, right? And you had very high level descriptive kind of English level descriptions, but then you could actually implement those steps with automation, right? So the three lines below the given a login to ultipro are actually pulling from the commands and the page objects from echo and allowing the person to write automation at this level and it's still very readable and so on and so forth. And we integrated a lot with it, the data-driven testing, we had test tagging and so on and so forth. So we're gonna run through some of these pretty quickly. But there was now the bigger or biggest picture that I talk about, the requirements integration, right? You could actually come and create a test that was based on a user story, right? So you create a new test and you say, hey, I wanna create a test that's based on an existing user story in a system of record like G-Rail. And then once you enter in the identifier, unique identifier for that user story, then your test is automatically kind of populated with the documentation from there and it's well-structured, the story is linked already and so on and so forth. And you have kind of like the beginnings of this descriptive test that you want. And then, of course, you can navigate straight to JIRA and see the user story that was used to generate this, right? And so the process again, right? For business level users was now in place. And of course then there's automation, right? You then needed to be able to take those steps and start to refine them by adding automation, right? Which we kind of talked about how then you start to now go through and you can, you know, give your context and if you're only logging page, you'll now from that point, see all the page objects and commands that are relevant to that page. And it kind of guides the user, right? The poker yoke kind of guiding the user towards what bit is allowed and allowable in this particular, scenario. So, you know, as you look at this, you ought to think, well, okay, this is good for abstractions. Like we could actually hide away everything and expand. And so that we have these multiple level of abstractions even at the language level. And then while executing, you had the power of being able to actually set breakpoints at, you know, the automation level, right? Which is still very high level here. And step through your tests, right? And, you know, here you just see an example where we kind of step over the application launches and step over again and let it run. But you can certainly think about debugging your automation at this level as well. And then, you know, later on drilling into it. There is also formal parameters and the data manager. So it wasn't just these hard coded strings, right? You still wanted to be able to leverage data-driven testing. And so the syntax itself supported just adding formal parameters to your test and leveraging the data manager from within the JOS Studio, just a widget that we created that let you, you know, cycle through the different tests that were there. And if there were any parameters in the test, then you could go ahead and start populating the data-driven aspects of your tests. So very convenient for everything to kind of be there all in one spot. And then, you know, part of this was then, entire process was thinking about, well, what about the test inventory and all the documentation, right? That someone shouldn't have to open these projects to kind of get to it. And so, you know, what we did was just say, well, okay, you know what? Legend can also be a source of truth to populate the test inventory because user stories will get outdated over time. It may not be revisited, but the test is something that kind of keeps, you know, the life of it as at least the automation as it lives on. And so let's sink that over to, in this case, is Microsoft test manager and all the steps kind of come over. And of course, you could think of running this task daily, nightly, but if you just updated the test, you could just push it and update that test, right? So that was pretty cool. And, you know, part of it was then also being able to share and communicate, right? So you could copy a link to the work item in the repository, share it with a colleague and say, hey, could you review this? And then they can click and go straight. All this was just the convenience of being in the same environment, but also having external stakeholders that may be in other systems like the test inventory. And then last but not least was the idea of really building on categorization and doing selective regression all from within the legend environment, right? So it was like, hey, how can we facilitate modeling the application domain and having that model available to test so that folks can leverage it, right? So if this is a test that belongs to the, you know, deductions category, or if it is also a test that is related to payroll, then you can, you know, add those things there. And then as you go through, you can see like it's showing you what tags are available. We did things like house synonyms and stuff like that. And then you could go through and search by browse, by tag, by author and by project, and then actually selectively execute tests by, you know, either of those things, right? And let's run the test. So that was kind of like the crux of legend where we kind of took echo to a new level by now thinking of the end user as someone who may not be super technical, right? And that may need more guidance and was able to kind of now successfully write tests because they had all the guidance. And then in addition, think about all of the process that actually goes into collaborating to write this automation, use it and so on and so forth. All right. So, I mean, we enjoyed building these things. We shared our experiences, right? So this is us at the international symposium for software testing and analysis, ISTA 2014. From my academic background, we always try to go to industry as well as academic conferences, especially those that were joined. So we published papers, paper on Echo was at Jamaica, which is a workshop on joining academia and industry contributions in test automation. And ISTA, I just mentioned, and then my first public speaking talk in industry was actually at Star West. And I wrote a paper that outlined both of these things and talks about both Echo and legend as part of that, right? All right. So why even am I telling you about this journey and going through this evolutionary tale with you? And well, the first thing is really that the reason that we were able to do all these things with Echo and with legend was because it was built on Selenium. So, you know, to the creators of Selenium, like Kudos and thank you, Jason Huggins, Simon Stewart, Jim Evans, all the committers, the entire community, you know, it's not even specifically just about the framework, but about the problem because Jason and Simon and Jim were focusing on the big picture that Selenium now represents it had a ripple effect that teams didn't have to worry about those problems that Selenium was solving and could focus on other problems and other big pictures, right? And that was great, you know, but it also uncovered, so what Selenium did also uncovered some of the challenges that were inherent with UI automation and it's not just Selenium, right? Some of those were inherent and some of those were because of the solution, but anything that's based on the implementation and anything that's based on the DAW is going to be inherently fragile. And we are still really lacking when it comes to automation as we call this because there's still limited reuse, there's not reuse across applications, there's not much generality, there's very hard maintenance that goes into, especially if you don't do these things right and follow good practices, right? And so that gets me to like, what is the future of this stuff and where are we going, right? And you know, in my role as chief scientist, like I've seen a lot of the application of AI to software testing. One of the cool things is the use of AI for things like element detection, right? So instead of using hard-coded selectors, actually training computer vision models to recognize these elements and identify them, even chop them up on the screen and use them as training data and so on and so forth. And we're already seeing some of that integration happen into Selenium and Appium and Cypress, right? Which is great because we need to see other innovations happening in this phase. And there's this just whole area of using other aspects of AI machine learning to drive the testing process and reinforcement learning, you know, having the box explore and do trial and error is also a very resilient thing, right? Because the resiliency of the framework depends on these methods used to identify things and locate things and to navigate through the application which could change, right? And reinforcement learning and computer vision help us to deal with a lot of that and move away from, you know, the fragility, right? Because if you think about it, even legend as great as it was still suffered from those fragility issues, right? And so, you know, our path to resiliency really is to continue to invest in a lot more innovations around bringing those bigger and biggest picture problems to the forefront of engineering teams that still need to be solved. And of course, Selenium still plays a role underneath the HUD as an interaction framework and as a community and as something that helps to scale and go on and kind of build that out to the best of your ability and continue to work on these other problems, right? And what we're seeing is we're seeing AI and machine learning start to enhance portions of the testing tools that are out there in frameworks. But I think as we move on, we'll start to see it look at entire toolstacks and different dimensions of testing but still require human input. But the cool thing about this technology is that over time with that human input being used as training data, the AI can actually take over some of those tasks. So I think we're in a very exciting time for automation for Selenium and it's not these AI-driven tools versus Selenium or Selenium versus Cypress. And it's like all of these things that we use to solve problems, it doesn't matter which tool you're using. If it solves your problem, then go for it. But then also think about what other problems do you need to solve or now do you have time and space to solve? So I encourage you all to do so and have fun doing it. We had a lot of fun creating those tools and then we went on to create more tools at Ultimate that use AI and machine learning and so on and so forth. So that's my time. Thank you everyone. Feel free to connect on LinkedIn or Twitter or shoot me an email. Yeah, thank you, Tariq. That was a really great presentation.