 Welcome everyone to the session working in the shadow dorm how to test web component by David Burns. So without further delay. What do you David. Thank you for that. Thank you. Hello everyone. I hope everyone is doing well. And so I want to start by saying thank you to kind of the whole Selenium comp India team for the hard work that they put into to kind of making this actually a reality. And so, I hope I do everyone a great service here. So with that, I'll get started. So, for people who don't know who I am. Well, actually, let me stop. I'm going to be talking about working in the shadow, specifically the shadow DOM and how we go about testing web components. So I'm at automated test on most social media. So if you ever want to ask questions, especially afterwards, if you, if there anything does crop up and I don't get to answer your questions straight away. Feel free to kind of ask there. There we are. So just a little bit about me. I head up to the open source program office at browser stack. We're testing in the cloud provider. I am the chair of the browser testing and tools W3C group. I've been doing that since I was at Missila. And I've been a Selenium committer for pretty much most of my career, I think we're bordering on about 15 years that I've been working on Selenium in one version, or another. And so I've seen how the industry has grown. I've seen how browsers have grown. And one of the main ones that came that I really liked was the idea of web components. So let me get with that. So just a quick agenda. And is that like, I'm going to talk about what web components are as a high level, and I'll start drilling down from them. How we go about writing them. And so I'm going to give you some code examples on the screen. It's, I haven't created a repository, mostly because like, they're really simple to make. You'll see a lot of different tools out there are still using them. And then we'll, I'll show you how we go about testing them. And there'll be a demo during that process. And then I haven't put it there, but like most good software engineers, I'd like to do a bit of a recap. And then hopefully we can dive into some of the questions as we as a crop up. As was mentioned earlier, like if you have questions, throw them in the chat. I'm not going to answer them as I'm going through, because otherwise I'll be really distracted and I really suffer from destruction badly anyway. But throw them in the chat. And I will try get to them as I go through. And if I don't get to your question and you still have questions or kind of concerns or anything like that. But later on, or kind of just hit me up on Twitter. And I will do my best to give you meaningful answers. And we're possible I so give you code examples with what we're going to do. Right. So what are web components. For a very long time, where like the web has been this like mishmash of building. And there was never any real thoughts into how we can make it better. Like this fads have come and gone. So like, when I started my career, the way people used to create their websites was like using Dreamweaver or notepad plus plus, or like tools like that. And what you needed to do is like you needed to handcraft pretty much everything. There were some was he were tools that allowed you to kind of drag and drop to get everything into place. But those tools only gave you like incredibly basic websites, and the web has moved forward in leaps and bounds in huge, huge numbers, right, like, I'm going to show my age but like, when, when I got onto the internet, the internet was pretty brand new. Like, I think I was able to get dial up when I was 17, the internet had been around for about four or five years at that point. And when I say the internet, I mean the World Wide Web had been around for about four or five years by that point in, in the way that like Tim Berners-Lee had created. And the browser wars were in full flight at that point. Netscape was still a company. And people built their websites and kind of just made everything the way they wanted it. And then like we, we hit the early 2000s and kind of Internet Explorer had won. And then because they'd beaten Netscape and then obviously out of the ashes of Netscape came the Phoenix Project, what we now call Mozilla. And so, like, the web started being rebuilt. And we started getting everything that we needed. But the one key problem that was around was like building a website is hard, right. Like, I know back end engineers will say, oh, it's only a bit of front end. It'll be easy. But it's not. CSS is hard. Building websites is hard. And then testing them is hard. Like, we know this is from kind of like we've all built careers out of this. But web components allowed us to allow us this idea of kind of taking all the best parts of web of like development. And so by that I mean kind of encapsulation. Can I build this one little thing and move it in and allow people to use it? Could I just like share it around to other people? Could I do all of this in a meaningful way that allows people to build stuff? And so this is where web components came from. It was an idea that came from Alex Russell. He first talked about it at the Frontiers Conference in, I think, 2011. And the Chromium team built out an example of it. To be honest, it wasn't very good. But at least showed promise, right. Like I'm not dissing them. And then Mozilla and Apple or kind of the WebKit folks. So it's Apple and Agalia and a few others got involved in like this is how we would make it better. And they did it and we got web components version two. And the web started like it changed some concepts of the web. But now I know kind of a lot of people thinking, well, what about front end frameworks, David? Like we have Vue, React, Svelte, Ember and so, so many more. Those do all those things that you've been talking about. They have the tooling that allows you to kind of build components, build smaller like sections of a website and allow people to share it about. And it works. And to that I go, well, that is true. Like, you know, I'm not going to argue that point. But when there's like a lot of these tools have also been built in the same for the same reasons that like jQuery was created. Interoperability between browsers is hard. This is the other thing that like a lot of front end engineers have to work with every day is like how do we handle kind of minor differences between different browsers? I remember part of my career, like there was one point where making sure that certain things with the same between Internet Explorer and Firefox were correct. And at that point it was like, okay, but like Internet Explorer can't do rounded corners. So like, do we really care about this? Like, and jQuery was still rounded was still really relevant. And then it's changed. But like the one key theme around view, react and all these other tools is they add more to the web pages. And it's in and not in a really, really good way, unfortunately. So I'm going to pause. And I this is going to be another shout out to the Selenium like organizers and how we've changed over the years. This is a screenshot of a YouTube video that was of I'm the kind of the blur blurry person on the left of what you can see. And my friend, Dave, who's on the right there. He's the other blurry person. How things have changed. I'm so kind of big shout out to the Selenium Conf people. But but back to web components. Adding view, react, all those other tools. Blows pages. It blows pages like you cannot believe. And like, even with them minified, we're talking like a megabyte or two. And to some people that's like that seems like not a lot, right. And on desktop, it's not a lot because kind of most people are on kind of fiber backed internet connections to their Wi Fi routers that allow high speed transfer of data. But when I was doing this. And so this is from GTAC, the Google Test Automation Conference in 2009. So kind of 13 years ago. At that point, it was kind of like, I think it was. And I'm open to correction. It was about like half a second delay in loading a page could cost you like 5% of your of your market. Like because people be like, I'm not waiting around for this. And that was, you know, when the internet was slow, the internet's got faster. But people's time attention spans have got less. And so like page size, what page weight load times is still super important. Because kind of when you're building for all these different tools or different web pages, you need to think about mobile nowadays. And like recently I went to go visit some colleagues in India. And like mobile phones are kind of fairly ubiquitous to like many people. But that's their only in access to the internet for a lot of people. And when I was at Mozilla. And I kind of met people during the Firefox OS project. Like their first access to the internet was going to be on a Firefox OS device. And so, you know, we need to think about all these different things and improving page weight and page load times. And this is what web components do is that it gives you that ability to recreate some of that front end framework. Goodness in simplicity and things like that. And that's a lot of the bloats and it's optimized by browser engine engineers who kind of like a lot of their job is just making sure that their browser is fast, right? Like Chrome has said for years that their browser is fast. It's got a bit heavy over the years. The Mozilla project has spent a lot of effort in making sure that Firefox is fast. And then when you look at like where the time, like the engineering effort that is spent on web kids by kind of Safari engineers or by WebKit engineers, it's just making sure that it's fast, especially on mobile. Like, like, I always might not have all the best features, but it's definitely fast. And so, and it uses a lot less battery and things like that. So it's important to kind of think about those things, especially as we move forward because we need to care about it. So I've done a lot of talking, a lot of preamble. But what does it look like? Like, how do I create a web components? And so here's a really simple example is that you just create your own elements on a page like this. So like, obviously you have a body. And here I've created a Selenium shadow element element. And I can create anything I want. And like, web components are so ubiquitous that like a lot of people don't realize that they're in the browser, like for Firefox, most of the UI, so kind of the back button, forward button, reload button, any of those types of buttons are web components. Like, before Mozilla, like went in on web components, they had a thing called Duzul, which is XUL, which is essentially web components and allowed you to kind of build out these tools and then ship them around different parts of Firefox. And then obviously there are times where we use, like the browsers use native controls, that would be like file uploads, where you get a, like a dialogue for the files. That's just a native thing that the browsers have hooked into. But pretty much everything else, there are web components and you can kind of build them out. And again, it's super simple. You create an element on your page to access it. And you can kind of use either some little bits of JavaScript, which we're going to get to in a minute to kind of make sure it loads up properly. But that's all you need to do. And so this is like a bit of JavaScript to make sure that that element knows what it is and how to do it. So I'm just going to go quickly line by line. The way web components works is that it exposes these HTML objects. These HTML, like objects, so this HTML element, there'll be like a paragraph element, things like that. And you can extend as and when you want it. You create this class and like, sorry. The JavaScript has got a lot better nowadays so that you can just create these classes, build out what you need. And then the custom elements, that's something that's on the window objects and you just define your element. And that's it. Like it's super simple, like really, really simple to be able to create these things. It just does it and away you go. So, so nice. And so with this great new feature, how do we test it, right? Like that's the key part here as we kind of move forward. Because we need to make sure that we can test it, right? So the web apps working group in the W3C and what WG kind of rejoiced. We fixed the web. And like you thought about testing, right? And they didn't. Like that's the thing, unfortunately. And I'm going to go through some of the reasons why. The web is not as fixed as we would like it to be. And for the main reason is who thinks about testing? When do we think about testing? And it's always that last afterthought. How can we do it, right? And to be fair to the working group, they did think about testing but purely for making sure that it worked in their basic tests to prove that you could do those. You could do those items. So there'll be what's called ref tests, which are like reference tests. There'll be some JavaScript tests and we can do it like the things like that. But like as a whole, no one actually thought about how we go about testing it. So the reality is, the answer is no, no one thought about testing. We're not thought about it in a meaningful way. Because kind of it's nice to test the component, but how do you test a website that has multiple components? How do you make sure that they're working together? Like this is our job. This is how we get our salaries each month. This is how we work on it. And so different groups try to solve this problem themselves. Like playwrights has a special locator that allows you to access elements within a web component. And you do that if memory serves with a double greater than sign in your locator. So you would go in like div greater than, greater than input. And that I would say find this div with this shadow DOM attached to it and then go find the input element. And then you could send a click or any form of interaction with it or just kind of query to get these things. WebDriver.io does something similar as well as Selenium base, which is a Python Selenium framework built out by Michael Mintz. And so these are really good. Sorry, I just lost track of it. But the solutions are not elegant or easily transferable to DevTools consoles, right? Like over the years the Selenium core contributors, one of the overarching goals is that like you can copy an locator from your Selenium test and put it into DevTools and it will work. You can't do that with these new locators that everyone's created because, and I'm not blaming these teams, right? Like they've done what they felt they needed to do. But because no one thought about testing and why testing is so important to do it. And so we got this kind of unelegant process. We got this nice elegant way of creating websites, not an elegant way of testing it. And the tool that they, the way they're doing it, only works for one part, one type of work component. It might be the majority of way people use it, but it's still only going to work on one part. And earlier I kind of glossed over this section in this, in that the line where it goes this dot attach shadow. So what that does is that says, like, when we define the custom elements of Selenium shadow elements, and the constructor is going to attach the shadow dom to that element and allow us to work on it. And the key part there is that it's got their mode open. So there are two modes, this open and this closed. When it's open, the way playwrights, WebDriver.io and Selenium Basework, it works because they can kind of, they can do some JavaScript, they can do some like kind of, you call that command, they then do some magic and then they're in it. If that was closed, you might as well kind of try access a flash object or a canvas and try work with it that way. Because it doesn't work well. And the reason is that like the Safari team said like, well, you know, people might want to put certain bits of information into their shadow dom and they don't want to like fully accessible to JavaScript on the page for security reasons. So it might be kind of like your, you might create a new element that always contains your CSRF tokens or something like that as a like an extreme example, right, like, but there might be other reasons why you don't want people to do it. And so you could create a closed one. And then the, the working group also decided that like there is not we're not going to create a mechanism to pierce shadow doms, especially because they see it as a security problem if something is closed. And so, at a high level, it makes sense for you and me, it doesn't, and it's painful and we need to work around it. Until on a steed road in Jim Evans to save the day. So for those who don't know Jim Evans is a member of the call committers group in the Selenium project. He is also a principal engineer at Salesforce who have been working with web components for a very long time I think they, they call it the LWC project I think that's lightweight web component project. So Jim wrote in and used his many, many years of experience to kind of think about how people would want to use it in Selenium. And how people would want to use it at Salesforce so he has this huge raft of engineers that he can access and ask different questions to. Jim came up and wrote the WebDrogo spec prose to handle this handle the situation. And the way he did that is fairly elegant but then like all solutions that Jim tends to make are fairly elegant and one of the key reasons why he's a principal engineer. We have something like this. So, as always, if you want to interact in Selenium with an element you need to find the element to start with. And so here you can go Selenium shadow elements. Now, the way shadow DOM works is that like, if you find the element, you can carry on searching through the DOM, or you can kind of dive into the shadow DOM, depending on what your need is. The way the analogy I tend to kind of help people think about it but it's not exactly right, it is kind of the way iframes are in that you kind of find something and then you dive into the iframe or you can kind of search the rest of the DOM. And so this is where he we created the get shadow routes API calls, which allows you to dive in. And then from there, you can kind of just use all the usual things that you would. So like, if you want to find a single element, you would just use find elements if you wanted to find multiple elements you would find it. And then if you wanted to do interactions you can. Another thing is depending on like what your web component is, is that like, if you just found your elements and you did your interaction so you'll click your send keys at that level. It would still go through to how you would want it to be so like, you can have multiple ways of kind of going in and out of things or kind of just being able to use it from the top. And that makes it really elegant solution. The other thing is, is that the, because of it being in the web driver specification, we've been able to make sure that it works on both open and closed. And that is a really massive thing is that like there might be times where you want to be able to test in in it. And so the get shadow route allows you to just kind of access the next level of the shadow DOM and then move on forward from that. Sorry, I'm just going to have a quick drink before we carry on. And so with that, I'm going to dive into a demo. And so here's some kind of JavaScript that I've written. I'll just close these. I don't need them at the moment. So I've got a kind of night watch test that allows me to, I'm just going to navigate to MDN. MDN. I'm sure everyone uses pretty much every day I do to look up things. I have a really good example. We're going to get the shadow roots. We're going to kind of find some things we'll do some assertions. We'll find some more things and a final assertion and we'll do that. I've just put these commands. So this is new commands in my watch that allow me to kind of other pause or debug. Like there's a pause command is a debug command to allow me to kind of step through some of the code. It's in a kind of easy way and allow people to kind of do things. So it's a fairly simple test. I'm going to, this is where kind of, as always, partly the main reason for doing the pause is also so that I can move the browser because if you've ever watched any of my Twitch streams, sometimes the browser just does its own thing and always goes to where I don't want it to go. So hopefully the code looks fairly simple and there aren't any questions, but as I said earlier, if people have questions, please do ask them in the chat. And at the end, I will kind of answer them and we can go from there. So I cheated. I used to reuse the file. So here's a widget. I'm just going to load. DevTools. I always forget where it is. Here we are. So here is our shadow. And you can apply styles to like the nice thing about web components and I don't think I put enough emphasis on that is that you can create your own styles inside that web component. You can kind of change it to how you want. And so this pop up info. So here it just looks like an image, right? We can change it, but we can then create like it's, and hopefully this is coming across. I know it's quite small, but the idea is that there's a little like widget that appears when you hover over and we can access any of some of this information. So we've got our test. I paused it. So let me hit space to continue. And so here we expected some information. And now we're going to kind of search into the icon and then expect some more information. But we've been able to kind of find the info. So here we can kind of, we want to make sure this has, because you can't see it, but like your card validation code, sorry. So like that's the expected code that's passed, because we've been able to dive into the code here and just make sure that like that is there. And as you can see, again, I'm pretty sure I'm reiterating what I said before, but like, this is the new element which is our pop up info. It's not standardized HTML you won't see this in the HTML spec. And so we've got that and then our test has finished and we were able to kind of just check that everything has worked as we expected. That's sorry. Let's go back to present to you. Oh, okay, cool. So we've had our demo. And that's it really like it's really simple. We've got like, I'll just go through what we've learned, which is like, we know what web components are and how they're impacting the web and how we can make websites a lot less bloated by using kind of native features in the browser rather than relying on front end frameworks. I'm not telling you not to use those front end frameworks do what you think is best for your users. But there are more native features now and they're any going to get better. We also looked at how we could go about interacting with them. I'm just going to bring that up again, which is, you know, finding our new element. In this case, I search by tag name so which means I could search by the tag that was there that I created which is a Selenium shadow element. I could dive into the shadow roots and then I could search around inside it. And if I needed to, I could do whatever I wanted. And so it's that. So it's really, really simple and it's not adding too much like what I call cognitive load like you don't need to think about it too much. Anything you need to think about is in the same way that like if you needed to move into a new iframe you just switch to that iframe and then you carry on with your normal test. It's the same here. Except now, you're not changing where driver is going to be, but you're creating a new shadow roots. So in Java, it creates a search context objects in the same way that Web element has it has some of those search context tooling. And then you go from there. And with that, I'm done. I appreciate I am possibly about five minutes early. I might talk, but if there are any questions, I am happy to kind of answer them and explain things. So yeah, thank you. Thank you. Thank you. So I can't see any questions in the Q&A. Okay, that's fine. That's fine. One of the things I've just realized that I didn't mention is this is only available in Chrome at the moment, but there are patches being reviewed for Firefox, which will hopefully be very short. There are some questions coming in now, so I'll answer those. And I spoke to one of these Safari engineers who said, because it's in WebKit and WebKit is like they do an upstream or downstreaming to make Safari. So like the code is there, but it needs to be just plumbed together is what they said, and you should be able to do. So there are some questions. Thank you for them. Is there any chance to automate a shadow route that is closed? It should work. If it's not working, that's a bug in itself. Like, so I've been working on the Firefox patches and in Firefox, it's definitely going to work on closed shadow routes because like it makes sense to do that. And you should be able to be able to do that. Handling of pseudo elements will be some will, I'm assuming will handling of pseudo elements be similar like shadow elements. I don't think so. So kind of the way Selenium generally works is that like whatever you can do in query selector. So if you can create a query selector, you should be able to find those elements like if it returns a list of elements, those are then serialized into web elements when they sent back to your side of things. And so they should be really simple to do that. So you should be able to handle it. And like it should just be a web element. If it returns something, if it doesn't return something, then it might be only searchable via kind of the way CSS does pseudo elements. And I know there are cases where they don't always match up. Will it not be raising exception? When we try to access the shadow element using find element. No. So the only, it shouldn't do. And I know from at least my testing. I've not seen a case where it has been doing it, but if it again, if it starts throwing weird exemptions, please, please kind of raise it with the various browser vendors. So, and if you're not sure you can always come to like the Selenium Slack, and we can always help you find the right place to raise that issue and be able to kind of get it to them. But you should be able to interact with it like it as either a fully fledged elements. So kind of just find the elements and then interact, or you can then dive down and work with it that way. Yeah. So we have one more question. Is there any chance to automate shadow closed? Yeah. So I answered that the first one, but I'll just go over it again, which is, yes. Definitely, you should be able to, if it's not working, then you should raise an issue with that, like with the relevant browser vendors. I've made sure that it works well in Firefox. So I've been working with the Firefox team, mostly because I know the code base in that area. So I was trying to help them out. And I know it works with closed shadow reads in Firefox. So if it doesn't work in Chrome, I haven't checked that, but I'll go check it. And I might raise an issue with the Chromium team to make sure it's sorted. And the nice thing is that like once it's fixed in Chromium, if you're using Edge or Google Chrome, you should be able to access that feature quickly.