 Hi everyone, hope you're having a great day. I'm having honestly a fantastic day. The weather outside is absolutely terrific. Unfortunately, you can't see a thing of that since I had to put my blinds down as you maybe want to see me as well. But let's get started. I would like to present to you today how to test Visual Studio Code Extensions. And let's start out with me. My name is Dan. I'm a software developer working at SUSE as part of the developer engagement program. So my task is mostly to build tools for other developers. I'm also a package maintainer for some open SUSE packages. I also maintain a bunch of packages in Fedora. And there I'm also a member of the I3 special interest group where we've created the new I3 spin for Fedora 34. One of my big passions is testing. And that's also one of the reasons why I have invested quite a lot of work into testing Visual Studio Code Extensions and why I'm also giving this talk. In case you want to stop me on social media, you can find a few handles further below where I occasionally post stuff about technology. But let's take a look what we'll do today. And as the title of the talk suggests, it's about testing extensions for Visual Studio Code. So this is really from a developer perspective. And the main agenda is, well, how to test extensions. And I'll be covering the three big approaches that there are unit testing, manual testing and integration testing. Well, and at the end we'll have a Q&A session and depending on how much time there's left, I can maybe also show you a thing or two in case there's an interest. So first let's get started with Visual Studio Code. I mean, I guess you probably know Visual Studio Code. It's a cross-platform IDE developed by Microsoft. It's built with web technologies. So this whole thing is built in TypeScript, which is a super set of Java script. It's extremely popular. So it started to be, I think it became a thing in 2015 and it breached 50% market share according to a Stack Overflow Developer Survey in 2019 and it grew even more. So this is an insanely popular editor. It has a extremely good initial user experience which also explains why it's so popular and it has a very rich extension ecosystem and especially a very well-documented extension API. So in case you have a feel like creating an extension yourself, go for it. This is really, really simple. And also a bit for my background, I have spent a good chunk of last year developing the Open Build Service Connector, which is an extension for Visual Studio Code to that bridges Visual Studio Code itself to the Open Build Service, which is the, in case you don't know it, it's essentially the combination of Pagua plus Koji in Fedora. So, but now let's get started with the actual topic at hand, testing extensions. And so in my impression, testing really just the extensions for VS Code, this is pretty tough and why is that tough? Well, the main reason is you are testing a UI and UIs have unfortunately the disadvantage that these things carry a lot of state. So imagine your visuals, your essentially any UI, but now take VS Code. If you have something open, all of the UI elements that are present, this is a certain state that's present. And all of the state influences how your extension interacts with that and what the outcome of that is. So this means if you want to actually write meaningful tests you will have to pre-create all the important state yourself, feed it into your extension, get the state that your extension produces out of that and check that it actually matches. And that's pretty hard because your VS Code editor window, there's a ton of information. You have to extract all the important information. You have to find the right one. You have to ensure that you actually got the right one and so on and so on. So this is not a VS Code specific problem. This is a UI testing specific. This is a general problem when testing user interfaces and graphically user interfaces specifically. Also, if you have created an extension, then usually the important part is that you have certain workflows. Imagine a Git extension or subversion or extension that connects to Kubernetes or to OpenShift or whatnot. You want a workflow to be working, to be functional. So let's say I want to be able to commit things. And in my impression in recent years, it's been very popular to write unit tests and to rely on unit tests, but unit tests are really only good for testing individual chunks of code. They are not good at testing really workflows because a workflow is a combination of a whole ton of things, but a unit test tests just one tiny, teeny tiny bit and you're actually interested in a big picture. Well, and last thing here is what you... So what you test is not necessarily what you see because if you start out pre-creating this UI state for your tests, you can easily create state that would never appear like that in practice. And also, if you then actually want to check the result, it all, it effectively means anyway that you'll be running your extension, you take a look at the output and then you check what comes out of it and then you test that. So effectively you still have to take a look at the actual stuff that happens. So now let's get started with actually testing and first, a general notice. Usually extensions have to rely on some kind of external service. So whether this be, for instance, external libraries. So imagine that you need to store credentials somewhere. These need to be stored in the operating systems key ring. That's how it's usually done from VS code. And depending on how you're exactly testing, you have to mock these out. So I'd say for unit tests, mock them out. I'll cover that later for integration tests. You can use stuff like LD preload and write fake libraries. I've done that for the open build service connector. I have a fake lip secret that just reads credentials from a file. And so it doesn't mess with the developers machine. If you have external services, so imagine Kubernetes or some client provider, take a look whether upstream has some sort of staging environment. Or if preferably a development environment. So usually upstream projects have development environments that can be just spun up for testing. Use these to test against them. You want to be testing against the real thing, especially for integration tests, test against the real service. Otherwise the service will change something or something will behave slightly differently than what you mocked out. And your tests will be green, your actual service, your actual extension will fail. Good, let's get to unit testing. So unit testing means you have one functionality. So you test one teeny tiny chunk of code and you test one specific functionality of this. So usually it means you take for instance a function, you take a class and you want to check, okay, does this function fulfill its purpose? And as I already ranted quite extensively, anything that touches the UI will need extensive setup and tear down and verification will be a pain in the backside. So if you want to really check that certain elements work the way they are displayed the way you want them, it will be a lot of work and I would recommend not to do that. If you have external services and need to communicate with them, my recommendation is for unit tests, mock them out. So if you have for instance credentials, use some mocking library for that or do dependency injection since your unit tests should be relatively fast and if you now spin up testing environments it will take a lot of time. If you want to get started with unit testing this is fairly simple. So the upstream documentation has an example if you use the official generator for a VS Code extension then that will set it up yourself and how these unit tests, these use the module as called VS Code tests and what that effectively does is it spins up a temporary VS Code instance, loads your extension into that and then it executes all your unit tests that you defined. So you will have a real VS Code instance running. You can do all kinds of stuff that the VS Code API exposes to you and with that your unit tests run essentially in the real thing. This is fairly straightforward. So it can be sometimes a bit tricky to get it run on a CI as you have to have to have an X server running but again, the upstream documentation is fairly good in this example. A tiny catch is if you want to extract test coverage the upstream documentation does not cover that and it's also a bit nasty to get that running but fortunately, Conor Pete has a really good example for that and it's linked in the slides. So if you want to use this code snippet yourself go ahead and just steal it. I did the same for my extension. Good. So I'd now like to go over some specific parts of unit testing that I found interesting, particular or surprising maybe. So one thing is extension settings. Depending on your extension you might have to write or read settings because you have hundreds of them in that case, this is fairly straightforward since you have a real BS code instance running. So you can just read, write, modify them to your liking since you also have a testing environment that doesn't mess with your real settings, that's nice. The only thing that you should really keep in mind is clean up after yourself. So if you use some post test hook and remove all your temporary settings that you made since subsequent tests could then otherwise fail, that's essentially it. Events, so if you have worked with the VS Code API then you'll see that there's, then you'll have seen events and an event is essentially, as you can see in this. So events are typically used in Visual Studio Code to signal to event listeners that something happened. For instance, a user selected something, user clicked somewhere, some property changed and how these run. So here you have the on the change value event and how this works is you pass a function into that and every time this event happens that function gets executed. This also can be used, so you should test these if they include a whole bunch of business logic. The only catch with them is the Visual Studio Code itself will not await promises. So if you pass an asynchronous function like in this example here, Visual Studio Code will just execute it and not wait for the promise to resolve. So my recommendation is implement some fake events yourself. This is really straightforward. If you want to take a look how this is done, steal my implementation from the open build service connector. It will be linked on one of the last slides. Disposables, so disposables are essentially a workaround for JavaScript being in garbage collected language and not having finalizers and these are destructors. So a whole bunch of the VS Code API returns a so-called disposable that's just an object that has a dispose function and this unregisters it from. So for example, if you have registered a command then the function that registers a command gives you a disposable that will unregister it and the idea here is you can't register a command twice. So just keep in mind everything that returns a disposable should be actually disposed of after a single test and for that most test runners have some kind of after or after each hook that runs after a test or after a test suit or after each test. And so just keep in mind to get rid of them. UI elements, as I said now twice, don't check them preferably and if you really want to check them really check only the interesting parts. So don't check the whole data structure, your test will otherwise fail on every VS Code update. You don't wanna do that. So really just check for the part that you really, really care about. For example, the icon. Check that you actually use the correct icon under a certain condition and keep this part as small as possible. And a general recommendation here is structure your code so that your main business logic can be very well tested and the UI part is as small as possible since otherwise you're going to have a very, very bad time with QA it's going to be very annoying. That should be it for unit testing. Let's go to manual testing. And now you might say, wait, manual testing you mean like doing it yourself? Yes, I mean exactly doing that. Do it yourself. Which might sound like a very weird suggestion given today's world where we try to automate essentially everything and automate ourselves out of our job. And of course that's not going to happen but so when does this make sense? Manual testing really makes sense if you have a kind of a one shot extension so you've written it once and you expect your future changes to be really, really small which is usually not going to happen but if you anticipate that you could go for manual testing or if you have a really complex environment that's essentially impossible to create automatically or you have to wait for certain conditions that take ages and doing that automatically is very, very hard. In that case, you can go for manual testing and my recommendation here is make yourself a test plan. So make a checklist, write down what should be done exactly in what steps, what are your expectations make yourself a tick box that you tick and before every release you go through the whole thing. If you have, you can even use some test management system like key VTCMS, if you are familiar with that you can plug that into this. But yeah, not necessarily my recommendation. And so let's go for integration testing since manual testing is effectively manual integration testing and it just means let a machine do it. So you test the main workflows of your extension and you effectively let something click through VS code and do all the stuff and verify that everything that comes out is actually what you expected. So for that, I found a very nice module that's called VS code extension tester developed mainly by Jan Richter from Red Hat. And this leverages the simple fact that Visual Studio Code is built on web technologies. The whole thing runs in Electron. That's a headless Chrome browser. So you can use all the cool stuff all the cool testing libraries that are out there to test web pages. You can use these to test Visual Studio Code. And one of these is Selenium WebDriver. And what VS code extension tester does is it wraps all the calls to the actual DOM. So to the actual HTML stuff that's created by Visual Studio Code and gives you a convenient API that looks like this. So you can say, hey, give me a new text editor then it can open a certain file, set the text and so on and so on. And this works quite well in practice. The API is also pretty nice to use. So what should you test with that? My recommendation would be find your main workflows and try to identify really the main and most interesting workflows that you really want to care about. Go here for an 80-20 rule. So try to cover with 20% of the work, 80% of your code. Since your test runs, they will take a while to run. And don't test corner cases or some minor regressions that are hard to reach. Since that's, I don't think that's something that integration tests should do. That's more something for unit tests. How should you test? So upstream, the VS Code extension tester uses Mocha. I would suggest you use the same as a test runner in this case. Mocha also supports root hooks. So these are essentially hooks that run before all tests and after all tests and use these to really set up your development environment. If you need, so keep in mind, you can override environment variables like the home directory, like LD library path, LD preload. You can inject all kinds of external symbols. If you need passwords from Libsecret, then use a custom Libsecret. If you need another library, use another library. If you need to sub-process into Git, use a fake Git and so on, so that you have more control. And then run each of your individual steps as individuals, so-called it steps. That's just how it's called in Mocha and to be done with that. So downside is you won't be able to run them in parallel, but you can't do that anyway. Good, so let's take a look at few of the catches. The integration tests, especially with VS Code extension tester, they are relatively slow and resource demanding. So if your machine is under load, then sometimes you'll get random timeouts, which is annoying, especially if you run your tests on a CI, because your beefy development machine is not comparable to the CI worker that you share with 50 other people. You should also definitely avoid explicitly sleeping in your tests. Unfortunately, the upstream examples include that in a few places. So that's, you should avoid that because again, that will fail on the CI. Then also another thing is certain elements in VS Code are invisible by default. So for instance, certain buttons only appear if you hover over another element with the mouse, but that's a solvable problem. It's just something that you should keep in mind. And the last thing is you will not get test coverage out of that, that is more or less effectively impossible. On the other hand, you shouldn't, in my opinion, rely on test coverage from integration tests anyway. So good. And since I'm already running pretty low on time, here are a few links that you can find in the original slides. The slide source code is here on GitHub, if you want to take a look. The obligatory legal slide with who owns what the copyright. And now I would like to thank you for your attention and I'm open for questions. Thank you Dan. We have one question for you from Brian. Hi Dan, thank you for your presentation. I was wondering if you ever tried to mock certain parts of the VS Code API and how would you go about it? Yeah, so I have done that. It really depends on which parts of the VS Code API you want to mock out. Yeah, this really depends on which parts of the API you are thinking about. So maybe we have a good idea to perhaps discuss this with Brian in Discord maybe in more detail because yeah, like you said, it depends. So Dan told us that he will be available in Discord right after the talk. I'll be there. And that's it for the questions. So I think we can wrap it up. Thank you, Dan. It was a great presentation. Thank you. Thanks for having me. And thank you for attending guys. And like we said, Dan will be into Discord server after this talk so you can reach out to him there. Thank you. Thanks, bye.