 Hi. Welcome to Visual Studio Toolbox. I'm your host, Robert Green, and joining me is Omer Raviv. Hey, Omer. Congratulations again on the perfect pronunciation. Thank you. Omer is with code value, makers of the absolutely fantastic extension, AusCode, and we have an episode coming up in a few weeks where we're going to review a bunch of the features in AusCode. But before we do that, Omer is here to talk about writing high-quality extensions. Exactly. For Visual Studio. So, if you are a user of extensions, and we're going to do this in two parts, because it's about an hour's worth of content, we're going to split it up into two 30-minute sessions. If you are just a user of extensions, this could be interesting to get a sense of how they're written, and it's always good to understand how your tools are written. If you're actually writing extensions, then these two episodes are definitely for you, because Omer is going to share with us a lot of tips and tricks and best practices on how to write high-quality extensions in Visual Studio. Absolutely. A lot of this advice is, I think, good for any.NET program. There are a lot of particularly extreme things about Visual Studio, in that it's a huge ecosystem. There are all these other extensions we have to share the same process with. There's a hugely complex API we need to understand. So, there's a huge challenge in writing Visual Studio extensions, but there's also a huge amount of possibility for what's out there. So, I think becoming a Visual Studio extension offer has changed me as a developer, because I really realized after the first year or so that there's really nothing you can't do. Like you could do, if you really set your mind to it, the API of Visual Studio allows you to basically do whatever you want. But with great power comes great responsibility. So, today, this is going to be a bit of an eclectic session where we're going to look at a bunch of different technologies and techniques and fantastic open-source libraries that are going to help us make sure that our extensions are high, that we bake quality in. I've broken this down into four chapters, testable, stable, performant, and memory efficient. And we'll start by talking about my favorite topic, which is testability. But before we do that, I'll introduce myself briefly. I'm a senior consultant at our company Code Value based out of Israel. And I'm pretty big in the open-source space. And today, we'll also be talking about some open-source libraries that the Osco team has recently published that help specifically in writing Visual Studio extensions. I'm also available on Twitter, but most importantly, we have a fantastic community for Visual Studio extension offers up on GitHub. I'm extremely active there. So, if you have any questions at all about the things I'm showing today, go ahead and reach out on the Extend-VS chat room where both other extension offers can see it and the PMs on the Visual Studio extensibility team can see it and we can chat and collaborate and so forth. Why am I so obsessed about testability and debugging and all that? That's what I thought I should explain to you first. And I think it has to do with my personal journey as a developer. I started out in the industry of medical imaging where I was working on the software that the doctors and the technicians all worked to operate the CT scanner. And every few months, they used to take us out to the hospital to actually see how our code is being put to use. And that's where I got real scared real fast because I was just sitting there thinking to myself, well, I really hope this thing doesn't crash because if it does and they have to deliver a double dosage of radiation to that patient over there, they're probably going to want a better answer than just now reference was not set to instant variable, right? So, I was very scared. And that's when I sort of realized that we need better debuggers and we also need better tests. And some people think that that's a real weird thing about me in that I describe myself as somebody who's both extremely passionate about debuggers and extremely passionate about test-driven development because a lot of the test-driven folks say, well, you're not supposed to use a debugger ever. But in my opinion, when you debug, especially if you're debugging extremely complex things like Visual Studio Extensions, you absolutely need to have a good testing suite so that the time you do spend in the debugger is quality time, right? Every single minute you spend in the debugger is a minute you squander away, right? Right. But there's two ways to use a debugger. One is completely and utterly wasting your time and the other one is having quality time with the debugger, right? And it's really important distinction to make, in my opinion. Because... I love that. Yeah, because wasting your time when you're using the debugger and I've spent like, I think probably months of time debugging a Visual Studio Extensions, Visual Studio Extensions are extremely hard to figure out when things go wrong. It's not knowing what you're doing, not knowing what your hypothesis is. But having quality time with the debugger is applying the scientific method religiously and then simplifying the scenario to the point where you can actually write a test. Again, because when you have a test around a bug, you've essentially got the bug locked in a room that has no doors, no windows, it's got nowhere to escape, you're gonna catch it. But that's good advice, but it's extremely, extremely easy to say it, extremely, extremely hard to actually do it. Right. Now, this brings up the question of, well, you know, nowadays people tell us you should be practicing test-driven, you go to any software conference, everybody's talking about how everybody should be doing test-driven development. But we had an episode with Phil Jepixi talking about that a little while ago. Yeah, and it's good advice, but then it sort of makes you think. So if everybody's telling you you should be doing test-driven development and unit testing and all that, how come when you actually go out to the real world and you look at real companies, you see that probably less than 10% of the people are doing unit testing and those people are probably doing it for less than 10% of their time. And why is that? And I think the answer to that question is complex. Part of that is because writing tests are is extremely time consuming and something that takes a lot of skill and mentoring. But the other part that we shouldn't ignore is that some people have very valid concerns about whether the time they invest into writing their unit tests or integration tests actually have a good ROI, right? We've all had that experience where we write a test and then something changes and all the tests fail, but they're not actually failing because something's wrong, but just because they're flaky and all that. It's also something that you're going to do. Oh, next project. Well, I'm right in the middle of this one. I don't have time. Next time I'll do it, right? It's like, oh, I'm eating now. It's the holiday seasons are coming up. But once they're over, I'll get to the gym. Of course, of course. So what we're going to do right now, I'm going to teach you a bunch of different techniques to hopefully convince you that writing tests, you can even test the untestable, right? Visual Studio Extensions are extremely hard to test, but I'm going to teach you how you can test them in a very efficient way and have absolutely terrific ROI out of that investment. But I'm going to do it in a bit of a roundabout way where I'm going to talk to you about two fabulous open source libraries, BDD-Phi and Visual Studio Test Hosts. And then I'll explain to you how we combine both of those open source libraries into what we think is an absolutely killer way of writing tests for Visual Studio Extensions. So first let's talk about BDD-Phi. BDD-Phi is a way to just write better tests. The concept behind it is that we need to improve the readability of tests. If you think about a typical unit test, there's a lot of boilerplate going on early. There you have to set up all of these dependencies, all of these mocks. You have to put a whole bunch of stuff into motion. It takes a lot of time and that's also part of the reason why people usually treat test code as inferior. They don't put enough tender loving care into their test code as they do into their real code. And this problem is exaggerated 10 times over when we're talking about Visual Studio Extensions because everything you try to do has a lot of boilerplate. You have to access all these different COM objects. There's all these different layers of the Visual Studio SDK, the COM layer, the MEF layer, et cetera, et cetera. So I'm a really big believer in integration tests because for things that are tricky to test, writing a lot of unit tests will oftentimes not be the correct solution. And when we started writing OS code to be completely honest, for the first half a year or so we had absolutely zero tests. And the reason that happened is because initially we found that our biggest problems are not the small parts. It's how they all work together, right? We have stuff that's talking into Roslin. We have stuff that's talking to the Visual Studio Debugger API. There's stuff that's talking to the Visual Studio Editor APIs. And the stuff that's why should we invest into testing if we can't actually test the things that's most likely to break, which is the integration points between those different components. And we decided to automate all those things. And for that we used several things. In order to be successful at testing, we need to write tests at the right level of granularity. So Mike Cohen, who's a big testing guy and a big testing offer, has this idea of the test pyramid where he says you should have a lot of unit tests which are fast running, very compartmentalized, just one chunk of execution that is under test. And then you're gonna have a smaller but still a significant layer of system or subcutaneous tests. What that means is that tests that do run inside the system, but they're actually talking to the services themselves, telling them what to do, rather than a UI test, which is the thing you should have the least of. You should have some, but only for testing the happy path. So the UI test is where we actually, simulate writing all those, pressing all those keys on the keyboard and moving the mouse and so forth. And that's extremely hard to do in Visual Studio. Now I'm gonna talk to you about some open source libraries that you could do for each one of these. So if you want to write unit tests, the best GitHub repository to look into is VSVim, that's a Visual Studio extension by Jared Parsons from Microsoft, which allows you to use Vim inside of Visual Studio. And that's the best way to do unit tests on Visual Studio extensions. Up one level is on talking about subcutaneous tests or system tests. That's what we're gonna talk about today, which is the OS code open source sample called Visual Studio integration test samples. And if you really want to do UI based testing, the best thing you should look into is Python tools for Visual Studio, which is open source up on GitHub. And they have a huge source base of doing UI based level testing of Visual Studio extensions. And my personal little open source contribution is that I extracted that stuff out of Python tools for Visual Studio into the open source repo, which is called test utilities, which you can click on these links from the slide deck and check them out. But today we'll be talking more about writing system tests for Visual Studio. So what is this BDD thing? BDD is basically a progression from TDD, where we can write tests using natural language. We say, given some initial context, when some event occurs. So before, what are these acronyms? TDD is test driven development. Yeah. And BDD is behavior driven development. Okay, and then what's ATDD? Acceptant test driven development. So ATDD is basically the idea that TDDs are fine and great, but we should focus more on actually phrasing the tests in terms of what users actually expect to get out of the software. Not the nitty gritty of how this class is written, but more of, I'm a user, this is my story. This is what I wanna get out of the system. When I do this, that should happen. When I do that, the other thing should happen, et cetera, et cetera. Okay. And BDD is about writing the test in a language that is just like the spec of the software, right? So usually most companies that I know keep their spec in this huge Word document. And a huge Word document is probably, in my opinion, the worst way you can possibly track of your requirements because what will happen invariably in about 100% of the cases is that the Word document will go stale and it no longer reflects what's actually going on in the software. And Mr. Joe developer here says to the product people, can I see the requirement doc please? And they just say, there you go and here's this 100 page Word document and you go on and do your job. So BDD offers us a solution where we can actually express the test mechanics using natural language and inside of C-Sharp. So what you need to do, you just go into your Nougat, you say install package test stack BDD-fi. It's written by these four extremely talented, extremely smart people who each have their own very interesting blogs about testing. And BDD-fi is something you could use with any testing framework. It does not require its own test runner. It's really simple to use. And let's see, let's finally jump into Visual Studio and see what the real benefit is here. Yeah, because I'm gonna be interested in how you get from, how does BDD relate to TDD and how do you get from the story to the test to the code? Absolutely. So there's two different ways to write a BDD-fi test one of them is a class pair test style. So here we have a test where we're saying the account has insufficient funds, we're testing out an ATM, we have a card, we have an ATM. And basically what you see in this code. Can you bump up the font a little bit? Absolutely. It's really small. Let's do that. So what we see here is that every method is a test step. And what we do is that the way you can read this code linearly from the top of the file to the bottom of the file, that's how the test is actually gonna execute. So first this method is gonna execute, then that method is gonna execute, et cetera, et cetera. So the names of the method are very human readable and they go given when then. So given the account balances $10 and you notice this attribute that's giving us a human readable way of reading this, but you can actually humanize it for us. And given the card is valid and given the machine contains enough money and we see that the code that correlates to doing that is actually inside the method, when the account holder requests $20, then the ATM should not dispense any money. And we got this assertion and the ATM should say there are insufficient funds and the account balance should be 10 and the card should be returned and so on and so forth. Now when we actually run this test something very magical is gonna happen. So we see that this test passed. When I click on output, you see that right here, the output of the test, it actually basically just reads out like a story. What BDDFI does behind the scenes is it takes those names of those methods that we saw and humanizes them into this very readable thing that we read here. So what we get out of that is that when the test actually fails, which is when we need help. So I'm gonna go ahead and change this instead of account balance should be 10. I'm gonna change it to some mistake. I'm gonna change it to 25 and run the test again to do. And now I'm gonna go see that this test fail. I'm gonna go click on output. Now look what happens. It tells me which steps passed in the natural language and it tells me which test failed exactly. So I can see actually match the test failure to the actual behavior that the user expects thing to get out of the system. Does that make sense? Yep, so just to review, you are using a particular framework, a particular extension, how are you getting this? So you don't have, you can choose whatever unit test runner you want. You can use every Sharper, you can use the Native Visual Studio, you can use any whatsoever, and you can use any unit test framework that you want. You can use X unit, N unit, MS test, whatever you like. This is all a library that you're using. Another cool library that I'm using here. So you notice that if we go to the test output, this assertion failure is not that useful in that it's telling me expected 10 actual 25, but it doesn't tell me what was the thing that was 10, but it was 25 but supposed to be 10. That's not all that useful. So another open source library that I really recommend is Shouldly. So instead of having this assertion look like this, I'm gonna change this to this. So, and we're gonna again add a mistake here. So it says card.accountbalance should be 25 and should be is an extension method that I get out immediately once I install the Shouldly Nuget package. So now when I run this test again, it failed. I'm gonna go and see the output. And now we see that the output is account balance should card.accountbalance should be 25 but was 10. And that's extremely useful because I don't need to really sweat to actually see why the test failed. All of this advice that I'm giving here is just general unit testing advice that's gonna come in real handly once I have a proper integration testing suite set up for my Visual Studio Extensions. Visual Studio Extensions tie into this picture. First, we said we're gonna write a Visual Studio Extension. Then because of my background in the medical imaging industry, I said I'm gonna write tests for my Visual Studio Extension. And then I found out about Microsoft Visual Studio Test Host. What is Microsoft Visual Studio Test Host? So this is something that used to be part of the Visual Studio SDK but is now part of the open source Git repo that's maintained by some of the guys who work on Python tools for Visual Studio which is called Microsoft Visual Studio Test Host. So what does Visual Studio Test Host does? It does something that's actually very, very, very crazy. So I'm gonna go ahead here and look at this test I have over here and you'll see that, so I'm gonna open this up. You'll see that I've got a start debugging test and this is a regular MS test test. Only it's got this really special attribute right here that says host type is VS IDE. Now I'm gonna actually, so you see that this is the very first Visual Studio test that I ever wrote and it's real simple. All it does is it takes this program, tiny program that just has a call to debug or break and it says given this program when we run the program then we break on line five. That's really easy, right? And the BDD-5 is this part right here where it's actually a way, a different syntax that we have in BDD-5. So yeah, I sort of skipped over that part. So as we said before, there are two ways of testing with BDD-5. The one part is the test per class syntax like we saw before where every step of my test is actually a method that's executed in linear fashion. The other way we can do it is we can use the fluent syntax of BDD-5 where I can compose these test steps using the given when and extension methods and pass the actual steps as a lambda expression. And this is really nice because it's a really good way of composing and reusing the same steps in different tests. BDD-5 itself is a NuGet package you added. Yep. Which one can you show us? So it's right here. Let's go back and make sure that it's not too lost. So you install it by typing in install package test stack.bdd-5. All right. All right. So now let's get back to our test. So I installed BDD-5 and I also have a Visual Studio test host attribute. So let's go back. There. All right. We've got a lot of Visual Studios here. We don't need this one anymore. This one. There you go. There it is. Oh, there it is. Right. So I'm gonna run this test. This test we just saw that's using the BDD-5 fluent syntax. And when I hit this test, you're gonna watch something extremely magical happen. What is this doing? It's opening up a brand new instance of Visual Studio. So that's the magic behind the test host attribute of MS test. So what happened just now is that the test ran inside of a brand new instance of Visual Studio that came up. So when you think about it, usually when we're talking about unit tests, the test runner process, it's just a boring old CLR process that comes up, loads up all our test assemblies and runs the test. But with the Visual Studio test host attribute, it actually brings up an instance of the Visual Studio IDE and uses that as the test runner. What that means is that we can write integration tests that talk to whatever part of the Visual Studio API we want and it will just work because the test is actually running inside of Visual Studio. And when I realized that I can do that, I was shocked. This whole notion of subcutaneous testing, which means the test is actually running under the skin of the program, means that you can test anything you want and there's no limits to what you can test. And you can write a test that actually captures the entire journey that the user is going through from the beginning to the end. Okay, so that was a Visual Studio test host and all of the code that I'm showing you today is up on GitHub like we saw before and you can actually go in and check it out. It's up on that oscode slash VS integration test samples GitHub repository that you can look at. Alrighty, let's continue on. Okay, we talked about making extensions testable. Now let's talk about making them stable. You've all probably seen this really, really nasty message box, sometimes a pop-up in Visual Studio, which is not all that useful. I call it the activity log XML message box of pain and suffering. It's painful for the user but it's actually equally painful for the Visual Studio extension offer because when the user sees this dialog box, it means two things. It means first off that the user is having a really bad experience. And it also means that actually exceptions that get up to the level where this dialog appears are actually, they don't go to the application, domains, unhandled exception callback. There's no callback that you can easily get to get a notification as an extension offer that this happened. There is an interface for it, but the interface doesn't tell you if that's a problem that happened in your extension or another extension. And that makes it really tricky to deal with. Now, luckily, the OSCO team has open sourced another thing that we like to use which is the OSCO Visual Studio extension analyzers. So what these analyzers do is they look up all the places in your code where an unhandled exception might cause a very bad thing to happen. And there's a huge variety of very bad things that could happen if your Visual Studio extension causes an exception to be thrown that's not handled. One is that this dialog box, also Visual Studio could crash. But the worst thing that could happen is that if one of your MEPH components, because Visual Studio extensions are usually written with MEPH, if one of your MEPH components throws an exception in the constructor, that means that probably what will happen is that your entire Visual Studio extension just won't fail to load and the user won't able to even use it and you wouldn't even know what happened. So the good thing, the best thing to do is to use an exception monitoring tool. So there's application insights, there's Raygun, which is the one I personally use, there's Exceptionless. There's lots of really great tools on the market nowadays for just sending them a notification whenever an exception happened and then later on you can actually track it and see what's what. So, I'm gonna jump in here, in here, in here, first one's the charm. And this is a tiny Visual Studio extension that I have and I have a MEPH component in my Visual Studio extension. So it's got the MEPH export attribute. It's using an important constructor. And so the first phase, and I also have a method that's actually being called whenever a Visual Studio text view is opened up. So I'm exporting myself as an IWPF text view creation listener. That's just a way for me to tell Visual Studio, hey, whenever somebody opens up a text editor, I wanna know about it and then Visual Studio calls this method right here. That means that if I were to throw an unhandled exception inside of this method right here, then the user will get that annoying message box that we saw before. And I won't ever even know that the bad thing happened. Most people will probably just uninstall my extension and say, I'm never gonna use that again. So because I have the Auscode Rosin Analyzer installed, I can just add a try catch here. Auscode will detect that I really need to have a try catch here. So it adds the try catch for me. And over here, we have the same problem. We're actually a more severe problem because again, if an unhandled exception happens in this line of code, my entire extension might fail to load. So I'm gonna go ahead and add the important constructor thing here. And there we go. It added try catch. It goes to this method down here and I can configure the name of this method. But the idea here, what you wanna do is you wanna log the error. You probably don't wanna show an error notification UI just because you might get a lot of these exceptions all at once. And what you would put in here is this code to report the exception into Raygun or application insights or exception list or whatever it is you like to use. And I highly, highly, highly recommend that you pick and choose a good exception monitoring tool and you know it, use it, love it. What you get out of that is that you can assign an unhandled exception to someone on your team. That person can access the call stack, see exactly what happened. And one of the nicest things is that you can actually say an exception has been fixed in a particular version of our extension. And then if the exception happens again, it will only tell you about it if it's a regression, right? If the exception came back after you thought that you already fixed it. Yeah, okay. Okay, really, really good stuff. Okay, the other really unfortunate thing that could happen if you as a Visual Studio extension offer neglect to have try catch where you should is that Visual Studio might crash. And in Visual Studio 2017 update three, Microsoft added even a notification to actually let users know that, okay, that crash that you just happened. It's probably those guys' fault. Right. And that's why dealing with crashes is extremely important. We need to know why crash happened and fixed immediately. So one bit of advice is of course, install that NuGet package that I showed you before because that way you instead of Visual Studio crashing, nothing bad will happen because there's a try catch and you'll get that notification into your exception monitoring tool. The other thing that's really important is that you should, every machine that you have, both you as a developer of a Visual Studio extension and your QA team and any customers that are complaining of crashes, they should just install this red key. And what that red key does is says, hey, whenever devend.exe crashes, we need to create a dump on the folder C.dumps. That's really, really important thing to have. It's just one click to install it. And then whenever your Visual Studio crashes while your dog food in your extension, you can go ahead and open up C.dumps, open up that dump in Visual Studio debugger and see why that happened, make sure it never happens again. All right, so that wraps up our part one where we talked about testable and stable Visual Studio extensions. In the next episode, we're gonna talk about how we're gonna make our Visual Studio extensions both performant and memory efficient. Fantastic, all right. We said this is gonna be about an hour. It's been 30 minutes. Perfect. Woo! So in episode two of this, we'll continue the discussion.