 Hi, welcome to Visual Studio Toolbox. I'm your host Robert Green and joining me today is Kendra Havens. Hey Kendra. Hello. Welcome back on the show. Thanks so much for having me. And we're going to talk about unit testing. Yeah. Which is not the most exciting subject maybe people are thinking, but that's why you're on the show to add it some excitement. It's just because they don't know about all these cool features. That's right. We're about cramming in. So this is why we're here to talk about it. Unit testing is a tricky subject because it's like everybody knows you should do it, but what are the tools? How do you write unit tests, et cetera, et cetera? We're going to do a series on that later on in the summer. But in the meantime, yes, we'll build your pixie back on to do a series on unit testing. Excellent, that'll be really good. In the meantime, you're going to show us great tooling in Visual Studio. Yeah, so the purpose of unit testing is actually one of the first things they want to talk about. And I can talk about our Roslyn team example. Then I'll go over learning resources that people tend to miss a lot. And just in general, this whole episode will be about announcements that are really easy to miss during conferences and all of our big releases and in blog posts and stuff. So I'm just going to be highlighting all of the ones that have to do a testing and more of a focused. Okay, you know what I'm saying. So learning resources, latest testing improvements, I'll demo all of those. And then more of our Visual Studio testing tools that maybe we haven't talked about in a while that I'd love to bring up again. Cool. Cool, so purpose of unit testing. I love using the Roslyn example. I work on the Roslyn team, which is the C-Sharp and Visual Basic Compiler. So as many people know, we went open source and we architected the compiler in a way to be extendable. So we wanted other community contributions. And as a result of that, last year 60% of .NET contributions were from open source. The only way our team could possibly keep up with that many PRs is through our tests. So we have 86,000 tests on Roslyn and learning how changes were affected in a pull request. Looking at your test is a great way to do that. So it helps the reviewing experience a ton. So that's a really big example. That is our bread and butter on the Roslyn team is actually having tests. That sounds like a awful high number of tests, is it? It is. But we got millions of people who use it and thousands of scenarios. It's kind of an important feature of the product, C-Sharp. It's definitely paid off with our team having those. It's the only way we could have kept up with those changes. Right, so here are the learning resources that I think a lot of people might have missed. I get a lot of questions about like, hey, how do you test legacy code? And I'm like, we've actually done an episode on that on Microsoft Virtual Academy. So yeah, just go to Microsoft Virtual Academy and you can look up our sessions on testing legacy code and test driven development. That's also a really nice one. A lot of people also miss that MS test and VS test are open source on GitHub. So you can look at docs and issues and they're very informative, especially if you're running into a bug. A solution actually might be on the repo or at least the discussion to know what's happening with it. So that's kind of a pivot like you could either go to developer community and just file the bug. But if you're actually trying to like find the workaround or see if we're already engaged with the community on that, there could be issues in those repos too, so that's really helpful. And of course, if you're using .NET test, the .NET CLI is also all open source on GitHub. So that's a third repository to look at. So very latest testing improvements, massive performance improvements. I've showed you a few of them before, but we'll go over to it and just focus on testing. A lot of that's due to real-time test discovery is the new source-based type of discovery that we checked in that no longer lies on assembly base. So it doesn't require a build, pretty cool. There's also been huge test framework improvements in in-unit, ex-unit, and MS test, qualifier for this episode. It's probably going to be a lot of .NET focused stuff because that's my life. So if you want to learn about C++ testing improvements, they actually wrote a blog post not too long ago with some of that. But that's what it is. So I do want to urge Big Push to update your test adapters because as you can see, execution, this is in our labs, test execution of over 10,000 generated tests by Visual Studio Update by Framework has gotten so much faster. So just updating your test adapter is such an easy thing to forget to do because you're always focused on the framework version. Don't often think about the test adapter version, so please do that. It's awesome. And that's your test discovery type of improvement as well. So just updating your test adapter there will also improve it a lot. Cool, so latest test experience improvements, we've got the hierarchy view in the test explorer. Latest being. Latest. Ooh, so the hierarchy view was added in 15.6, that update. Yeah, and that's all I have covered in this slide. So it sourced your tests alphabetically and it organizes your tests by project namespace. So in other words, it's there now, it's not coming in the Fable 15.8? No, it's here. Excellent. It's here. It's been here for months and people love it. Cool. If they know to update to get it, because they still get requests. Yeah, so we're still designing the hierarchy view and we're open for discussion, but enough talk. Enough talk. Geez, that was a lot of talk. Robert didn't want me to use a PowerPoint, but I was like, please. What if people forget what I said, because I talk too fast? Anyway, let's increase the front size there. Okay, so now you can see the Test Explorer over here. That's what I want you to call your attention to. So I have a large test project. If I go ahead and unload that project, you can see the Test Explorer will update automatically. And if I reload it, it can discover 5,000 tests within just a few seconds. Boom, done. Pretty sweet. So I'll go ahead and use the hierarchy to go ahead and navigate. So this is my project namespace and my class. And I can show you how the source-based discovery works. I can go ahead and add that test, and then it will also automatically appear right in the Test Explorer. Pretty sweet. And if I change the name. You don't have to build first? Don't have to build first. Oh, man, that's awesome. Yeah, pretty great. So if I comment it out, it will also then just disappear. That is extremely cool. Because in a large project like this, it can take some time to build. Yeah. And just having that context and having to rebuild and then refind your tests and then run it, big time saver. So what I also wanted to call out was this summary line is also something we added. So this is sort of a global view of how many tests are in your project. And then it says to failed, right at the top of the Test Explorer as well. That was also a pretty popular customer ask. And the last thing I wanted to show you is updating your test adopters. That's really easy to do. Just type in NuGet and open your NuGet package manager for your solution. And go ahead and type in test. Click on your test adopter and move to the latest table is my high recommendation if you want some sweet, sweet performance improvements. OK. OK. So next slide. So I also get asked a lot, how do we make writing tests easier? And I find a lot of people might have missed some of the awesome features that we've checked in. I'm completely automating it. By completely automating it. Right. That shouldn't be too hard. That could be pretty easy, especially if we've actually checked in that feature to Visual Studio. So we always want to kind of bring more attention to it, especially when people ask me for it. So right clicking and generating unit test stubs for .NET Framework projects, that's in all versions of Visual Studio, just really easy test method stub generation. The stub. Yeah. Not the test itself. Yeah, the test itself. How hardly difficult would that be, by the way? Well, that brings me to our next feature, IntelliTest. So this is also only for .NET Framework projects. And IntelliTest is only available in Enterprise. But that will actually generate and look at all of the logic in your method and generate a test for each logic branch. And I can show you that in just a minute. All right, cool. And the next thing I'm going to call out is code coverage, also an enterprise-only feature, and how it can integrate into the VSTS pipeline. Cool. Okay. So we'll escape PowerPoint again. Sorry. Now I feel like I'm using it way too much just because I know you're grinding your teeth. Like this is a cool conversational demo show. What are you doing, Kendra? Geez, sorry. Your word's not mine, I love it. Gosh, Kendra's the worst. Okay. So I can generate or create unit tests, just the stubs really easily, but from the class or the method level. So I already have a home controller test. And that's been in there for a very long time. Yeah, yeah. So I already have a home controller test class. But again, it's easy to forget these things. And we definitely get comments like, how do I discover all of these cool things in Visual Studio? And we're working on it. You just did an episode with Justin with his tip of the day thing, right? That's another good way. Maybe I should get this in there. Okay. So let's say I already generated the test for this class and I just added a method. And I want to now create a stub for that method. I can also go ahead and append a generated test to a file that already exists. So I'll go ahead and do that. And now it has my throw not implemented exception and it just depended tests to my method name. Super simple. You already snore in. It's okay. I'm just thinking that that'd be nice to have an extension or setting that automatically creates the unit test for you when you create the method in the first place. I like that feedback. I'm going to file that on developer. I'm just kidding. Sure. I got you. Okay, so the next thing I want to show you is IntelliTest. Now, let me show you what this project does really quickly. We put an episode on IntelliTest a couple of years ago, but I haven't really heard that much about it since. Exactly. So we want to talk about it more. Yes. Because people ask me for a better test generation. I'm like, have you checked out IntelliTest? And they're like, what's that? I'm like, gosh, I got to go on one of those cool Channel 9 shows. Do you know of any? No. So this is my obscure calculator. So it's taking normal measurements into obscure measurements, like days to fortnight, miles to furlongs, et cetera. So pretty simple methods right now. And so IntelliTest wouldn't generate very interesting tests for these methods. So let's look at a method that has a lot of branches in it. So this is my calculate trust method. This is from the old adage. I only trust them as far as I can throw them. Right. Right. Obviously. Excellent demo code. But it does have a lot of logic branches in it. So for every conditional branch and code, a case analysis performed in IntelliTest. So if statements, assertions, operations that can throw exceptions, all of these are analyzed. This analysis is then used to generate test data for parameterized unit tests for each of your methods. This also gives you really high code coverage. So I'm going to go ahead and right-click and generate IntelliTest. So I'll create my IntelliTest right here. On the X unit test project or on the obscure calculator. Oh, I don't think that's selected. It's going to generate a new test project in general for me. Okay. But does it matter what solution is highlighted when you right-click? No. It matters where in the editor you right-clicked from. Okay. So I'm going to right-click and create IntelliTest. So I want to put this all in a new test project. So I will call it my generated project. And it will go ahead and run that analysis right now. So now you can see that it generated a .cs file. But it is actually, when I run IntelliTest, will generate more .g.cs files. And those are all of our generated classes. So I'll go ahead and do that. So that's going to run the tests that you created through IntelliTest, not your entire suite of both IntelliTest unit tests and other unit tests. Correct. Yeah. I can still use the Test Explorer to do that. But yeah. Right now, definitely just focusing in on the IntelliTest. And you can actually see in the Test Explorer, I now have all of those tests that it just generated for me. You can see they are kind of randomly generated names with exception numbering after aid. So you can see in my solution Explorer that I now have that .g.cs file. And this is where all of those test cases live. So the test data for this is generated every time I run IntelliTest. So I don't want to actually spend my time editing any of these files because they'll be overwritten when I rerun it. So you can rerun it every time you run. Yeah. Okay. Yeah. So. How's the performance of that? Good. I'm not sure how often people will be regenerating everything, but you can rerun them without regenerating any tests. Okay. Sorry. I don't think I made that clear. Yeah. So you'll see this .cs file is sort of the higher node of the test suite. And these all rely on the PECS method, which would be from the Microsoft PECS framework, which was developed by Microsoft Research. Okay. So let's look at my results really quick. Looks like I have some divided by zero exceptions, some null reference exceptions. I have some problems with that code. Luckily, I have a handy-dirty code snippet because I'm a cheater. That's okay, right? You'd be cheating if you've already written it. You don't need to watch me throw out all of that stuff. This is basically just adding a null check and making sure that I'm not going to divide by the projectile person's weight, if it's zero. So now if I rerun those tests, and actually before I rerun, I do want to point out I have nine tests here right now, seven of which are failing. I'll go ahead and rerun them. So IntelliTest is looking over my logic branches again, and since I don't have as many logic branches or cases that it shows would cause a divide by zero error or something like that, you can see all my tests are passing now, but I also only have seven because I changed the conditions that we're generating the test to begin with. Okay. Interesting. Pretty sweet. Yeah, so that's basically IntelliTest. IntelliTest also provides great code coverage, so it's an easy way to kind of check that box. And it would seem also to be a great learning tool to learn how to write unit tests and what should be in unit tests. That's a very good point. Yeah, so it kind of takes maximum values and just use cases and certain exceptions that you would throw and kind of figures out what is most likely to be seen in this scenario. Right. So I'll go ahead and run code coverage in Visual Studio. That was just in Test, Analyze, Code Coverage, all our selected tests. And I'm not doing too great, but let's look at specifically the conversion calculator, which is what I just generated the tests for. And it's at 100% because it looked at the entire method as well as multiple blocks within that method. So the ones that you have low numbers for were not the ones you created with IntelliTest. No. I have to run those over the other Calculate Fortnight for a long time. And the ones that IntelliTest created would have much better coverage than like zero. Yeah. That was 100. Excellent. Which is great. Yeah, so it's a good way to bump up your code coverage score. Very cool. If you want to get code coverage results in VSTS, that's actually really simple to do. Just go to your build definition. So you can go to, whoops, actually I'll just stay. Build and Release. And, oh it gave me that message because I had been clicking in this. And you can hit this code coverage enabled button and then it'll run code coverage for you. Okay. And that'll be a part of your report that you see on that build. So that's not relying on the IntelliTest, right? Or is it? No. The code coverage is a different tool that the IntelliTest would help improve your code coverage score, for sure. Okay. So that's just a... So the point here is that the code coverage you get in Visual Studio in local development, you can also see in VSTS as part of your pipeline. Okay. Cool. Cool. One thing I also want to call out in VSTS is this newly added option. It's running only the impacted tests. So what are impacted tests? VSTS is actually able to look at your commits and what code was actually changed and figure out which tests were affected by those code changes and only run those tests. So if you've only changed like four files and maybe like 30 tests were affected, you don't need to run all 5,000, 86,000 of your tests. So it can save a lot of build time. Okay. Really nice feature. Cool. So speaking of only impacted tests, let's talk about live unit testing. Let's. That's the last feature I want to talk about. Okay. So if you haven't heard, live unit testing listens to your code changes and runs impacted tests in the background as you work. So test results are shown right in the code editor, right beside your code in like line by line feedback. So this is an enterprise only feature. But the recent improvements that we've checked in are better performance and easier configuration. So if you had issues opening your project in the past with live unit testing, it could just work. That was always pretty easy to begin with. Oh, that's fantastic. Yeah. Bye. I had some. I'm sure there were issues people were running into. Yeah. I can share internally. We have like customers in office who have odd project types. Well, don't tell them I said that. But they had configuration issues and live unit testing is just a lot smarter now and it can easily drop in the files that they need. So I asked them to retry it because they hadn't tried it out in a couple months and it just worked. Cool. Which is exactly the experience that we want. So I highly suggest just trying it out again if you ever had any issues in the past. So it now supports MS-Test V1 and .NET Core. We also added a test center notification which just gives more feedback on what live unit testing is doing in the background as well as a skip category. You can include and exclude test projects and we now have test method icon so they pop out a little bit better. Let me show you what all that means though. Enough talk. Okay. So I actually have a very large test project. That's almost 5,000 tests. So I don't want to include it when I start live unit testing right off the bat. I only want to include a certain test project. I can now go to the solution explorer, right click and only include that. So as you can see my large test project isn't running in the background. It didn't try to re-execute or anything super helpful. Another thing I called out earlier is the skip category and there we go. So my chocolate sentiment test, long story of why it's named that is basically using Azure Cognitive Services to analyze the comments that a guest left in our Hotel 360 app. Really negative we'd ask housekeeping to leave a few more chocolates on their pillow to sweeten the deal. But since this is Azure Cognitive Services I didn't want that test running every single time I made a code edit because that's way too often. I could probably rack up the bill. Stomping it out and doing a mock. So you can just say don't run it. Yeah I can just add the test category skip when live unit testing and now I have a little empty beaker instead of the usual beaker test method. Okay so you don't have to go to that and explicitly exclude that you can just say always skip it. Exactly. Cool. Okay so I'll go ahead and show you how live unit testing works really quick. Let's say I have a few failing tests. I can click on the icon in the editor. I can hover over the test that's failing and actually read the error message. If I click on it I can actually debug this test to learn more about why I'm getting a null reference. Though I'm pretty sure it's this one string message that I'm using. So I can go ahead and add, yep message was null. So I'll go ahead and add a null check for that. And I can do that using our handy dnd code fix for it. And of course I can highlight and drag it where I want and I'll do that just kind of clean this up a little bit. Let's only run message.normalize if message is not null and you can see in the background while I was editing live unit testing was re-running those tests when I typed. So now I can see that all things are passing and I feel way better. Cool. And then of course you can turn it off for that method while you're writing it. I could. Yeah. And if you're. I could have added a skip category right here. You can also write you can right click and do it as well. No, the right clicking was for excluding certain projects. So at the method level we have categories so you don't like lose track of them. But for projects since they're much bigger blocks we just have right clicking in the solution explorer. Got it. I want to be able to better navigate what tests are and are not a part of live unit testing maybe with like a new group by in the test explorer but we're working on it. Okay. Yeah. Cool. So those are testing improvements. They shared new resources for learning why we should be testing. We talked about teletest, code coverage and live unit testing. Sweet. And now my life is happy. So let me make it more let me make it difficult. Oh. Okay. So we had we had talked about IntelliCode which is the AI inside Visual Studio. So one could imagine a world in the hopefully not too distant future where as you are writing code Visual Studio is writing your complete set of unit tests for you. That is something I can definitely imagine. I think we'd still want because I think the organization of tests within test projects is really important to people because they probably already have a way of like even when I was right clicking and generating a test method stab it asked me what project do you want us to put that in and what class. I think that aspect is really important and I'm not sure if we're ready to decide how everyone's architecting their test project solutions. So if I have a choice between doing all that myself or having Visual Studio do it for me and then I'm not that crazy about the test high project hierarchy is that's a pretty small price to pay. Oh you think? Yeah. I think. Gosh trying to please everyone that's the that's the real gotcha yeah we're definitely taking a look at how to make test generation a lot easier because that is always a top ask in the testing space. So I'd love to incorporate more IntelliCode like features and actually use machine learning that because IntelliCode is a lot different from IntelliTest in that IntelliCode looked at 2000 open source dot net repositories that were out in the world and IntelliTest is just sort of more of a logical analysis of what cases you might hit and what would be good test data to plug in but if we took both of those and combine them. Like I test you know method per string message right yeah right then and there Visual Studio goes oh well I wonder what happens if the message is null oh I wonder what happens if you know whatever there's a couple things I mean IntelliTest is doing that right yeah in one big pass when I run it I'm just thinking that you would be cool if you had that kind of going on at the same time. I agree. In a future version. Keep working on it stay tuned for Visual Studio 2019 yeah. That's not a promise though. No. All right. Did it sound like a promise? No it didn't. I can't make promises. It didn't. I'm just a PM guys. Cool. So cool stuff. Thanks. So all that stuff is shipping today yeah additional enhancements that we might see in 158, 159. Wow. I didn't prepare any future looking stuff. Yeah we're thinking of we're always thinking about better test explorer design so stay vague about it but yeah there's a lot of asks that we've gotten especially targeting multiple frameworks and how the test explorer handles them today is not ideal. So that's a popular ask so getting a new like group buy that is by multi-TFM or something like that could be really helpful. All things we're thinking about it no promises. Okay. And then what's the state of how this works with .NET framework versus .NET core and what about potentially other languages you might be using? Yeah a top ask is for making and tele-test work for .NET core because I did mention it's only for .NET framework right now. That's definitely something we're thinking about in the 2019 timeframe. Okay. All right. It's on our plate. Okay. Cool. Well I'll keep you posted. I'll have you back when there's something to show for sure. Thanks for doing this. Thanks. Yeah this was fun. Hope you enjoyed that and we will see you next time on Visual Studio Toolbox.