 Good morning and welcome to this week's episode of the Visual Studio Office hours. I'm your host, Matt Christensen coming live to you from my garage in Redmond. And today is a really, really interesting topic because it's one that I am really curious about, but I and I have no idea how any of any answers to the questions. I'm going to ask because this is about being an engineer at Microsoft, and I'm a program manager. I work with a bunch of engineers over the years, but I don't actually know what it's like being an engineer and how they work sort of when you get down to the details of it. And so I'm very excited to get some answers to this and. There's no better person to answer these questions than Tina from the Visual Studio team. Good morning, Tina. Good morning. Can you introduce yourself to the audience please? Yeah, so my name is Tina Shropford. I am a engineering manager at Visual Studio. I've been on the Visual Studio team for about six years and then before that I was in the Windows engineering system team for about seven years. So I've been an individual contributor engineer most of my career, meaning that I don't you know, I don't manage anybody. I'm just an engineer kind of working on on on issues right? Writing code and I've been sort of moved into a manager role about 10 months ago, so I've been a manager since. And I'm part of the Visual Studio platform group which means that we build the APIs for other people to build more features on top of Visual Studio. And specifically my team owns the Visual Studio extensibility charter. So our goal is to improve the experience for both extension developers and consumers. Awesome, so before we dive into it, there's there's something there that's kind of interesting that I think maybe most people are not aware of, which is that the core Visual Studio team that builds the APIs. We don't really build a lot of the features that people are using every day. We're building the platform underneath, which other teams built upon, like the ASP.net team. They build the you know the web stuff or and you know the wind form stuff and the C++ stuff, but we don't really build any features on the platform team. Is that is that full? I think to the most degree, right? Like my team owns like things like service help right? It's an infrastructure for for components to be out of proc in Visual Studio so we can kind of take advantage of other target frameworks or 64 bit processing so there's not really like it's not really any feature that you would think of in Visual Studio that you interact with. But our team also owns the integrated terminal, which is a feature that users interact with and it's part of the platform. But for the most part, the platform team provides APIs. So if you take editor, for example, if you think about when you're typing in the editor and that brings up completion, well, the list of what's in there is provided by a language service team like Rosin, right? And we kind of just provide them with the API so that they could fill out that list and then bring up that completion session so we don't really own any and I think like if you think of like the data in completion, we don't really own any of that. OK, yeah, that's exactly what I thought. So. So that's kind of interesting because we get a lot of feedback and questions and people go to the Visual Studio team and says, hey, my you know this doesn't work in windforms or in whatever component and we then redirect. We have to redirect that to the windforms team because it's not a core Visual Studio thing. So I hope I cleared that confusion up for some people that might have been wondering about that. OK, Tina. So one thing that I've you know, I've seen over the years, you know people join teams, they move to other teams, they move around and there's a lot of new faces in the various teams that joins all the time and I've always been wondering. The new engineers like that joins, let's say your team. Like what is the first thing they do? What is the thing that an engineer do when they when they show up for the first day on your team? How do they get to the code? How do they start writing Visual Studio features or infrastructure pieces? Yeah, so as a new engineer, what you'll probably do first is familiarized with our Azure DevOps. Repo system right? We mainly deal with three types of repos. There's what we called a micro build repo so if you as you can imagine Visual Studio is a huge code base with many, many thousands of lines of code, right? So before we used to have this one central repo where we call the BS repo where every single line of code was checked into that repo. And the problem with that is the build time is insane, right? Like it's impossible to get anything done quickly. So over the past few years, what we've done is we moved a lot of components to be outside of the BS repo. And what we do is we build, we'll build them in a micro build environment, meaning like the DLLs and the XCs that they produce is built in a different pipeline. And then we insert those built assemblies back into the Visual Studio repo and then that gets compiled into the Visual Studio that you would install on your machine. So most of the code these days that people write are probably in micro build repos. So a good example of that would be on my team owns would be like the language server client. So language server protocol is a protocol that was developed by Visual Studio code that provides a single standardized platform for people to provide language smarts across platforms. So we own the Visual Studio client portion of that and that code is in a micro build repo. So it's just like any repo that you would enlist on GitHub, right? You have your own targets, you probably set up your own way of building, but essentially you can just launch a developer command prompt and then you can go MS build your solution and then that just builds your thing. And it's very easy to F5 debug. You just kind of do what you would normally do in a regular solution outside of developing a Visual Studio. And then there's the BS repo that I mentioned before. This is a kind of a special repo where we have this thing called Razzle and Razzle is basically just a customized build environment where we insert our own targets and command prompts with environment variables to sort of customize our build environment. And that's where a lot of our kind of legacy code lives. So for example, all of our native code, right? What you would see like in Devon.exe or MSN.dll that goes into the VS repo. And you would have to use at this command prompt we call Razzle to you would have to launch that in order to build any assemblies that come out of the VS repo. So for most folks, VS repo is mainly reserved for like native code writing. We do have managed code in there, but the problem with the managed code in there is that there's really no good integration with Visual Studio because of the custom targets that get set up by Razzle. So the building of it is a little hard. And then it's a little bit different when you work in VS repo in that you can't really a five debug. You can't open up a solution because you're missing the custom build targets that Razzle provides. And then you can't, you can just say, I want to debug this assembly. So what we do is a lot of people actually fill debug bits or sprinkle in debug.launch or debug asserts in their code when they want to debug something and then attach the debugger when their scenario is hit. So it's a little bit more legacy. It's a little bit more cumbersome, but we do have a lot of code that central to Visual Studio in that repo. And then the third type of repo we have our GitHub rebos. So we do like a good number of our projects are open source. So Razzle is a good example of that, right? It's a huge project and it's open source. So for people who are involved with the Razzle team, they probably primarily interact with the GitHub repo and then more of an open source community type of development. Okay, that was a lot of information there. So one thing, so a few things stick out here. So one is that it's really funny that we have to put like debugger launch into those statements that, I'm an old web developer and it's sort of almost like a console write line or it's an old school way of, it's a debugging mechanism, old school style. So that's kind of funny that we still do that in something as big and fancy as Visual Studio. And so that makes me feel good when I do it in JavaScript, for instance. I don't feel alone. I feel it's okay. It's as dirty as I thought it was. Yeah. That's good to know. So all of these repos, so the GitHub of course is Git, but you mentioned Azure DevOps. Is all of it Git or do we use any TFS or TVS or whatever it's called nowadays? We don't use TFS anymore. All of our source code operations are Git operated. I think the only place you might encounter TFS is probably if you want to dig back into the Dev 11 or Dev 12 where the source code is managed by TFS, then you have to dig a little deeper, but I don't recall a recent scenario where we have to do that. Okay, hang on. Dev 11, Dev 12, what is that? It's just older versions of Visual Studio, like Visual Studio 2008 or things like that. What does it stand for? Why do we call it that? Oh, I think it's just a code name, right? Like basically every single major milestone release that we have, we used to have code names. I feel like we used to have, don't we have Whitby or? Yeah, yeah, yeah, yeah. So I think Dev 11 is more like Dev 11, Dev 12. I don't think there's actually a Dev 13. I think there's a Dev 14. Dev 14 is Visual Studio 2015. Yep, yep. Not very confusing at all. So it's okay, so with the Studio 2019, what is that? That's Dev 16. So that means the 16th version of Visual Studio, is that what it means? Yeah, well, if you skip over 13, I don't think the math is a little, because we didn't have, yeah. We skipped 13. Yeah. I asked about that one time, like why do we skip 13? I think the obvious answer that you think of is, oh, it's because superstition. The same reason that in hotels, there's no 13th floor. Like I think my mom, she whenever there's 13 dinner guests, she actually puts a 14th plate out, just so that there's not set for 13 people. But that turned out not to be the case. What I heard was, I wrote a blog post about this a while ago, and I had to talk to the people that was in the room when they decided to not do Dev 13, and go directly from 12 to 14. And it had to do with like aligning the numbers. So you know how Dev 12 was actually Visual Studio 2013. Just like Dev 14 is Visual Studio 2015. And so if they went to 13, like Dev 13, straight after 12, then that would be off again, because they would now be 2015, right? And so it would be, so now there's two years apart, that wasn't good. So they decided to skip it, but now there's still a year apart, right? So now 14 came out, but that was 2015. So it was, I don't know exactly if it was miscommunication by the people who made the decision or how that came to be exactly, but they was decided to skip it, and it had nothing to do with superstition. It's interesting. Just a fun little anecdote. Okay, so developer comes in, learns how to set everything up, right? But it's so far, you've described it, well, they need to know Git, it seems like, because you need to be able to do a Git clone, or do they have to fork it? Do they have their own like good branching? How does that work? How do you do, what do you do there? So what we do in most of our Azure DevOps Git repos is we create a topic branch. So we basically take a branch, either the main branch or some release channel, and then we basically create a topic branch off of that. It's kind of like a fork, but I don't think we use fork that much within Azure DevOps. If you're working with GitHub, I think the pattern so far in the repos that I've been involved in is you do fork, and then you merge your fork back into the main branch. And then we do have release branches versus the develop branch, right? We operate on a three-week sprint cycle. So every three weeks is a sprint. And then the last two days of the sprint, we lock the branch down, the main develop branch, right? What it means is any changes that you wanna go through would have to go through more rigorous approval process because we wanna stabilize the product. So we would only, we would kind of take a look at what are the changes that need to go in in those last two days, and then decide whether they can be moved out to the next sprint. So we have more time for stabilization, or if it's a critical fix or a critical feature that we should bring it up for upper management approval. That makes sense. Is that what they call ask mode? And maybe explain what ask mode is in the next row. Yeah, so ask mode, there's actually two types of ask mode. There's what we call an M2 ask mode and then what we call the QB ask mode. So before I kind of talk about that, I wanna explain the concept of QB. So QB is quarterback, right? So basically every release of Visual Studio, so 16, we're on like 16s eight right now, right? Preview one, preview two, preview three. The entire 16.x release is managed by a person who we call quarterback. And the quarterback is kind of responsible for reviewing all of the features that are planned for going into that milestone. And then coordinating the release as well as figuring out, okay, when we get into the stabilization mode, if we have critical fixes or critical regressions, what needs to be fixed, absolutely fixed. So that's kind of how we manage our release cycle. So back to the sort of two types of ask mode. So the last two days, the branch is locked and we go through what we called an M2 ask mode. M2 just means it has to be approved by a group engineering manager. So a group engineering manager essentially means they manage other managers, right? They don't manage engineers directly. So they're in charge of an engineering group. So what we do is whenever you want to submit code, you would basically send this email to your lead first and then the lead forwards it on to the M2. And we kind of talk about, okay, what is this code change? What validation was done? What scenario does it fix? And where's the PR? So we can do a more rigorous review. So after the two days is done, we go into an even more strict stabilization phase where now if we have anything that needs to go on that train for the release, it needs to go through a quarterback approval. So it just means like we have more documentation that we have to do. We have more rigorous testing that we're requiring and more hoops to jump through. So I could see how like if you find a bug and you say, I definitely want this bug fixed before the next update to Visual Studio goes out, then that might be approved by the M2. I think M2 stands for manager level two or something like. Anyway. And so it has to be approved by them and then the quarterback as well. And see, that makes sense from a quality perspective. You don't want something that will crash Visual Studio and so you want high quality. So you want to fix those important bugs. And so you want to take them up to ask the quarterback, can we get this through for the next release? But what about if it's like features or tweaks to features? Can any get so important that they will also go through and be approved by the quarterback, let's say? It kind of depends. We have, you know, certain tenants, right? Accessibility is a tenant that we really care about. So let's say you implemented a feature and it was discovered last minute that, by the way, this button is not accessible. So we have to, it's not really a bug because that feature is not released yet, but it's part of improving, tweaking that feature. So for issues like this where it's an important tenant to sign off, those can be submitted. What we don't want to do is we don't want to introduce a completely new feature two days before we ship and then discover 10, 20 bugs associated with that feature that we didn't have time to stabilize. So those are kind of the scenarios that we want to prevent. Yeah, that makes sense. So there's a little bit of flexibility depending on its priority and it could be any type of fixed tweak or whatever. Okay, so now the engineer has gotten the code down to their machine. They know how to do the branching and all this sort of stuff. Are there any other tools that they use to open the solution? And is it just a solution in Visual Studio? Just like anyone else using Visual Studio, you have solutions with projects like .NET projects, I assume, or .NET core even sometimes. Or do you have to use other tools, command line tools? Like what's the next step there for the engineer? Well, I think you should talk a little bit about the types of work that engineers do, right? Most of the time, if you're doing feature development, it's looking at solutions, right? And then working through solutions and projects like as you would normally do in a non, in a project that doesn't build Visual Studio, right? Like if you're building a website or if you're building a Windows app, it would be the same thing. But in terms of the types of work that engineers do, there's bugs, there's feature development and then what we call house cleaning, right? Bugs are just issues that come through from various channels. So one of the most common bugs that I think a lot of people dread on the team are Watsons. And what Watsons are are basically crashes that get reported through the Windows error reporting system. So anytime as a customer where you have Visual Studio which is kind of die unexpectedly or crash, we will record that through the Windows error reporting system where we would kind of take a look at your stack of the crash and then it would also upload a memory dump. Sometimes a memory dump is like a mini dump. Sometimes it's a full heap dump. The difference between a mini dump and a heap dump is just how much information you have. So a full heap dump would have a lot more information for you to go to bug versus a mini dump. So for those, what people usually do is those get reported and then we have a centralized kind of data insights team that takes a look at those in the long bugs against the really critical ones. So while the system will report how many hits we have what's a percentage of hits and how urgent this is and then kind of create bugs accordingly. So what's a hit? A hit means at any time a user encounter that sets a hit, right? Oh, so when they have this, when the same reason for the crash happened, not any crash, but like that specific crash, like that was caused by the specific call stack. Yep, yep. So anytime you have a crash, the Windows error reporting system will take a look at the calls of the crash, right? It will take, basically take a look at the stack trace that was resulting in that exception. And then it will bucketize. So similar stack traces will get bucketized into the same category. And then every time you hit a similar bucket is basically another hit on that bucket. Okay. So some of these have like, I assume just a single hit and some of these Watson bugs will have thousands maybe. Yep. And then we fix them all or do we start with the one that has a thousand and then eventually we'll get to the one that has one hit or is there a limit to what we kind of go for and fix? I don't think we fix them all. What we prioritize the most are the top crashes. So for a given release, let's say the 16.6 release or the 16.7 release, we have people like the quarterback of that release, for example, right? Constantly monitoring what are the top crashes for this release? And then we would log bugs against those, those crashes and then get engineers to fix them. So we're constantly monitoring telemetry and Windows air reporting and then fixing the most critical ones, the most egregious ones. So and those fixes are the ones that go out into the, we have these service releases. So after each new update to Visual Studio, let's say, I think the latest released version was 16.6. And so then you have 16.6.1.2.3 and those are service releases. And typically they will contain like critical bug fixes. Is that right? Like the Watson crash fixes? Yep, yep. Okay. Yep, yeah. So they'll include Watson's, they'll probably include what we call Perf Watson's. So Perf Watson's are not crashes, but they're hangs or UI delays. So anytime that you see Visual Studio kind of freeze up for a little bit, like even a second, I think there's a threshold that we monitor whether that counts as a UI delay or hang, right? So anytime that happens, we have our telemetry system also locking information. So we also look at those, that type of data to see how responsive Visual Studio is. And for the Watson's, what engineers typically do is they open up the memory dumps in WinDBG and then debug the kind of root calls that way. For Perf Watson's, what we do is we collect ETL traces, which is kind of like data markers in the code where we measure performance. And then we use what a tool called Perfew to look at the ETL traces and then determine, okay, well for this delay that happened at this time, what was the system doing? What are some of the call stacks that are in the system and why it's causing that? Yeah, so we're basically just using the windows, the build in Perfew and Windows for handling that. Okay, so that's actually really cool. And that maybe also explain why. So I looked at the numbers, right? With something that we follow, like the number of hangs over the years have gone down significantly and the number of crashes has gone down significantly. I'm sure that people watching this are like, oh, it happens to me all the time. And that's probably true, but generally speaking, it's gone down significantly. And so that's based on the Watson and also the telemetry. So when people are sending us telemetry, we actually are able to fix and improve the product based on what they sent us. That's very good to know. Yeah, yeah, exactly. Okay, so I think you've explained about how we're doing a bunch of this stuff. And so you were explaining about the different types of work items that the engineer had and you got around to doing bugs. And was there another one? Was there other other things they do? Oh, yeah, yeah, sorry. So there's feature development, right? Like we can fix bugs until the end of time. I think the nose piece of software is bug free. So we can find as many bugs as we want and fix as many bugs as we want. But at the same time, we also need to move the product forward and that's what we call feature development. So for us, since we are on the platform team, we're basically providing APIs for our partners, both internal and external, mostly internal. What feature development for us essentially means we're developing APIs and infrastructure for people to use. So it kind of starts with, we know we have this gap in the product. Like for example, we know we wanted to move some of the components to be out of proc a few years ago. So we started this platform feature called Service Hub. So we can start with an idea first of what's missing in the platform, either it's Service Hub or Language Server Protocol. And then we kind of talk about it. We communicate with our partners to see what exactly is it that they need. And then we come back and we do a spec. What's a partner here in this context? A partner is whoever would be using that piece of infrastructure. So it could be another team. Yeah, other team on the division, like developer division, right? So for Service Hub, it would mean either language services or like test explorer, right? Those would be people, other teams that would need to use Service Hub. For a language server protocol, it would probably be other language services, like for example, Roslin. Do you want to, would you want to potentially use LSP as a mechanism for providing language smarts? So it kind of, who the partners are depends on what the feature is. But we would talk with the partners first, set of a communication channel. And then once we figured out approximately of what we want to build, we want to go write a spec, a specification, basically a document outlining what is it that you're trying to achieve and how on a high level, you want to go build it. And then we would, most of the time we would have a spec review. It kind of depends on how big the feature is, right? If you're adding a new, let's say the terminal, right? Like we're currently working on some features for the terminal where, you know, just more niceties of like integrating the terminal further into Visual Studio, like maybe into Solution Explorer or supporting some of the niceties. Like for those type of features where we call more like bug level features, we probably don't do a spec and we don't do a spec review because they're just so small. But for big things, like we want to support a new target framework for a service hub or we want to, you know, like expand on some of the major infrastructure for LSP, what we would do is we would write a spec and then we would have a spec review with the kind of stakeholders, what we call stakeholders. So whoever's interested in consuming the platform and using it would be part of the spec review. And these would be other engineers from around the Orc. Yeah, mostly engineers, mostly engineers, yeah. Okay. And then after that, we would go into the implementation phase, right? So the spec would offer a general architecture, right? Of these are the components I want to build and this is how I want to structure things but it won't go into detail of, these are the, you know, the five or 10 classes I'm going to write and how I'm going to write that. So that's more on implementation. And the implementation, and I think spec can kind of interchange in the order of where we do things because a lot of times you can't really do a spec without going into the code first, right? You can't, like you kind of have to write the code and then get an idea of how it's going to work before you can confirm whether this works or not. So what we also have is what we call spikes, which are essentially prototypes. So for example, if I want to build a new feature, I don't have a clear idea of how I'm going to build it yet. I'm first going to prototype it, just write maybe hacky code, maybe throwaway code, but not very clean, not very production level code, just to make it work, right? And then after that, I would get an idea of, okay, this works or did this didn't and then I would write my spec based on that. So I think the coding and the spec writing is it's more about making sure that you have an idea and then communicating that idea and then finally implementing the idea to a production level quality so that it's stabilized. Right, so there's a huge level of, of course, coding when you're a software engineer. But there's also sort of the process aspect of getting it reviewed, writing down a spec for it on a Word document, I take it. Sharing that out, facilitating the meeting. So you get to learn a lot of other skills that maybe you didn't get in college or wherever you came from. So maybe it's a little broader. You don't just get an office and are put in the corner and told to code, which is kind of nice. Yeah, you can actually see from my background, it's my, I'm working from home right now, as it's with anybody, but this background is actually what my office looks like. So we have a team room. We have engineers in the team room. We don't have a single offices anymore, at least for Visual Studio. But yeah, this is what it looks like when we someday get back, get to go back to work. Oh, it's just about to say almost, right? Because all the monitors are gone. People have been taken home to their home offices, all their equipment, except there's a few monitors behind you there. So you took this photo, like after people had left and stayed home. And right behind you, that's my old desk. Oh, yeah. Yeah. Who's sitting there now? Nobody. Okay. In honor of my memory, my legend. Cool. Yeah, so most teams in building 18, which is where the Visual Studio team is, work in open offices. I think all of them work in open offices, just like what you see in Tina's background there. Okay, so a lot of it is like, okay, their engineer is getting a task. So they get a ticket from Azure DevOps, which is where we keep all our work items and our bugs and everything, right? So they get a task assigned to them, and they might start coding immediately because they need to investigate something or it's a spike. And so let's just take the terminal as a feature area. So that project is it, so the terminal is inside Visual Studio, we're not just so everyone is on the same page here. A couple of releases ago, we introduced a built-in terminal, like a console basically, inside Visual Studio. It's been a thing that people have been asking us to do for years, and it finally came in and uses the new Windows terminal under the hood. So we kind of had to wait for that to make it really good. And that was part of why it took a while, but it's there. And that's its own feature. It's an example of where we're not just providing the infrastructure pieces, but we're actually providing a feature to Visual Studio. And I would imagine that because it interfaces with the new Windows terminal, it does all sorts of things that it's a relatively large project or solution. So how does that actually, if you explain how it's laid out, is there an SLN file, a solution file that contains .NET projects? Is it, or what? The terminal is very interesting because I think it contains both native and managed code. So what the Windows terminal, I believe, is actually a native project. And what we do is we re-host the Windows terminal within Visual Studio. And the Visual Studio shell is actually WPF project, so it's managed, right? So we're basically hosting a native UI piece inside managed framework. So there's little bits here and there. So most of what you see when you interact with a terminal in Visual Studio is probably comes from the Windows terminal itself. So for example, if you see issues with localization, for example, right? Like I'm trying to copy and paste Chinese characters into the terminal. And hey, when I paste it, it's not what I expect. It's a bunch of weird Unicode characters that I didn't expect to see. That's actually probably part of whatever shell that's hosting that UI piece underneath. So we don't own that part, right? So depending on what you do, whether it's a developer command prompt or PowerShell, that kind of routes to them. And then the piece on top is sort of the Windows terminal that we re-host. So for that, I think it's a little bit different because you don't really have a solution that you could open in Visual Studio that say, hey, this is the Visual Studio terminal because we re-host. And the code is kind of distributed across both our team for the re-hosting part of it and then the Windows team for the actual meat of it. And then probably the PowerShell team or the command team for some of the shell command processing part of it. Okay, so it's a bunch of different projects containing different types of code that all compile together to make a single feature. It's not even compiled. So like when we get the Windows terminal, we don't get code from them. Like we don't rebuild their code, right? We get assemblies from them. We get NuGet packages from them that we reship. So that's how that works. Okay. Yeah. So okay, so maybe that was a bad example because that seems like it's a special case. So I'm just trying to get a feel for like, what it is when you sit in Visual Studio and you look at your solution explorer when you're working on one of the, maybe one of the newer feature areas. Like what does that look like? Is it just solution containing a bunch of projects? Do we keep hierarchies of projects in folders? Like what does it actually feel like? Is it just like any other solution in Visual Studio or do we, because we're like, the thing that's interesting is when you're building Visual Studio using Visual Studio, like is that a different development practice? Like when you hit a five, what happens? Like normally it's a webpage that opens up our console window, but when you're working on Visual Studio using Visual Studio, what does that even look like? Yeah. So one of the great aspects of Visual Studio is pretty much everything in Visual Studio can be an extension. So pretty much everything that we do to build Visual Studio and an extension developer can do as well. So our workflow is very similar for extension developers who write tools for Visual Studio. So I'll take an example of language server protocol. So the code base that we write to support the LSB protocol in Visual Studio is a solution. And we have various projects in it. And the most important one that you would use to debug or build is probably what we call like a container object, or a container project. What that is, is it's essentially a, it's a V6 project, it's an extension project. So it's a special flavor of the traditional CS project you would see when you're developing console apps or libraries. And what that does when you F5 on that V6 project is it would open a new instance of Visual Studio, what we call the experimental hive. It's basically an isolated instance of Visual Studio away from your main develop scenario. And your extensions get, your assemblies get loaded into that instance of Visual Studio and you could put debug breakpoints and kind of hit different things there. So a lot of people on our team do F5 debug, right? Like you basically launch Visual Studio, the debugger gets attached immediately. And whatever you breakpoint you put in your code gets activated when that scenario is hit. What other people also do is, let's say I don't want to start my experimental instance immediately with F5 debug because I would need to download a lot of symbols. Like maybe my bandwidth connection is really low. Like maybe I just don't care about all of the other scenarios of startup. I just want to focus on my particular scenario. So what you could do is you could start the experimental instance without debugger attached. And then when you get to a place where you want to attach you would use the debugger and attach yourself, right? That's also another option. And so it's important here to just to remind people that the experimental instance is not a separate thing of Visual Studio. It is Visual Studio where you just pass a specific command line flag to it. And so if you have Visual Studio, you also have the experimental instance. Just make that clear. There's nothing special that we do there to make it work. Yeah, yeah. Okay, yep. So this seems pretty straightforward. This doesn't seem like it's a different type of setup from let's say other companies. It's pretty standard type stuff. You need to know your source control, I guess, or you will learn it when you start. And solutions and projects and debugging sounds very similar to any other .NET project. So that's very cool. That's good to know. I always kind of found it a little bit scary to think about, oh, because I used to be a developer and I was like, could I be a developer at Microsoft? I always thought that was too complicated. But maybe it's not. Maybe it's kind of just similar to what we may already know. So that's kind of nice. I like that. So OK, so now you've got a job. You've been there for a while. You're getting really good at it. Maybe you feel like you want a promotion. How does that work? So promotions, we have what we call bands at Microsoft, right? You're either in sort of like the entry level band or you're in the mid-career level band or you're in senior and above band. And usually what promotion means is it's kind of giving you credit for what you've achieved throughout the past few years or whatever time period that is. But it's really about, are you able to take on more challenging products, right? It's more about, because the problems that you face as an engineer kind of becomes bigger and more vague and more complex at the higher you go and level. So for an engineer out of college, kind of just starting in the workplace, what the type of work that the engineer might do might be very scoped. Like we need a button in Visual Studio that does X and it's very well spelled out and the code involved is probably more simple, right? And then you move in, you do that for a while and you prove that you can do that and you can take on a little bit more complex things. So then we promote you into kind of a mid-career level stuff and the mid-career level is more like, I can really independently do something, right? I don't really need that much hand-holding anymore. If I give you a task, I can figure out who are the right people to talk to. I can figure out what's the general architecture, maybe not myself, but at least Elizabeth feedback from other people and then go implement that and then make sure it's of high quality and I have sufficient testing and it's stabilized, right? So then you do that for a little bit and then you just kind of move on to bigger and bigger problems. Like when you get to the senior and principal level bands, it's more about like, we kind of know we have a problem here somewhere, like we don't know exactly, go figure out as an engineer, what's the exact issue that's facing us and then come up with a solution? And then if you move even more, it's more about like, okay, what are the problems here? Like let me find out what are the problems that need to solve? And then I can either formulate a plan and solve it myself or I can elicit the help of more junior developers to go help me solve this problem that I found. So I think promotion is more of a sort of a way to say I'm able to handle more and I'm able to handle things that are more complex. Right, so if you, so the entry, so if you're coming out of college, you come into that entry level band, but if you're coming from an industry from another company or something like that, you can come in at a higher level, you can come straight into senior band or even higher, maybe principal or something like that. But then the same thing applies, you are then expected to be able to handle those larger, more complicated things. Is that right? Yep, exactly. Okay, so at some point, a lot of people they, and actually a lot of people don't, but some people might I say, they also think, well, maybe I want to try to become a manager. And you're a manager. So I know this is going to be, there's going to be a lot of different answers depending on who you ask. But if I ask you from like your perspective, your experience, what you've observed in your years at Microsoft, like how does that, how do you go from being an individual contributor, like a regular developer, let's say, and move up the hierarchy and then you move to manager? Do you, can you become a manager like early on before you move up the hierarchy, for instance, if that's what you really want or how is that progression? That's interesting because I think over the years that I've been at Microsoft, I've seen multiple versions of this, right? From, I'll just take myself as an example, because I don't think I have insights into how other people got into their manager roles. But for me, I was never, I never really wanted to become a manager as my true, like that's my goal of what I wanted to do. I think it's more of like, it kind of naturally fell onto my lap and this opportunity existed and it felt like I was in the right place at the right time. And hey, why not try something new? But what I've observed is I really wanted to become a good engineer first before I become a manager, because I think in order for you to understand what engineers are going through and what are some of the problems they have and even to approximate like, okay, well, this task should take about this long or you're blocked on this because I think you need this type of help. I think I really needed that foundation of being a really good engineer. But I think there are other people in the company that are more oriented towards a management job, like maybe that fits the personality more, maybe that's sort of the career that they want. So they have ways to sort of help them get there. So for example, within Visual Studio, within the developer division, we have what we call culture clubs. It's basically people who share an interest, whether that's like a mom's group or an LGBT group. And then we also have a group called aspiring managers. So what that group does is for people who are interested in eventually becoming a manager, they kind of get insights from existing managers on what that life is like and what helped them become successful when they transitioned to that role. So I think it's more personal. There's really no one-size-fits-all solution for people who are interested, but yeah. Okay, so don't you think that that's potentially also, not dangerous, but like if you take the, like in your case, like you became a really, really good engineer that knew everything about your space and now you become a manager, so you basically take the best developer out of being an engineer and put them in the manager seat where I guess you had a little experience at the time you started. Doesn't that just then productivity drops and is that the right way of doing it? No, I mean, I think as a manager myself, I'm able to make more impact by helping other people kind of learn what I know myself, right? It's more about, I feel like my impact as a manager is actually broader than when I was in IC because as an IC, you just kind of code and then you kind of come up with solutions yourself. But since I became a manager and I've had the time and it's my mission and my job's goal to kind of mentor people and then coach them, I'm able to kind of teach them to do what I used to do. So now before it was me working on one project, right? But now I can kind of translate that knowledge into two or three different engineers that are starting out. So I think that aspect of it is amplified, yeah. I never thought of it that way, that's really cool. So it's kind of the opposite of then what I was suggesting in my somewhat leading question. So we got a question here from, and someone that's anonymous, he says, will we ever see a 64-bit Visual Studio? So we actually did an episode of The Office Hours a couple of weeks ago with Andrew Arnett. So go ahead, go back and check that out. I think it's called Inner Workings of Visual Studio. He has very good explanation of how it all works, where we are with that and so on. So go check that out. I won't talk about it here. It's a very complicated thing. Matthew asks, do you build Visual Studio in Azure Pipelines yet? And how long does it take? We do. So that question kind of, yeah. So since we switched to the micro build solution where a lot of our components are built outside of the VS repo, right? It's more of a parallelized build process. So for example, Rosin gets built within their own pipeline in GitHub. And then our components like LSP or terminal, for example, gets built within our own micro repos. So depending on the size of the individual repos itself, the build can take anywhere from an hour to two hours. And then we have the build of the main VS repo. And I think that takes, I can't remember the exact time. I wanna say at least a few hours to build. So it's a lot faster than it was before. We have CI builds, continuous integration builds coming out of the VS repo. I think every at least 10 builds a day, like every few hours we'll get another CI build every day from the branch that you're currently working on. And those CI builds are actual installs that you could put on your machine. So they're not like binaries that you would have to patch. Like that's the one I got installed, right? I have our master build, which is updated. Like I don't even know how many times a day I get a notification. Like every time I open Visual Studio, I get a notification that there's an update. And so that is the, when the full Visual Studio repo has run its integration tests and it's full build and all this sort of stuff, the output of that is a fully installable version of VS. And we actually use that to dog food to find issues early on the team. I got one of those, so that's kind of cool. So, okay, so all these different micro bills. So each team has a micro build, but of course only the components that you are working on is being built. So if there's something you haven't worked on for six months that's not being built. And so, but what about when you do the Visual Studio, the VS repo, the big build, will that still then call and build those little micro bills or is that all, it just grabs the built artifacts from already run builds of those micro bills? Like how does it stitch it all together to create one installable Visual Studio? Yeah, so I think that's exactly what it does. It's basically built, grabs the artifacts of all the other micro builds. That's basically the artifacts get pushed to a cloud storage somewhere. And then when you build VS repo and you compile that setup package for you, the installer, it basically grabs a location of all those inserted bits and then downloads them and installs them on your machine. What's cool about that is, let's say I wanna work on a feature in Terminal and that's from a separate repo or let's say I wanna make a change to the editor, right? I wanna change the finding files, look and feel a little bit, like maybe I wanna make the button pink. What I could do is I could make that change in my micro build. I would kind of kick off an Azure release pipeline that would build the artifact and then insert it, upload it to a cloud storage. And then I would insert that into the VS repo. And before I even merge my PR to say, hey, this thing has been inserted and now in the product checked in, our engineering system actually produces a PR build that you could install. So it's basically a full on Visual Studio and store based on that single get commit that you have potentially going in. So this makes buddy testing, test sign off really easy and it gives us the ability to test before we check in. So for a lot of our changes on the platform, like let's say we have a platform change and then we also require Rosin to make a change. What we could do is we could kind of coordinate an insertion into a single PR and then we could have our testers test that kind of PR build before we even check it in to find issues faster and to make sure we don't break dog fooding scenarios for internal members who are kind of using the latest master build. So every single PR that kicks off a new build of the whole VS repo, like the entire thing. Yeah. So I assume we would get, we have like hundreds of PRs per day. So we have a server farm I take it that can handle this because we have to build all this in parallel because you said it takes like several hours to do a full build. I don't actually think the PR build is that, I think they did some optimization. I think the longest duration for PRs is actually running test. So once we run our full, what we call DD RIT, it's basically like a set of tests that like a set of maybe 10 scenarios that we want to run through like to make sure, hey, I can open a C sharp solution. I can start editing. I have IntelliSense. I can bring up the debugger. Those core scenarios that we want test. Once those tests are run, you have a full build. And then what we also do as part of PR is we run regression test, like performance regression tests. So we measure UI delays and we measure hangs and of the key scenarios. So if you want it installable build, I think you can actually get that within an hour or two because that's how long it takes for the DD RIT to run a test to complete. The regression test take a little bit longer. And I think when I looked at it last, it was like six hours or something or potentially, but I think they did some optimization recently to decrease the amount of time there too. Okay. So we got a question here from Steven. Can you explain buddy testing you were mentioning earlier? Oh, buddy testing. It means, let's say I develop a new feature, right? I'm working on the editor team and I implementing a new feature called search, like yes, search that was actually implementing a long time, a while ago. And I don't really want to check it in yet. I kind of just want to give my partners like C++ or Rosentine to try it out before I check it in because I'm a little bit wary of breaking a core scenario. So what I could do is I could kind of stage the release and the builds and then do a mock insertion into the BS repo and then produce this installable link or basically this installable executable that we could go have our testers install on a VM or on a test machine or something and then try out our test scenarios on that specific build. So it's not something that you would check in but it's more of a thing that you would use to test to make sure that what we wrote is ready for check-in. Yeah, I've heard it described and on a previous team I was like as a sort of a sanity test just making sure there's not anything obviously bad or wrong or inconvenient or something like that. Yeah. Here's another question. This I guess is a little bit back to when you were talking about Service Hub which is a way that we can pull things out of Visual Studio and out of process to run inside a Service Hub instead. Here it goes. Why did companies like JetBrains have to outsource part of their components in their extension to an external process in terms of performance? What's the obvious problem from an architectural standpoint of VS? So why do we do this? Why do we even wanna run code out of PROC? I think there, from me, I think there are two major reasons that we do that today. One is Visual Studio is a 32-bit process and so the memory footprint you have here is limited. The second reason is really reliability. Like let's say if I'm writing a component that's run within the same process as Visual Studio and I have a very nasty bug there that basically just throws a fatal exception whenever bad things happen and that crashes the system. If you're run in PROC, it just means Visual Studio just dies and you lose all your work. If you move that component to the out of PROC and you hit that same bug, it just means that the out of PROC component is killed but you might not have some of the features that were once available like code lens, for example. Maybe you'll lose all your code lens when this bug gets hit but you still have the editor loaded. You still have your unsafe changes. You can go and kind of do your cleanup, relaunch the machine in a better situation. Yeah, you could even restart that service hub, right? So if there was a crash in a component that runs, the code that you have in your extension that runs inside Visual Studio can respawn that service hub and continue the work in case something bad happens. So it can recover itself as much as possible before and as you mentioned, that process can also be 64-bit. Exactly. And we already have a bunch of 64-bits processes just to that question earlier about 64-bit. We already have a bunch of 64-bit even though Visual Studio itself is not 64-bit. Okay. So what other testing are you doing before you can say that a fix was correctly solving the bug or a feature is well implemented? Like how do you test your code? So we have three types of tests. We have unit test, which I think most engineers know it's just kind of testing at the code level, right? Testing a method, testing a class, testing a property. And then we have what we call integration test. And in Visual Studio, we have this framework called Apex and Apex is just a way for people to write UI automation integration tests that get run as part of a PR merge or locally. And then on top of that, we also have manual test suites for our CTI team to run. CTI just means it's a team of testers that we have offshore and they kind of work over the night to run our manual test suite and then report to us if there are any issues found. So I think there are a lot of gate checks along the way, but of course, we can't possibly test everything. So we rely on a lot of community feedback. So for example, anytime that you log an issue, like you use report an issue in Visual Studio and say, hey, this thing is not working like, let's say, I'll give you an example of toolbox, right? I get quite a few of those. So let's say I launch toolbox and there's nothing in there, there's no control. So I go to report an issue and I say, I can't see my controls in toolbox, hey, what's going on? Those feedback tickets, those are what we call feedback tickets. Those eventually get routed to the individual team to own that feature. And then we have a team of people kind of responding for each feature, either asking for more information or creating bugs to track them or then just telling the customer like, hey, what you're doing is probably not a valid workflow and here's how you should be using the product. So we do take customer feedback very seriously. It's a good way for us to sort of measure the health of our system that we built. So yeah, that's another way. You know, I think we should do a full episode just on reporter problem and suggest a feature and how that all works. That will be super interesting because it does go straight to the engineers. Yeah. I know a lot of people don't believe that, but it does, it goes into it and ends up in that. When you create a ticket like a couple of minutes later, it shows up in Azure DevOps with that team. Yep. All right, we are at an end. It's 10 o'clock here, local time. So we've been at it for an hour and that's all the time we've got for the day. So Tina, thank you so much for coming on and help me understand how this all works. Thank you. All right, and for the rest of you, I hope to see you again next week. And in the weeks to follow, as usual we're doing this on Thursdays at 9 a.m. Pacific time. And you know, have a great day everybody and see you next week.