 Good morning everybody and welcome to a new episode of the Visual Studio Office hours. I'm your host for the next hour. My name is Mads Christensen and let me tell you something. I was writing some documentation the other day for my job. I'm a program manager on the Visual Studio team and I was writing a doc for some guidelines for how we internally but also externally handle suggestion tickets. So whenever you suggest a feature for Visual Studio that comes into my team and we look at all sorts of things and we don't have any guidelines for it. So I decided I should write some and that was kind of cool. I had to learn how to do it and we have this publishing system and I realized that there was just a lot about documentation in general that I didn't know. But I do know that in the past I have been choosing products based on their level of documentation over their competition. So I'm fully aware of its extreme importance in like there's competitive advantages and all these sort of things but in sort of a day-to-day type of situation like where I just had to write some documentation like I just had a lot of questions. And so I'm really happy today that we have with us here two people from the documentation team at Microsoft from the Visual Studio that handles all the Visual Studio documentation among other things and we can talk to them and ask them all these questions about how to write good documentation. Why do we need it? How do we do it at Microsoft? And how does the Visual Studio team deal with documentation in general? So without further ado I'd like to welcome Jill and Justin. And Jill if you want to introduce yourself. Yeah, hi I am Jill Reinhauer and I'm a part of the content and learning organization at Microsoft. Content and learning is the team that owns docs.microsoft.com and also learn content docs.microsoft.com slash learn and the Q&A platform and within that organization my team writes about Visual Studio and Visual Studio code. And yeah yeah I've been at Microsoft for about 20 years and working on content for 15 of those. Wow. All right Justin. I just learned something about Jill that I didn't know and we work closely together. So I'm Justin. I've been at Microsoft about a year and a half and I am in the developer division same team as UMads and we work on the Visual Studio product and my role is kind of a content liaison or advocate. I really work with Jill's team with training groups and with our program managers to basically help program managers connect with customers through documentation. So that's kind of my role. All right. So before we sort of get into all the tips and tricks and how we do things then I have a question for you Jill and since you've been here so long and you work in the documentation organization like how big is it? How many people do we have at Microsoft that writes documentation for their daily job? Yeah so there are about three large organizations within Microsoft that are completely focused on documentation. One is the content and learning group that I mentioned that's within cloud and AI and then there's also a large content organization in the business applications group. That's a group that the dynamics products fall under and then within office and M365 there's another content group and there's a few other content folks in various other parts of the company but those are kind of the big three centralized content organizations and then for example the content learning CNAI content group has probably oh gosh around 200 content developers and content pms so it's a pretty big organization and then that group also that org creates and maintains the platform docs.microsoft.com as well so there's some engineering resources as part of the org also. Okay so that sounds like it's a very large organization just for handling docs and I know Justin that you've been working closely with the feature pms so the program managers on the Visual Studio team that you know writes the specifications for the new features that Visual Studio comes out with all the time then you work with them and they for them to contribute to the docs as well so it's not just the 200 people on the docs team it's it's also the pms and maybe engineering teams as well is that right? Yeah there's a lot of moving parts. So like from a on a sort of a maybe not daily basis but weekly basis how many of our pms are involved with the documentation like how how typical is it that we have like engineering teams with the program managers and so on contribute to the docs? That kind of depends on the team um engineering involvement with docs I think largely depends on you know how how much it makes sense to have engineering involved with docs um like I believe the VS Code teams there's a lot of engineering involvement with docs they write kind of what they call quick and dirties which is a quick overview of a new feature or changes to a feature and then they work with content folks to kind of edit that clean it up and actually publish it. Other teams might have more dedicated content resources that are doing most of that and then other teams it's kind of up especially smaller teams or new products that may not be in GA yet those might not have any content resources yet and so that's kind of the thing that Jill and I work together on a lot is figuring out where those handoffs happen and how to kind of not drop the ball in those handoffs how we move smoothly from a product that's in preview and then as that begins to scale out and starts having more users how do we onboard you know content people to help out with that and just kind of managing those transitions that's that's something that takes a lot of communication between our teams to get right. Yeah so um maybe one of the questions I have in all of this is um like when when should a content author like Jill one of the on your team when should they write documentation versus when is it the job of a of a of an engineering team to write the documentation is there is there a sort of a rule of thumb on that? It depends on a lot of things um one of the ways in visual studio that we've kind of looked at that is that um very often for for preview features for things that maybe don't have a wide audience yet a lot of times the the PM organization starts the documentation on that they're very close to they know exactly what's going on with those preview features and and so they kind of see the documentation and then once the feature maybe moves to a public preview or to GA or gets wider adoption um then at that point sometimes it'll move into the content organization owning the content so that's kind of one way that we do it but I mean it really it depends some of it's based on history and some of it's based on resources and availability and you know all sorts of things but just for some some numbers to throw into that um the content and learning organization has an OKR this fiscal year um a key result around external contributors so it's very much a goal of ours to have more contributors from folks outside of the content and learning organization so for example the the key result this year is to go from 1200 and seven external contributors to 1800 roughly um so so it's something we definitely are putting a lot of focus on and something that Justin and I are partnering on and and that's the big the big push is to get to get more people contributing to docs from outside the content org so when you say when you say external contributors you do you mean uh like Microsoft employees working in engineering teams for instance or do you mean like because the docs are open source so it could be new community members so can you clarify yeah yeah both um yeah we welcome contributions from Microsoft folks and from from external and we do get pull requests from the community um all the time and and we love that yeah right all right so um what are some of the like you you alluded to it a little bit Jill you said that um oftentimes it's the engineering teams or the program managers on the engineering teams that start writing the documentation or begin in the feature area that they're doing um so is that something you think would be a way to do it for other teams as well like in other organizations like if you have a if you work at a small company or a medium company or whatever like would this be the way to go like you you start with engineering and then maybe content writers or maybe you outsource it to content writers um sort of to do the final editing or how could a flow like that um sort of work for even for smaller teams Justin yeah so a lot of my career has been on the engineering side and um you know a lot of places that I worked documentation if it happened um was heavily dependent on engineers to document what they wrote but I think there's definitely opportunity to scale that as you grow as your product scales if you have you know not not everywhere splits the concept of program managers and engineering sometimes those those roles are the same thing or they might be split into into different ways I think there's definitely opportunity to start small have the people who are building the thing write about the thing um and then as you get bigger scale that out maybe have you know not every engineer or or program manager is necessarily a great writer it's there's a reason that's a job is it's it's a difficult thing to write and keep things like accessibility and readability and localization all these kind of scale problems that you have as your product scales at some point you reach a point where it doesn't make sense for engineering to have to keep all of that knowledge and program management to keep all of that knowledge and so that's where you know the content team is really good at understanding those things and giving feedback on the platform to help improve those things and be aware of that and I think that is kind of how you scale as it probably starts with engineering and program management contributing to um the docs and then at some point you have to level that up a little bit and bring in content professionals to to improve that and scale it out I think you're muted meds yeah sorry that was a garbage truck that was outside I really should move this from Thursdays to like any other day because Thursday is the uh it's garbage day here so okay so if you're a small company and you don't have you know the budget necessarily for content writers or outsourcing or something like that at what point is is a is documentation good enough like at what point does it add value versus adding like positive value versus negative value like I would imagine if you if you write like docs really quickly and you don't pay attention to grammar or something like that and you publish that that might have a negative impact on from your customer perspective or something uh Jill is there is there like some sort of rule to that hmm I I don't know that there's it's a good question and something that we wrestle with a lot what is what is good content what what makes healthy content or what are the measures you look at to say this content is performing well versus this content is not performing well and there's so many so many parts to it um some of the things that we look at are um feedback that we get from customers so if we get comments or ratings that indicate something is it's not what the customer wanted that has a very big impact on on you know how we prioritize but we also look at kind of the behavior that users exhibit when they're when they're looking at documentation so the things that they click on a page or how long they spend on the page do they do they dwell on the page as long as it should take to read the page or are they going someplace else quickly um that would be an indication that maybe they're not finding what they want on that page um we look at bounce rates if they they go someplace else within five seconds that's how we are defining it um and we look at the path you know where they came from to get to this content and then where they go after um so there's a whole bunch of metrics that that we pay attention to and then there's also some operational metrics that kind of play into that not so much behavioral but things like when was the last time we updated the topic is it is it still up to date or has it been um ignored for for too long and now the screenshots are out of date um so kind of a freshness metric um and then things like broken links or um terms that are are better for accessibility so there's a lot of things kind of on our side that we look at too that that help um show what content what healthy content looks like yeah so can I expand on that a little bit too go ahead yeah go ahead um so one thing to back up a little bit um one thing that I think about a lot in my role is ultimately as a as a division that produces software as a software delivering division we're trying to create good experiences for customers that's what most software companies are trying to do right ship software that is a good experience from end to end and so if you think about what that what that actually means like docs are part of it the product the feature itself like the apis the sdk is whatever it is that's part of it but it's this kind of whole journey of I heard about the product from somewhere maybe a forum maybe some piece of marketing maybe a friend um I downloaded that like maybe I downloaded visual studio community edition I engaged with all or some part of that product maybe I ran into an issue or needed to learn I went and read the documentation and so this this is kind of a whole journey and so really like what I'm trying to do is make sure that the docs is thought of as part of that journey that we're not just thinking of you know what was my experience in the visual studio solution explorer and um but really thinking about kind of that whole journey as the product surface right and the reason I wanted to call that out is because there's a lot of different customers and different types of journeys in that and so when you talk about are the docs good enough or how good is good enough if you've got a product that is in beta sign up and it's um you know pretty rough around the edges not polished at all not enterprise ready it's kind of a you know barely more than proof of concept your customer in that case is probably a lot more fault tolerant right if the docs are not super well written not great grammar maybe they're not even localized into multiple languages the product is rough around the edges they're going to be a little bit more tolerant of that because they have said by signing up I'm kind of saying I want to try this I know it's going to be somewhat broken by the time you actually get to an enterprise level where people are building big scale stuff that depends on this that's a very different customer and the documentation you have to surface and the journey of that customer could be totally different and so a big part of the complexity of this is is figuring that out like I was thinking when you ask that question like if you find the answer that let us know because we think about it and talk about it all the time like what type of customer is this where are they in their journey I think Jill maybe had something more to add yeah it's a good point and I I think that the kind of the polish on docs like you mentioned if there are grammar mistakes and and that kind of thing while it maybe doesn't hurt the experience like you can you can figure out what the intent was but it does there's a level of trust with the company if you if you go to documentation and um you see broken links or you see grammar mistakes you sort of you can start to lose trust in the value that's placed on that um and so and so there's kind of this balance of what is good enough versus what builds that trust and and keeps people wanting to come back yeah so so you brought up an interesting point about freshness like is the is the documentation still relevant uh when was the last updated does this does the screenshot match the latest version of the product and so on um let's say that you have a piece of documentation that two years after it was written is still completely accurate but it's two years old and so you know when you when you google something you know I do that all the time I look at the dates usually there's a date of when it was last opted in the google search result either because the URL has the date of the year in it or something like that or or just just print it out and and I would always go with the latest just because I don't know what has changed and so I trust new content maybe more especially like in software right where everything moves so fast so in the case of that two-year-old still relevant documentation would it make sense to go in and say okay let's just update the like make a minor change just so that we can say it was updated you know now uh to to make it fresh for the search engines and and on the page and stuff like that or is that considered cheating or how does that work Jill yeah I think it's if we do a freshness pass what we call it on on a duck on a topic and you know run through the procedures check out the screenshots and determine yep this is all still up to date even though it was written two years ago then changing that metadata for the last updated date is perfectly fine I don't feel like that's cheating at all it is still up to date um you know as opposed to you update a typo and don't look at the whole topic what it's trying to say but then say yep this is updated because I changed that typo that would be kind of the wrong approach so um yeah and and you know we we've thought about right now I'm pretty sure that that last updated metadata is a manual field writers choose to update that um you know in a way it would be nice if that were updated automatically but then you'd have to set some thresholds on like well if it's x number of lines that have been changed in this topic then we can automatically update so that it wouldn't pick up just a typo or a minor change um so I don't know that's something we kind of thought about um but for now it's it's manually updated the writer chooses yeah okay um so I was made aware of this uh a while ago um that we we basically don't have the capacity to to like write documentation for every feature all the time and so we group things into tiers so we have the most important features fall into tier one and they get like dedicated content authors and then we have tier two they get like somewhat of a of a content author associated with that to help evolve the documentation in that area and then tier three is the one sort there is hardly any content authors and it's up to the engineering teams themselves to maintain the documentation I don't know if I got that right Justin or is that how it works um kind of I don't know if it's so straightforward I guess I can give you kind of a concrete example um code spaces is a product we've been talking about a lot and code spaces has largely been in private and then public preview and because it had a very small group that it was exposed to it kind of fell into that market of these are maybe a little bit more fault tolerant users because they've signed up early um then they want to try this new thing out and so that content was pretty much the responsibility of pms with some light additional resources to figure out as we move towards ga and more people start using that we reach a point where it's like now this really needs some some uh dedicated content resources the other thing is is that's not necessarily constant right like you could have okay we've got you know 30 docs that were written by pms that were kind of you know quick and dirty they're about to be formally released they're about to be localized we need to do a pass for translation we do a pass for accessibility so that might be a project that needs more resources for a period of time but then after that maybe we're not doing big doc changes constantly but there's just a slow trickle of updates and and a new uh doc now and then then maybe those can ramp back down so i think jill might have more to add on that yeah one thing to add i think the decision on when something is owned by a pm or engineering team versus content you know there's a lot of factors that go into that and one of those factors might be if if there's a pm or a pm team or engineering team that really wants to own the documentation that's great we welcome that too you know it could be somebody that has a background in writing or just really enjoys writing or wants that additional interaction with customers and and then you know they they can they choose to say hey we we want to own this ourselves and and then the content team can give support for some of the the back end things the repo management or things like localization and accessibility and those things that just mentioned okay so so with these different tiers or or you know ownership maybe rather uh like who owns the documentation then there is the the third aspect which is the like you have the content owner sorry the content author you have the engineering team but you also have the open source community that wants to contribute and you know that's one of the things i thought was the most exciting thing when we started out doing docs.microsoft.com which was what five years ago or something like that it seems seems like it's not that long ago but i think it might have been when when it all started out but the open source right so we can scale out and and i really like that aspect and and you were mentioning you get a lot of feedback or sorry a lot of contributions both issues and pull requests and so i guess it doesn't matter if it's open source it doesn't matter who the owner is on the engineering side like any you know you can send a pull request to any of these to fix up some documentation if you want to why why is it that people do that Jill why why is it interesting for for externals to contribute to our docs yeah i think we're lucky that we have customers that really care about the product and really love it and have really want to contribute and you know somebody sees something that they can help out with and they jump in and do it and it's it's pretty amazing you know i think some of it might be i don't know if there's a recognition piece of it like having your name on something or or if it's a lot of people really feel they want to give back and you know i learned a lot from the documentation when i was starting out and hey here's something i can contribute that i think is a motivation for for sending in pull requests just in any other thoughts on that yeah um i mean kind of piggyback on what you were saying i think there's it's also a good way for people to kind of get their feet wet right like if um if you want to contribute to an open source piece of software there might be a lot of kind of overhead of setting up your development environment and you know 17 projects and the solution that you have to kind of learn your way around and there could be a lot of overhead to just ramping up especially if you're a newer developer without a lot of experience and i think docs are a great place where people can get some experience contributing getting get experience being part of uh like an open source community um and just kind of start building you know a network of of people that they work with or getting some name recognition it's it's a really great place to get started when you know it's i feel like a lot easier relative to jumping you know into a big engineering project with both feet to jumping into a doc set um so you could go out and find you know typos or things like that in in docs or just find you know a little place where maybe the bullet points aren't that clear for a beginner and you can actually add value to that and have an impact on other people and on um you know Microsoft's fairly quickly um especially if you're you know kind of newer to engineering and don't have the the confidence and experience and all of that to just jump into a much more complicated project right away yeah that's a really good point because it looks like all you need it really is as a markdown right you it's all just markdown so if you know markdown or maybe you don't even have to know markdown you can just look at what's there and make the edits right in the in the text and you might be just fine and just and i know that uh you came prepared and you actually are able to show us how this whole system works like how did the what did the docs look like how do you edit the docs and i guess that's a sort of the same whether you're an external or you're an internal contributor is that right yeah that's right and um i can absolutely do a super quick overview not like a training session at all i'll try to keep it quick and and not get you uh into the details but um yeah i can give a quick overview i would love to see it so if you just share your screen i'll make sure that everybody can see it i keep getting some warnings that my internet quality is not great so hopefully this uh this all works so you should be able to see uh one of our docs i have a big monitor so i've just blown up so i need to scale it a little bit more let me know yep you're live cool so um so this is you know this is a doc uh i picked this one because i worked on it um back when i was directly contributing to docs and so i'm kind of familiar with it and and a little easier to prepare to talk about it so um this is you know the doc itself um but the docs platform provides quite a few other little things that are worth calling out one of them is you know the bread crumbs you can see where you are there's this we call this the table of contents of the toc it also renders a little collection of shortcuts here so everything that's an h2 will be pulled out and a shortcut created here so you can quickly skim a doc contents docs contents and and jump to a specific area so when you say h2 you mean like the headline size too yep yep sorry um all of the anything that's a heading of some sort will get a quick link so if you want to send links to other people um to a specific part of a doc like you want to see you know i want to send somebody a link to xamarin essentials i can grab this and copy that url and send it to somebody for reference you can give us feedback on whether or not um a page is helpful and that can be as simple as binary yes no or there's an opportunity i believe to to add some comments as well um and then i think kind of the coolest thing though is this edit button right here and this isn't in every doc set by the way each each doc set can be configured a little bit differently but um many or most have this edit button oh actually one other thing i meant to mention this filter is really useful if you're working in xamarin forms and you're like i want to you know see a checkbox um you can quickly filter i want to see a um progress bar or um whatever this helps you it's filtering on title and it helps you drill into um kind of see quickly what what is available in a big table of contents and then the search up here searches all of our doc all of our doc sets so if you're searching for something really specific you can get good results here but because it's searching a lot of things um you may get less relevant results than filtering on titles um but anyway the cool stuff is the edit button and then the feedback buttons at the bottom of the page so if i want to give feedback on this page like maybe i found a bug um i am not prepared to you know do my own contribution and fix it i can click this and i'm going to yeah this is the the two factor off that we have in microsoft yeah um so it's started an issue here i can give it a title i can enter some feedback in here um it puts in some metadata so that this can be tracked back to a specific page and then that issue will show up for docs writers program managers engineers to answer update that the doc and close that out so that's one way that um you know a lot of people don't think of issues as contributions but they absolutely are issues that that help us find and identify problems or weaknesses in our documentation is absolutely forward progress but then you can also just click the edit button and you're taken right to this open source you get kind of a preview here there's some yaml metadata that helps our docs platform understand this page's organization and things like that um you can view the raw markdown or i can click edit right now and be editing this file so the so when you click just click edit right there then github will automatically uh create a branch for you that's kind of hidden such that at the bottom there when you're done with your edit you can submit a pull request without cloning or forking or opening the the file on your local desktop it's all in the browser it's all right here so any typos any small edits you can just do it straight on directly on github is that right that's right and this is kind of a special case as special permissions on this repo so its behavior for me might be a little bit different than its behavior for just a public contributor but in most cases what what actually happens is github is creating a fork and a branch for you behind the scenes automatically you can make your changes in here you can actually preview your changes um and make sure you know it reads the way you expected you can do a commit and then once you commit those um you can turn that right into a pr and that will go back to the um the doc's repo kind of owners to review and approve so that's the kind of light workflow um the more common workflow for regular contributors and kind of deeper contributors there are limitations like i can't add images with this i can't add new folders i can't there's there's things that i can't do just in the browser so there's also um there's also a workflow where you you know use like github desktop or git bash or whatever your favorite git client is you fork and pull um the repo down and then work on it in vs code and this is the workflow for most of the content contributors and pms that are doing regular edits um this provides a lot of helpful tools like spell checking um markdown linting um we have something called the doc's authoring pack you can actually see everything that's in it here if this text is large enough to be readable but there's a markdown is this an extension for vs code yep it's a v s code extension it has some yaml editing uh validating stuff um some markdown linting uh some spell checking a variety of things that just help write quality content and my personal favorite feature of it is i can hit and shift or control shift v depending on mac or pc and i get a preview that you can see is really close to the doc itself so i can be confident when i push this up like this is roughly what i'm getting if i've changed an image or i've reorganized headers or something like that um i can get a good idea before it goes through the whole build process once you commit and pr once that pr is accepted there's a lot of cool stuff that happens first of all when you create a pr there's a lot of automated grammar checking spell checking readability checking checking for you know things we're trying to avoid in terms of uh things like inclusive language or confusing acronyms and so a lot of that is checked and that that will call out in your pr as comments um things that are recommended to fix and then once that is fully accepted and the pr is merged it actually goes out and is in most cases kind of again repos can be configured in different ways but it's translated into 13 different languages and published there as well so there's a lot of things that our docs platform is doing that are pretty cool i think a lot of people don't realize kind of how sophisticated and how many moving parts there are um so yeah i just kind of wanted to show that off that's very cool thanks jason um you know when i when i look at that you know the first thing that just popped in my head i don't know you maybe you thought about this already but i just thought about it is uh code spaces instead of you having to um you know install extension on your own you know install visual studio code if you don't have it already then install that docs authoring tools extension and all this sort of stuff what about code spaces what if they're on the docs was one that says edit and you know we had a we would create a code space for you with all those things installed so you just go straight in you just use the browser you just uh because it basically runs via code in the browser like a code space um is that something jill that you're looking into i love it that's really cool uh i don't think we're looking into it yet um um let me make a note of that um well i guess there is um i guess there is a potential issue in that you know that would i think by default and that would um be something potentially that whoever contributes will have to pay for and i think for something like like this we probably want to foot the bill so we get those contributions in and and anyone who contributes are not you know forced to pay that would be horrible yeah and i think we'd have to look too at what um how much use that would get you know what are are enough external contributors wanting to do the bigger more in-depth contributions um that would benefit from having vs code and having the extension available versus something that you can do on on the web in github just a quick edit here or there yeah or the internal p.m. organization right because just and i know that um that you've been working on like getting them to contribute more and take more ownership of the docs and and all this sort of stuff and and that there's been some challenges with that um and i'm guessing that one of the challenges is that there's some moving parts there's some things to understand about git and um and the workflow and what extensions to install and how do i actually do this that if if that was if we if the barrier was as low as possible then that would be easier to get internal contributions is that right that's absolutely something that uh jill's team and i talk about a lot and the teams around jill and um the docs platform team i think we're all really interested in finding ways to reduce complexity um finding ways to train and scale out that training because yeah the overhead of training and not just training but keeping those skills from atrophying right like if i taught you how to edit docs today and then you didn't do it for two months i basically have to retrain you because you're going to forget all you know all of the at least the details you might be able to muddle through it but you need regular use of those skills to kind of keep those from atrophying so all of those things are are barriers that we're trying to look at and reduce um yeah i as you were talking about the code spaces i was thinking like yeah we haven't actually talked about dog voting that and uh that might be a good idea yeah i think that it might be really helpful i think another part of the complexity sometimes when we are addressing github issues for example a question might be about a particular version or dot version or a preview release and so the person who's responding to that issue in addition to getting their doc environment set up may have to install a different version of visual studio or you know go through you know various pieces that to kind of mimic the the scenario that the customer had when they submitted this issue and so you know resolving any single github issue can take a long time because you have to get all that set up so yeah potentially having code spaces that that get the authoring part of it for you could be really helpful cool yeah that might be something that would be interesting for you know any viewers here that is considering doing more documentation or open sourcing or something like how can you lower the barrier to have more people contribute i'm sure there's a there's a bunch of ways so um all right jill so you talked about in the very beginning like just how many people we actually have and just in your group there's 200 on the content theme team and you said there were two other groups around microsoft of similar size or something like that now that's a lot of people and so how many how many docs do they actually then right if we just start with visual studio just to keep it in the you know to the to the audience here they're all visual studio users i'm sure like how many docs is that pages do we measure that in number of pages do we have for visual studio i don't know who to ask this question is that you jill yeah i have some numbers on that gosh do i have them i don't think i have it off the top of my head um visual studio docs well there's lots of ways to slice and dice it but roughly 3000 articles are just visual studio just the latest version there's another set of docs for visual studio 2015 that's probably about the same size um 2017 and 2019 kind of share a doc set except for a handful of topics that are different about one or the other um and that's just conceptual docs there's also reference docs there's visual studio sdk um there's then some of the other repos that uh like uh code spaces is in is in a its own repo and live share docs are in its own their own repo um and then the localized versions of all of those yeah it yeah just in uh what are your thoughts so um one fun number i think in the microsoft docs hub org there are over 5000 repositories um so that just gives you a scale of just the number of repositories um and then yeah like jill was saying i mean there's things that you might not even think of as a product that needs documentation but so visual studio is a product right but then you might be paying for an enterprise subscription of visual studio and that itself is a product um and we have a doc set for visual studio subscriptions we have a doc set that is just for roadmap and release notes type of stuff that's separate from the doc set of how to use visual studio because it's it's covers conceptually different things and so yeah there's there's a lot of i mean like i've said a couple times there's a lot of moving parts there's a lot of people involved a lot of you know just effort going into trying to consistently um deliver and improve our documentation as part of the whole kind of product experience jill i think you had something more maybe yeah just in terms of you know the scale that's a huge number of docs and so then on the flip side one of the things we look at is are there are there docs that we need to retire um and so sometimes we will work on a we call it weed the garden project and um look at things that have had zero page views in the last three months for example and can any of those be deleted and redirected and you know in some cases they can't it's it's part of a whole or you know we're documenting the UI and so if you took out one piece that wouldn't make sense but there are cases where you know either the technology has changed or for whatever reason this doc is not useful to customers and so instead of letting it continue to be a weed in our garden can we can we get rid of that yeah so uh i think i've seen sometimes that some portions of the old documentation is is it archived is that what we call it we it's basically not showing up on docs.microsoft.com but you can still download them or how does that work correct there's an archive or a previous versions area within docs and um some of that content is not indexed for search engines and so you wouldn't find it but if you go to the archive then you can search through that TOC and get to you know a previous version of Visual Studio if you need that yeah yeah so we want to make sure that we always have the latest and greatest up and available through search engines and so on and i guess that's super important right you get a lot of traffic to the docs from search engines i take it i'm guessing yeah i think roughly 50 percent of the traffic comes from search engines and so that's another metric that we look at just how much is coming from organic search how much traffic comes from referrals from other microsoft.com pages or from within docs specifically and then direct traffic is the other bucket that we look at yeah is there like a correlation between the things that people find in docs like they want to learn more about and then you know the and then the individual features in Visual Studio so let's say as an example the new find in files dialogue that we have in Visual Studio 2019 if people go and search for it and find it and read the docs on it or they may even open bugs on on the documentation do we then kind of understand that it could be that we have some issues with the not the docs itself but with the feature like maybe people don't understand the feature and therefore they they go to the docs website for more explanation or how do we how do we manage that relationship between the feature and the and the doc. Can I take this one? Go for it Justin. This is a this is actually like kind of the driving thing of my role is i have this belief that you know traditionally the majority of issue review and issue closure has happened by content people. I I have a hypothesis that I'm working on testing and and pushing you know improvement to I believe that the way that our customers interact with docs has a lot of potential to tell us more about product problems or how to improve that kind of high level journey that I talked about. I feel like docs are a critical part of that and that is actually my job is to try to improve how our program managers engage with docs engage with the content people engage with customers to use an example you know if you have docs on just use Xamarin forms again because it's the docs that I'm I'm actually still the most familiar with if you have a huge search engine traffic for one specific control in that doc set well what does that mean it could mean that it's really really popular everybody's using it it's one of the first things that people want to learn about it could mean that the API is not very good and engineering or or program managers need to take a look at that how customers you know are filing issues you know so that's that's looking at traffic how customers file issues a lot of times docs issues might be reflective of an actual issue with the product itself not just you know there's a typo on this page or I don't understand you know the steps one two and three it might be that we find bugs or flaws in the product itself or just opportunities to really improve that experience and that's that's really what I'm trying to focus on and improve is that that integration and the workflow between all these people that we've talked about we have a lot of program managers we have a lot of content writers we have a lot of tech involved in this how do we unite all these things how do we reduce the friction of that communication the the friction of those workflows how do we bring that down and bring ourselves closer to our customers and and understand them better that's kind of the whole the whole goal that Jill and I and a lot of people work together on yeah and so I take it there's the relationship between the docs and the engineering teams producing the features that the docs are about is sort of an iterative process it's not just like you know the best thing would be if you start writing the docs as you define the feature I I think that would probably help you define the feature to begin with to write the spec if you were to write it in a way that you were like showing people how to use it before it even exists and keep that iteration going as you develop and as you get real screenshots instead of mock-ups and so on keep keep improving it and then after your shipping your feature keep modifying and running the analytics to figure out on the docs to figure out like do we have to make product changes to that you know that was made visible through the use of docs and so on is that so I take it that's it's sort of a holy grail that that doesn't exist fully but is that kind of like where we want to go we talked a little bit about this before that's why I'm asking you this very specific question what's your take on that Justin yeah um that's complicated right it's really complicated to figure that out and scale it like how do we you know how do we do this well without having people just sitting in meetings and reading email instead of doing useful work I mean that can be useful work right but if you're an engineer the time you spend sitting in meetings and communicating and stuff is kind of overhead where you're not doing actual engineering I mean that that's a hard problem and right now in my opinion what what we have is teams have largely evolved their own workflows um and some of those have less friction than others some of them are a little better we're trying to find opportunities to improve that across the board without taking away like usually those evolved for reasons right like every team is not the same every product is not the same and so we have to strike this balance of creating standardization and efficiencies that we can scale across teams without causing major disruption where we we come in you know it's it'll be super arrogant to come in and say okay now you have to all do things this way because this is the way and you know that's like that's kind of my customer my customer is program managers content people and indirectly the Microsoft customer because I'm trying to improve these processes for the Microsoft customer without destroying the day-to-day workflow of program managers and so yeah it's a tricky problem um we have a lot of different evolved solutions some of that can be standardized some of it can't it those solutions involve that way for a reason like I was saying sometimes engineers are heavily involved in docs sometimes they're not involved at all sometimes program managers write most of the docs and like Jill was saying some of them love that some of them hate it some of them are great at it some of them are like I'm not a writer I'm terrible at it how do we accommodate like individuals and the unique context of different kinds of product development and still improve efficiency and deliver a standard experience that like a quality experience across the board that's that's what we're working towards that's like there's no easy answer for that right it's a it's a line somewhere on a gradient right it kind of um begs the question a little bit to have um is if there are any areas that does not need documentation and therefore free of resources to concentrate on the areas that do as an example let's say we do a new feature the new I'm just going to use it again the find in files uh feature so you probably know this in visual studio you can you can search through some some folder and you can find all the files in that folder that has some text in it that matches your search term it's it's sort of very simple in the way it works uh I'm sure it's not simple simply in its implementation but the concept is very simple right uh does that need documentation like is that not self-explanatory at what point does the feature have to have documentation Jill yeah I mean I think it depends one consideration is something like f1 help so if there's a UI element and and there's a possibility of clicking f1 to get more info then we have to have a target for that and so that may be a reason to have a page that that users can land on um but yeah I think there's definitely cases where the the UI the design is such that you shouldn't need documentation and that's fine it kind of gets to the the point of like these huge content sets where can we reduce where can we where do we not need content that's great right it's like any software project right what what uh what code can I delete uh you know and sometimes less is more right it's maybe it's easier to find it's easier to reason about um and and that could be the same for docs I take it yeah for sure yep okay so we're we only got a a few minutes left here is there anything that you would uh encourage our listeners or viewers to do when it comes to documentation like what is what is the the tips to give them for their own docs Justin so probably uh better for Jill she's more in the uh in the day-to-day writing I think my my big tips would be um we spend a lot of time talking about accessibility we spend a lot of time talking about um ways to reduce friction and make it easier to get from I have a doc idea to I have a doc published um those are those are big things I can think of so make it as easy as possible like make it easy and you know remember individuals like it's easy when you're looking at especially at scale big numbers and stuff to kind of lose sight of individuals and and you can really go wrong that way right like you could write your docs for um beginners and lose the experts with you know too much uh kind of hand holding you could do the opposite and fill your docs with acronyms and um assumptions of existing knowledge it doesn't exist so it's yeah it's hard to come up with a like one line of like this one this one tip yeah um one thing I'll share is the um we have at Microsoft three voice principles for for any kind of documentation content that we write and and so we want all of our content we want the voice to be warm and relaxed crisp and clear and ready to lend a hand so those are kind of just principles for for any good content um to have that sort of voice um so that's in the writing that's in the writing style yeah yeah so um you know warm and relaxed um try to be human we're people talking to other people um it doesn't have to sound robotic um and the crisp and clear you know that gets to some of the things like sentence structure and um scannability of a topic um making sure that um just that that it's it's easy to read um and then ready to lend a hand you know we we want we want to sound like we are you know we're here to help and so we want the documentation to come across that way too that that that whatever we can do to to lend a hand to make your experience better that's that's what we want to do um and then I would say what what helps us the most is if users leave feedback and rate topics um leave suggestions submit pull requests all of that is super helpful to us and and we we really we look at all of those as signals so yeah all right sounds like I need to uh to do more of that um and I hope that our viewers is gonna do the same so um you know when you visit docs.microsoft.com don't be shy hit the like or unlike button if you were helped or not open issues and pull requests um and it makes it better for everyone it's a it's a really a great way to contribute right um it's kind of easy you can do it just in the browser if it's just a simple simple little fix that seems like something worth doing where you can help a lot of other people it's not about helping microsoft i think to me it doesn't sound like it's about microsoft it's about helping other people that are in the same situation as you that need information to unblock them to continue their development work and so this is a great opportunity to do that and you know make make the world a little bit better I guess for some people so um awesome I just added a an okay r for you to do 10 con contributions per month for the rest of the year so okay okay I gotta I have to sign up for that I guess all right well thank you so much both of you for joining us we are out of time and this was wonderful having you on so thank you so much and thanks for all the great work you're doing on the documentation thank you this is fun and to all the to all the viewers we're gonna move the show from thursday to monday starting already next monday 11 am uh pacific time so two hours later and um that's gonna be super interesting and uh there's some good reasons why and i'm gonna talk about that um monday and um so with that uh thank you so much and see you then