 music and shake things up a little bit. So the title of my speech today is Increasing the Awesome. And as you saw by the first video, there's a lot of jobs out there where you're trying to decrease the suck. You're just constantly fighting off negative things. You're just trying to keep up with all the issues at hand. And I'm in a lucky position, and several of us in this room are in a lucky position where we get to increase the awesome. And that's not just working at NASA. I think you should consider the work that you do with Wikis as a chance to increase the awesome. Think about the way that you have transformed your organization's opportunity to do different things with the same knowledge that was always there, right? And so the goal of my presentation is one, just to kind of give you another recap of what NASA has done with MediaWiki, our experience, but also two, to just kind of share some examples of how to celebrate the awesomeness that we've achieved and encourage you to take a look back at what you've done within your organization and use that as a motivation for your people. Don't just get stuck in a rut saying, hey, we need to use this Wiki, but go back and look at how cool it is and what it has afforded you that you didn't have before. And I'll just clarify that I'm speaking on behalf of the NASA JSC Wikis. Of course, we've also got Rich from Glenn and he's, we're now starting to figure out how to work across centers and for the whole agency, but I'm just gonna speak on behalf of JSC today. But before there was awesome, there was definitely a lot of trouble. There was a lot of negativity when we first started things. It didn't just happen magically. When I first started working at NASA, it was the same story as many of you have heard. It was very difficult to find the resources I needed to certify as an astronaut instructor. You'd go look in a folder and there'd be like three different versions of the same file and Rev Bravo had an older time stamp or a revision time stamp than Rev Alpha, but then there was also Rev Dave. And so you're like, well, which one do I trust? And many of you have heard the same story. And so I said, well, why don't we use something like Wikipedia at work? And so my immediate boss said, okay, sure, make it happen. Okay, well, but how do I do that? I have no idea. So it took quite a while before all the magic sauce came together. There were times where I was evaluating confluence. It was readily available, but it was for a different program. It wasn't for the ISS operations. And so I couldn't really figure out, well, can we really use this? Because it's not in our budget. We're not paying for it. And we don't officially have access to it. And so that kind of didn't work out. Eventually, James came along, unfortunately he's not here this morning, but he had some more experience than I did on setting up web servers with MySQL and PHP. He had actually built some applications at his previous job. And so he was familiar with the tools that were necessary to run MediaWiki. And just luckily, he also had set up a scheduling tool that we use for our classes at the Neutral Buoyancy Lab. And so he had access to a server. That was kind of critical. And so kind of on the side in the corner, he secretly installed MediaWiki and we started tinkering. But we kept it real quiet because we didn't want to get busted. We didn't want to get shut down. It kind of was a skunk works thing. And over the course of the year, we just started pumping information in there until finally people started catching wind of it and asking us, hey, I want access to that. And all of a sudden it became the cool thing. So eventually we got approval from management and so it took several years. But in 2011, that's where we started actually officially using a wiki in our workplace. And as I said, it took about a year to build up that critical mass. It was, it's really, as Mark was explaining why we're creating this kind of meta wiki or example wiki to showcase, it's really hard to explain to people the power of a wiki without showing them examples. And so we had to build those examples before people really got it. And so that took about a year. And we've covered this before. So this presentation is linked off of the schedule for today. So feel free to pull this up later or whatever and you can get access to all the videos. So I tried to link some of the prior presentations. So in Vienna a few years ago, we gave a background of the history of how we got started. We explained some of the examples of how we used semantic media wiki. So I won't go into too much detail on our beginnings today, but feel free to watch those later. One of the things that really helped kickstart our usage of a wiki was meeting minutes because that's what every organization does is they go have meetings and they talk about stuff, but we'll why not just feed that into a wiki and link those minutes to the topics about hardware and things that you're talking about and that way it's just all recorded there. And that definitely helped us organically grow more quickly. After about five years, we started having a lot of people come up to us and say, we want a wiki. And so Robo also, I mean, you've met a lot of the other NASA people here. Everybody wanted a wiki and we were just handing them out left and right. We were like, this is great. We're gonna have so many wikis, wikis are great. So by year five we had, we could actually say we had tens of wikis and this was a good thing. We had a lot of people using wikis. I decided to put together a kind of a retrospective of just kind of analyzing the usage and the growth of our wikis. And so linked off of here, I took a page that I had originally created on one of our internal wikis and I got approval to post it publicly. So now it's on a media wiki.org. So the page is five year wik, NASA wik anniversary. And so I'm not gonna go through everything on here but just kind of briefly, I did a bunch of SQL queries to just kind of analyze like, okay, how many users do we have and how did that trend over time, this kind of stuff. So there's lots of little stats in here. We broke down our articles by category to kind of describe what kind of different things are we talking about here. Made lots of plots comparing how we started off with just a core set of admins making most of the contributions. And then over time we had other people besides the admins starting to fill in the contributions and then eventually we had people even outside of our organization helping to contribute to our collective knowledge. So it wasn't just the EVA group, it was people outside of the group contributing. So lots of different analytics here. I won't go into every one of them but I just had a lot of fun with R. There's a program called Exploratory which makes, I'm not sure if you guys are familiar with R but it's a programming language for statistics and it's got kind of a steep learning curve but there's a program called Exploratory which makes it really easy to just throw some data in there and then spit out some different plots and just kind of take her with it and find out what you really wanna get out of your data. And so a lot of these plots were created using Exploratory. This one here I kind of like because, so this is kind of like over time, the number of contributors. So you can see like of course when the Wiki started there was just James and me and then some other people started contributing and as time goes on you can see that it truly becomes a collaborative effort. Everybody's contributing their piece of the puzzle. I talk a little bit about watch analytics. We heard a little bit about that yesterday and I'll describe that a little bit more in a few minutes here. Let's see. There's also kind of something else to note is that we started off with just the EVA group and then Robo and Osso and some of these other disciplines joined and when you do it in that fashion, when you do it based on not just the Wiki for the whole company, but you start off with a single org and then you start adding other orgs kind of from the bottom up, you end up with kind of a step function every time a new group of people joins. So it might not just be a linear growth scale over time. You might end up seeing kind of step functions as they joined. For example, when Robo joined they had a SharePoint Wiki and James found a way to just wickify that. I think it used Pandoc or something like that. And that way they were bootstrapped with the same content they already had but now with the extra functionality that we have with the Media Wiki. We did a little comparison between our viewership and contributorship between comparing us to Wikipedia and of course the scales are way different but the plots were interestingly were very similar. The shape of the plots were very similar. So I'm not gonna dig into that but it might be interesting to take your analytics and compare to other Wikis and see how your growth is over time compared to others. So another thing we looked at, so there's kind of the 1% rule. So out of everybody you might consider that only a small percentage of your viewers are actually creating content. You might have a small portion of people editing content but then like maybe only 1% of your users are actually creating content, like creating new pages. So I don't know, does anybody actually go in and look at this kind of stuff and see across your organization are you actually measuring how many of your employees or how many of your workers are actually contributing versus just consuming the information? To me that's kind of one of the biggest differences between the public Wikipedia and a private enterprise Wiki is that Wikipedia is kind of voluntary. You're not gonna get every single person to contribute. I mean it would be nice but you're just not gonna get there. Whereas in a corporation you really do need every single person contributing as much as possible but you still end up with this sort of small percentage of people that actually contribute. So it is kind of a problem. This is, yes across all the wikis. And then we just did some kind of fun plots to just look at things differently. So here's a heat map showing the x-axis is the hour of the day and then these are the different groups, the different org codes. And so you can see that most everybody works between 7 a.m. and 4 p.m. because that's where the contributions are. I mean that's, I don't think that's very surprising. But then here's another one where we split it out by org. So each color is a different org. And then again we've got the hour of the day but now it's normalized to 100% of the contribution. So what we see here is a big red chunk, right? And so what is red? Well that's HSG, that's Greg's group. And they spend a lot of time over in Russia. So they're not in central time zone. And so they have a disproportionate amount of the contributions at 3 and 4 a.m. central time because they're over in Russia. I mean, I don't know, maybe this is silly but I think it's kind of cool to explore this and recognize, oh yeah, so that group actually is working over there. Their contributions might be masked because they're a smaller group but when you look at it in the early morning you better make sure you're not doing maintenance scripts that are gonna shut the wiki down. Because they're using it at that time. And then finally, I made an animation to demonstrate the growth of a wiki page. We had an EVA that was a very quick turnaround. We had to get out the door as quick as possible and go fix something. And so we just immediately threw a wiki page out there and just started adding pictures and doing research. And so this animation is a screenshot of every revision of the page and it just shows the growth of the page over time. So over here is kind of like the zoomed in and then on the left is the full page link because it had so much. It's just, you can't show it all on the screen. I just think this kind of shows the power of how simple MediaWiki makes it for everybody in a team to throw their information on one page and let's just solve this problem really quickly. And then I'm not gonna go through these but there are some testimonials at the bottom of the page. So maybe later on your own time, take a look at this page and I just asked all of our users for volunteers to please tell us your feedback whether it's positive or negative, whether you hate the wiki or love it. And we got a lot of really good feedback and it's just really interesting to see the different perspectives of each user because you might not anticipate how each person is using the wiki until you get that direct feedback from them. So I just thought it would be useful to share this. And when we shared it internally, of course initially after our five year anniversary, it was also good for other users to see what other people were saying about the wiki because you might get some people that are like, I don't really like using it but then once they saw these testimonials, you could see that it kind of had an impact on them and they thought, oh, okay, I guess I never thought about it that way. Let me go try what that person's doing. Just another plot showing that we're still continually growing, we're definitely still in the growth phase. I suppose eventually we might taper off but things are still going well even six and a half years later. And as I said, we kept adding more and more wikis but we didn't have a digital strategy and Mark's chuckling because this is kind of an inside joke but we didn't really plan this. This just started with the EVA wiki and then we were, as Oprah showed you earlier, we were just handing out wikis left and right but we didn't really think about the consequences of that. And all of a sudden we ended up with a whole bunch of wikis and at first we thought this is great but then we realized, okay, well, if a certain piece of hardware breaks and we need to fix it, well, the OSO group is gonna get that hardware ready by changing out the internal components and then the EVA group is gonna package that thing and take it outside and then the robotics team is gonna provide the robotic arm to get us to the work site and then once it's installed, you've got the Cronus team that is gonna command that box and check it out and make sure that it actually functions and so you've got multiple groups that have information about the same hardware but it's across different wikis and so now all of a sudden we've got the same problem. We've got multiple silos possibly with conflicting information. So, yeah, we kind of realized that we were kind of back in the same situation as before where we had a whole bunch of, where we created new silos and things were still disconnected and that's not what we wanted. And so now, something we're working on currently, we had actually hoped to have this finished by the conference and tell you about all of our success but it's actually a lot harder than you'd think. Eric has been doing the bulk of the work of trying to merge some of our wikis so we looked at, you know, which wikis can we merge based on access levels? We have one wiki that is open to basically any NASA employee and that's mostly just like printer information and how to use your time cards tool and that kind of stuff but then the robotics and EVA and OSO is like the internal maintenance so a lot of these wikis had the same permission scheme basically that you had to be a NASA employee and you had to be either a US person or a foreign person that has a need to know like if that's part of your job so like a Canadian flight controller or a Japanese crew member they need to know that information and we realized, well, if they've all got the same permission scheme let's go ahead and merge those together into just one ISS wiki based on the program. So we're working on that now. It's harder than it sounds. Basically you've got to look at a lot of the templates when Robo and OSO started their wikis they copied our templates for hardware pages and meetings and that kind of stuff but then they kind of made their own flair their own version of it and so when you merge back of course there's gonna be conflicts there and so you might think there's a scripted way to do it but really it comes down to manually comparing things and figuring out how to conform again and I don't know if we'll have time for a lightning talk for Eric to go into more detail on that but it's been a lot of work hopefully we'll get to report the success at some point. So I did want to share briefly like the fact that we're not we're trying to not just be consumers of this awesome technology. I mean definitely we're using media wiki and we're using a lot of extensions and we're enjoying the benefits of that but we're trying to also feedback into the community so one of the ways that we feedback the community is our participation. We're here we're trying to share our story we're trying to help other people out you know if you have something you're struggling with just chat with us and we'll see if we can help you figure it out. We've gone to at least one hackathon we went to a developer summit and kind of voiced our opinions there that was in January. We've also been growing our number of wiki savvy people so I mean it just started off with James and me and then Scott Ray he very early on got active in creating a ton of content for our wiki and then now as you can see we've got a lot of representation here from NASA and it's a little bit like the Fight Club Army you know you get this underground army of people that are savvy with the wiki but I mean everybody just continues to grow they level up in how they know how to use templates and properties and then eventually they start tinkering with extensions and so we've got several people now that are developing new extensions they're developing integrations we've got several people that never contributed to wikipedia before but now that they're comfortable using media wiki at work they say oh this is pretty easy I'm gonna go to wikipedia and so now they actually go contribute to articles in wikipedia we even have people that are now donors to the wikipedia foundation so I mean I think that's cool. Some of the software that we've created that everything we do we try to well everything except for our custom NASA specific stuff we try to share publicly so I'm not sure if hopefully James will get a chance to share more details about MEZA you've probably heard rumblings of this it's a set of scripts that we created that tries to simplify the installation and configuration of media wiki just so that you could keep that standardized and keep it as simple as possible we've created a lot of custom extensions for what we do at NASA but then there's several of them we've created that anybody could use we've modified existing extensions and we've also integrated things that were already out there at NASA I think it's important to not just try to cram everything into the wiki I think there's definitely cases where just leave some of these resources where they are because they do a good job of what they do but tie that into the wiki find a way to tie into their API and merge that information into a single search result within the wiki but let that other database keep that data as the source so I mentioned MEZA there is an animation here I won't show it here but it's kind of a fun way to show kind of the growth of the project it uses Gorse I think is the name of the tool it basically takes a GitHub repo and just shows the timeline progression of all of your commits which is kind of a fun way to go back and look at the growth of a project but just to very briefly describe what MEZA is it's a few years ago we had an opportunity where David MEZA one of the people at the chief knowledge officer office at JSC he gave us a couple servers for development and so in his honor we named the project MEZA and we later sort of forced the acronym back that we called it media wiki easy admin but the concept started off as a bunch of shell scripts to take a baseline Linux server sent to us and install Apache and MySQL now MariaDB and to install MediaWiki and a suite of extensions and to set it up as a farm if you want to have multiple wikis with just a couple commands on the command line since then James has converted it into a set of Ansible scripts so a little bit more on the technical side there but the kind of the big picture is that if you can get yourself, get your hands on a baseline sent to us virtual machine and you run just, I think it's like four or five commands and then wait ten minutes and then all of a sudden you've got a wiki farm so it's, I think it's pretty cool I think it's gonna help some people out to just get started and if you guys wanna contribute and participate in the project then by all means please as I said we've tried to integrate with some of the existing resources out there so we have imagery online there's a database that has thousands of pictures and so why try to upload those in the wiki why not just write a parser function that where you can list out the image IDs and then display those in line in the wiki page several other resources where we are able to pull information from another server into a wiki page and then we also started tinkering with, we call them bots I'm not sure if that's technically true but basically like scheduled scripts cron tasks that go off to a database and then compare that database data to the wiki and copy some of that information over and the reason we did that is that the wiki has a great built in notification system whereas a lot of these other resources don't have that so you kinda have to manually keep up and you might not get notified if something changes so instead since all of our workflow is in the wiki if we pull some of that information in nightly then you're gonna get notified automatically when that page changes and you can use the diff tool within media wiki that's already there to easily see what changed I'm not sure how many people here use Slack but you know it's just another chat program but we started using we started writing some different scripts that give us kind of analytics and real time reporting on our server performance in Slack and that way if we're just down the hall and we get a notification on our phone and it says oh hey by the way Eric just created 5,000 pages then you know he's probably working on merging the wikis and maybe you wanna go review that stuff but yeah it's just taking a look at how can you get notifications outside of the wiki whether it's using Slack or IRC or whatever just so real briefly something else I've been working on recently is we have a new class of astronauts coming in and so we have to train them on all the basic skills of EVA and so it's kind of hard to see this but basically every row is a skill that an astronaut has to learn how to perform when they're doing EVA and then every column is a class and so in our wiki we actually have a wiki page for every lesson plan for every class that we teach and then we also have a page for every skill that they need to learn how to perform and that way the new astronauts can go to those pages and watch videos and pick up tips on how to perform those skills properly and then this page is just a matrix that kind of shows us every time that skill has been practiced and so you can see these skills here they see it for the first time and then the second and the third time and then they've seen it more than four times then you think they probably haven't figured out and it's I don't know if this is real impressive or not but it's just you can use wiki pages in such weird ways where you just are able to report you know a whole lot of information just using a set of queries on a single page so kind of bluntly though but does anyone care? You know we do a lot of this stuff and we're pretty impressed with ourselves but kind of taking a step back and thinking about okay but does the rest of the community and the rest of the organization do they actually care? You know you heard about watch analytics we've done presentations on watch analytics before so there's a video there if you wanna watch it later I think watch analytics is a great extension it gives you a lot of insight into the activity of your users on whether they're actually keeping up with their watch list and reviewing changes it gives you a much simplified and more direct feedback on what pages have changed and then of course the diff view there's nothing new there but these links will take you right to the diff and so it's just a very easy set of directions for the users to go see what has changed and then it also suggests you know which pages need help being watched so kind of like your if you've got a bunch of pages that just nobody's paying attention to it'll say well I know that you're watching other hardware pages so maybe you should watch this hardware page and you know as Brian talked a little bit you know this tool gives you the opportunity to measure your success on actually reviewing the content as it keeps changing so he described this yesterday but just briefly to re-describe it so the outside here on the outer ring these orange dots represent people the blue dots represent wiki pages and then the lines indicate people watching wiki pages and so in here you know you would expect like this a lot of wiki pages are being watched by a lot of people but then out here you've got pages that are only being watched by a single person which is a problem right so like if I'm watching 100 pages and I go say a bunch of wrong things on those pages and nobody's watching them then they're not gonna correct it they're not gonna catch that incorrect information but then the plot actually also includes red lines to indicate a red indicates when someone has not yet gone and seen that latest revision and so this was our initial we had that same as Brian just said yesterday we had that same assumption we just said hey people are watching pages they're reviewing it and if it's wrong they'll go fix it and we just thought hey this is great and then we when James created this extension and we actually saw what was going on we realized oh wow they're not not only are people not reviewing pages but what the heck's going on here so once we started vocalizing this to our management for a brief moment they were really responsive and they said all right people go out there and do your reviews and as you can see our status got a lot better briefly this is current this is our I took this a few weeks ago so I think we're kind of back into a bad situation and so that's kind of why I put that slide of that page in there does anybody care initially people said oh this is great we need to take care of this but then it just quickly became forgotten and we try to raise this up with our management periodically but everybody's busy and yeah we'll get to that eventually but I mean how important is it to you and your organization to have your documentation current and correct I mean to me it just seems like that's so important but I don't know how to better convey this I don't know what we're doing wrong so approved revs your own described this yesterday we use this a little bit we try to only use it where absolutely necessary so like lesson plans and some of our console handbook pages that people think are super critical we put this extra layer of protection where it has to be approved before that revision will be viewed by everybody so we do make use of this but when you do that you add a layer of bureaucracy you add a bottleneck so I created this little script that every week right before our weekly group meeting it goes off and it checks all of the pages that use approved revs and it sees if there is a revision that has not yet been approved so basically that means that whoever owns that page needs to go review it and either approve it or fix it and then approve their revision and so it spits out the date of when that revision was made and then the link to the page and then if possible it lists who the owners are if it doesn't have the owners then it's just the management team and at first it was kind of nice because it spit out this list every week and I thought maybe we would address it but then I started noticing that at the bottom it was truncating in the middle of a page name and I thought oh man I got a bug in my script but then I just realized I was exceeding the limit of a string of whatever the data type was for passing all this data through in one value so our list was so long it was running out of memory for that data type and so I said to management like hey our list is so long it's breaking the script, come on I mean the oldest one is 2013 seems like we should address that but these things they just kind of linger and I don't know I'm not in a position of management I'm just in a position of providing data and trying to encourage the group to do better with their knowledge with these analytics but I don't know it's kind of weird being in a powerless position where you say hey we've got these problems we should fix them but then where does it go and so we've got a ton of other ways I mean there's so much data if you go into the tables of MediaWiki and you start analyzing all the different things back there along with watch analytics you get page scores Brian touched on this yesterday but we don't really use it in our group but it's there it's an opportunity to take another look at how well are we managing our information but we just don't use it you can even look at the timeliness of reviewers like how quickly do they go check their pending reviews but I don't think anybody really goes and reviews this I even started tinkering with some gamification stuff I made some plots to kind of show like compare that so every color is a different person and this is over time showing their contributions and so you don't necessarily want to say the most page revisions as the winner but you definitely want to see is everybody kind of continually contributing to the wiki but yeah I mean we have all this information out there but I don't know if it's not being used then maybe what's the point I don't know it's just kind of frustrating I guess is my so how can we motivate our group to make use of this information how can we motivate our group to make more use of the wiki instead of just letting it sit there and stagnate so we did one doccathon which is where our management set our rate no classes, no astronaut activities today we're just all going into a corner and we're going to work on the wiki we're going to split up into groups and each work on these pages and it was great I mean I think we got a lot done that day we worked on our lesson plans we got a lot of kind of the grunt work that nobody wants to finish we actually got that done that day but then that was just one day and that was two years ago I think so I don't know why aren't we doing more of these every week in our group meeting we do what we call a wiki-quickie it's just limited to five minutes does anybody have a question on how something works okay let me answer that real briefly and then we can talk more later or hey we have this new feature I just want everybody to know about this I think that kind of helps to get people's attention but and then I mentioned gamification several years ago in St. Louis I gave a presentation about how I was really excited about adding gamification to media wiki and it's easy to just throw some points in there like everybody gets a point for editing a page or everybody gets a point for thanking someone using the echo and thanks extension but then you can easily game the system and what are you really trying to reward you don't want to just reward page revisions you want to reward quality of content and that's a much harder thing to measure so I mean I tinkered with showing this is a plot showing over time different people and how many contributions they're making this is I tried looking at so this is for me this was like in 2013 these were the top 10 pages that I edited so like I guess my to-do list I edited 126 times I guess I used that quite a bit that year but then categories of pages that I edited the most and I don't know this was out there for everybody to be able to go look at their own information and see like well what did you spend most of your time in this year doing and I don't know maybe that's useful it's a little bit less on the point system it's a little bit more on kind of introspective kind of learning more about your workflow I tried putting together like a little bit of a story like you're a hero over the past 1605 days Darren has made 13,000 revisions to 3,967 pages let's estimate that if it just saved one person five minutes every time he did that and he has a time savings of 330 hours I don't know is that motivational to some people maybe but that's the trick is trying to figure out like some people are competitive and they want those points but that's probably only 10 or 15% of your group like what actually motivates everybody else some people are kind of social animals they want to pair up with other people and work on the same thing so I thought maybe if we look at the categories of pages that I edited and then find other people that edited a lot of the pages within that category maybe we can be wiki friends and edit the same pages I don't know I don't know that that would necessarily motivate me but it might motivate some other people and so I'm kind of just sharing all this just kind of I don't know if you have ideas later and you want to pitch some ideas and if you think your users would actually be motivated by any of these different aspects let me know because I've really hesitated to like actually release an extension and put it in use because I don't want to do it the wrong way I don't want to ruin the wiki and turn it into a system where you're just doing it to get points like I want it the whole point of the gamification is to get people to participate and make the wiki better so just a couple of final things here this is another plot from that I like looking at the time this is from the beginning of the wiki to I think this might be like maybe a year ago that I made this plot but it's you know here's just the number of contributions and every color is a person so you can see over time it generally goes up this is that same blood where it's like normalized to 100% so you know here we are at the beginning of the wiki and one person and then two people and three people were contributing and then here at the end you've got hundreds of people editing and I don't know I just think it's kind of cool to share these things with your users and show them show them the power of what they're working on so again here's the animation from before so if you want to take a look at that it's in the presentation and then I've added some links to some other videos we've got a lot of different presentations already out there on YouTube and I just tried to consolidate all of them and link them from this single presentation so that's it does anybody have any questions we've got microphones right turn it on when you're ready go ahead I was wondering what your sort of bottom line is because I'm seeing the activity measurement your graphs and the various things of what the activity is do you have any bottom level it appeared that your objective was an educational one from what I could tell in other words you're looking at skill development at one point and therefore using the wiki as a support tool for education as opposed to a factory call you know that you go to something to go and find what is the specific fact or specific information that you're looking for do you have any bottom line if you're going to upper level leadership and you're going to say what's the effect of all this on your education program is it improving the quality of the education or the efficiency of the training process do you have any bottom line feel or how you could get to that bottom line so I think I understand your question and so I'm going to I kind of skipped over this because I realized I was running long but this plot I showed you earlier the way it works is that here's an example of so an NBL run when we do a training run at the NBL we will perform several objectives and here is just one objective and that objective is to practice APFR APFR is a foot restraint it holds their feet steady while they do work and so you have to practice getting your feet into that tool and so you know for 30 minutes we might practice that skill and so there is a lesson plan wiki page that has all of these objectives and it has lots of semantic properties and then on another page there is a page that describes here are the steps to get into that foot restraint so this tells an astronaut this is what you need to do to succeed eventually we'll have videos and other resources but that's kind of like that tells you how to do it so there's definitely a lot of content in the wiki that is educational that is your standard kind of encyclopedic material but then the point of this was more the wiki can be more than just encyclopedic this is more of a reporting tool so what this did is it helped us figure out hey look at this skill we're only training it twice in their entire training flow is that enough? do we need to add it? like we might not have missed that had we not had this reporting tool whereas we can easily see so like this one oh man we train it pretty much every class we're good to go so it's sort of looking at you you're getting towards the human resource issue of people and skills and their training and the whole management of that but I'm thinking what's the bottom line of this it's obviously got to be something that's being done better than the alternate mechanisms I don't know if this helps answer but the way I saw this the bottom line was the previous groups of astronauts we've trained three or four years later we determined we're not what we expected in the skill level but we had no data to show that so the bottom line of creating this on the wiki was we actually now have data we can show our management and say we can analytically show how we're training and eventually increase that to to benefit it so we use the wiki in many cases but the bottom line for this specific skill I think that was kind of the bottom line of how do we get more proficient students and show that we are doing that with the data we have you're getting to holistic skill management well it identifies gaps in our training in our data set okay so you want to pass the mic back to her in your room I think you're next just briefly I think there's someone in a previous talk who said that you know wiki activity was integrated into performance reviews I don't remember I think it was Brian right oh yeah okay right right I don't know if that's an option for you guys it seems like I mean if I was a manager absolutely right but I'm not you know so I'm just this guy that you know trains astronauts and builds the wiki and I try to I try to provide this information to management and say here look what everybody is doing and I think to a certain extent they do look at it but I don't know I guess they're not they don't because there's a lot more to our job that's not necessarily tied to the wiki right like that's not everything we do so I don't know so my question builds on the first question so you're building or capability here and my question is can you tie what you've done on the wiki to a cost savings because if you if you are sending all these people out for training but now they can do it on the wiki you know there there must be a cost savings that management would pay attention to and if you could take that to them they could help you socialize market the wiki you know put it put other training on there to build OC definitely I I I think I agree with what you're saying I I don't know how you exactly formulate the the quantification of that though like well so for example this is what we're doing in our company I'm gonna put oil full essentials on our wiki okay so for each person that takes that class it cost let's say it cost about a thousand dollars okay but if I can have a class of twelve go through each quarter then I'm saving my company money and I think that's what management pays attention to is what you know what is that return and you do wonderful graphics and you know graphics tell the story you know that's the narrative so I think maybe I mean it's something to consider do you have a microphone for James because I think I know what he's gonna say so yeah I was just gonna comment one one area especially in training especially in this job where it can take three years to get somebody certified and they're basically they're non-value added in the beginning and then it's a slow ramp up as the percentage of work that they're doing they're just building building training time non-value added training time versus and then their percentage of contribution start to build and then they're certified let's say they're a hundred percent contributing one of the concepts that's been getting some traction is just in time training so it's especially with astronauts but it's also with flight controllers and other people is the idea that you get this class one year but you don't use that skill until you launch a year later and so you have to find out which pieces of training have it treated by then what they actually need to get refreshed on and then also especially from a flight controller training perspective so much of the old system was learning how to look things up learning which dusty bookshelf has the technical manual that you need and so when we we be examining and doing evals on these trainees sometimes we would say no you don't need to memorize that you just need to know how to find it but now as things are moving to the wiki how to find it is just typing in search box and so it's really interesting seeing how much of that entire training flow has been eliminated where okay you can look it up instantly sure the time savings I mean that yeah it's got you know how much of the hour and then versus looking things up the thing is the metrics were never tracking how long it was taking for people to find that dusty old bookshelf and even if you could sort of generalize you know and even yeah you could show them you know that value and I think that you could get more buy in James did you have a response to her or was it separate so we have kind of standard training flows different levels of certification is it's different for each group but for our group specifically it's about a year after you start you get you get your certification in in our instructor flow and would be really great if we could say by use of the wiki we have produced that down to six months because then we could go to our management and say you know number of new employees times six months times average pay this is what it's worth but unfortunately it's not that that straightforward and I have no data to back this up but my general feel is that that our training flows are not any shorter they may actually even be a little bit longer but I think that our certification level is actually a lot harder now I think because it's so much easier to find all of the information now when when instructors go in that are certifying people they have a wealth of information that they just didn't really have access to before and it's like now I expect you to know this too which you weren't really even required required to before just because our entire corpus of knowledge is so readily available we kind of expect new people to know more of it than we did before the other element of savings it's tangentially related to saving to training but when it came to updating documents what was interesting in my group is that using the old paper system in which you had the serial signature process so you have all these people review the latest revision to the handbook and they would all be collected into this one rev and then this technical expert had to sign off of it and this was the second signature and then and just goes up for signatures they had a system of tracking that so you could see it was taking on the order of a year or longer to get the next rev to the document published and so just by switching a wiki not only not having that linear signature process sped up publishing new information on a paragraph level but if somebody wasn't doing their reviews and wasn't doing their signatures the rest of the team was sort of moving over like water flowing around the bottleneck so my next question is do you have lean signal trained people in NASA that you could leverage because you know they're trained to help do these small measurements you know reduction of cycle times and that's what I'm hearing from you and actually they can help you with those metrics I mean we use them all the time in our company and you know you just a quick meeting you tell them what you want and then they come in another quick meeting and then they because they're trained and they do all the metrics and stuff and then you talk about them and then you normalize them but you get at least you get a baseline of where you are and if you you've had the wiki for six years six seven years so to me it's a success it's just fine-tuning it and getting you know sort of that monetary number that people are looking for yeah I would love to have anything monetary out of this