 Like, I often don't know if I'm going to get a book of your thought and if I'll have comments and I'll not. Me too. I'll not ask for it. Right. Yeah. That's fine. Well, I think that's a good point. Well, and they don't understand this. They're like, why did you come to this? You really have a problem? It's a problem. You know what I mean? A diffusion is not, it's not, it's not a squirt. It is, right? It is. Well, I would argue that an idea is a game. Oh, right. It pertains to pure speech for me. Right. I mean, there's nothing in your numeric about it. Definitely. But I'm sure the situation in French will be worse. Is that okay? Yes. No, in the French, when you get to the second meter, you mark the situation and you start making new comments. I don't know if I'm doing it right, but I've been wrong before. I know that there's still a lot of things to do. I know that there's still a lot of things to do. I think that's a good point. I think that there's a lot of things to do. There's a lot of things to do. There's a lot of things to do. I mean, if it's no other unidentifiable. Pure speech. No, it's not. It's totally just something I've never heard of before. My understanding is that it's just a huge, intractable problem. Intractable problem. Where every time they approach it, they realize... It's just a matter of... It's like we should know these are both... It's a very annoying piece of... No, I mean, what I mean is that... Gid is Gid. And then you put your eyes on Gid. So it's a lot to me because I'm multiple. Oh, except that Gid is a... Gid really has no concept of this sort of social kind of... I actually have a lot of my life. There's some things to do someday to... I mean, it's crazy because that's what I want. I want to see the conversation that it was all about. That's what I want to see. And it's a true thing to do. Amen. Thank you. Yeah, I mean, I've not seen that done. But I'm sure they make sense. Maybe it's Microsoft. It's me. Any time you want to start. There's nobody here. Even better. Even better. That's right. Well, yeah, we've been doing that. We have Cordo. So Chase and I have a presentation. That's right, Chase. You're part of this presentation. Yes. Last year... Is somebody supposed to be here directing this? I think it just goes and whatever it sees, it sees. I don't believe it's recorded. I think we're supposed to have somebody try to take notes underneath your pad. There's probably some lists somewhere of things we're supposed to do. There's my card. I don't know if I'm still looking. It's smaller. Yeah. Somebody can yell at you at the same time. Yeah, if you find the schedule page on wiki.org, I think there's already a hotline. Do you mind watching IRC? Yeah, on the off chance that there's anybody on the planet. Hello, planet. Yeah, so about the time of Wikimedia, whatever it was, 2016, and... We were all in an IRC channel being upset that some tools were dying. And I said, God damn it, I should make a presentation and give it at Wikimedia. And this is my second dry run before I try to do it in Montreal. But the format here is quite a bit different, I think, than the Montreal one, where I will be talking out to people who are largely the editor community on wiki. What I really like from this talk is more ideas about things related to what hopefully we'll set up here at the beginning. So, developing community norms provide lots of tools, or don't let your brother-in-law be the only person who keeps your community running. At the beginning here, Chase is going to have to talk, because he needs some slides. We can go quickly. Some slides about language norms. What are the terms we're going to be using? Everyone, I think, here knows, but it's better to be explicit. So we have OpenStack Cloud, which we use as a service. You can have a DM or a project that has quota. And we, being the foundation, have our own sort of projects, which we run officially and unofficially, I would argue. Next slide. Tool Labs. Tool Labs is one of these projects. And it's a low-marrier-to-entry environment where you can, in theory, run write code and run it with the minimum amount of pop-up certain states. And then there are a lot of things running on the platform. We have long-running bots, short-running bots, one-time bots, recurring bots, which I guess would be scheduled by jobs, head-off process. You've got a bunch of people doing research, and then you have somewhere in excess of a thousand different web services that are written by everyone. We use Tool as a generic for all of these things. I think officially, when we send out the annual survey, there's officially a sense in there that says, we don't have a definition for what Tool is. It's all the things. If it's like a distinct entity that is meant to be run on the platform, then we call it Tool. And then it's really ham-fisted to put open source up at this conference. But the reason I did it is because as far as making our code available, we just stink at it. We have tons of people who have written all kinds of really important bots. And wherever the code lives, if they've deployed some binary thing half the time, we don't know. And the worst part about that is that historically it's just been fine. We've just kind of rolled with it. If you wrote something that now with media Germany depends on for all of their work every day and no one has the source code to it, historically that wasn't viewed with sort of the evil eye. Now it is, that's what we're saying. And then just what norms are basically expected behavior, community standards, technical, non-technical. So this is kind of a thesis for the talk that was basically happening. The tools are actually a vital, it is to say critical instead of vital, and MC gave me a whole bunch of run around about the word critical. So the word critical became vital. But a vital resource for automatically content creation and curation activities. This was kind of the rallying statement that got me the job that I started in April 2016 as community tech's outreach liaison slash helper, slash whatever to the tool labs environment. And I have some stuff here that I think to some extent backs up my thesis. Maybe it doesn't prove it. This is a number from a 90 day period from September to November of 2016. 24.67% of edits across all the keys on all the media run properties originated from IP addresses in the labs environment. If I had my speaker notes here, I've got some breakdowns too, but it was wiki data was on the order of 50% of wiki data and it came out. And I think just under 8% of the wiki edits. Is 53% of wiki data and 8.9% of the wiki. Thank you. And another for basically the same time period of 90 day window, 3.8 billion individual API requests originated from the labs IP space to the wiki media farm. And again, I don't have my speaker notes. I think this was about, let's say on the order of 10% of all the API requests that were made in the Action API. I like this number because any time you can say billion, it's pretty quick. And here's something that's a little less like Lucy, let's throw up some random numbers. So, Africa was a co-author of a paper in 2013 about CluBodyNG, which is just one of the up team Zilean bots that runs in labs. The time period measured was the first quarter of 2011. And the bot at that time was having some, it would go up and go down and go up and go down. And so he had this data that he could look at when it was up and CluBodyNG was doing what it does, which is basically revert things that look like obviously bad edits. The meantime to revert, it's about 12 and a half minutes for an edit that was eventually going to disappear to disappear. And when it was down, that time came close to double, so the bots a lot faster than human beings. But one of the interesting things in the paper was that things did eventually go away. So the bots aren't critical, but they're critical. Well, this is just one bot, right? I know most of the people who watch for edit revision history do some of the other bots. Yeah, this is just whenNG is just one very, very, very weak AI. It's just a little collection of characteristics that basically, if it sees an edit that blanks a page and says no, you can't do that. If it sees several other kinds of things like that, it backs them out. So, all tools are unique snowflakes. We have 100-ish tool accounts. Some are maintained by full-time professional software developers. Some are maintained by somebody who figured out how to use Stack Overflow last week. And I don't mean that to be disparaging. It's just the reality of it. Some tools have multiple maintainers that have 10 or 15 different people that will show up and fix them. Most tools have a single maintainer. Some tools use version control. Most tools don't. Well, that's maybe a little bit too much. Some do, some don't. They're written in just about every programming language that I've ever seen actually used to do anything in the real world. We have one that's written in tickle. But not just annual tickle. It's written in a custom compiled tickle that does something different than any other tickle that ever existed. They don't mean any of that to be disparaging about how anything is built or why anything is built or who does it. Everybody who's doing things in tool labs is doing awesome things that are within the bounds of capability and time and other resources that they have. But there's no, every tool is E-neased. There's no prototypical tool. There's no, this is how they all work. In this great and varying ecosystem, I've noticed a few things that go wrong more often than I would like them to. And so I've got two stories. And again, in the Wikimedia version of this talk, this is the meat of the whole thing. It's telling these like, there was this thing and this thing and this thing. We're going to kind of fast forward through that, but we'll get a little bit of it. So the first one is just the tiniest little thing, but it's the hugest thing. There's a tool that exists, it's a web service, and it does KML data files. So it makes data representations that can be overlaid on Google Maps or Google World to look at things. And it's used to extract geocordinates from articles and then make these overlays. And it is, there's a template on Ian Wiki, there's a template on German Wikipedia and several other Wiki, pretty large Wikis that have pretty highly transparent templates that make direct links in this tool. And it started having some problems. It was having frequent out-of-memory errors, it was crashing. People cared so much about it that they came on IRC to find us and say, hey, can you restart this web service because it's down? UV and Alhalla both over the course of about two months spent at least a week's worth of time trying to make this thing better, trying to keep it up, moving it. It was one of the first things that was moved under our Kubernetes service because we thought that the Kubernetes system would be more robust and stable for it. And things got better, but it wasn't great. And it got to the point that somebody filed a fabricator task proposing basically a rewrite method to fix it up. And the reason that they went that far was that they had tried on Wiki and on Fabricator to contact the sole tool linkator and give them to provide some information. Nothing happened to that initially. Eventually I noticed the fabricator task and I went and used my powers of bugging people every five minutes until they pay attention to me to get a response from the original author. And he had a perfectly fine story. He said like, hey, you know, I made that thing and it's cool that people use it, but I'm not interested anymore. I'm over here working in Wikidata now and I think they're doing better than some geo-coordinates. So I don't really care about that thing. But hey, if anybody wants to take it and take it over, that's fine, they should do that. And one of the problems with that that I pointed out in the request to him, there was no license for the tool. So it was a single Perl file sitting in a directory with no license information in header, no license information in body, footer, inside and anything. And his, their response on Fabricator was, yeah, software licenses, I don't know, just do whatever you want with it, I don't care. And the reason it's really bad is that we can't just do that, right? Just like I can't take some random thing I find on the street and put it on commons and say, well, I don't know, I found this thing on the street, nobody was looking after it and we need to take care of it. So it's on commons now, right? Like the communists would, rightly, you don't have license, you don't have ownership, you can't do that. In the United States, if you don't provide a license and you don't somehow release your intrinsic ownership copyright, I have to wait until 70 years after you're dead to take over your stuff. And that 70 years number is just going to keep getting larger as long as the mouse is still around. That's a little thing, that was supposed to be short, but it was really long. So this other one, I keep making eye contact with Brandon because I knew this slide was coming. Not a little thing, this particular death of a tool story is about one where every little thing all happened over the course of 283 days until there was no more that we could do for it. So there's a bot that's a very active editing bot on German Wikipedia. We noticed as we were doing the wind down tasks for the conversion from HTTP to HTTPS that this particular bot was pretty high on the leader list of things that we're still making on a regular basis in HTTP connections instead of HTTPS. And so I think Brandon made the first contact, Brandon pinged the author on the mega tracking bug that we were doing about all the things that were on the leaderboard. And the author showed up and said, hey, yeah, I'd love to fix my thing, but it's written in Java and you guys have a crappy old version of Java, and I just can't fix my code with the crappy old version of Java that you're running that thing with, 1.7 running on tool apps and they wanted 1.8. So that was written down, kind of got lost in the noise a little bit. I don't think we condemned it the author very well at the time, but we had already by this time investigated a Java 1.8 upgrade and had a pretty detailed task that laid out. These are all the things that we look for about how we can fix it and we can't do it right now. A couple months went by, we looked at the scoreboard again. I think at that point this particular user was the number one. I think we knocked off the ones above them and they were left at the very top. Pinged them again and fundamentally got the same response, moved on with their lives. Another month went by and another user who had been watching the mega tracking bug filed a new bug again asking for the Java upgrade. Again, I think the haul of this time went through and did try just about every idea that we could try to try to figure out how to get Java upgraded. The story kind of goes on like this in these bad circles and I'm not telling it with near the force that it makes my stomach hurt. We could upgrade the OS to get Java 1.8, but I think Redemption wouldn't work on it. If it stayed with Redemption, it would work but not get its job. Right, it was Redemption based. Yeah, that was basically it. So then we were getting near the end of the time window when we said we were going to keep the secret loophole open that kept all this stuff working. And I finally looked into what did this mock do because it was making a lot of requests. Were these requests actually useful and that's when I found out that it performed like two-thirds of the archiving tasks on all of German Wikipedia, like daily, weekly, monthly archiving and all kinds of other things. So we realized this is going to be disruptive if we just kill this sucker. So I tried reaching out to the author again and I didn't get any answer. So then I actually went and contacted Wikipedia Germany and see if they could hook us up with the author or with somebody else who would know something about it. We were at the Anyone Who Cares stage. Yeah, this is about probably about two weeks from shutdown. The last, like, really big blocker that we were in though. And we found out the author was on a wiki break, not a wiki break that they really wanted to be on, but they were on a wiki break and there wasn't really anything they were going to do about it. So we weren't going to be able to get their help for a while. So after much hemming and hawing about this isn't my job, but damn it, somebody's got to do it, so I rolled up my sleeves and jumped into the tool using the superpowers I had at the time as a co-container on tools. And found out that, yes the tool was written in Java, no there was no visible license, and no there was no visible source code. So there were compressed JAR files and shell scrapers that executed the compressed JAR files. So the first thing we tried was we tried to write our first proxy. We tried to make it not be Brandon's problem and instead be my problem and push him in between. That almost worked except some requests were getting redirected. Actually no, the majority of the requests were getting redirected, some were getting proxied. So I decided, okay well as long as I'm playing the hero, let's really play the hero. And I decompiled the JAR file and figured out what libraries he was using and found that he was using an Apache wrapper over the HTTP connections that through the beauty and wisdom of whoever wrote it in the Apache project didn't recognize, by default did not respect the standardized proxy flags that you can give to Java. But instead offered an opportunity to write your own custom shim to tell it how to use a proxy. That's pretty much the end of the story. At that point I was not willing to... There's one more minor point that you did too. At one point I was taking the author to claim that he switched over his license, then he was in a lower license. That was his whole statement on it, but then we're providing an indirect switch for us as well. Correct. Yeah, I think that came in the day before I said fuck it. Shut it down. There's an even longer version with more parts about this, but the perfect storm is that there were lots of little places along the way. The shutdown date was 283 days after the first contact date and there were lots of places along the way that something could have been made better by somebody if we'd done something. But it didn't happen. The bot got shut down. Luckily, during Wikipedia it still survived. Some people came in, wrote some new bots, took over the tasks. They were boosted savers to kind of loop some people in and add desperation loudly enough and also said... So the whole process of reverse engineering one single bot, it's completely not scalable and there are so many bots tools in this sort of state that if there comes a time when it's just not possible for the Royal League or people to do this continually. And so we essentially told that we will do anything to help you, but we don't have the ability to reinvent those. It must be important because they found people. Yeah, it must not have been that hard to fix because as far as I know like a week later... It was quite fast, yes. So yeah, thinking about that, that was about the time then that this talk came into being right around then. And one of the things that I talk about as those couple of examples and we have some others that have existed is that there are a few, what I think are pretty simple norms that we can borrow from most open source culture and try to promote that might help with these things. So this is kind of the list of things that... Again, I'm going to be speaking at Wikimedia too. Trying to get the body prover groups and the various groups that are on the Wikis that interact with these things where they turn into a giant pile of myths to try to kind of like adopt and promote to people. Pick a license. I'm going to work on... It's on my personal team roadmap for this January to March quarter to actually make that easier to do. And actually to try to integrate it into the act of creating a tool account itself that when you say, my tool is named foo, its license is bar. And if you don't put in a license, the former just keeps coming back and saying, like, no, I'm sorry, I don't see a license here. You need to have a license. And obviously, you know, like a lot of tools are hard to pick any license for so it won't be like you can't have another file that's in the directory that's underneath another license. If there's no license stated, this is the default license. Well, and it's worth noting that it's technically always better this way. We don't run code and it's not available to the source. It's just no one ever really worried about what that meant. You know what I mean? You can't run code source code on the platform, but we have code which is not available. So the distinction has become really meaningful when you go to actually pick something. Yeah. And it's fundamentally an enforcement problem, right? Like, we don't self-merge except, well, we all self-merge. So we're going to try to make it harder to get around that particularly. To publish the code, I'll make a tar ball once on every third Wednesday and put it somewhere. Use version control. Hey, I've got tools and I'll be glad to help you use version control. Do something. You and I have some ideas about lowering the bar for this even farther for people who don't like version control systems. Have multiple maintainers. This is one of the big ones for that. Both of them really, but fundamentally for the second use case. If there had been one other person on the planet who had any idea what this tool did and any idea how it did it, they probably would have also had source code. And they would have been able to either work on fixing it or help us help them learn. And that leads, I think, to multiple maintainers and to write some documentation kind of code hand in hand. We've made a special namespace on Wikitech in case there's nowhere else in the world that you can figure out how to document your stuff. Write down, how do I start this thing? How do I stop this thing? And ideally, again, some idea of what it does, what other resources does it use? Does this melt when redness is gone? And Chase said this last one, and it's a good one. I feel bad that I hadn't seen this last bullet point when I was writing things down before. Participate in the community. And by community is a loaded word, especially in the Wikimedia Foundation. But particularly here, we mean the Tool Labs Developer Community. And I think the next slide, here's the Labs Developer Community. We've got some mailing lists. We've got an IRC channel. We've got Wikitech Wiki. And once a year, two years running, so we can actually say annual survey now. That's great. We try to do a survey to ask you, like, do we suck? Do we not suck? Do we do something good? Do we have some feedback? It may not be highly visible yet how we're actioning on those surveys, but we do talk about them a lot. We're using them to change some of the direction as slowly about where we're going. So then, we've also been working, the whole Tool Labs community's been working with me kind of being the ringleader over the last six to eight months now on filling in some policies that were never there. Just stuff that nobody ever got around to make it up before. So we've got two that went through drafts and comments and reviews and all the way to RSC votes on Meta approved by the community. In the final stages now of both of these policies, I'll talk a little bit more about them, involve a volunteer committee to help make them happen, to help get action on them. And we're in the final stages of putting together that committee. So hopefully over the next couple of weeks, we'll be able to send out some nice announcements about yours or shiny new committee and your shiny new policies. Let's do some stuff. The two that came up with the right to fork policy is kind of a codification of our existing open source requirement. And what it fundamentally says is that every tool has to have an OSI approved license. You can do a license if you want, if you want to do it under MIT plus what the public license. You can do that, but one of them's got to be an OSI non-crayon license. That's the only must in the policy for a tool maintainer. And then there's a whole bunch of shoulds. Should publish your code. Should have multiple maintainers. Should, should, should. And then the place where the committee comes in, then it defines for anybody who's not the maintainer of a tool a path to say, I would like to make this tool better. And I've talked to the maintainer and they don't want me to fix their tool. So I'd like a copy of it to start from. I'd like to fork it and go do my thing. And it gives a procedure, a general checklist procedure about how to do that. And then the place where the committee comes in is that the committee is kind of the arbiter between the current maintainer slash copyright owner slash whatever, and the people who want to do it to try to facilitate that and get it done. And the abandon tool policy builds on copyright fork policy. I would really like to see people fork. Yeah. So part of that whole deal is sort of born out of some social etiquette particulars that come up quite often, which is one user has a tool, someone shows up, it's down. We haven't seen the primary maintainer in who knows. But that's not unusual. We have people who do it only when they're at college. We have people who disappear for six weeks during the holidays. Every schedule you can imagine. Not being able to get a hold of someone for a month is not crazy. But what do we do? It's somewhat not polite to just add a co-maintainer for them without talking to them, but yet their tool made me try. And historically we've sort of gone with a desperation slash consensus model, which says, if a bunch of people show up and say this is important and no one who knows about it is around and you're willing to fix it, then we're gonna add you and we're just gonna do our best to let the original people know because that's the best we can do. But with some kind of, I don't know that oversight's the right word, but with some committee in place to kind of keep everyone honest and so that folks know up front. If you wrote a thing and it's awesome and people are really using it, try to be communicative because if not, you may end up with a co-maintainer. So I think it's sort of around just establishing these social norms that are so odd because we have so many state projects and the people who control that may or may not be available depending on the place of the living. Yeah, so that forcibly adding a co-maintainer is step one in the abandoned tool policy. That's adoption. And then if you adopt successfully and everything goes well and the original maintainer never shows up or the original maintainer comes back, definitely. Is there a kind of co-maintainer if you're more responsive when it's at 60 days or something and you're gonna start a co-maintainer? Yeah, we put some stuff in there and we tried to strike a balance between long enough that it was annoying that people wouldn't come in like five seconds and sit down and say, like you need to put me in there now, but not so long that the tool is down forever. What is that first way of time before we might look at the co-maintainer? I think we... It's written in the policies on what you're taking. It's 14 days, which three various drafts over a year and a half that went from 90 days to three days that I think we did at 94. So that's... Yeah, I mean... And so that's for the... There's like two stages of the abandoned tool. There's adoption and usurpation. And so adoption is you get at it as a co-maintainer and you can go in there and try to fluff it up. And then usurpation is now it's yours and you can kick the old person out. And that can't happen. There's a much longer timeline for that and a whole bunch of the things that are shoulds in the right to foreign policy have to flip to musts. So you have to basically show that you're an exemplar maintainer and you're doing everything just right and okay, now it's yours. I think that we... The tool at that is not all of which by any means are foundation employees pretty much always deserve the right to go and kick it then. So that's just part of the deal. Yeah. And we'll see how it works. Like this is a policy written on a wiki. So we can make changes to it as it goes along. That a big part of both of these for me too was putting a volunteer committee in control. This is not WMF action by fiat. This is your peers who are doing the same things you are who are all the comedians and they're taking care of it. Speaking of which, there are people on these committees who are here. They're not in this room. So... You may watch this video later. Oh, you're here. No, no, I'm not. I had a question. So would this be a long wiki process, some of it like cross-prediction or something like that? Yeah, we think very similar. So that part is not strictly codified yet because I want to get the committee together and let the committee do it. I basically want to treat them like the storage group. These are really smart, dedicated people who are going to do good things. So let's give them some space and let them figure out what's going to work. Because again, it's not my radio. It's not the WMF's baby. This is about full-labs community building. Yeah. And there have already actually been suggestions that it gets expanded beyond full-labs to all of labs. But we want to... Let's crawl before we run. Extensions. Yeah. Yeah. But on wiki, it's not such a big problem. And extensions, it's not such a big problem, right? Because an extension doesn't go on to clustering lessons in Gary. And when it's in Garrett, it's owned by everything. Somebody may be the one that we look to, the plus two things, but it's owned by everybody. Tools, again, because every one's a unique snowflake, they're a little different. So yeah, it's better to adapt, right? We don't know that these rules are going to work. I don't know that the proposed FOSS kind of guidelines are going to work. But we want to get started and then fix it as we go along. I think the committee ended up being seven people. Yeah. None of whom are foundation employees. And we got to turn... Oh, wait. Okay, one person. I assume that's Nick. Yeah. And we turned people down. So there's interest. That's the interesting part. Yeah. Yeah, this was all really much better received than I had been led to believe. It was going to be when I started off til they get the window. So this is really... Now we're at the part, and hopefully I didn't burn up all the time just sweating and yammering up here. Now we're at the part where I want feedback from anybody who's interested in giving feedback in this room about, like, beyond what I've talked about, what are we missing? What else can we do? Like, what kind of tools? So, like, one thing I think would be cool is to implement all this, which all this is really awesome to see happening. With me, I don't know, maybe we do this categorization or something, but the idea of, like tying an SLA to a checklist of practices. Like, sure, we have all these norms and there's just things we want you to do. But, you know, if you do the bare minimum, you know, then we make a certain effort to keep your tool alive. But, you know, if you have checked all 10 boxes and been verified, they have a published source code that's home sourced with a list of group languages that have a repository, blah, blah, blah. You know, we put a different flag over it, so if we guarantee more maintainability, then we can work with this. Right. And then that flag system kind of gives also the Wiki communities, you know, D, Wiki can say, well, we rely on those 10 boxes and we look at that list, eight of them are in a really good SLA category or two of them are not so well maintained categories. Maybe we should be concerned about them, totally. When some kind of concept like that I wrote, so I have this task that I've been assigned to me for like two years, and something in me just won't let it go. I have no other tasks that I haven't unassigned because I haven't closed them. With this one task, I just, something in me can't read the English. And that is essentially two defined support levels across labs, which is more general, but that's the idea we came up with, which is markers that indicate basically your commitment, which allows us to mirror, you know what I mean, or at least ideally. The barriers there are, we honestly probably don't have time to reflect the goodwill of everybody. If everyone showed up and did everything right, we'd be like, oh, we're screwed now. Which may be a decent place to be, I don't know. And then, I think the other thing is, we're working towards that checklist, and we just haven't written it down. I think we all have, right, we've internalized some version of that. And what happens is, some bot tool goes down, you go and look at it, and you run through, okay, there's not a license, and it's written in mono, and whatever, written in C-sharp, and it's running through the mono, and whatever. And maybe you're thinking, oh dear God. But, eventually, we usually come out the other side, telling the person who maintains the tool, like, here's what we can offer, you know, and it might not be much, but something. Well, it would just be good to have those up front, right, because eventually when you hit that emergency, and that tool is down and someone is trying to fix it, you're going to run into well, here's the slowdown, here's the bottlenecks for why we couldn't fix it for you very easily, because you did all these road blocking things that are not necessarily the best norms. At least you can, instead of flagging, instead of figuring out how to flag those things on the front, you can categorize to the support levels, you know. Yeah. If you check all these boxes, not all of which are strictly required, but you can go outside that, you can check all these boxes, you know, we're going to provide you a higher level of support and flag your bot as more improved. Do you think, I don't, I don't, I like the idea of having a video camera and it's like that for people that can fly, but the thing is that it's, certainly in Brian's examples, the people that suffered when the tools died were not the people that were able to use, and so, so the incentives aren't aligned very well. But see, if there was some way for, I think that we could do that by having, ultimately the status being a flag on that tool, like when it makes edits on DEE, it's by whatever bot, and it's got a little icon next to it that says like, super good bot versus questionable bot. Right, so all these evidence are mapping from this not very well maintained bot. Yeah, so I use this financial app to track our stuff and it analyzes your stuff, like, do you have any money, say, do you spend more than you make and it does this, like, yeah, or, oh no, and it also has an exit, it has like, how do you compare to your peers? We broke, and some version of that would be cool. And as far as like, defining what those checkboxes are, but obviously, these are two things that we're committing all of that and they're kind of norms and standards and these policies are just starting to come. And at the other end of the spectrum, we can look at where we, at least in some areas, define production practices, right? Because the ultimate goal, moving up that chain is your code is so important and so well maintained and it moves into, like, immediately with the extension as part of our internal stuff and everything, right? So that's, I'm just saying that in the second half of the spectrum, it's like a drum list of languages that we support without production developers and like, we're able to get prepository that they can hear. Right. It provides it's hard to attain the checkboxes toward the categories, right? Well, the most extreme example would be if we got the waiting people to decline a lot of water. Yeah. And that's, that's the one that we found out that's the, like, yeah, right? Like, but it, like, but, and your, your idea is a good one, like, and then we, uh, it actually came up, uh, Valhalla was at, uh, an Edithon, a well-known volunteer. Yeah. Sorry. Sorry, yeah, yeah, he's a, he's one of the two volunteer admins in Toollabs. So he's gotten full-routed in Toollabs and he does all kinds of awesome stuff for us. Um, he was at, uh, an Edithon somewhere around and I had him, he was like, I'm going to meet these people. What should I say, though? I was like, ask him about this stuff. And that was one of the suggestions they came back with, was some sort of, you know, like A plus to D minus sort of rating system that could somehow be paid-visible. Like, how to make it visible, that's, that's a whole other question. Yeah. I, I really feel the pain. And so, what I worry about is I end up causing more pain in that room. Like, before, you know, I've heard that, like, there's a year-long pain for the whole foundation. But like, those changes are coming even more rapidly. You know, I've now wrote it. And like, when we're looking at those from a public-facing perspective, we're not trying to be very opinionated. Usually, when we finally kill something from a public perspective, it's like, and there's this .01% of crappy browsers from the world that aren't compatible with this and we're really feeling like, things that haven't been passed in eight years. But now we know that, like, some of them also fall that far behind on software levels and so on. And, I mean, the pace of innovation to change and the necessity of change for security reasons and all that will continue to increase and we will have more of those purposeful moments. You know, you know, totally. Well, it's all happening in a way, because we're, you know, in science. And so, yeah, I guess. Yeah, yeah, we're going through another long-tail phase of it right now. So we have this, I'm sorry, go ahead. Oh, no, I'm just going to go through the internal narrative and like the labs team, which is, we're really overloaded and there's just no way we can get all of this done. That's a good thing. The platform's really popular and like, there's too many tools for us to support and, you know, all these people are showing up saying, these tools are down and it's really impactful to their workflow and we're like, I'm not sure how we're going to get to this, but the platform's really used a lot. Like, that's really kind of amazing to me. So, we have this sort of so, I'm thinking about the issue around like, a lot of people groups in the different projects and like, a lot of them, I don't want to require open sourcing code and what I've written along these pages. Like, a lot of the arguments are made and it's like, oh, this could be too dangerous for anyone to be able to use in the vandal on this amateur game, you know. But I'm wondering if there could be some sort of offering like, private repository or something like that. Do you have to offer a lot of people groups and say, if you want to develop the code not open source to the world, but at least like a central thing, here's what you could require as opposed to saying, I don't buy into that. So, I mean, it's a topic that's come up before and the potential for a code escrow service, I mean, it's there, but I do not believe any of the arguments that there is such super-secret mega-sauce in any of these tiles and red-axes. I mean, I'm not in the position they are to actually affect any of this that I'd be my patient with as well, those are the lowest arguments I mean, we probably shouldn't entertain those arguments because we should set people straight and that's not the way to do this. Wait, wait, and I think the other way to look at it is to help people understand like, honestly, a lot of people don't want to share their source code because they're ashamed of their source code. Right, and so the other thing to really help people understand is that we're all ashamed of our source code. Right? So one of the more returning points as a developer is when you realize your home for the socks they always build and everybody's for the socks they always build and there's no reason to be ashamed. Right. I have talked to a lot of okay, this is random hearsay, but I've talked to several foundation employees that that was the scariest thing for their first six months of working here was that every code review so we don't I'll push you then for like, why don't we work with them and require open source in the code it has always been a requirement Well, so there's a difference between licensing and publication Yeah and it's always been a requirement to have a license license Ryan, if Ryan Lane was here he could tell us what his intent was when he wrote some big words on the wiki or something and somebody had that license to it or else there's no license license to it Yeah It's hand-waving you know what I mean Yeah but yeah I mean that would be my suggestion Honestly, in rare cases where there is a need for a secret it should be secret code it should be secret data and that's a whole different nation a resource code that loads the time instead of secret data Yeah, and there are some things we're hoping to work towards that especially with the Kubernetes migration and the whole system built inside of it for and when that's separating Yeah I have one I just want to give you a Yeah five Thank you I have one too and now I'm going to do that because we started service and I discovered that and it's probably my own and I don't get the results so I'm not sure I'd love to monitor you monitor your metrics Yeah and you'd love to have it on that So in theory the big brother system tries to keep great engine submitted jobs that are set up with it right in theory in practice it may or may not work for years but funny thing about trying to write your own reconciliation system on top of a great engine is that web service has it now built to Well I think he's primarily talking about verification regardless of auto-detection Yeah, because if you don't don't restart it We're terrified Some people have written their own and there's a project which either we just created and coded to remember where several tool apps were basically trying to solve the problem on their own but it's been we don't have a few things in this arena or a lot of things we don't have like a hey this would be a great release process for you to do on this platform you know what I mean like here's sort of the vanilla version of that maybe you need to add something to the beginning at the end but it makes it predictable for everybody we don't have a good log aggregation we don't have any log aggregation system and that's like a ginormous main point and then we also don't have like real deal user tool monitor I mean we have people who have written their own we have a lot of ad hoc stuff and then we have the grid has some function for like if your job gets killed unexpectedly they'll send it in whatever but it's all pretty reading so I could say that you're just right what you want is perfectly reasonable yet it doesn't exist yet and in my mind those things have standards like so everyone because one thing about even if someone has a dog hadn't been proactive and written something to monitor their tool that's awesome but then like I have to not only figure out your tool but like how you're monitoring it what that means and everyone's doing this slightly different or we have a lot of people who have maybe they've used like the native I found wrong rotation but more than likely not they probably wrote some thing that just offers a bunch of files and who knows what shuts their own service down and doesn't move on a directory and you know eventually that falls apart and they don't know if it's us or them but the real life value of each there is that there's no one single sort of curator way to do it and that is in my mind sort of on this list next if there was another slide here definitely and we know it's not you it's us yeah but I think the future is one of these future possibilities it's like a just a standard for monitoring yeah totally like we expect everybody to you know if they want good support to write a tool that conforms to that like a script that does a nodgeo style exit codes and checks whoever needs to check about process states and provide you an email address right or something like that I mean we standardize something we can write a script for the exit code and how does whatever you want to talk about the tool for writing an ability yeah the hardest part for me about that like I totally like that is an answer right like that is the answer we would use in production that is the that's a one percenter solution right like I know that most of the people that are using tool labs to do stuff they can't do it they can't do it because they don't have the time they don't have the energy they don't have to communicate right like if they could do that stuff we would have already hired them and they would be there's stuff they could produce right or they would be constantly fending us off from trying to hire them yeah but so some some things we're not going to make any promises because making promises is there's a way to just make people sad but we were trying last quarter and I don't know that we've necessarily succeeded but we've been trying to collect a series of requirements for a true platform as a service product so replace open grid engine with we're replacing open grid engine with Kubernetes that's that is a known thing that is happening that's in progress what we would like to do is rather than make our own collection of pearl and Python scripts that figure out how to put jobs into Kubernetes which aren't called jobs anymore but let's just say jobs into Kubernetes we're talking about the push to the play a second yeah we would like to get a true a third party developed open source well supported platform as a service layer to put on top of Kubernetes and that's certainly one of the criteria that we've got on the checklist is what kind of monitoring what kind of notification what kind of what kind of feedback loops does it have to help people we've got a whole lot of eggs of hope in the basket could open shift or DS or some other product that we haven't yet identified that exists in the marketplace give us you know give us a big boost right like not just like a little incremental like it's a little bit better but like no look at all this stuff because when we bring that out it's gonna hurt people right like it's gonna be worse than the tool server to lapse migration it's going to be a whole other migration where I have to put my stuff in some different format do some different things I don't have any time and 20% of tools are just going to disappear because of it right when you get it you're just going to be like I don't even care anymore so it needs to it needs to fulfill lots of some pretty good things for us to make people that made for my mind that maybe one of the better meta things going on we have an occasional sea change to shake things up and everybody would look at their stuff about data you know yeah it's when the infrastructure goes stagnant for like five years running and it needs to get really out yeah shake things up in the long now view yeah like in the we're a movement that's trying to distribute collect all information and knowledge so that we can redistribute to everybody stuff all washes out like all these short term like oh my god and half of people left and they're mad at me like that's fine because 75 years from now those people aren't around to be mad at me anymore anymore right that's this you eat much better than I do also one other thing is is that kind of what you're talking about all exist natively in kubernetes like right now if you write a web service and you go to run it on kubernetes we essentially like lobbed up a deployment which then creates a single replica set which tells the kubernetes stuff that basically the manager the reconciliation loop that says is it running should it be running and we basically keep one instance going at all times right and the way that it does that is essentially is the container up and then is there a health check which is a native mechanism that allows it to be like more contextual about what is it of meaning so we have that baked in but part of the issue is once we present it the first time to users as if we want them to use it it better be the way that we want them to use it for the next X number of years because that's a really expensive thing to give your mind around and the underlying mechanism right now it's pretty low level I'm not saying that people aren't smart enough but I'm not sold on sort of evangelizing either but I mean it's there hopefully the there's two kind of contenders for a pass like solution Red Hat has one sort of a gorilla in the room called Open Shift which I believe is something I've seen presentation or whatever and then there's another place that is sort of like the Open Shift model is let's wrap everything up let's put a pretty little bow on it so we can charge people we have you know the Red Hat it's their whole thing right like we have an Open Shift origin which then becomes our product which we can now sell and there's no real difference and in fact when we went to Qcon whatever that was over in November a lot of the coming access control stuff is all being backboarded from Open Shift to Core and Kubernetes so I think it's pretty tight integration there and it's possible but the alternative is a third party called Deus and I think their product is also called Workflow and their whole model is essentially you should run all the components that enable this push to the play model as containers and it's all just kind of baked in you know what I mean like they eat their own dog food for running the sort of layers that flush this out I don't know which is better I think that one might be more flexible but Open Shift is cool I'm hoping that one of those two things makes that model that you're talking about like one thing that might be done today is there's plus oh maybe not but I know one that will do a health check for free on like 500 points is that something that people write or is there something that like no that's free if you if you can find some uptime monitoring as a service thing out on the universe and you want to point it at your web service do it yeah and that was the project that Chase mentioned earlier that there's a I think at least a couple of volunteers that are involved in trying to spring that up they're basically trying to build a labs project next to tool labs that would just do that that its job would be to be at that the problem there is you can't tell if there's no connectivity into labs outside of labs right that kind of that's awesome and I hope they can succeed I'm not sure that's the long-term solution I don't know that it's magically scalable but it's worth letting somebody give it a shot basically if you're interested in solving this problem we're happy to like sure yeah well so the Kubernetes VMs pieces are OpenStack themed so basically OpenStack is the infrastructure to service it handles the networking networking compute memory storage stuff for us today and then Kubernetes sends in pieces on top of that and it it does that's on the step about it yeah there are probably standard things all of them in Kubernetes it's regular it's not that's not all of them and I think that's not necessarily like we could run Kubernetes on very much like so in the grid that they're setting up in production I believe it's on very low also we could use NEET which is a container orchestration you know kind of a narrow Kubernetes on but besides it can be it's and it might just be it's it's I guess we rambled through all of it great time now but if there are things people want to keep talking about these things I'm around for the rest of the day I'll be here tomorrow Fabricator IRC email BD808 Fabricators probably the best for a lot of these kind of like longer theory like the monitoring things and so then I think if you go I don't have the numbers memorized because I'm not that guy but I'm pretty sure if you go look at the tools labs project you'll see a big ticket that says Jesus Christ why don't we have a monitoring and I don't think there's a good place to vent on these people tomorrow but on the other side you can stall for three years right at least it's persistent so the next time we have the conversation we can go back and say hi we're going to be right beside you okay thanks yep thank you