 Okay, all right Welcome everybody. Thanks for interest. Wow pretty crowded This is Kim Jill. I'm under clapper. We are 50% of the engineering community team at the wikimedia foundation and The talk is called wikimedia adopts fabricator deprecate seven infrastructure tools So this is basically what we've been working on for the last maybe nine or 12 months I I looked at the talk description again. I realized there's a few aspects that you might be interested in There's there's a part that's the social process decision-making There's the migration process itself And the functionality of this tool that we're using now fabricator itself So you might be here for different reasons I tried to cover everything a bit and you might all be disappointed in the end because it wasn't in depth enough So first of all wikimedia It's a project and community behind wikipedia Commons and a few other ones. There's volunteers. There's stuff of the wikimedia foundation. We tour stuff Development engineering teams They all have the different needs workflows tools or heads depends Of course also projects like core extensions So what I want to say is there's quite some diversity in teams and attitudes and workflows and tools But I think we can say that free and open source software and free knowledge are kind of like shared core values So to to explain where we were about one year ago The main tool was bugzilla instance You see bug report here with lots of interesting ui fields. It's just the top who has who has used bugzilla before Hands up I see Next one is rt Hands up Okay, rt was mostly used by by the operations team in wikimedia foundation This is mingle not very popular. I see well at least a few Yeah, it's yeah, okay wikimedia stuff So you so you see the card view here It's this is not a task or a ticket like the two previous ones This is a function that for example bugzilla doesn't have and you see the columns and you can move the cards and Um, we also had a I think this is trello. Yeah a trello instance trello anyone. Well, that's more popular. Okay, cool And for code review we still have a garret instance Garret anybody well also quite a lot. Okay And uh, we have and had all these tools and uh in between these tools we had some interactions for example a hook to create Comments from garrets in bugzilla tickets And as some of our teams, uh, like to use mingle and trello for the project management We also had self written custom scripts like called mingle and bugello to sync a bit between bugzilla and mingle and bugzilla and trello and all these funny things You might see the problems with the setup Collaboration between staff and volunteers and also in between teams Communication openness transparency accountability Um, for some stuff you needed a login even like like mingle or trello was harder to reach rt. Some people said it was a black hole Duplication divide we had to maintain these scripts Things broke sometimes for example once we upgraded bugzilla then the garret notifications didn't work anymore for version 4.4.5 or something um programming languages media wiki the the main thing is php, but um That was pearl and java and uh, we weren't it wasn't that easy for for people in the community to even do what they want Uh, then onboarding new contributors Learning curve is pretty steep the more tools you have and people nowadays are a bit used to github um, yeah and bugzilla simply Isn't for project management. So everything all together slowed us down So the idea was to find something better or uh, at least discuss if we can find something better with clearly developers as the main audience and not for example Editors or or readers of wikipedia or something um, the the problem as usual with These things is if you want to come up with a new standard, it's uh in your community You might end up with just one more tool. So that was something we wanted to avoid um, so what we did uh in i think around May march or april last year was we invited teams and developers to describe their needs On the page that was december january. That was december january. Okay All right Cool. Uh, thank you. Okay. Um, so so we asked a few simple questions. Uh, we had 24 replies Uh, by the way, um All the stuff you see here is of course on media wiki.org. So, uh, you can look it up 24 replies but from 24 types of profile that you think that you know, we have more than 500 people But really like we we we went into depth into different roles in the community in our organization, etc um, so so we collected these that feedback, uh, then Uh, we consolidated it into must versus would like to do this or that in in the tool I'm using for my project Split that into several categories. So we we got a basic idea basically what what are the needs? what what do the people want and um Then we asked people to propose tools that could be options because Developers and in general people have opinions. So Normally, they also have ideas, uh, which projects they like And I have to speak like this. Yeah, I'm sorry. We only have one microphone. We are really good colleagues So no and we did that because uh as if you if you enter into A discussion like this in your community probably the first thing you will get is oh, I like this tool I like this tool and you're discussing basically your preferred tools and you're not discussing what you really need to discuss Which is what you want to do. What's the current problem, etc, etc. So that was the reason for the survey um, and after that Yeah, we had several tool options at the beginning. Um, we Encouraged people to discuss to investigate the functionality too And then we were after decreasing the proposals the items in that list that we considered We encouraged people to set up test sensors. We have wikimedia labs Where we can run or install any kind of stuff um And and the feedback quickly Turned into this one question like shall we move to fabricator? We we had looked at seven or eight other tools sometimes also combinations Uh, shall we only use that part or shall we use this part together with that from from some applications? But Really we went we went two or three weeks. I guess it was like fabricator seems to be Best fit for the needs. Um And then together broader community support and opinion Because yeah, we just had a few dozens of people discussing this We had a three-week request for comments, uh, which is also a wikimedia specific thing where we Asked the community put up banners on pages Communicating widely basically And that rfc after three weeks, uh, it seemed that there was a general support for moving From our infrastructure tools, um to fabricator So that was as I said, there was also another proposal Proposal just use a part of fabricator and keep some other stuff But generally everything so and fabricator is according to wikipedia if you can trust that suite of web based software development collaboration tools um originally internally at facebook it's in php and it's under apache two And uh, I'll have more screenshots in there. Uh, and if if you're Can't wait fabricator dot wikimedia dot org, but it's it's a software forge So it has several applications well integrated into each other And you see there's there's a few core applications for example for for tasks and bug reports where something called manifest You have you have code review, uh notifications At some point Continuous integration Yeah stuff But the thing is it's well integrated. So it things linked to each other Um, so as we had this rc, uh, let's move to fabricator. Uh, how to approach that. Um, we Created a team basically it was cross team, but you needed you needed, uh, some expertise from some areas So, um from from the release engineering, uh core team We have somebody with php skills and and he also had maintained a fabricator instance before Lower level stuff operations. Um The migration script itself somebody had to write it. There was nothing So that was also done by uh chase who's who's in operations. Uh communication Wide uh in lots of places with with lots of stakeholders With the community for example with with the upstream project because hey, we're interested and we'll probably come up with some needs Yeah, the debug management aspect itself and, um, trying to find all these other small bits that might fall through the cracks Because you probably always forget something Because nobody has done something like that before with with that size and there was no migration script anywhere And of course, um product management release management that that you have contact partners there, um That you can discuss the needs, um And what what I consider is especially important the relationship with upstream. Um, you must understand their model their attitudes Uh discuss what's feasible and for fabricator upstream. I I really have to say, um, they're really clear on priorities They're They answer unbelievably quickly, but they the answer might not what you want to be what you want to hear They're pretty clear with one fix this is not in our plans Our priorities priorities currently are different if you want to challenge them. You need really good arguments But if you want to hack this up yourself custom here's the explanation and Really well technically written the eight or nine things you need to consider So upstream is very clear on priorities And very helpful and very responsive Something amazing about upstream is that you see fabricator and all they have and this is basically three maintainers and It's amazing. I just have to say it's uh You have to you have to see to believe and three maintainers that they develop all that stuff But they also give you a great support. So even when they tell you know They tell you know in a way and with an explanation. I say, well, okay fair enough So it was an amazing it's an amazing experience to work with them Um, so the basic steps we performed was um, we had this test instance instance still, um, but uh Our first aim was set up the production instance Then also Yeah, after this number one, uh use that production instance itself for planning. It's like dog fooding Um, but that also means that you already create tickets in there and these tickets have a number So, uh, then, uh, we set up another test instance with Buxilla data migrated so people could see like what does this script do how would it look like What is all the data you dropped? You can have wonderful bike chat discussions about this like which priority values do we want? So you also need to manage that part somehow And then the actual migration of the Buxilla data We did that in late november over four or five days And we also migrated rt in december, but that was not too complicated But just took eight hours or something Um and way less box only one thing we didn't think about before with rt with Buxilla. We had a few more things that we didn't expect it Um, we identified the task for each of these steps You have this list, uh, we have a list on wiki for for the migration itself step by step who's doing what But uh, we also had this on fabricator, of course because we've been using that tool for planning this actually and uh communicating we asked Weeks before we migrated Buxilla our users, please please register already in fabricator Because we somehow needed to link the accounts and the data so We can still do this with people who who haven't Registered yet, but the more we could already handle in these four or five days of of users the better Um, yeah, this is pretty close to how how fabricator looks I mean this is a test instance So some colors are different and and some some decisions how how to tag Things we've changed in the meantime and the ui might have changed a very little bit, but um This was the test instance. We we set up. We asked the community for two or three weeks To play with it. Uh, we got 45 buck reports 25 uh, we fixed so that was really helpful. I recommend that um Yeah, and then this uh weekend came where we Migrated this was the wiki page. Uh, normal people got uh redirected to while a few people still had access Um So these are just the first steps to the complete list is on the wiki, but switching back to that to read only Was an interesting hack Yes, please interrupt me so here's the Before you was the bug master now, this is the fab master. Uh, can I ask you for some numbers because I don't I don't The volume of the whole thing is what counts. I mean, so just uh, how many monthly users we have more or less Yeah, you've got a good point. Um With with bugzilla we uh had about We had 20 000 user accounts. Um We had 75 000 tickets. Uh, we had on average 500 users per month logging in I mean it's it's been running for 10 years. Um And How many bugs created on a daily or weekly basis? More or less On a weekly basis. I think in bugzilla it was like 250 would be my guess roughly This has increased uh, since since we're using fabricator. So, um All these two stress all these two stress that we are talking about a big machine Used by a huge amount of people very different people and basically we are telling them So we're going to change your car. Uh, by the way, we're going to be offline. Maybe four days five. I don't know Uh, it might even might even be that we have to go back because nobody has done this before So this this was the real context that we had so don't don't only look at the technical work Look at the social situation. Um, where you can lose your head easily just for I don't know if you have ever attempted to change the homepage of your project And the discussion is my ring only the homepage So now saying, you know, I think this is an important point that you need to take into account We get more tickets per week now simply because we've invited way more teams. Uh, also non-engineering teams, uh, into fabricator Um, it's not about bugs anymore. It's not about bugs. It's tasks now. So it's it's a wider field and, um, so so we we've been running fabricator with with uh, bugzilla and our team Well, we grade it now for, uh After that for full two full months and so far the numbers are too early, but 620 670 active users per month This january was 670. So so far it's it's increasing compared to uh, bugzilla That felt like slowly going down because less and less teams accepted it because you couldn't really do project management Um, yeah, so so all these funny steps, uh work around known bugs Um fetching the tickets uh from bugzilla uh took like five hours creating tasks Commons attachments and fabricator took 25 hours Uh, we had a separate database like not directly uh, also, um creating tasks, uh, for example one wonderful. I think slightly bike share tickets was uh As as we already had tickets in their production instance, we couldn't map like Bugzilla ticket number one will be fabricator ticket number one Manifest the the bug task tool in fabricator t1 But so we we try to at least have have it linear, but uh, that wasn't too great for performance, but uh Chase from operations who wrote that script uh somehow managed so, um May I ask a question? So how many of you are in a project where bug number one is something emotional and significant? No raise your hands if you are Well, our bug number one was Fix all the docs so fix the I don't know broken documentation and it's talking back with And we had really a lot of discussion because we are going to lose our bug number one But apparently we just We ended up with the rule a bugzilla number plus plus 2000 Is the fabricator number and we also set up redirects, uh for bugzilla IDs. Um, and um Of course that required to change the bugzilla url from bugzilla wikimedia.org to old bugzilla wikimedia.org And then uh, yeah claiming accounts as we asked people to register before we migrate bugzilla We had a script for that For the most active bugzilla user this took like about 15 minutes um Here you see before and after so this is our bugzilla import script Um, and on that side on on the uh, right. Yeah, right You see, um The the actual accounts after they've they've been connected to the comments and attachments and everything Yeah, okay, this is slightly edited here. So it's on the same level Um, so yeah, we we uh on that weekend we had to test a lot of stuff like access restrictions We we have a common a custom extension to For for access restrictions to certain tickets that might be security vulnerabilities or so That's not an upstream. Um, we also have a custom extension For dealing with sprints now You have lots of other small bits to do around in your environment on wiki templates I think I I edited dozens of of pages of documentation that link to bugzilla that now need to link to fabricator And you run into problems in bugs. You never ever expected to happen. Um, which can be, um Interesting. So we didn't think about moving bugzilla like, uh, yeah, we need a new certificate for that subdomain. Well Um, you had something like private comments in bugzilla and our script when, uh um Putting the the account data together or claiming it again Uh, it's overwrote that flag made invisible again, which Uh, wasn't too good. Um, well, we fixed I really got to say we uh, we are mostly chase and mukunda As as developers with the help of daniel another developer in the foundation and also with the help of upstream upstream because, um Even from upstream he was available on a on a sunday morning for him to, uh, fix our issue with the search indexer performance uh, it was an upstream issue, um And yeah, so, uh, I I blocked about details and you had something. Yes So this was for people. We had never met face-to-face Like, you know always online And this was one of those weekends where you end up being really good friends, you know one of those So, yeah, and we finally met last week in the same place of four of us. That was nice as well true Yeah, it's people you deal with basically every day you chat at least for an hour and yeah um, so All in all it went surprisingly well I was afraid of two things after the bugzilla migration because that was really the biggest Tool we had and the most popular one when it comes to numbers of users and opinions um, so I was afraid of two things, um One is after migrating bugzilla that everybody will come up with I have this great idea I can I do this in fabricator and the second one was Maybe some vocal people somewhere whatever staff community That might there might be some backlash. Oh, I didn't like I don't like this Why did you do this or that and and the second one? Didn't really happen which made me happy because then you can still get some sleep at night And if you only need to deal with the first one um First of all compared Okay compared to bugzilla we have a unified login So my naive idea so far is that's maybe also why we have so far a higher number of of users compared to bugzilla and other stuff um That that was together with uh exposing email addresses the two main complaints are ever overheard when it comes to bugzilla Uh, when you compare front pages, this is the latest upstream development version of bugzilla And this is the front page of fabricator where you have the tools here on the side Um, and some feeds over there with with latest stuff. I I have notifications up here telling me about about stuff. I'm watching Um, and if I click here on manifest, uh, which is where I spend most of the time, um You you you get to uh the default page will be uh your queries that you have saved And I have the the tickets that are open and assigned to myself on the top So that's what displayed here and uh, I basically go there and see in the morning. Okay. What to work on And you can have your saved queries for other stuff as usual um Yeah, this looks complicated But but the ticket status workflow has also has gotten easier. I think because the statuses are really just like the brown orange ones And these are more more like actions nowadays. This this will go away once we mitigate code review to to fabricator We haven't done that yet but all in all there's just the status and fabricator not the status and resolution anymore and um, I think we simplified it a bit by by having these four statuses over there, which are all closed Uh, and a bit easier than the workflow we had before And maybe the most important thing or criticism, uh There's project management. You you have work boards. You have columns. You have cards Uh, they show you the assignee here. They don't oh, yeah, they they do here. Okay The color refers to the priority, but you can Backlog needs discussion ready to go needs design Doing this implemented needs review whatever you can come up with your your custom Uh for each project with your custom uh columns So for example when you have a developer summit like last week, you can have the idea to Assign the rooms by using columns, which who had that idea? It was interesting. I liked it So you can basically plan quite some things Um, thanks to wikimedia germany. We have uh an extension custom burn-on charts for sprint projects Which many teams wanted we had a scrumbugs extension before for baxilla, but that was also like not really maintained One interesting change I like but but it's challenging is there's a flat namespace. So you so you have only projects you can use different colors and different tags for things Sprints target milestones teams Like keywords with a cross project basically like accessibility or performance actual Code projects or code repositories. Um, but it's a flat namespace So in baxilla you had like tags keywords status whiteboard product component and all different fields and um I I prefer it and here you see also the column that it's in on that work board But you have to have some idea at least how to organize that first You shouldn't just start So where we are right now, um We're still like after increasing the acceptance across teams. We're pretty happy pretty fine so far But of course, there's always teams that have some specific needs still that we haven't covered yet that they yell like Oh, yeah, I want to move everything now to fabricator. So they're a bit reluctant um We have a team practices mailing list to ensure effective and consistent workflows because like oh, okay I have this work board now which columns so that creates and what works for my team uh maintenance We are pretty selective with local patches because upstream moves so fast that it's Last time we we pulled a new version from upstream. We needed the second try because uh, yes We were we still haven't totally sorted out the testing yet for that Upstream doesn't have versions. Yeah, upstream doesn't have versions. That's true. So you just pull and see what you get But normally it's good. So it was more on our side the problem. It was on our side So Just to give you an idea Within a month we upgrade a month after and there were 14 Tasks that have been reported by our users that they had fixed 14 in a month now go back to bugzilla and your experience and let me know if there's anything similar to that Yeah, I agree I'm continuing the immigration from from mingle and trello for trello. We don't have a script yet I think our colleague s will work a bit on that from mingle some teams have already migrated. There is some script um And migrating the the code repository browsing currently we use a git blit for that moving that to fabricator and uh Begrading code review and continuous integration. That will I think be the fun part So this is this is a screenshot from upstream fabricator. So, um Code review. Yeah, you see reviewers. You can require lint unit tests Test plan, uh, it's uh, where's the is here a link? No, not yet To the well to the ticket. It's here at least this is this is the bug report the task number Um, and and to commit as this is merged And the lower part of this, uh Yeah, this is how it looks like Well try it. I mean it's upstream. So you'll find it Um, if you're here wondering like oh, is this something for our project? Um Do understand uh the upstream's understanding of what they do and what they reject. They're very clear on this Want to add something? Okay um upstream They in one ticket recently they explicitly said our api is considered unstable for the next two or three years So if you do customizations yourself, you have maintenance costs. You need somebody who's willing Uh to maintain your stuff and they won't announce their api changes either So you will realize once you pull but it's better if you realize a bit before If it has a public or rest api If the rest api is also unstable, um, I would assume so I'm I'm not sure. Uh, I haven't checked that they don't have an api promise at this point And they explicitly they're promising you that they will break But it's software. We like it, right? Um, they have a cla in place and um, some people Might not like that, but I I'd say like read it and decide on your own um Noven heats of your projects and developers Because you can't just push something down there if you don't have acceptance And uh involve your community and and the stakeholders like like project management Release management in your discussions Communicate early often wide and transparent I sometimes wondered like if we spam people too much when we had Two irc office hours per week where people could ask questions and put these banners up on media wiki orc And put this banner on bachzilla and this mailing list and better also this mailing list and um, but I think it worked well You will never be surprised you do all this and still people come three months later saying you didn't say anything so It's the internet. You didn't personally ask me for permission to change this website. Yes Um, so yeah, that's it. Um, you can find a lot of uh documentation help that we wrote over these months on media wiki orc slash wiki slash fabricator the sub pages like How to get the code uh our immigration code it has bugs, uh, but I've written that down now There's the project management page, uh, where we discuss workflows how to create projects. Um, there's also, um, how to set up a test instance How to pull our changes that we have on our production system and uh Yeah, if Send me an email or if it scales we'll see Or ask questions now Thank you Yeah, I'll I'll detach this and uh you Um, so so the question was uh, how the migration script was written if it was just like In one or if it's in certain steps that you could basically restart again uh, it was So, um, we we separately pulled like tickets and comments from bachzilla And then separately attachments because we had to work around an upstream bug in the xml rpc api Um Putting it into the database then again a very different process like putting that stuff from the database into fabricator a totally different script actually Um, we we had to restart Realizing after six hours Completely restart that our attachments were still Scrambled because of some encoding issue that was the upstream bug. We thought we had dealt with it We hadn't so at that point we had to completely restart at last six hours But our database engineers somehow made some things faster. So in the end we were pretty fine and we had enough buffer So yes, there was this this, uh, the man for something more incremental that less disruptive But really I mean you're dealing with 73 000 bucks is your whole history And you say guys just take a week and enjoy this great weather Come back on monday, uh, you know, that was that was what we did So one two We still use Jenkins. Yes, that was the question We still have Garrett for code review and we have Zool and Jenkins still but we'd like to kill that in the next year The next year is we are in February already. So it's this year actually I mean, sorry 12 months Uh, if we did work to integrate media wiki and fabricator Um, not much yet, but but I'm not I'm not sure. Are you after something specific? So so what we had for example were on wiki templates that uh on on certain wikipedia's and other uh sides People discuss things sometimes these things are bugs and our tasks So we have a template to link to that task for example But what we do not have in place yet, for example is automatically Pulling the the task status and displaying that in media wiki We need a bit more setup on laps with Some jason format that was special Um, if you can pull media wiki into fabricator, um, what's the use case? No, I wonder if you have just something specific Uh, it uh fabricator has a wiki. We just disabled it So, uh, it has a lot of things integrated. Yeah Yeah, so basically uh, and it's an important question actually so we decided Uh wikis are good for documentation Uh fabricator is good for project management It should know move to fabricator. I'm not going on. That's something a community should discuss So, um What I recommend, um Involving him or without involving him As I'm also active in gnom, um I don't know yet. I want to run this a few more months. Uh in in wiki media. Um, see how how things evolve How how happy people are and You need uh, both like Yeah, manpower to to maintain these things. Uh, that's something to sort out first Let us summarize this comment. Um, so yeah that that enlightenment uses fabricator and When uh, encode review when patches are a bit more complicated or longer cover more files that More patches together that it gets complicated and also with the command line tool arc, uh, probably arc, right? Yeah arc, uh, to use that part Yeah, um, I'll cut you because we still have some questions only 10 minutes left So, uh, we decided to have everybody registered Uh, I we have heated discussions and anyway, we want to give feed I mean, we know for many reasons we decided it's registered you have the policies not to do so And it might be that you find some surprises because the main use case for fabricators also with registered users But technically you can uh, but you know No There was another question down there and one So the question is if we're a bit afraid that there's only three people maintaining this upstream Um, I think the situation is is a bit different compared compared to bugzilla because so so I I've been following upstream bugzilla for quite a few years and um it's it's mostly um There's a few other companies, but it's mostly mozilla, um working on it and uh, they have their own special needs Their custom extensions some are upstream some are not Um, but it's it's not the main aspect that that mozilla focuses on And bugzilla is sometimes some people in the bugzilla community see themselves as a separate entity While we're fabricated It's a company behind it. Um, they probably want to succeed. Um, so I'm pretty positive that They're interested in Yeah, I mean, I think there's I think that's a another good question You're going to bet your your house on that. So I personally took two factors one factor is Really go to the repository see the speed of development there. So yeah fabricate on my day one day But like I tell you it's not now It's really fast and also like there's three main maintainers, but there's also a matter of adoption They are getting a lot of adoption from different types of projects. You can see how A ring of more experience contributors are getting more and more involved and you have people from other companies or other As of now, it's more about companies I think in the open source committee is not that well known, but I have no doubt that more people will just dive in And then well, there's another factor which which is subjective is who are these three people these three people are Five minutes these three people are Former facebook employees. I guess they had their options when they facebook goes public I don't think these guys have they love what they are doing. They don't have any any I'm I'm sure they have don't have any pressure to pay the rent or something So, yeah, they they could have stayed in facebook and have a safe life. They just said let's create this company Let's bring fabricator to everybody. So they really love what they are doing. So I think I I don't know. I feel safe at least in with this option One more question Which other options we had to choose from Of there were several things. I I really recommend the Review page, but there were several things like I think rat mine was brought up a fulcrum Oh, I don't remember. Yeah, but but but we at least had like track nine different ones That were brought up and combinations between them, of course The main point was that it was The alternatives are not very exciting So they solve problems that we had with vaxila, but they come with other problems that you quite know I mean like track for instance. Yes, it's a different type of beast gives me integration some points But we also know that beast and so there was really nothing It's unfair if I say nothing exciting That's not true But simply not not enough exciting to leave the place where we were And that was really an option where I was like, okay, should we go for this? And there was a sense of excitement of the opportunities that that this type of game would bring Okay, well, thank you very much