 So, I'm Martin Dahlberg. I'm the other Martin. Just to make sure there's no confusion. And this is Jeff Webster. I am the Learning Technologies Coordinator at NC State University. And so I am responsible for being both the Rosetta Stone, kind of translating between developers, system administrators, help desk, faculty students, and making sure that we're kind of moving in the right direction. And Jeff Webster is our Director of Educational Technology Services, aka Manager of the Magic. And so we're going to talk... Oh, I'm sorry. Yes, of course. I'm used to roaming. That's my problem. So, one of the... I guess one of the issues with open source is that one of the wonderful things is that you can modify it. You can change it any way that you want. But that's also a big burden as well. So if you go ahead and, you know, innovate and modify and whatever, when you upgrade or even just add patches, if you're going to upgrade core, you have to maintain those things and you have to patch each and every time. So I don't even know how many modifications we have. Jeff probably knows, but it's not really important. It's quite a few. And so we try to be very judicious in how we handle this. So, you know, it's possible to attach to different data sources, to modify, customize things to meet our particular needs or what our faculty need. If you're kind of, you know, not in education, there's other things that you can do. But, you know, we try to encourage our community to be involved. You know, all of our educators and even the students can submit feature requests to get involved to actually feel like they have some ownership. And so, as a result, we obviously don't want to just say yes to everything that people request that we hear. So we have a process for managing that. And, you know, you have to keep in mind that, you know, where somebody may want, you know, people have suggested that we have, for example, as a default, editing on as a default when a faculty member goes in or instructor goes in to work on their course. And since the default has been no, we typically don't want to change things where it's going to irritate most faculty that are used to a certain way just to accommodate one person. So we have a process for dealing with that. So, faculty, staff, and students want new technology. We have a system called User ECHO. You know, we just took a little survey and picked something where they can go ahead and enter a feature request. They have to answer a few questions, particularly, we try to get them off, you know, people have a tendency to latch on to a tool. I want this. So they see something that's flashy or, you know, a vendor at a conference or something like that. They say, I don't want that. And so we try to force them to say, what are you trying to accomplish? Don't tell me the tool you want. Tell me what you're trying to accomplish. Maybe we already have something. So, what would you like to see changed? How is this going to help you, you know, perform, you know, teach better, connect with your students better? And if they have any external resources, so now if they want to tell me about that wonderful product that they saw or if there's a Moodle Doc page that they want to show up here for a particular plugin, they can send us to it. So, why do we do this? We want to promote engagement with the learning community. We want to make sure that we understand the needs of our community. It's actually also a great way to spot future trends. Our 2,000 faculty know better than we do, okay, they're all out there. They're all, you know, they're seeing what's going on. It's supposed to many more conferences, many more advertisements, many more colleagues from different institutions. So, they have a much better idea of what they're looking at, of, you know, what's going on. For example, document annotation. The first time that somebody, you know, a document annotation and collaboration, the first time that a faculty member came to us and said, you know, I want to be able to have my students work collaboratively on a document, like take a PDF or something, you know, students have a conversation on it. It's kind of like, okay, I'm not sure I understand why, but you could use Google Docs for that. And, you know, I got two more feature requests, and at that time I started to see a trend that I realized, you know, we need to, we need to really look at this, and then we found a tool that allows them to do that. So, how do we do this? So, we iterated through several processes over the last 10 or 11 years that we've been doing this. The first time that we did it, I, the coordinator, review the request. I make sure that it's a well-formed request that they've answered all the questions. If there's an error, if it's really a bug that they're trying to report, I'll deal with that appropriately. I then send it, we have governance committees, and I'll tell you where all these resources are at the end of the presentation. You can go and take a look at what our governance committees look like, but we have basically a best practices and support that looks at it from a theoretical point of view and a customer needs and policy that looks at it from an institutional or organizational point of view. And then they report to a steering committee and the steering committee makes a decision. So, document the results, and then I send the approvals, approved requests over to the systems team. And what ended up happening is we were spending all of our time on feature requests. Governance became feature requests. We didn't really look at policy. So, second iteration, instead of as they come in on a rolling basis, we started to batch them twice a year. So yes, we'll be responsive to you, but it's going to take you several months to hear back. And that at least allowed us to kind of, in September and October and then in January and February to go ahead and evaluate these feature requests, kind of push them through and make decisions. But it's still taken a lot of effort. So, third iteration, I got some power. Now I have the ability to fast track. So if I see a particular feature request that I think is either really arcane or really good, I have the ability to go to an internal committee that meets every two weeks and says, I want to approve this. We may have a little discussion about it and we make a decision and we move on. If we can't agree, then we send it through the regular process. And we do that. I like to have a check on me. I mean, I think I know everything and I think I'm the smartest person in the world, but clearly I'm not. And so, sometimes I have a good idea or I think that I'm on the right track and the collective wizard of the room sometimes is a little humiliating, but gets to be sent in the right direction. So, when do we do this? Our director works with staff and projects. I think this is yours, right? So, yes. Marty gets to oversee the fun part of reviewing these requests, talking with our customers and looking at the new stuff and once they're approved, I get them to actually try to implement them. So, I guess the first thing we do is we add the new request to our backlog for those that do either agile or it's not a traditional backlog. It's just a list of items. They can be really tiny. So, an example, install the matrix question type. They can be really ugly. One that's still an open one is have hidden groups within Moodle relating to privacy of groups where you don't want the students to be able to see who is actually in the group with them. That one actually may be getting close to a solution. The tracker is actually getting action. But we have those. We evaluate them. We look at, you know, sort of how many people is it going to impact? How hard is it going to be to work on it? And that gets us a waiting on it to put it on our priority list. As we go through those, every once in a while we'll review the backlog, move items over to the Kanban Board, which is where we track the actual active work that people are working on. One thing we have to balance is we've got these feature requests for development that needs to happen in the regular processes that are out there. So things like, okay, we've got a new feature request, but we also have to actually deal with the new version of Moodle that's coming out. Balance the workload because kind of need to do the new version of Moodle. We want the feature request, but you can't spend all your time on the new stuff. And then at regular intervals, we actually will also look at that backlog and look at things that have been on there for a while and say, this one's been on the backlog for two years. It's still low priority. We're never going to get to this. We throw those back over to Marty. He takes them back to the government and says, okay, we did approve this. We thought it was good at the time, but we just can't get to it. We want to decline it now. And so far we're actually doing a pretty good job. We've had over 600 feature requests come through the system. I've got about 250 of them that were declined before they got well, I guess some of them were declined before they got the committee and some of them have been post decline. We've got 55 that are still actually in progress that we're trying to work on. So I think that's actually a pretty good turnover. Yeah, so since we have a little bit of time, so yes, Jeff's mentioned we have about 600 that's over, what, about 12 years? Yes. So the pace has slowed down a little bit until we upgraded to Moodle 4. So you can find out, you know, we'll talk about how we upgraded to Moodle 4 tomorrow. But we did get a flurry of requests to basically make Moodle 4 like Moodle 3.9. So we're kind of batch those together and we're going to deal with those all at one time, I think next month. But, you know, that's a natural effect of people not liking change. But, you know, the nice thing about it is that it's important to at least give your users a voice. So I may have to say no to them nicely, but at least they've been listened to. And I think that's kind of important. So we have a number of resources. You should be able to download the slides and there's two links there. One is specifically to our SOP on feature requests. And by the way, I'm standing next to the author of the SOP on SOPs if you ever need something. But we have feature requests and in general we have our NC State Wolfware Governance site and the feature request, the feature request SOP is on there. But there's a variety of things. You can take a look at our governance chart. We have SOPs on a variety of other things. And as far as I'm concerned, that's open source. It's available for anybody to use, steal, modify, you know, use in your organization, what have you. Or, you know, scoff at and laugh. And now, open up some questions. So it's really interesting. I love your processes and how you organize it. Yeah, I like things that are well organized. Yeah, definitely. Okay. We have some time so if there's anyone with a question, okay, I can see one over there. So, well organized. That's a relative term. We'd like to think. Hello, I'm Mariah Stegerman. I have a question. You started your story about how people can enter requests. How users can enter requests and how users can use user echo. Is it something inside Moodle or it's a different thing? No, unfortunately, user echo is something that we pay for. I think the license is relatively inexpensive. Do you remember how much it was? ours is extremely cheap because we got in early. Oh, okay. I believe an annual license is less than $1,000. I wouldn't concentrate too much on the system. I don't think you need to just find what works for you. In general, we were looking for something that would allow people to submit a request. Would allow others to upvote them. Would allow us to make comments or other people in the community make comments like, yeah, I really like this or I'd like to see it like that. And then for us to process them and put them in different categories based upon whether they were entered under review or completed or rejected. So I think you just need to find a system user echo I would not say is a perfect thing. It's a tool that seems to work. And it was relatively inexpensive and so we stuck with that. But I'm sure there are plenty of other tools out there that would do that. Any other questions? I was going to say one quick piece. Being able to see the status of those requests is a piece of feedback we got from the faculty member. Especially things like the declined request so they could see that it had already been requested before. And it might be that, yeah, it was a decline two years ago. We will let them submit it again and that's some of those where it's okay it actually makes sense to do that now. And so some of them will get reconsidered. I'm keen to understand the support structure that you have. Is your team in the middle between academic and IT or are you IT? Is there a design authority that is driven by academics? Is there a filter for these requests to say so that's already a capability that we have and not a new request, not a bug but go to our team for further orientational training. Thank you. So if you can talk to that and describe how that adds to the success because if we are to implement the same we'd love to learn from you, thank you. Alright, so I guess the our organization is only responsible for academic technology so we sit outside of central IT we actually are kind of nice, we actually have some additional people that are non-technical so like we have instructional designers for those people within our same organization so that's great for the filter Marty is the filter on those requests where he looks at them we talked about looking at them when they first come in and so he'll look to see yes oh I want a tool that does discussion wow there's a form in Moodle you might want to look at this, it does that you know, obviously simple example but he will filter through looking for things that we've already got in our toolkit so I'd like to say that I have the best job at NC State because I have essentially the product owner I have no direct reports, I have no budget but I work with all of the different teams in our educational technology group so both the training training help desk the tech people, the designers and the faculty and the students so any other questions? thank you very much