 So hello, my name is Simon Thornit. I'm a senior software developer at Catalyst I'm here today to talk to you all about an open-source plugin that we've been working on That will hopefully help many of you here plus the wider community And that plug-in is the data import tool Now over the course of this presentation, I'm going to try and cover a few different things What the plug-in is and how it works the problems that we try and aim to hope solve with it How it's been developed the approach that we decided to take in this instance and hopefully how that can benefit everyone in here even if the vanilla solution doesn't quite meet your requirements and Finally the Q&A session at the end So First off, what is it and how does it work? So you'll have to excuse me reading my script constantly The data import tool is a plug-in that will enable you to connect your Moodle to an Sorry, will enable you to connect your Moodle to an external data source and Automatically keep categories cohorts courses users and users enrollments synced up And up to date You'll understand more about how this works as we step through the plug-in and the various settings And then how we actually manage this is via the settings interface, which I'm gonna bring up now That is a horrible resolution. I am so sorry. This looked great on my laptop Damn right one minute. Okay, so just imagine you can actually read that and I'll step through it So here you can see the general tab That's what the first word actually says this tab contains the global configuration and connection settings and you'll Hopefully see at the top Global connections So this allows you to enable and define the settings for external connections to a database A local file or an SFTP file They are there. I promise and This will determine the shared setting that each import task will be able each import task will be able to use Again, you're gonna see that on the next screen in a little bit At the bottom. We actually have the log settings So here you can define the types of import logs you want to store whether it's just successful logs error logs Or whether yeah, whether you want everything or custom and finally at the bottom there You can define a list of email addresses of people who will be notified with Yeah, any Thoughts like a theater. Am I being flagged off to get off the stage? No No, okay So, yeah, what was I emails? Yeah So basically you can define a comma-separated list of emails of people who are gonna be notified at the end of the schedule task of Basically, how messed up it was any errors any problems warnings and so on so Let's have a look at the core step That's even worse right It was a really big page. So I split it into three screens. So just It's like a magic eye thing just squint and you'll be able to read it So starting with the left image, you'll see that we have the option to enable the task pretty self-explanatory We also have the schedule This uses the same Unix Cron format that you would have seen in the schedule tasks in the rest of your system And it allows for Individual customization of each of the tasks now again right at the very top You'll see that there are the different tabs. So category Cohort course user and user enrollment. I'm only gonna go through the user one in this level and you'll kind of see why on the next screen in a minute Now this is where under the connections You'll see where the global connections that we were just talking about on the previous screen get used So within each of the tasks you have the option to choose whether or not you want to use the global connection that you've Just defined so you can have all of your tasks picking up all of that data from the same place for consistency Or alternatively you can define at each task level where you want each individual set of data coming from So your courses could come from an external database. Your users could come in through CSV files. There's being exported from your user management system The it's completely configurable however you want to use it Under so under all the connection settings on to the middle screen, we have the general settings For courses we have the options of what we're going to do with the external if the external course has been removed So in this instance, we've got hide selected So you can hide it delete it or do nothing with it the task itself actually defines this so You could choose to back it up. You could choose to reset it You could extend this and make those modifications however you want And underneath that just a boolean flag just to say whether or not we actually want to keep the courses updated Do you just want to pull in the courses once and then forget about them? Or do you want to constantly keep courses users and everything in sync with the external system? Finally, we've got the mappings. So we've got the in the middle of the bottom You've got local mappings and then remote mappings up here Local mappings tell us what field the remote data will use to determine the linked to Moodle So in the case of courses, we need to know both the category field and course field that is there The remote mappings let us know what column in your data maps to the local field within the system So you don't need to worry about restructuring or reformatting your data We can just pick it up and modify it and handle the transformation at the task level Okay, so now with the exception of the mappings which are obviously really specific to each area of the system I haven't got screenshots for all of those The main difference between all of them is the import task settings import our settings, which you can see here So we've got the category I Can read that the category the cohort the course the user and the user enrollments Category has the most complex one Simply because it isn't just the category it obviously contains all the contains all the courses and everything So here you can actually specify the action that you want to take against those courses So if the category gets deleted You can then choose what you want to do with the course you can move it somewhere else You can hide them and so on Similar kind of functionality to what the course one has um Excuse me Okay, so Cohorts and courses can choose to be deleted or hidden users can obviously be deleted or suspended User enrollments you can unenroll the user with all of these you have the option to do nothing If the data disappears off your remote system if you don't want it to you can just leave it on the LMS You don't have to delete everything out the system But the idea is to give you the configuration the choices to how you want to configure this and how you want to use it okay, so Why did we create this? We created this in conjunction with a few of our different partners a lot of them are having the same problem of their user management systems and so on they wanted to get data Share data between things so we have started having a lot of discussions and trying to work out if we could build a Singular solution that would meet everyone's requirements No, you can't do that So what instead we tried to do was to create this plug-in that was completely configurable, but importantly it is fully extendable Which we're gonna get to in a second We did talk obviously about trying to use as much of the core functionality as possible in terms of the front-end interfaces but they can be Scattered across the system some of them don't enable automation and if anyone here has tried to do that Obviously, it can be quite a time-consuming and complex task to actually try and integrate into the system And also tends to break with upgrades as well So our hope was to remove some of the stressors when dealing with data imports by using this plug-in And it interacts directly with the core functionality of Moodles Now to get on what to what I was just alluding to about how it's been actually been developed During the presentation. I've referred to this data import tool as a plug-in, which isn't which isn't actually accurate It's 13 Or more precisely. It's actually one plug-in. Let's get that thought it was actually one plug-in and then 13 sub 12 sub plugins which are structured like so So everything from the settings the connections and the import types are Their own sub plugins that define the logic the form settings They all as you'll see have a base sub plug-in this handles most of the heavy lifting So all of those forms that you saw on the previous page Just an array of information. It's all passed through and generated to each of the base plug-ins to handle all of that So the main plug-in actually operates more as a framework for loading these These particular sub plug-ins the settings connections and import types And obviously the reason that we chose to go with this is for extendability and customization What may work for your average person or may work for one person one client isn't necessarily going to work for any of you In fact The chances are of a vanilla plug-in working out the box for everything is probably going to be quite slim You can use it for a good amount of things, but for everything You'll you'll want to be able to customize and extend it and that was why we went for with this and What I'll do is I'll give you an example of how this could work my favorite one so So you have a HR system and it's exporting a user see the user CSV file onto the mood of server For you to load in with a local file connection Now obviously being security conscious you want to have the file encrypted Now by default the local file connection isn't going to be able to handle the decryption side of things It doesn't have an area to put in the decryption key or anything like that However, with the sub plug-in framework, you can actually create the new connection sub plug-in that handles this with about 25 lines of code so So by creating a new sub plug-in that extends the existing file sub plug-in You just need the few lines of code to add the settings. So the ones that you saw on the course page All of those settings down in the center You would obviously want to add in say something like a password filled with a decryption key that you'd want to use and Then you just need a few lines of code within the actual Process itself to handle the decryption before it gets passed back to the vanilla to the vanilla local file plug-in So you're just handling the decryption not actually worrying about any of the process you give that to the parent and Make the setting changes pass it back to the parent and it loads it and it loads it into the system It should be relatively straightforward Something that originally could have taken days of development work to do is Down to a matter of hours and obviously that's not including testing deployment and everything else But you know for the most part it's a relatively straightforward process because we have the framework We have the sub plug-ins and you can build upon them And it's not just limited to those five tabs that you saw if you wanted to load in grade information again, we could you could have a great great sub plug-in type and anything Everything is handled by the framework in that regard so You could create new collection connection plugins so you could load it from an external web service that's something I've been asked a few times Can we load Use the data or course data in for from our web service? Well, yeah, you just need to create a new connection sub plug-in that handles that you define the settings And it all builds in with the actual framework to run the request regularly and pull all of that information in One of the reasons we focused on this approach was to reduce the need to modify core code Modifying core code whilst seems easier and can usually be quite minor Can have lasting and far-reaching implications as I'm sure many of you are probably aware if you've had to do anything like that so For starters, you're no longer on the maintained version Which means that update security leases bug fix and everything can't just be pulled in as they are so Instead of using a vanilla plug-in and then making the modification so that it suits your requirements By using something like this and by using the sub-proget plug-in approach You do early By using the sub-plug-in approach You don't need to you can keep the vanilla plug-in as a core instance That is update updated and maintained for you And then it's just your small sub-plug-ins that you need to be aware of when it comes to doing upgrades and anything's coming in and With a lot of the heavy lifting being handled by the framework It should be relatively minor in terms of the work that you actually have to do to keep these in a step with each other I Think that's about it. Oh So before Q&A I'm a developer. I don't I don't do talking very well So it's time for the cute a bit. Well before we get to the QA I want to preemptively answer the first question that I get which is Where is it? I want it where can I have it? Where can I look at it? Where can I pick it apart? so On the screen now you're gonna see a link to the catalyst repo where the plug-in will be housed in quotes The aim was to try and have a beta release already for this. I'm gonna step aside Have a beta release already for this for this week Unfortunately due to various different reasons that didn't happen We're still in the alpha and going through some of the local testing within our actual development team But you will see that there are a couple of discussions on there There's the beta release discussion and the full release discussion What I would ask you to do is that if you are interested in this and you do want to Be a guinea pig and run some testing as well Please go and subscribe to the discussion And I will be posting updates on there with regards to actual release schedules and release plans I am sorry that we don't actually have the final code But I didn't want to push up what I have got and it yeah No one wants to push up half finished stuff. So yeah Those are the discussions and Yeah, that's everything Any questions? Can I make a recommendation the next time can someone move this under the AC? It's boiling up here Do we have any questions for Simon? Sounds very very interesting. I was wondering if it's also available for workplace, or it's Currently only aimed at core. It's currently only been developed for Moodle That doesn't mean it won't work with workplace It's certainly something that we can have a look at if it does require any Modifications to it to make it work specifically with workplace, then we would probably fork off of this in a separate repo But there's no reason why it couldn't From what what I'm aware of in terms of how it functions and how we've designed it But certainly something we can have a look at I'll actually make an Sam, can you make a note of that check workplace? Thank you. I've left my notebook at the table anymore Hi there one question that I have is Are you trying to aim to replace the already existing external database enrollment that is already? present in Moodle core because I feel like 30% or so of the functionality is kind of covered by this however having used that functionality I know it's kind of difficult to get it up and running and your Approach seemed to be like Way more usable. Let's put it this way and so that's where my question is coming from No, I can appreciate we're coming from The thing is is that the the issue that we had with it and certainly with the discussions that we were having Was the problem was in the name is that it's database only obviously you've got the external database That you can use for things like the user enrollments Which runs in and picks up on sync when the user tries to log in it will go through and do those bits We found it quite difficult to do it preemptively as well when people want to report on things before the users even access the course as to Who's on the course? but also trying to Bundle it all in one package so that you only have to go to one place to deal with things and Also, whilst you have the external database tool If you wanted to put something in there where you want to load in say grades or something else from somewhere Something custom the option to extend and add all of those things Is difficult to actually do and to maintain that going forward? So it was just as a way to try and give everyone one place to go kind of a one-size-fits-all Most solution and then add the customizations in from there. I appreciate that so if I understand that correctly it currently does not have the Like the functionality that the external database Authentication because it's two different things actually has where if a user is present in an external database and locks on to Moodle The tool can check if that user is present in the external database and then migrates it over to Moodle Is that something that your tool is going to cover as well or is that planned? Not at the moment But that actually ties into triggers and events which is kind of a phase two thing that we've been looking at obviously We've got the schedule as you saw with the schedule task and be able to define the frequency for something like that You could run it every cron run if you wanted to But one of the other things that we have been discussing is triggers and events such as course created course updated And use a login events for example being able to select from them and choose to pull information around That sounds super interesting. Awesome. Thank you very much. Thank you Thank you very much as well. I was wondering maybe you could point out For short What's about it? Is there any kind of documentation how to write sub models for this and? Whether in the future there might be breaking changes how do you have plans how to communicate them and Okay, so most of the communications that we're going to be doing will be through the actual github in terms of any announcements and releases and bits and pieces Backwards compatibility for any major changes and major breakages I'm a big fan of Actually going through a deprecation cycle of keeping things in a deprecated stage So we ended up with both and then slowly phase that out to give everyone a chance to go through and make those corrections With regards to the actual documentation for the sub plugins. We have templates for the sub plugins. They are Incredibly lightweight as I said the actual building that whole settings page For the course for example is only about I think that 80 lines of code And most of that is just an array of information and language strings the heavy lifting is mostly done by the base plug-in But we will be providing Documentations and templates as well And obviously more than happy to have questions and feedback and anything slung at me on the on the discussion Yeah, so I hope that helps