 Onto a discussion panel. So as many of you would know, learning analytics has been a focus of our set at Moodle and about March this year we introduced the open solution Learning Analytics with Project Inspire So with your help, we've worked through phase one and we've collected valuable data and in May the first public release of Inspire as a third-party plug-in was made available and that coincided with the Moodle release 3.3 In 3.4 the Inspire system will become part of Moodle Core making it automatically available to all Moodle sites with a locally managed partner hosted or hosted on Moodle Cloud So today for our learning analytics panel discussion we have from Moodle HQ Damien, Tom and Martin and Grant Bevers from Moodle Rooms, Andrew Bogue from Kettleist and they'll provide some more updates and take some questions and share about the latest projects and their learning analytics within Moodle Thanks guys. Hello. I'll kick off Thank you Rowan. So Rowan just mentioned Inspire but there are many many forms of analytics that we can cover today lots of different projects in Moodle and out of Moodle and so I thought we'd just all have a little spiel here a little chin-wag So let me tell you a bit about the history of analytics in Moodle the very very first module that I wrote in the very beginning so Moodle just had one module and it was the survey module and that's all you could do was analytics pretty much there was nothing else So that survey module that no one likes to use because they can't change the questions was a very specifically designed collection of surveys like the Collis, the Constructivist Online Learning Environment survey which was designed as an analytic tool to understand what the group of students in your class were doing and how they were performing and how they were feeling and by diving into the the graphs and the results of that survey you could really see immediately outliers and you could find particular students who are struggling in some way and you could zoom in on them and then go and talk to them and in the very beginning when Moodle was only that survey it was used in companion with other ways of teaching so you might have a face-to-face class and you would give them this survey or you'd be teaching on I don't know a mailing list or something and you would give them this survey so it's sort of interesting that that was there right from the beginning and there's a collection of other survey tools in there as well and in the very first Moodle course that was ever I used to run that analytic every week so I used to put the students through that survey every week sometimes twice a week and it was just this constant feedback of trying to understand where they were at and then modifying my teaching for the for the following week so that's a historical thing now there's other tools in Moodle still to do with analyzing what students are doing and trying to give you some something to act on as a teacher as a as an educator I'll probably leave some of those maybe maybe these guys want to talk about some of those too but I know who wants to go next who wants to say something okay cool thanks Martin and hi everyone you know the definition of analytics is wide and varied and in different institutions that we work with and and I suppose if I look back at last year and to a certain extent this year but last year I mean I can't think of any institution that that Blackboard or Moodle rooms works with that wasn't doing some sort of analytics project I think if we added all of the dollar value of all of these projects we could we could probably feed you know a few different nations quite easily the amount of effort that's going into that and I think a lot of that's driven by the fact that you know there's a lot of regulatory change going on in the sector there is a greater need to report to government about various things that are going on the way in which we're funded especially in the vet sector going forward and and certainly you know that will flow into higher ed is requiring us to ensure that we see our students through to completion so as a result of that you know we're wanting to use the analytics to track retention rates to identify students at risk and you know and be able to I think probably do a much better job in all honesty certainly in the vet sector you know around ensuring that we've done the very best that we possibly can to identify students before they become problems and so I think the emphasis on analytics as a tool to allow us to do that is probably you know more important than it's probably ever been and so you know I said a lot of people analytics for the sake of analytics is just data if we need to have some sort of call to action around that analytics and and be able to you know take that analytical data and then be able to act on it straight away and we want the tools to allow us and help us to act on that as well and we're starting to see a lot of those sort of tools in built into learning management systems you know I'm hoping that something inspired is is kind of like doing is is not only collecting this data in a way that's meaningful for us to do something with but but is also allow is going to allow us to then reach out to the relevant people that are going to allow us to take action with respect to that data we're doing a lot of work certainly in middle rooms in blackboard we're doing a lot of work around what we call at at the point analytics so we we at blackboard we talk about big a analytics and little analytics the big analytics is the is the bi you know the big data warehouses where we're sucking data in from all around all of the applications we've got in our institution and we were crunching that data and we're spitting reports out for management and whoever needs them the little a analytics we talk about is that those analytics at the point where it's happening inside a course so if we've got a discussion forum or an assignment or a quiz can I there and then click into that environment and see some meaningful analytics about who's doing what where they're doing it who's having trouble who's not having trouble as opposed to kind of like the old style of going out of the system going into a reporting structure running a report getting that information printing it out or exporting it somewhere and then doing something with that data so I think you know we're investing a lot of time in that little a analytics work making sure that whenever you need that data and whenever you need to get access to it it's there when you need it not having to go looking for it and I think you know that's where we should be focusing our time because the time it just takes to chase that data down and then do something with it we could be spending our time doing a lot more useful things and yeah so pass on thanks very much look I'm really interested in this topic personally and we've spent a lot of time talking with clients and amongst ourselves and you know hypothesizing about what data analytics means and and you know what we could use it for look I was really impressed with the middle room's sort of dashboard that you guys just demonstrated I suppose data analytics at this stage has two broad applications in a learning sense and there will be more and there are others but from what I sense is this whole idea that both Martin and Grant mentioned of custodial care right being able to detect when someone needs a pat on the back some help support you know that's this very real reasons why institutions need that right I mean at the very least the finance from a revenue leakage point of view because you know they want students to start to finish in the other end of the spectrum of the certainly from what we've seen in a wish list from our clients is this idea of you know learning analytics that allows us to improve the the educational methodology that we use or the questions that we ask or the quizzes that we deliver or some meaningful feedback saying we need to change this in the course or we need to use this and that's I think I think at this stage a little bit harder because it's just got more data points and there's a little bit more subjectivity about you know what how you measure these things etc and some food for thought around learning analytics because often these things can be really massive from an organizational point of view and they and the tool set itself costs a million bucks and there's a huge procurement process around it but you can do so much with so little if you look at analytics especially in the custodial care component if you look at these things like collaboratively and iteratively like you can do a lot of damage with the right data in an excel spreadsheet and someone who's pretty keen to find the answer to some questions if you really do socialize the outcomes of what you discover and you iteratively look at what starts falling out of it and ask new questions now if you get that flow happening with any kind of analytical process you're doing well right and that's something to bear in mind before sometimes before organizations launch into huge processes you know in search of this analytics tool is to really think well come on let's just see what we can do with a little bit with people who are with stakeholders that are really interested and motivated and know the way around a few tools and I suppose another bit of food for thought I've just thrown there because this is such a big topic and I think I'm really interested to know the questions you guys are going to ask is there are some tools set aside what you want to see aside it's quite quick that we've seen you know you bump into some issues that are challenging for universities I talk about universities for example one of our clients who we've we've got a we've had a number of engagements with on on analytics outputs you know they're talking about it was more custodial care focused actually their outcomes at the beginning when they wanted to look after their students and sort of detect the outlining students and there was a discussion around attendance of tutorials right so from your organizations here who tracks attendance to tutorials okay so less less more often not to track these things now I understand I went to quite a few meetings about the mechanics and should we and shouldn't we and how do we do it and all these sorts of things so I was present in these discussions and I can understand why it isn't a no-brainer but from if you're talking about your ability to ask the question does people going to tutorials correlate to success or failure you need that data point right and from the outside as an engineer who I don't work in universities you know I sort of see that I sort of thought well of course you want that data right but as you can well understand sometimes those questions and sometimes those things get really complicated organizationally so that that's an ongoing challenge that sort of has nothing to do with the with the technical choice but is is very real in order giving in order to give you the data points that you need to provide you with sort of predictive outcomes so I think that's all I'm going to say because there's there's other people on stage and I'm really interested in hearing what you guys are going to ask hi I'm the developer little developer so from our point of view what we've been focusing on recently is this project inspire so the whole idea about that is that it's sort of designed we're building an open API and we want to collaborate with the community in as much of an open research type of way as we can so we've built a really good platform where we can take in all these data points and then feed them into something and then get predictions at the end so we're trying to generate meaningful insights the insights get sent directly to somebody who they're relevant to and that person then has a meaningful action that they can take based on that insight and they can see where it's come from and they can see the data now the thing that generates the insights in our particular case is interesting to me because I'm a tech nerd kind of person and it's about machine learning so we can interact integrate with TensorFlow which is an open source machine learning library from Google and it's really powerful and it's getting used in all sorts of different places in the world now to generate all sorts of crazy things because it works so well so it's really interesting from that machine learning side to see how much progress we've made in the last few years and the idea behind it is that we don't know in advance what kinds of things are going to be the predictors for the insights that we're interested in but what we can do is we can hypothesize and we can say this thing it might be useful let's create an indicator that can calculate a data point based on the raw log data in Moodle and any of the Moodle database tables and there's lots and lots of raw information in Moodle but we want to try and get some meaning out of that data so we can then we can take the raw data we can calculate an indicator and feed it into this machine learning engine and see if it can predict something the machine learning engine might say no that that information is irrelevant but it will kind of work that out itself which is why it's very interesting and then at the end it'll be able to say based on the it can only work with the data that's given so the more data you give it the more accuracy it will predict things and so then it will be able to sort of say with a percentage of confidence that it's actually predicted an insight and when that accuracy goes up high enough that might be something that we're interested in sending a notification to a specific teacher or a student who can act on that data we want to be very careful about sending predictions that make themselves true like we don't want to tell a student that they're doing terribly and they're going to fail we want to we should be sending that kind of notification that a student might be in trouble to somebody who can step in and look at the data and make a real decision about what to do about it so yep so that's from me I'll hand over to Tom. How many of you come from schools that have evaluations at the end of each term or semester for the teacher? Okay I used to do that. How many of you have schools where there's a process where that data is analyzed and some meaningful steps are taken from the feedback that we received from our students? Cool yeah that's not true for everyone there's a big black hole on information that we gather from our students frequently it takes a culture change to think of that information which is very meaningful I like using the word meaningful because we have to make meaningful change based on what we learn about ourselves and what we learn about our learning and what we learn about our our students so when on Monday Joey comes to class and spends two hours in Moodle and then Tuesday logs in and spends four hours in Moodle and on Wednesday spends three hours in Moodle and on Thursday fails a quiz miserably and then doesn't log in for six more days we have to find out what meaningful things we can do whether that's reaching out to him thinking about what we did there with the quiz thinking about a number of variables which are hard to think about but worthy of it so I'm on this panel because I'm really interested in the impact of this new knowledge that we're going to be able to collect and culturally I'm wondering about the readiness of our organizations to to listen to that meaningful data and make meaningful change well that was cool thank you it's open to anyone who wants to ask a question or have a re-observation or a thought or something that you know some news and let's just follow where this meandering discussion takes us we'll need some microphone runners okay and we've got how many one two okay 73 one thing I really like is seeing data presented visually uh in different patterns and you know different people interpret visual data in different ways depending on what it is you're trying to ascertain um I find that Moodle's not that good at presenting patterns the data patterns visually can you make any comments on that um sure it probably could be better of course there are graphs in like logs you know basic stuff like that but yeah pattern identification is definitely something inspire is going to be good at um we've been doing a lot of work with shiny technology I don't know if many of you are familiar with that but the ability to really interact with the data so bring the data into a tabular form to start with and then really start to interact with the graph change the parameters of the graph zoom in on parts of that if you've got very large cohorts looking at large cohort data in a graph can can be quite problematic because of the size and so forth so that ability so um you know I love it the technology looks great I'm I'm understanding that it's a standard for the way in which we we make more graphical stuff more interactive um but I think you know dashboards dashboards are becoming the the way in which we're presenting that data and then the ability to to then get to the raw data is is kind of like the secondary thing a lot with with a lot of that sort of stuff but I think we're seeing a lot more of that than we've ever seen before so um yeah as Martin said there's there's there are graphs in Moodle they're spread about here and there but they haven't really been a focus we did update all the graphs recently so that the data in the graphs is presented in a nicer way and then you can actually go in and see the raw data that is behind each of the graphs that's generated in Moodle so it's good that we've improved that side of the API but so now that API has been improved it would be great to see people are sending it more plugins and putting more plugins in the database there are actually quite a lot of plugins for generating dashboards and all different kinds of reports in the plugins database but I understand that actually there is a challenge there where people sometimes can't install plugins on their own sites and also just the fact that there are so many plugins available it's hard to go through and mall and see which ones are going to be useful to install and things like that so yeah one thing that maybe we can do is we can go through the plugins database and find the very popular very useful plugins and look at adding those to core that might be a good use and also our education team so Elizabeth has gone through Moodle and identified a bunch of places where we could add some more reports that would be really useful for people to put them up front obviously there are other things like in Teleboard it's something that you can just you can plug in and they have great dashboards as well and so there's a lot of options that people can install on their sites but out of the box at Moodle probably is quite a little bit bare on the reporting side yep you lead me right on to what I was going to say that there's a difference between reporting and analytics for me I think reporting is a subset of analytics but analytics should I think go further I mean it's about the analysis it's not just presenting data for you to analyze although that's one part it should hopefully be doing some of that analysis and giving you actionable feedback some sort of feedback that you you can take an action on that's my definition I would say anybody disagree no no it's going to say it's going to say similar I mean we you take a graph and you look at that graph and you say well yeah that's a nice graph but then you look at the graph and you then say wow there's there's four students out lying in in that area there that there's way out from the mean of where everyone else is there there's something happening out there that I need to maybe do something about that to me is the analyst the analytic part of where the reports and analytics kind of like fit together in my eyes and then like I said the really important stuff is in doing something with that and and being able to then you know reach out you know can we can we directly from that analytics reach out to that student who is in the outlier of that grass and click on him and say we need to get together we need to talk about this because there's a problem going on here somewhere so I think it's always got to end in some sort of action that's going to bring about some sort of change well that's not always but but I think you know for the benefit of of the sort of work that we do with our students in retention and so forth it's certainly key um with inspire will you have any plans so that's I think the main thing for me is that making something meaningful and having an action that's meaningful from the data because I think our teachers they're really just overwhelmed you know so then they get all these reports like we have a number of plugins like gizmo inactive user alert I think we've got another one I can't remember off the top of my head but then they've just got all of this data that they're freaking out about you know they're like there is this beautiful looking report but what does it mean and when do they intervene so I think from our perspective it's really educating them on how to use that data and when might the points be so I suppose my question about inspire is will you have some sort of I don't know a rollout plan or suggestions I know that you probably don't want to direct people fully because it's all about what we make of it but I think sometimes that would be really helpful knowing what your intention was you know because we just get something and then we're like what is the intention behind it apart from having all this wonderful data well I could tell you about the intention when we launched the project in in my mind was that the system is creating events so it's saying something's happened I'm going to notify somebody about it and we have a quite a we have a messaging system in Moodle and if you have the mobile app they come through on your phone or an email or other devices whatever but the intention was that you would get these notifications then you obviously want to control them and so we had talked about I don't know how far away this is going to be but some sort of a way of saying oh we actually do have this you can say I don't want any more of these right you can you can kill them on the report page currently right yeah they came in you can say so what happens is the teacher will get a notification that an insight has been predicted and they'll see the details about like this student is at risk of dropping out of a course and right there it'll have the data that was used to calculate that insight so maybe they they haven't logged in for or they haven't completed their profile and but they did participate in the week one activities but they haven't participated in the week two activities and then from that insight directly on the insight you have a list of actions that can be taken and those this is what we call a model which we want we've shipped this students at risk of dropping out model in 3.4 and we want people to come up with the other models and contribute them back to Moodle so that we can all have all kinds of different models in Moodle but it's like the collection of the things that might be useful as indicators of something and then that's going to generate an insight and part of the insight is the suggested actions that you might want to take from that insight so in this case the suggested actions could be to go and look at the students grades in the grade report or to send that student a message and you can also acknowledge the the insight to make it go away or you could say this wasn't actually useful so when you say that the insight wasn't actually useful that in itself becomes an indicator back into the system so it's like a feedback loop so the system is less likely to generate that kind of insight in the future as well. Where I would like to see that going is become like an assistant so Moodle this is what I was saying this morning Moodle's like a voice talking to you in a chat basically and saying hey what about this and you go oh stop telling me about that or tell me more about those or stop bothering me at night or you know you adjust its behavior and the assistant kind of learns what you want and you and you it's like a slider bar you're adjusting the sensitivity of it until it's just what you need. I think that seems to be the kind of anybody could learn to use without any training right it would just be something you would just learn by doing it. You're just following up on that it's a very good analogy Martin so what sort of my perception of those sort of challenges as well let's draw it to something that's more of an outdated existence which is you know running a managed services company and getting support requests and escalation requests that sort of come in through the front door and we have to deal with those the the successful when things are running badly there's too much noise there's too much noise it's coming all the time there's too many things coming to too many people you're not really sure which what you're absolutely expected to deal with and what you're not and there's a lot of there is effort and process around optimizing the way in which those that messaging is directed how it is structured and what the expectations are around action and resolution right I mean great example is when you're monitoring systems for example for if they break or if they fail automated tests I mean you you the perfect monitoring system is one that you never get alerts from right that's the perfect monitoring system but in reality and the very bad monitoring system is one that there's so many alerts that you just look at the you look at the wall and there's all these alerts and you go yeah I'm not even sure which ones I need to bother about because hey that one's been read for two weeks and someone's going to fix that one and all that's I mean that's then that's a reality and I've we've had those systems before so I mean you an assistance great way of looking at I mean often my some of my great team members because I'm not very good at being organized about anything to do other than sort of IT stuff and if they come and ask me questions all the time then I just give them random answers about what to do but if we can sit down for an hour every couple of days and work through things and I don't have to think about those things at all outside of that then we have much more sensible decisions getting made and and that's because you need to sort of get in the in the mode of these things and working through working through these things in a concentrated way but not being overwhelmed and not context shifting and having clear definitions of who is supposed to and I never say care about this but whose whose responsibility it is and what the expectation is around taking actions and and also talking amongst yourselves and saying there's too much there's too much like there's no there's no value in us as an organization getting 50 notifications every Monday about things we've got to do because we simply don't have time so we need to we need to change the way things get escalated so that perhaps things need to go to a higher level before before we're expected maybe the first step is you just send emails to the student saying oh not good enough it doesn't come to us we have our actual involvement in action is required until it hits a higher a higher level for example right but and those are decisions that need to be made organizationally and they sometimes some of them aren't easy and some of them don't necessarily align with what the business expectations are compared to what the resourcing might be available Martin alluded in the keynote this morning to you know future technologies or technologies that are with us now in AI and so forth and I know that some work that we're doing at the moment around IBM Watson I think many of you might be familiar with IBM Watson as the computer that beat the chess champion and and so forth and there's there's Watson projects going on in in many of the universities in Australia and around the world many of you might be familiar with Deakin University student support environment students actually don't talk to a student advisor at Deakin they talk to Watson and Watson gives the answers back so it rolls through all of the data and it comes back with a search that that's been modified and presents the the student with the answers and those answers might be look it sounds like you need to talk to a student advisor click here to create a calendar invite to a student advisor so we've started some work with a number of universities around this concept of a nudge okay and and and mobile driven but the ability to nudge a student or nudge an advisor or nudge a teacher or nudge an administrator based on the the the communication that a student would have with Watson and so you know the student will will will ask questions via the phone or or or some other mechanism and then you know Watson will then determine who should be nudged as a result of that and it may be that the question the student says look I'm really struggling in this subject I don't know really what I've you know how I can how I can see myself getting to the end of this subject without failing what am I going to do you know Watson comes back with some options the student continues to have that conversation and then it leads to some sort of actions about getting that student in front of a guidance counselor or the lecturer or whatever is the case and and therefore informs those people as part of that whole particular process so I'm really excited I've seen some sort of introductory work on that that we're doing I'm really excited to see where that goes it may not be Watson it may be some other AI technology that that that leaves that way but but it's really exciting because the computing power needed to manage and troll through all of that data you know is is not something that's that's available to us in our typical service structures and stuff we have in our organization so you know we're needing tools like like that to be able to troll through you know gigabytes terabytes of of data to get those answers back first I went to a presentation today where Mike from Catalyst was showing this really great little use of Watson and great little use of sentiment analysis and he was picking up data from forums and just simply indicating what some of the overwhelming feeling was in those forums whether it was negative whether it was positive whether it was full of disgust you know and and and I think that kind of nudge nudge your class finds this disgusting you know it's really interesting and useful and and a thoughtful application of of some of that technology so Mike at Catalyst it was great hi guys Zach here from CT in part I have a question which is kind of specific about inspire and one is more more broader about your thoughts my question is is any of the data that's collected through the inspire let's say machine learning engine is any of this data shared with a third party like Google or anyone else who uses that technology is that stored locally on the server because again I don't know all the details and how this works I'm not an expert in the area and the second more broader question is what are your thoughts on on data probably something that Martin talked about this morning where is the line between collecting enough data in order to make those processes machines better and keeping that privacy so I would like to hear your thoughts on this so about the first part of the question about where does the data go so with project inspire it's all run on your Moodle server so it comes with a choice of two machine learning backends initially and we can add more in future one is TensorFlow which is the the Google machine learning library but it doesn't actually go to Google servers at all it's a Python library that you install on your Moodle server so there might be challenges for some people installing things on their Moodle server the other option is PHP machine learning which is a PHP library for a similar thing and that just you don't there's no installation requirements for that one so anybody could use that but again all of the data is kept on your Moodle server and processed on your Moodle server so it's not shared with anyone in terms of data privacy there are some implications just so it depends on what whether this is enabled and what kind of actions are being the data is being used for but it's good if you have a site privacy policy to tell people if their data is going to be used in this way to generate things and what those things might be used for I know it's part of the EU general data privacy regulation changes that are coming up next year that's one of the requirements is that you need to tell everybody not just students but you need to tell people when the data is going to be processed and what it's going to be processed for and give them the choice to opt out of having that data processed and what the implications of opting out might mean as well that's clearly a challenge with any analytics going forwards so yeah I don't know if anyone's got any if anyone had a special interest or looked into this kind of question before here and wants to add their two cents but I think it's a problem we all need to solve and in the inspire case the research forum that we have around it is exactly the place to discuss these issues and bring that research together and best practice so yeah it's a it's a complicated one this one so we were actually being directly affected by the the new EU rules because well we're in the process of understanding what this means because of the way we've implemented our follow the sun support for Moodle so we have a team in the UK a team in Australia and team in New Zealand and we watch each other's Moodles and the when one person is awake and the other one's sleeping pretty simply and this these new regulations have implications in the details around really what the definition of access and data even though data isn't moving around so it's it's it's it's an ongoing challenge and from someone and I've been quite in quite a few discussions around privacy and data sovereignty and those sorts of things with various clients and understanding what contractual obligations you're getting into and what I will say is is that if you have a lawyer a commercial person and a technical person sitting around the room talking privacy they might as well be speaking different languages right like it is complicated so do not think for a second that your that maybe because you haven't had a lot to do with these these things before that you are not qualified to ask any question and there is no such thing as a silly question in these things because it is complicated and different people have different takes on it and and and and also and I hate to say this but from some angles it's sort of a little bit absurd as well I mean some of the some of the requirements are too much and some of them are just broadly ignored but it is something to be paying attention to because the failure in an organization will be you know if data rings up in the place it shouldn't be and something bad happens the failure will be because people weren't asking questions because people didn't go through because your your inability to demonstrate that you went through a process of asking questions and reviewing implications and just sort of saying look we were paying attention is what will be the downfall that's it's sort of like a compliance challenge but it is it is not I mean my personal take on it and our our perception of it is there are some quite curly challenges here and you know hopefully it'll get easier yeah and I think you know with the move to the cloud for for any application it brings with it a whole lot of new challenges that we never had when we were running our Moodle you know in the back office in in on the campus or whatever and that data wasn't really going any further than than that particular environment so you're right you know get a good lawyer if you're concerned and and and work your way through it there's there's not only is there the EU EU regulations as FedRAMP in the US there's there's a whole range of Australian we've just gone through a massive workload with Singapore government which has got the tightest privacy registration of any organization in the world when we brought five polytechnics together in in one environment with with a whole suite of you know there were 12 layers of security protocols beneath each other just to get to the data and and so forth and and you have to meet both hardware-wise as well as software-wise each of those and get signed off and all that sort of stuff so you know it's only going to get worse then it's going to get easier I think is as we move and when we get a Moodle cloud and all that sort of stuff there's going to be implications around that as we have people from all around the world accessing those Moodle clouds as well and where that data is going to live and move and all that sort of thing so one thing I will say though is you've got a much better chance of solving these with open software than anything else just from a everyday user kind of a perspective perhaps what the lady said earlier which was there's an interim step since it's going to take me well for this to to come on board it would be nice to have more data like with from assessments like quiz and assignments we would like to see more data come out of rubrics marking guides and things like that which we're currently not able to get get get a hold of and having that data would be much more it would probably drive our decisions and what we actually deliver as opposed to more and then be able to act on the students so we're not getting any data out on how good our assessments are because it's all buried in marking guides and rubrics and and quizzes and quiz types so we'll be end up being able to have an assessment database and have the analysis of that sort of thing as well so where are you getting it out to are you pulling it out of the database yourself well that's the thing is we can't get access to that so there's no access to all the data from all our rubrics and marking guides from our assignments and that's really rich data which we could so no access from what we're within Moodle itself so from the front end we can't get to that unless we were went to the back end of the database so it makes sense so when you use rubrics and marking guides and once you fill them out what it does is it then generates a score and the score is the thing that goes in the gradebook that's great so what I understand you're trying to say is that you don't actually from the gradebook for example you can't then go and see the breakdown of the rubric or the marking guide that made up that individual score or see like aggregated data about the first question in this marking guide most students got a two out of five and maybe there's a that that topic wasn't actually covered in the course material or something like that yeah or yeah or particular whole parts of the syllabus are done poorly by the students so we should you know address that issue yeah I mean it sounds like a fair comment some places in Moodle do give that kind of data when the the competencies changes that we we added that does give you a sort of a report of a breakdown per competency of perhaps things that aren't being completed by students and things like that but I think yeah for marking guides in rubrics that does seem to be something that we don't do great yes UTS has a room dedicated to analytics called the data arena it's a physical room that you go into and you put on some goggles and you get presented vast datasets and you can actually engage with them in a very tangible way moving your hands and also the goggles give you almost like a virtual reality experience do you think there's any scope for bringing virtual reality and analytics together with Moodle well I mean it sounds awesome sounds like my nollie report or just on that I saw a I saw slightly seagwaying but I think it's relevant I was driving through I was walking through Sydney here yesterday and there was a big add on so on the side of a bus for I cannot remember the university with my university with a person standing there in reality glasses and saying we have made accounting more exciting with VR so if you can make accounting more exciting with VR you can certainly make analytics so it's problems been solved by someone they just need to adjust it a bit I think I don't know how they've done it though I don't believe them look you you won't find a much bigger fan of VR than me here we have we have a VR room at Moodle HQ and we I have one at home as well and I have had since the vibe came out so for quite a while however I don't think it's useful here the computer should be doing the analysis in my I would I would love to boil it down like I said to just little messages like get the computer doing the work I think throwing heaps of data no matter how fun that people was not it's it's not practical it's I think AR is where it's going to get very interesting but again you don't want all that data overlaid on your life I think one other important bit of analytics or bit of data that we're missing and I talked about it in my session and it's something that I know we all deal with but we don't think of the importance of is that as it relates to retaining students and so forth is what we call week zero data so it's the data that you know about a student before they even start you know first in family ethnicity where they've come from are they mature students are they undergraduates or did they finish in year 10 all of that sort of stuff and I think when you start to meld that data with their learning data the actual meaningfulness of that is much greater so I think you know you need to be looking at systems that are bringing not only the data that's being generated out of noodle but adding into that that other data because it can tell you so much there is a lot of historical data out there and there's a lot of things that we know about students from where they come from and where they lived and and all those sorts of things and I think we can learn a hell of a lot from that now that data is sitting in your student management system it's a case of bringing those two together and it'd be great you know if if Inspire had some mechanism of bringing those two things together starts getting starts getting creepy though doesn't it does yeah I was just going to comment that that does once you do start doing things like that that does bring a lot more privacy concerns in so if you start predicting that females are more likely to pass the maths test and males and you send an alert to all the males and things like that might be a problem but it's actually mentioned in the EU data privacy regulations if you start processing that class of data that's classed as highly private data and then it has flow on implications for your reporting and how long you can keep the data and the kinds of things that you need to mention in your privacy policy when you're communicating that data to your users so there are a lot of you have to trade very carefully it brings the other aspect of minority report in which was that he was accused of crimes he hadn't done yet so you know our university is going to start you know cutting students go we know you're going to fail hey I haven't I haven't failed yet like I mean it's an extreme example but like are we are we not giving people a chance also we close the time all right getting the getting the wind up every last question down here is there going to be any facility with the inspire to be able to add in information about once the student has left or completed a course and you can feed that in and see where they've gone on for work purposes or gone to other courses or any of that kind of thing so that you can get that more longitude and all information anything like that well it's designed as a an API platform for everybody to send us examples of things that we want to feed into this engine in future so no not at the moment but that is the intention that people can come up with suggestions for things that might be useful and send it to us and get it incorporated into the system yeah I'm hoping in a year that we'll be seeing dozens of analytics plugins for inspire that take it in different directions as you need to okay I think we're at the end all right thanks Damian Tom Martin Grant Andrew good on you