 Hello and welcome to the Enriched Community of Practice first Zoom call. I'm Ann Buchhous from the PAWSI Supercomputing Center and I'll be introducing today's webinar. As context, Enriched Community of Practice is a community of practice of educators, teachers and trainers in technical areas such as research, e-research and computation. It's a place where we can gather and chat as educational professionals and practitioners to share challenges, ideas and information. It's open to all and everyone is invited to join. The topic of our first Zoom call came out of a suggestion at the ARDC skills summit a few weeks ago and the webinar will focus on using data metrics to show training value. Kim Gerwitz from Elixir will be looking at this topic in two contexts. The higher level context is one of sharing impact or value of training to your institution or business and in the second context or the lower level context, Kim looks at the value or evaluation of particular courses or trainings. We're lucky to have Kim with us today from Elixir. Elixir is an intergovernmental organization that brings together life sciences from across Europe, including bringing together life sciences training across Europe. Kim holds a role of training impact coordinator for Elixir. She's joining us today from the University of Cambridge in England. Welcome Kim. Hi Anne, hi everyone. Thank you so much. Thank you for the invitation to present today and for that kind introduction. Would you like me to begin sharing my slides now? Yep, that sounds good. Thanks Kim. We'll jump straight into it. Okay great. So today I would like to share with you all Elixir's strategy for assessing the quality and impact of training as well as an interface that we developed, an online interface which we call the training metrics database which streamlines the way in which we collect and store this data as well as report on it. So Anne did mention a bit about Elixir but just to give a little bit more context. So this is a pan-European research infrastructure comprising 22 member states across Europe and it's structured around five platforms of which training is one and is, we see as very important to the capacity building efforts of this collaborative initiative across Europe. The training platform in particular aims to establish and implement best practices in bioinformatics training and for this we're developing a toolkit which we hope will be available soon. We hope to support training providers across Europe and we do this by focusing on delivering training at three levels so for developers, researchers as well as trainers and lastly we aim to build a sustainable training infrastructure across Europe. I've included a few icons of some of the organizations and groups that we collaborate with and we really try and do link up as much as possible with what's happening elsewhere. In terms of the objectives for training quality and impact in particular we focus around I guess three main areas. So it's to assess the audience demographic being reached by our training events. In the short term we look at determining the quality or assessing the quality of these events so this would be directly after training and then we look at impact in the longer term so six months to a year after training contacting participants again and looking at how things might have impacted their their careers and all of this information is collected to firstly improve training quality but also to feed into best practice for elixir but as well as the wider training community. In order to to achieve these aims we collect metrics various sets of training metrics the first set being metrics about the events themselves so these are things like the number of participants that attended how the course was was funded what other what other structures were involved etc demographic metrics things like what is the career stage of the participants so gender what is their country of employment and then quality metrics so these are things like how would they rate their satisfaction with the course would they recommend the course to other to others etc and then lastly the impact metrics examples of these would be do you feel you're you're able to explain to others what you learned in the training would you would you still recommend it even after all this time that has passed maybe did the training lead to or facilitate a publication or submission of a dissertation or some kind of collaboration etc so those are the types of things that we look at and in in developing the strategy for what metrics we'd like to collect as well as the way in which that information is collected we've worked very closely with the elixir training coordinators group so I mentioned that elixir comprises 22 member states each state has a coordinator of training and this group I guess forms a really great community and sounding board to developers information so perhaps parallels could be drawn in the context of the enriched COP in terms of coordinating across centers and I'd also like to mention that our strategy as well as the metrics have been approved and endorsed by our elixir head of nodes committee which is an important structure who feed into developing and agreeing on the the technical and research strategy for elixir as a whole and then lastly just to give a sense of scale this the strategy has been adopted by 19 of the of the elixir nodes so we have and then and then perhaps just just to mention that each node might then comprise tens of of institutions you know within within each country so it is quite a large collaborative effort and then for the context of this particular group who might be interested in implementing something similar I've just included a quite a lot more detail so it's it's quite a text heavy slide but hopefully something to refer back to should you want to in the future so to mention we we use feedback surveys quite quite heavily that's our method for collecting these metrics directly after training has happened that's when we send out the survey about the how did you find the quality of the course all of those metrics and we took a lot of time to define these questions as well as the associated answer scales and that was to facilitate data analysis and it's obviously much much easier than mining free text while free text is really rich and we do recommend you know including a comment box for all the questions by having defined answer options as well it really helps one to to bin those answers and make comparisons across courses and that speeds to the third point which is all about consolidating this data collection and this could be at many different levels so whether it's across a consortium such as elixir perhaps it's even different courses within a single training program or even for multiple occurrences of the same training event by having very specific metrics and associated answer scales and one can you know compare across those courses perhaps see how things have improved or changed and so we found that a really important aspect in addition collecting the event data was very important so these were things like I mentioned earlier the funding source in addition you know the start and end date of the course and this helps to contextualize findings so you could essentially pull out a subset of the data maybe you're interested in a particular time frame or maybe you're interested in online online courses in particular so by having these training event tags one is able to do that I'd also just mention a few I guess tips things that we found very useful in order to get people to fill in the forms perhaps useful to allow time at the end of the training event itself full participants to fill in the survey you know if you tell them to do it afterwards perhaps they'll they'll forget but allocating 15 minutes at the end of the time really does seem to increase the response rate and in addition it's important to allow for provision of collecting of this longer term feedback or what we call impact assessment in the project planning stage so I guess it's very easy to send out a survey at the end of a training event people have just attended so you know perhaps they're likely to answer but contacting people again six six to twelve months down the line you know things people forget and maybe maybe they haven't even given consent to be contacted so it's important when when setting up the your quality and impact assessment strategy to think about that long term right from the beginning how are you going to contact people again do you need to consider anything in terms of data protection so in in Europe we're very concerned about the GDPR or general data protection regulation which stipulates that you know you can only contact people if they've given you explicit permission etc that might be something relevant to your context and so just to put it in a bit of perspective maybe giving a case study I'd like to speak about the elixir training program and and some of the metrics that we've captured for for elixir so and I think I think showing the sense of scale also provides a bit of rationale for why we decided to create an online interface to help us collect all this information as you can see it's quite the initiative was quite large between September 2015 and March 2019 we had over 850 training events that were recorded over 19,000 people trained over 2,000 days of training of those 800 events we collected feedback data for about 400 of them and as I mentioned 19 of the elixir nodes were involved in this in this effort the reason why we can report on these numbers is is because of our insistence of collecting those event metrics for for each event so perhaps that's it's an incentive to collect that type of information because when pulled together it really is quite impressive you know you might just be collecting it one by one for each event and things are all over the place but when you compile that and consolidate all the information I think that the numbers really do speak for themselves so that brings me to this training metrics database that I mentioned so for the for the training coordinators it's all about streamlining the collection storage visualize visualization and and reporting of our training quality and impact data and then for stakeholders or you know perhaps for anyone who's interested in in viewing what we've done it allows individuals to view and generate dynamic reports of our training data and have a look so at the bottom of this slide I have included the URL to the training metrics database and you are welcome to visit that site it's it's open access and you can have a look at the type of things we collect and and view some reports um I'll just show a few a few of those slides now um as a demo so some screenshots from the website when you land on on the homepage you see a dashboard which shows um summary statistics of the um of the initiative overall these slides are perhaps a month or two old so I think the numbers are a bit um a bit larger than what's then what's shown and I think that that's important to to say because the um the training coordinators have really been very central in shaping this initiative and and keeping them engaged has been really important so the only reason the numbers go up is because the training coordinators are using the database to upload their their information so any type of coordinated initiative like this I think really needs the buy-in of of all the the the partners and if they can be involved in developing how that looks um I think that it definitely strengthens the the longevity of the initiative so on this dashboard there's also an interactive map which shows the distribution of of training events elixir training events across across Europe and actually across the world and one can zoom in um again I invite you to go have a look and see one can click on um specific training events and uh have a look at information about those events in particular but if one is interested in in just all the elixir events maybe kind of seeing an overview one can go to this all elixir events page on the left hand search bar and that page looks like this it's essentially a table that lists all the elixir events uh training events that are in the database and one can apply various filters to have a look maybe one is interested in a particular funding stream or event type etc and and those um lists of events can actually be uh exported which might be of of interest to perhaps a elixir uh heads of nodes for for example um so that's that's a view of the training events that one can also view reports on um on some of these statistics an example being the events reports so this this would be all all the information shown in the table but just um represented graphically and again one can subset the information using the filters at the top the page that's just showing some of those graphs that we can generate perhaps you're interested in looking at the training quality again um interactive reports can be generated showing those those results um within this database we have also included um a list of all the metrics that we capture as well as the defined answer options and this might be a good resource for some of the members of this group that would like to see the the details of what we collected if you go to the um the help tab on the database and um follow the references link you'll you'll come to this page that shows all the references um and you can get a better idea of the of exactly what what we captured um i'd also like to make mention of training in terms of the wider socioeconomic impact um what what we focused on in elixir up until now has been very much about um uh i guess the um participant view of how the training has impacted them um how they found the quality etc but i think it also uh training is also important to consider in terms of the socioeconomic impact and um how this relates to financial investments so elixir training has been involved an initiative called ri paths um which is a research infrastructure impact assessment pathway they're aimed at developing a framework to describe the socioeconomic impact of of research infrastructures there are various strands here that you know economic um social political and and then human resources which i think training falls falls falls within so we're we're collaborating with this ri paths initiative to try and inform um how we determine the socioeconomic impact of of training um with that i would just like to thank those that have been involved in this initiative again it's very much been a community effort and there are some of the the faces that have contributed and lastly um thank you all for for your attention and i'd be happy to answer um any questions thank you thank you kim and we do have a few questions i might just uh not pop up my final slide yet as you may want to demonstrate a few things um so are there any publications on the training metrics database or publications using these statistics to demonstrate the value of elixir training so we have just put together a draft which we hope to submit within the next few weeks um so hopefully there will be a publication in the works um imminently and um i can for sure share that with with ann and and and this network that's great that was a good uh when it is published that was a good advert for the upcoming publication um and as well a thanks from that participant as well the second question your infographic indicated that you collect feedback from about half of the training events can you please tell how you decide which events you want to follow up is there a minimum length or for example would you seek feedback from participants of a three-hour workshop etc so that's that's a very good question and i think speaks to um the fact that this initiative developed alongside training programs that were already taking place right so we have all these statistics about about events but maybe um feedback was not um systematically collected for for those events so part of this initiative was getting people to almost getting them in in the habit of collecting this information so the fact that roughly half um only has feedback that's because perhaps for the for the first year or two um the way in which the data was collected was not as as rigorous um you know so it i guess shows that to that progression um we do kind of think about which events to follow up with um i presume by following up this person is speaking about the impact assessment components so that would be contacting people in in the longer term um so we do try and follow up with courses that are quite narrow in terms of um their objectives um relatively shorter courses and that's just so that we can then link the measures of impact to specific skills that they might have acquired during the course um whereas if you know it's for perhaps there there is a way to do this for longer courses but in in the context of elixir most of the courses we run are three to five days in length quite focused in terms of um the objectives or tools that are covered um and and we found it was important to try and link those responses to quite specific course course types um because that helped us with with with the analysis i hope that helped there are a couple of other questions um around the training toolkit that you mentioned um early on where would we look for that when it becomes available so that would most likely be made available through the elixir website um i think i did include it in the slides which i'm happy to to share with you all i can i can send that to to an um i'd have a look on the training platform page of that website okay um i might stand corrected but yes no i'm i that that is likely where it will be made available okay and then likewise around that platform of the dashboard is that a homegrown platform or is that something that is a an application or a package that's readily available for for use so it was um developed as as as a bespoke uh platform for for us although it was developed on open source using open source tool so it was developed using drupal um pantheon site i think is is the host okay um but yes it was it was um i i worked very closely with a developer to develop the site um specifically for our community okay uh there are a couple more in the chat does the australian node of elixir the embol abr use these training metrics that you're aware of um so i know that embol is yes so embol is one of the nodes of elixir um i'm not sure whether the australian component of of embol uses these i know that embol ebi which is based um in hingston in the uk they we've we've uh collaborated very closely with them actually in leading the sub this project so they most definitely do i'm not sure about the australian aspect of of embol okay that actually leads to another question um and that is around so you talked about 19 nodes and multiple institutions hanging off of those nodes um so it sounds like nodes can choose to participate and or institutions within nodes can choose to participate um what what is the uptake roughly of of all the nodes is it 50 uptake is it uh so so is the question with within a node how many institutions um i i think it would likely depend on which of those institutions are involved in in providing training uh because because elixir is um training is only one component you know we have we develop services and tools resources compute infrastructure perhaps some of the those institutions are more focused on those activities and don't actually provide much training okay so it would it would likely just depend on each um okay and i'll see of the elixir training nodes what kind of lead time and buy-in time design time development time did you have to did a take to develop this i guess yeah uh it was so work on this project project started before i joined elixir um so including those initial discussions i'd said probably about two and a half years um from first discussing okay who is collecting these metrics what what are we currently doing how can we try and harmonize that across groups um agreeing on a set of defined metrics agreeing on how that data should be collected centrally and then and then shared you know obviously you required quite a lot of back and forth and um and negotiation um and then getting individuals to to comply and and um systematically collect this information for these courses it has it has taken quite a bit of time um to get it to the stage that everything is now in a single database all the data is you know nicely there and can be viewed in in a single place um there were probably steps along the way where we where we could show things you know it's it's not that it's it took two and a half years before there was anything to to show for the initiative but in terms of getting to getting it to a place where where i feel comfortable that um everything is is in one place we don't have a bit of information in this random spreadsheet and then on that website and or or consolidated and can be viewed probably about two and a half years i'd say okay and who led that who led that discussion who pushed that that as a priority um so it was one of the key objectives of the training platform okay which is um actually why they hired me to to coordinate that effort because i i do feel that an effort like this really does need a champion someone to coordinate be be persistent um and um really try and listen to what to what the community wants and um develop it in in a way that it will be used in the long term you know because there's no point in developing all these things and then and then it's too difficult for people to collect or they don't see the value in it which i think is is a really nice aspect of our database and that as soon as information is imported one can generate reports and um subset the data in various ways so that's almost one's reward for uploading the data one can then see yeah that's definitely and like you said you've got a very rich a very rich and deep and broad perspective now um and there's a final uh can we get involved in any elixir training community if we're in australia because it appears that the embol ABR is not a node i i had i had thought that i just i wasn't i wasn't sure i didn't yeah i don't think i know that there have been some discussions um with with some parties within australia wanting to perhaps join elixir as a node um i could try and find some more information on that if um somebody is interested uh i don't recall the names of of those people but would they be able to join an existing community if they're in australia um i think i think the way elixir works is that um in order to be an official member there actually needs to be um agreement at the government level and things need to be signed by the minister of of science and technology or so i think it's a bit more complex than than just joining um and i think that that has to do with the way in which the infrastructure is is funded and who contributes various various things there's you know local funding and then across across the network so um but but i but i but i do think it is possible for groups um outside of europe to join elixir and if i'm not mistaken there have been discussions with some groups within australia already about joining in in the future okay um yeah i i i don't recall those direct those specific contacts though to to share with the group now but i could could have a look into that okay um i don't have any other questions but while i am sharing again i will just wanted to thank you again um and i i have to say that i went out and i looked at your at the the dashboard and i did go in behind it to look at some of the questions and we've already started we've piloted a series of questions with a couple of our training events to uh fine tune them for for our specific situation and so we plan at the beginning of the year to roll out a standardized set of questions before and after um our key training events so thank you very much for uh providing that and we are often often running that's that's really that's really great thank you for letting you know about that okay so thank you and a virtual round of applause thank you for your time kim and thank you paula for connecting us and thank all the attendees for attending this webinar was recorded and we