 So welcome everybody to tonight's Australian Evaluation Society seminar. It's a pleasure to be here with you all. My name is Jessica Haley Brown. I am a member of the Victorian Australian Evaluation Society Committee and I'm also a director at the Centre for Evidence and Implementation based in our Melbourne office. Before we kick the webinar off for real tonight. I want to start by acknowledging the traditional owners of the lands on which we are all meeting today and pay my respects to elders past, present and emerging. I myself am on Wurundjeri land this evening and I invite you to type in the chat the name of the of the land that you are on this evening. So this event tonight is hosted by the Victorian Committee of the Australian Evaluation Society, or the AES. And the AES, as I'm sure you know is a member based organization which exists to improve the theory practice and use of evaluation for people involved in the evaluation space. And this includes evaluators, managers, teachers and students of evaluation as well as other interested individuals. And today we are really fortunate to have Dave Taylor, my colleague and friends with us today to lead us in an exploration of tech tools and apps that can support evaluation work. Dave is a senior advisor in quantitative methods at the Centre for Evidence and Implementation, the same organization that I work for. Dave is based in Sydney. He is an experienced economist, quantitative analyst and systematic reviewer with a background in public policy consulting. He applies these skills across a range of complex evaluations in health and human services, and he's also a PhD candidate at Monash University. At CEI Dave has a reputation amongst his colleagues for being an early adopter of tech and he enjoys experimenting with tech tools and apps to enhance his workflow and evaluation products. So there is really no better person to guide us through this today that I know than Dave. But before I hand over to him, I just want to mention a few housekeeping points. So of course, if you're up to your eyeballs with homeschooling and you're cooking dinner and you've got a million things going on, I totally appreciate you might want to keep your camera off. But it's so much nicer for our presenters if they can see you and interact with you. So if you're in a position to do so, I do encourage you to put your camera on if you're in a position where that's appropriate. Stay on mute the whole time if you can, unless we call on you to ask a question and answer a question. It just helps the presenter to not have that background noise. However, I encourage you to make really good use of the chat function with your questions along the way. We'll pause along the way. We'll also likely have some good time at the end. I'll moderate the chat and monitor that. And I call Dave's attention to particular things that come through, so make really good use of that. And we'll have plenty of interaction using some of these built-in Zoom tools. So without any further ado. Oh, one other thing I'll say that's quite a lot of us on this session now, more than 50 of us, I believe. So if you use the hand up function, I'm likely to miss you. So please type it into the chat and I'll be able to keep my eye on that a lot more easily. So without any other hesitation, I'm going to hand over to Dave to take us through his session. Thanks Jess. I'd also like to acknowledge the traditional owners and the lands in which we're meeting tonight, which for me is the Gadigal people. I'm here in Sydney. And as Jess said, I'm Dave. I'm a senior advisor at the Centre for Evidence and Implementation. I've been here for almost five years and I've been lucky enough to work on some really interesting evaluations in the human services space, particularly in youth homelessness and child and family welfare. Also, I've done a chunk of evidence synthesis work. And as you can imagine, you get to wear a lot of different hats, these sorts of projects. It's certainly not a dull day. Some days you need deep in project management. Other times you're running focus groups. Sometimes, and these are the days I love the most, I get to sit down and write a lot of code. And over the last couple of years, I've tried to look for some tools and apps to try and make the work that I'm doing a little bit easier. So that's sort of where I'm coming from. And hopefully you'll find some of these things a bit useful. Cool. So what I'm hoping you'll get out of this. I'm hoping that you're going to log off tonight and have an idea of some new things that hopefully you hadn't heard of before. And they may give you some inspiration for ways in which you can apply them in your work. What, you know, I'm not going to do is, you know, convince you that, you know, as the cartoon says you can automate everything and suddenly this is just going to make your life fantastic. I'm also not everything is going to be relevant to everyone. We all do different types of work. Some things are going to resonate and some won't. But I will say if you have some specific like use cases and go, you know, I've heard that you can use this for this, you know anything about that. I encourage you to ask some questions. I'm by no means exhaustive. There are there are many ways in which we can use these tools and tricks, and I may just be scratching the surface. But I'm clearly not free of bias. I have certainly have my favorites here and I'll make a point of calling them out. But that doesn't mean that the other other things I'm talking about are not bad. I just might not have as much experience of them. And there may certainly be things I've missed as well. So, before we kick off, I'm keen to get a handle on a few questions just to know where to pitch some of my content. I've got a couple of questions that Jess is going to pop into the chat. I've launched it in a poll. I hope everyone is able to see that. Thanks, Jess. We have about half of our participants responding. The numbers still going up. What if it's two? What if it's both? You can only pick one. Oh, you have to pick one. The primary one. Just a handful more people to respond. Give you 10 more seconds. That mixed methods option didn't show up in the poll. If you scroll down slightly, it's heading just off the bottom. Thanks, Dave. Okay, I'm going to end the poll now. Can everyone see the results there? I actually can't see it, Jess. Yeah, okay. Let me try this. Oh, here we go. Yeah, great. Okay, cool, cool. All right. So about two thirds of us work in a team, some of us on our own. We've got a few people to commission the evaluation work too. Nice. And good chunk of mixed methods, which I put my hand up in there too. Awesome. Like, thanks team. That really helps to know just sort of where things are. So each of the things that I'm going to talk about tonight, they sort of fall into three buckets and they're certainly not mutually exclusive. There are some things that are, you know, might be really speak to you to working alone. There are some that are going to really be really helpful working in teams. And then there's a couple of little things that are specifically, I think, relevant to the world in which we are living in right now, which is particularly sort of collecting information when we can't work face to face, or when we can't, you know, meet other, either our colleagues or people we might want to engage with to support our work. So I've come up with eight topics that I've developed after chatting with colleagues and having a think about some of the problems that I've come across over the years. So each of these, so, you know, some of these like a startup with some quite simple ones just just arranging life admin or project admin. So finding times for meetings. How do we create collaborative agendas. And I'm not going to say automate, but take some of the drudgery away from some of these tasks that can take up a bit longer than we'd all like to admit. Keeping track of our, you know, our articles of things of interest that I think is a really useful tool. Project management, keeping up with our tasks. How do we communicate with our teams, particularly in pandemic times. Making the most of our institutional knowledge, whether that's just one of us or our teams or organizations. And that then as I hinted at before things that like, you know, running and engaging interactive focus groups and workshops. And finally, it's a couple of things here that I did in my heart from making the most of open source programming languages to support your quantitative work, and then also how to manage your code and your workflow that goes along with that. So, I, I spend more time than I care to admit sometimes trying to arrange meetings. I work with some particularly time poor people. And, you know, I'm sure I'm not the only person who has been on the phone to multiple. I've always been trying to to the upper meeting and been just going backwards and forwards and rounded around going, there's got to be a better way. That's just much more efficient. And it turns out there is. I'm sure that many of you will have, we know will be aware of of tools like doodle and straw hole. But for those of you who are not aware. Hopefully this this is something that could be quite useful. So basically arrange your web based tools that you can use to sort of coordinate meeting times of the, you know, your colleagues. This is particularly useful when you're working with external partners or perhaps you're working with a client, and you can't just fire up, you know, outlook or Google calendar and just see where there's a hole. Why should you do this? Well, you know, the less time you spend organizing meetings, you know, the better. This is just one of those like no brainers that I feel like. Yeah, it's super great. Are there any particular like programs or apps here that we look at and how and how can you use them for so for the arranging of the meetings is that there's two that really stand out. Doodle is the ubiquitous one I'm sure most of us here will have will have copped a doodle poll at some point. But recently I've come across straw poll which is on a slightly cleaner interface they haven't got as many ads. And I think it's just a little bit neater and a bit more professional. Up to you though they both have three tiers so you, you know, and then there's also a sort of a freemium tier where you pay a bit more and your movie ads and you get a slightly nicer look at feel. The other app that I wanted to mention was was hyper context. And this one was previously known as soapbox. And this one is really quite useful for creating agendas and tracking meetings. So maybe you've got a standard project meeting. Maybe you've got a one on one that you want to keep track of and then suddenly, you know, you're digging through your inbox trying to find what were the actions in the minutes from last time. And the beauty behind something like hyper context which I've used in a couple of projects is just to it brings these things forward and centralizes them. So at one point you've got who turned up last time. What were the things you agreed on. How many of the things are done. And then they just get bought forward to the next one. So you can save time. And the best bit about it is everyone can sort of add in all of the users can add in their own actions or items, which means not necessarily one person gets stuck with the very love job of taking actions and minutes. As you can see down the bottom, where do you get it. Plug those into your into your URL and you can download and or actually just their all online apps with with free trials. Dave, before you move on, I've noticed a question in the chat and I think it might be helpful to make comment on this earlier rather than later. Some people in their organizations have quite a lot of restrictions on the kind of software they're allowed to download and tools that they can use. You mentioned just then that those three tools are all web based apps. Can you talk a little bit about what that means for people who might have kind of maybe perhaps we work for a government organization with restrictions about where they can use. Yeah, that's a really, really good point and it's a really good question. So, yeah, by a web based tool, I mean, you can log into that as you might log into Gmail or Facebook. It's, you know, you don't need to necessarily download a tool. There are apps that you can get. And they are, you know, some a lot of a lot of these apps are mentioning have like apps for your iPad or your phone. And that many of these are also accessible on just, you know, through your web browser. Okey dokey. I've just got a little bit of feedback. I was wondering if everyone can go on on mute if you aren't already. Awesome. So, I'm sure I'm not the only person who has like a downloads folder, which is full of like lots and lots of articles with names like what I've got on the screen here. Maybe they're things that your colleagues have shared with you. Maybe they're articles that you downloaded multiple times. And it can be a bit intimidating trying to just wade through them and find the one you're looking for. And you can end up just going back and looking for it again, which which can be a bit of a time waster. For those of you who don't have access or to or easy access I should say to to articles whether they be you know reports or academic articles. It can also be really hard just to get your hands on them. And it's got to be an easier way to sort of share these things amongst your teams. And it turns out there there are. For those of you who have come across, you know, tools like end note and mentally and Zotero during your studies. They also offer like an incredibly useful way to manage a library. And this is something that we've done at CI to great effect that where we can use we can we can create teams can create a bunch of articles store a bunch of articles in a folder that are relevant to that project. And we can read them. And we can also use them as you would referencing software so you can cite these using a plugin available to word. Now mentally and Zotero are both available and actually and note all have web based versions as well as actual apps that you can download. And yeah, they all have the sort of pluses and minuses but they're they're really really great to essentially add is like a as a library is like a knowledge management tool or say no knowledge management tool but a way of sharing sort of resources amongst your team. So this works great both whether you're alone or in an organization. The one thing I would point out, you know, the for IP reasons you know you got to sort of maintain fair use if you work for a very large organization and you're making an organizational wide library with 2000 users and you're sharing academic articles the copyright police maybe a little bit upset with you. So you just get a sort of make make sure you're complying with fair use there. That will depend on your circumstances. There are three big players and really they they offer similar features and note is the big player it's been around for a long time, it's primarily referencing software. And it does that perhaps the best. It's not the greatest in terms of like collaboration and and and sort of being able to share this with your team if you're working alone it might be a great option and some, you know organizations, particularly academic affiliated ones might have an institutional subscription to it. But for those of us who don't, it's an expensive piece of software and if you're getting that a few times over. That would be the right option for you. Both mentally and Zotero on the other hand, offer what you call freemium versions. This means you can download it, use it. But then once you get to a certain size, you no longer you might have to pay to get to another tier. I'm a fan of mentally. I've just used it for a long time, but I've heard great things about Zotero and people swear by it. So that both mentally and Zotero doing quite well is that you can have a library locally on your computer, and you can also sync it to the cloud. So, over time, you might start to collect a few a few articles and if you're anyone like like me, particularly because I share this with some of my colleagues, we have, you know, quite an extensive library and you don't necessarily want to have access to this so locally, you can sync this to the cloud, and you can sort of, you know, either download the articles you need, or, you know, have all the ones on hand that you want access to. And as I mentioned earlier, they also function as referencing software, which is their primary goal. So that it's a really useful tool to help sort of take take charge of your, you know, all the articles that you might have on hand or one access to. And you can find them all at the URLs on the screen now. Dave, there's a bit of interaction going on in the chat about these. I might pose two questions. One is someone has indicated that they've had an experience where some of these platforms more than others have problems with dealing with grey literature, but then you might need to take a more manual approach when putting grey literature in versus academic articles, can you make any comment on that based on your experience. I sympathize. I've had that experience too. So the idea, I believe the comment is, when you import a PDF say intermendially, it tries to understand from the metadata in the PDF, who the author was what the title is, and where it was published, it tries to make things easier. Sometimes it gets that right. Sometimes it gets in a horribly wrong, particularly if you're dealing with part made articles. It's not the greatest. In those cases, yes, you'll have to sort of manually make adjustments to that. But it's not terrible in, you know, in terms of getting most of it right. Thanks, Dave. And there's another question, which feel free to say, you know, we'll get to it later. But someone has asked which is better for text analysis. Oh, by text analysis, do you mean that like, can I search the articles? Andrew, do you want to jump in with a clarifying question? Yes, that's what he means, Dave. Yeah, cool. I can't speak to Zotero. I know that if when you open up Mendeley, it's got a PDF you have baked in. And when you open that up, it will search your ability to search the article is dependent on the PDF. So, you know, sometimes, you know, a modern article published in the last 10 years, you'll have great searching capability. It starts to get older than that. And that's just the limitations to the PDF itself. So I think it would depend on what you're trying to read. Great. Thanks, Dave. Cool. So for those of us who are working on a whole bunch of things at once, which I'm sure is all of us. And you're looking for a way to help you prioritize your time. We're living in a golden age for project management software. There has been an absolute explosion of this in the last couple of years. And ideally, they might help your team work smarter. Now, this can be really great for, you know, whether or not you're alone or at a team. So it's really good to have a to-do list. And with both within a project and a team-based level at an organizational level. At a, you know, if you have a project, you can quickly get a zoom in to see what members of the team need are working on. What's, what needs to happen. It hasn't happened without sort of, you know, doing something more manually like, okay, let's have a project catch up. You can see and assign tasks and mark things off. And not just from a sort of a project management way. There are also a lot of them have different templates where you can use for all sorts of things, whether it's like HR or onboarding or meeting agendas or, or specific project types as well. The classic one here we're looking at is a Kanban board on an app called Trello, where you can see like, you know, this is a to-do list. This is my list that I'm doing. And these are the things I've done. You can color code things based on, you know, like maybe you want to put them in traffic lights because there's a, you know, things might be at risk or running behind. You can add to these, these tiles, you can add files to, you can assign them to individuals, you can add due dates. That's Trello is a particularly, you know, customizable tool. I quite enjoyed using that one, but it's by no means the only one. And as you can see from the screen, there are many, many options in front of us. Now they, these are all web-based tools, so you don't have to download these apps. And your, which one may or may not work for you is going to be very dependent on you and your organization's preferences, the style of work, what you want it for. Each has their strengths and limitations. You may not like any of them. That's completely fine too. And at CEI, we've had, you know, particular, my colleagues have had particular success with working with ClickUp. ClickUp is a, has a lot of free features. And I think that's sort of really sparked sort of increased interest. Summer is a really popular tool as well. I have some colleagues that are using this on a multi-year randomized control trial where they have to manage data collection. So it's a great way for the project manager to oversee where are each of the things, who's doing it, when are they doing it, and the research assistants in the field can mark out, like, this is what happened. And so just feed the information back to the project manager. And it provides, like, great visibility with just, you know, where things are up to and they can sort of problem solve on the fly. All these apps as well, like as well as being online, they've all got phone apps, apps your phone and your iPad as well, which you may or may not like because that means it works following you around everywhere. So I find that useful. Just choose where the terminal notifications on. Trello is also a useful tool. I showed just earlier in the previous slide, you can see how there's three columns here, you know, to do, doing, done. You can make those whatever you want. And I've spoken to some colleagues to actually use this when they're doing, you know, qualitative content analysis. And they've chosen, you know, columns of different buckets they want to put, you know, different information in and they've used this to as part of their analytic process. And so that's, you know, that there are multiple ways in which you can use these tools, not just for sort of your sort of standard to do list. And I will say though, before you start running off and googling these things the moment you do, you will get nonstop ads on YouTube for one of these things I get Monday.com. Every time I want to watch some Premier League highlights, it's just it's relentless. Once they know that you're interested in project management software. It won't stop. So that's that's the word of warning I might provide. Okay, so how do we communicate with our team in a way that's not email teams chat or text message. I feel this is especially important in the times that we're living in. How can we sort of like not just support collaboration but just sort of check on each other. And these days is like a, you know, a couple of like, you know, tools that people have tried to market as email killing that and as well as that support collaboration and help teams sort of communicate and work together better. Now, why, why would you want to do something like this. So what I'm going to do is with particularly intimidating email inboxes. This is a way of sort of like reducing those sort of like couple of line emails that could suddenly to be a chat. You'll never leave someone off an email thread again. If you have everyone in your project team on the same sort of communication tool. One of the most important thing, one of the most important things is, is he in a lot of these tools have a much, much better search functions. So you can go back in time and find out or what did you agree on on this project like 18 months ago, which is surprisingly hard to do when you're looking through outlook. So what are the, what are the main players so there really is the elephant in the room here is slack. It's an absolute behemoth. It's the industry leader. And I'm also a slack user so I'm, I'm highly biased. I really like it. It's, it's a really great tool. It's, and also the, I should mention that the, there is also an open source alternative called rocket chat. And just in the interest of like having competition. I haven't used rocket chat, but it's based on the same sort of principles. Both of these tools offer similar features. So you can have like an organizational wide everyone in your organization can join the workspace. And within that workspace you create a channel. So channel is a bit like a thread you might have on WhatsApp or messenger with a bunch of your friends. So you might say have a project channel or a channel for people that's just dedicated to a particular topic. Or particular like members of a particular office where you can just sort of put in chitchat or ask questions related to particular, particular topics. Like the slack on what I particularly like about it is you can, you know, you can set privacy on it. So sometimes you might be working on a sensitive project and you don't, you know, you need to control access to information. You can make that private. You can invite, you know, people from outside your organization to join a channel. So maybe you're collaborating with people and you would like them to have access to it, but not necessarily to other things. You can do that. And you can also join other organizations like channels if you're invited to so it's a kind of thing you can use to collaborate both internally and with other partners who might be using it. Slack has a map you can you download it for fear foam iPad and computer, but for those of you who might not be able to download that for organizational reasons you can also run it inside your web browser as well. Now the only downside with that is you lose the ability to make phone calls, which is also another useful feature as well. I would add that like tools like this are not just sort of great for, you know, communication and productivity reasons. At CI we found that like a great sort of morale booster during the pandemic times. It's a really friendly way to like just fire from message to a colleague that you might not feel like writing an email to. I think, you know, if you're not feeling like it you can react with one of the many silly emojis that have built in its own. It's really customizable and dare I say, almost fun. All right, so this is a project that's also a topic that's very dear to my heart. As someone who's sort of been at the same organization for for many years. You know, conscious that you know there's a lot of stuff stuck in my head. And I'm sure that for many of you there are there are lots of like a similar boat. Previously, institutional knowledge management used to be sort of the domain of large corporate firms where you'd have to have like a custom built tool that you know, specifically through, you know, previous cases if you're a law firm or previous projects if you're a big consulting and it was a little hard for like smaller organizations or individuals working alone or smaller teams to sort of make use of tools like this it was just cost prohibitive. In these days, there are some solutions that are much much cheaper. And I, you know, I think provides a bit more off the shelf alternative that you can really might be worth looking at if it's something that you haven't considered in a while. I think that the reason for doing this is fairly self evident. But if it isn't like this is a great way for you to sort of centralize your organizational information. So like, what are the some of the projects that you worked on. What are some of the things that you've like learned about them what be the central source of truth. If you if you're going on to your wherever you keep your files and there are like six versions of report mark final, which one is the one that's really marked final. You know, this is where you would store the information so that someone who's just starting out joining your team can jump on and go like all right so this is what I need to know. These are all my, you know, organizational policies. And this is also our organization's view on explains it. It's a, you know, a really, I think, you know, vitally important tool, then there are there are two apps that are available that I just make this an absolute breeze. One of which here, you can see a screenshot of an app called the notion. The notion is, you know, it sort of functions a bit like a wiki like Wikipedia in that multiple people can edit it. And that that's sort of a strength is it's not sort of reliant on one person who needs to make a, you know, take charge of the entire thing and run it like you might have previously imagined. Another is confluence confluence is highly customizable and goes beyond the wiki sort of function, and you can start to use that for collaborative projects. And it sort of starts to blur into the into project management as well it's a very, very powerful customizable tool of which knowledge management is one of the functions. Whether or not either of these sort of meets your organization's needs will really depend on what you intend to use it for. They're all, you know, web cloud based services you don't need to download these would add though that, you know, some of your organizations, you know, you might have rules about like where you can store, you know, content and access to it. So I would say investigate like the security arrangements of these apps to see if they meet what your requirements. Dave, can I just jump in on those security requirements there are a few questions earlier on about what might be some of the considerations that we should keep in mind if we're using for example apps to put agendas up there that might be a sensitive project or the way it about confluence, it could apply to slack as well could you make a comment on data security with these apps and tools that you're talking about. Yeah, absolutely. So yeah information security on data security is something that is really often underappreciated checking out like the terms and conditions or each of these things and whether or not they, you know, they meet your IT requirements requirements is, you know, is important. There's a reason that they often don't let you install things willy-nilly. Some of the reasons like, you know, if you're working with particular information, maybe you work in health, maybe you work in human services. If you have access to particular or have been provided with access to particular information, you will have signed up to things saying that you're not allowed to take, you know, store this information externally. So, by that I mean like, you know, these apps run on servers that are, you know, outside of Australia. So, for example, Google documents might have a data warehouse in Singapore, which might mean that you can't use Google documents to store, you know, transcripts of a focus group that you did because you only had ethical approval to store the data in Australia. That may not apply to you, but if it, you know, it's worth bearing in mind for things like, you know, whether or not you have what your data security and sharing arrangements are for your various projects and your work. Okay, thank you. So, I feel like this is one that's really important in pandemic times. Who here has had to try and run a focus group overzone. It's, it's a lot harder than doing a face-to-face isn't it. You know, some of the problems that we like, you know, face like, you know, with the loudest person in the room are amplified in things when we're in a situation where we're having it online. Also, if we, you know, not everyone may want to fire up the camera and really engage. That might find it really hard to, you know, or really just not very comfortable with engaging on an online medium. Thankfully, there are like a range of tools that may make this a little easier for you. No one really wants to get stuck in Zoom hell and some of these may offer, you know, ways to, you know, I wouldn't say increase engagement, but I'd say maintain it. Keep everyone sort of engaged, particularly, you know, if you're trying to like engage people over multiple, you know, an extended period of time. Now, I will add right that, you know, these apps are not going to like revolutionize the way you, you know, run an online workshop in that it's suddenly not going to make six hour online workshop sound like a great thing that you want to do. But they can, if you're breaking things up into smaller chunks, or you know, they can offer a way of making them a bit more interactive. Okay, so there are a range of tools here where so we've got Miro, Kahoot, Poll Everywhere, Mentimeter and Mural. I have limited experience with these, but from conversations with colleagues who have used them, they get rave reviews. The kind of features that we're talking about here, we got things like interactive whiteboards like you saw on the previous page, this here is a screenshot from Miro's promotional work. You can see that, you know, there are a bunch of individuals who have logged into this. Maybe they're meeting on Zoom, but they've used this as interactive whiteboard, and they're dropping post it notes on here so this is a way that we can still do our post it network on Zoom. And you know people get to, you know, drop in pictures, emojis, you could customize this to meet your requirements. The other builds in things that things like Poll Everywhere have been inbuilt. So an app that you can add onto PowerPoint, and you can inbuilt some sort of things like live polls and quizzes into your own PowerPoint slides. This could be particularly useful if you want to get engagement about like, you know, ask a question, and then, you know, get information and bring that forward. So if you hear like word clouds, most of these, you know, have added that as a way that you know you can sort of create word clouds and mood boards. And Kahoot, which I've used, had been participated in workshops where people have used Kahoot is, is quite a fun tool. It's, it's often used in education settings is like is like a quiz sort of knowledge reinforcement tool. It also does all these other things as well all the whiteboard and stuff but you can actually just, you know, sign up for a free account and create a quiz. And everyone can sort of do that on their phone, while they might be looking at a presentation on their computer. So these apps have their sort of freemium versions. If you want to get more features. You know, you have to start paying a little bit more, but they're all available to work through your app, that's already through through the web. And, you know, may offer what you're looking for in order to sort of engage your audience. So this is, this is like my favorite topic, making the most of a open source program. So why would you want to do this. So for those of you who do quantitative subjects at uni, they often like, you know, train us on things like SPSS and Stata. And why, because they get free licenses. And it's, it's cheap for students. And the companies do that so that when you leave you have to then stump up a couple of grand to get SPSS or Stata and all your organization does. And that can be really hard to justify the expense but you know you've spent a couple of years training on this thing and you're like I need to use something. I need to do something like SPSS Excel. What, what can I do. Now, over the past, you know, 10 years has been an explosion in interest in data science. And with that, have become, you know, sort of the rise of the programming languages so things like are and Python and Julia, and now pretty standard. You know, tools in the quantitative analysts toolkit. You know, abandoned Stata SPSS or MATLAB. Well, it's entirely up to you if if you, you know, you want to run with that. And, but for those of you who might be struggling to justify the expense, or running up against limitations. The, the, particularly, these open source tools offer a whole new world to engage in. They're open source by that I mean that they're free. They go through rapid development cycles. So you're there, they're constantly being updated. There is a, there are packages available for every conceivable type of analysis that you may be interested in. Whether it's just, I just want to make pretty charts, or I just want to clean some data through to structural equation modeling, deep learning, or like text analysis. Absolutely anything. There is usually a package or often multiple packages available to do this, based on your preferences or interests. One of the things that I've, I've really noticed over my years using them is that there's a really enthusiastic community out there that provides support. For those of you who might be intimidated by the idea of like coding. It's, you know, a journey that many people before you have worked, walked, walked on and, and then there's a lot of support available out there and resources that are really well put together that can help you introduce you to these sorts of things. And finally, and I think one of the most important reasons why coding is perhaps superior to the old point and click work is that it's support through reproducibility. So for those of you playing along at home, who may have noticed that, you know, over the past decade there's been a bit of a crisis in psychology research in particular, where, you know, people have been really struggled to replicate results that might have been done in other, and other settings. And it's been really hard to verify results, particularly because like the, you know, my little raw data might be there, the particular analysis that might have been done is is not available because it wasn't done in a way that allowed people to record what they did at each stage. Coding sort of literally is the opposite of that. You can, you have to describe at each point what you're doing. And so the idea that I, you know, I could write something, I can share it with any of you. And if you have the same data, you should be able to get the same result is really important, you know, tenant of science. So this, you know, this sort of approach, you know, really does support that for those of you who care about reproducibility and research. You know, this office and have avenues for that. So, what are your options, and does one offer an advantage more than the other. So I'm going to admit a very, very strong bias here. I'm an R user, and I'm an R advocate, and I'd recommend R. That's not saying that like, you know, Python and Julia have, you know, dedicated users who swear by it. And they, you know, you could have someone else saying exactly the same thing about either of them. Where can you get it. This, you know, really easy to find fire research engine. There are links here to R. And it's most popular. That's what's called an integrated development environment. This allows you to actually run our code. And they're both free, easy to download, updated regularly, and really, really quite good. In terms of like, you know, that seems like something you're interested in, and you want to get, you know, some help. If you typed in our help into your favorite search engine, you would be intimidated with the amount of resources. And that actually might make things difficult because there's no easy place to go to start. There is a free, there's a textbook published by a argue called Hadley Wickham called Alpha Data Science, which you can purchase as an actual book, or you can purchase the online one. So don't purchase just access the online version. And, you know, there's a great introduction where it literally just tells you how to install and go from there. So say you've decided to go down this route, or you have gone down this route, and you're working with, you know, a colleague on an analysis, and you're trying to share code. Maybe you're trying to work on the same thing. And you want to make sure that you're working from the right version. So maybe you just want to, yeah. So how can you make sure that you get on top of this there. There's a particular like a platform and a language that sort of come from the world of software development that is really useful for just, you know, data science and quantitative analysis in general. It's called Git. And it's essentially a version control platform. It supports, you know, collaboration. This rather complicated diagram, which we really don't need to need to know the nuts and bolts on shows you how principles it works. But the idea is that you have a central repository. Normally, you know, it can be up in the cloud, where you store the centralized version of your code, you grab that you make some edits, you send it back, and then your colleague can then grab it from the central repository. So the central source of truth remains up in the cloud. And the best bit about all this is that you can go backwards and forwards in time. So if you realize that actually, you wrote something last week, it was really, really good. You accidentally deleted it, you can go back and bring it back. And you can really, really support working collaboratively, particularly, you know, on the same piece of analysis. You can write something, you can send it up to the repository, your colleague can have a look at it, give it a quality assurance. You know, to check if they get the same results as you. And then, you know, either, you know, make some edits or mark that is done. But it's not just really good for, you know, organizations, you know, it's really just good practice for anyone working alone. And the reason I mentioned earlier, like you might accidentally delete it. Or like, you know, in the case of my dear dad, who called me this week and said, ah, I've lost the code for my research project, can you help me find it? So where is it on a thumb drive. Okay, it's gonna be fun. This is like a lesson in why, you know, if you use GitHub to store the code, you wouldn't be having this sort of conversation about like how can I restore something from Dropbox. It's a really, really powerful tool that's free. And there are multiple ways to sort of engage with it. So at a higher level, so like, you know, get is the language itself. But there are like four platforms you can use to make sense of it. GitHub is the most famous or well known. There's also GitLab, Bitbucket, SourceForge. There are, you know, free and freemium options and a range of like, you know, tools and programs you can use to sort of take your code and send it up to one of these things. So GitHub's got, you know, for the, for the novice user, the most like accessible documentation, and if you so can GitHub help, he would be like, you know, it'd be a good place to start. The thing I would add to bear in mind, though, is these these platforms are based on the concept of open source software development. So their default is to share with the outside world. So it's hard to do that. You know, maybe you're doing some analysis that you don't actually want to share. It might be, you know, code that you consider your intellectual property. It might be draft code that you don't want anyone stumbling across judging you for. So you can set your projects to private. Some of these platforms they offer that free some of them you have to pay for that. So create like, you know, an organizational account on one of these platforms and then invite your colleagues to it or collaborators and then control access. What I would mention though is that you should never store your actual data on these platforms. For the same reasons we mentioned earlier, your data should live somewhere safe and secure but this is just a place for your code. To get it, fire up your favorite search engine. Really easy to get up and go. All right, so we've got a few minutes left. So some questions. I know we're probably not going to be able to get to everything. So just preemptively say that like, you know, if you don't have your questions answered, or you want to have, you know, you have something that you are wondering about a bit later, you can click me online on Twitter, or LinkedIn, more than happy to have a chat. So just throw that out there. But yeah, thanks everybody. Thanks so much Dave, we do have a couple of lingering questions in chat so I might start with those and invite other people to pop other things in chat and or follow up with you on Twitter and LinkedIn given the time. So to backtrack into the open source software like our Python and Julia. If you've been trained in SPSS or starter or MATLAB. Is there a particular option that offers the lowest barrier entry so if you are coming at it with skills from SPSS or skills from starter. Is there one that you would recommend starting with. This is really down to personal preference. And I personally think that because R is, is designed primarily as like a quantitative analysts sort of language, I would suggest that's a good place to start. Julia and Python are things, you know, they're high level programming languages, you can use them to, you know, people. They're more than just sort of they go beyond sort of the statistical analytics kind of thing so R is probably a good place to start but I don't want to just count the other ones. If someone has gotten pre-skilled in using the syntax function in SPSS, do you think that would be a skill that translates to coding in R? I think it would just translate to, you know, coding in general principles. Yeah, it's. Thanks, Dave. And another lingering question in chat from a while back was about Slack and about whether or not you'd recommend that for use with externals like clients and so on. That's a great question. As with all these sorts of tools, it would just depend on what you want to share. You know, it's the same, I think the same thing would go with say things like Trello. You could have an external facing project that you wanted to track like, you know, maybe you used it to track your action items. Maybe you wanted to centralize one of those like risks and action registers that, you know, you often see doing the rounds and email inboxes and spreadsheets. You can try putting them one of these apps to make them, you know, a little more user friendly. It's really up to your personal preference. If you want to do that is absolutely no reason why you could. Thanks, Dave. We might draw a line under it there. I can see that there's still comments and questions coming in but in the interest of time we will let people log off and go and get dinner. And Dave on behalf of the AES Victorian committee, we want to thank you for for your excellent session today and I want to thank all of our attendees for their engagement for their suggestions that came through during chat as well really cool to have the live mind getting activated and for all of your questions and comments so really appreciate everyone's participation tonight.