 Fantastic. Thank you so much. My name is Christina Ismail and I am the director of primary and secondary education for open ed global. I will be the facilitator for this session. I'm joined by my colleague and friend Marcella, who is also going to be here helping with technology and as our tech facilitator. So if you do have questions or concerns, please feel free to ask those questions in the chat. You can also direct message us through the chat if you would like to. Otherwise we have nothing else to say because we're going to turn it over to our presenter Byron and hear more from him. So take it away Byron. Hi. Hopefully everyone can hear me just yell at me if I, if you can't hear me. Start my presentation. So yeah, I'm joining you from Calgary, Alberta today. It's a little bit darker than I expected this morning, but I was able to find a lamp to use for the presentation so hopefully you can see me. The title of my talk today is called the importance of ethics and data science students and teachers can learn the importance of being human through computational thinking. So what I'm talking about today are some of the major human centered ethical issues facing data science today. I'll give you a few examples and maybe talk about the good side and bad side of those things. Why you should introduce data science in your classroom starting today. Hopefully you'll take that as a takeaway from today and also some data science resources to get you started as well. I'm here representing the Callisto project. The Callisto project is a Canadian government funded can code project and the mandate of the can code program is to provide our students and teachers with the skills to thrive in the digital future of tomorrow. But which is probably actually quite relevant for today with all the online learning going on these days. The organizations behind the Callisto project are actually two of them. So Siberia, which is the company I work for it's a nonprofit it accelerator based in Calgary, Alberta and Edmonton, Alberta, and the Pacific Institute for the mathematical sciences which is a group of Western Canadian math researchers at Western Canadian math researchers who are like dedicated to research and excellence in mathematics. A little bit about myself I guess before I jump into the talk today. I am a project manager and data scientist at Siberia and my data science journey really didn't start by doing anything formal and data science so I'm a, I guess I did grad school as a computational biochemist at the beginning and the computational part kind of came after so I am really like a what have a wet lab background in my history. But then, as I was doing sort of my grad school, I started collecting a lot of data, and I would process it and analyze it and try to get results from it using maybe typical means like a spreadsheet. But I quickly realized that that was becoming unmanageable and I was having too much data to deal with. So I had to learn some techniques to to make my life easier basically. And so I started to learn a little bit of scripting and computer programming. And then as I did more of my grad school studies, I ended up doing, I collected a lot more data and involved sort of multi dimensional techniques as well so there's no way you can actually do that by hand. So I had to figure out a way to do that and process things and visualize things using computer programming and once I joined the professional ranks, I couldn't continue to develop those skills and sort of ended up where I am today. But also realizing that I had to pick up all those skills on my own, and there was no sort of formal education, either in K to 12 or in university that I'd taken or undergrad that I had taken that would have really like that could have prepared me for all the stuff I had to do in grad school and myself pick up myself, sort of on my own whim so I thought, you know, taking those experiences and really wondering what we can do sort of at the four K to 12 folks was sort of what an inspiration behind this cluster project as well. So, love for you to introduce yourselves I know a lot of you already said hi in the chat but feel free to introduce yourself. What brought into the talk today, things like that love to just connect with you a little bit and kind of this virtual setting. Okay, so this is sort of the thesis for my talk today so why are we scared of data science and AI. Now you may or may not be scared of data science and AI or artificial intelligence. But maybe we should consider this statement. So the biggest data companies in the world have grown so quickly that no one has stopped to think about ethics. And in general, there should be awareness about releasing data scientists into industry with weapons that they don't yet know how to use and that's from a UC Berkeley undergrad, a quote taken from a UC Berkeley undergrad. So that's something to think about and I guess the concept of weapons is kind of interesting as well with respect to sort of software. And so, before we get into maybe why we should be scared of data science I also want to share with you why I'm excited about data science and why I really got excited about data science. And so this is sort of some of the inspiration that kind of inspired me to really learn more about his sense and skills in this area. So we did the software which displays it like this. Every bubble here is a country. This country over here is, this is China. This is India. The size of the bubble is the population. On this axis here I put fertility rate because my students what they said when they looked upon the world and I asked them what do you really think about the world? Well, I first discovered that the textbook was tinted mainly and they said the world is still we and them. And we is Western world and them is third world. And what do you mean with Western world? I said well, that's long life in small family and third world is short life in large family. So this is what I could display here. I put fertility rate here, number of children per woman, one, two, three, four, up to about eight children per woman. We have very good data since 1962, 1960 about on the size of families in all countries. The arrow margin is narrow. Here I put life expectancy at birth from 30 years in some countries up to about 70 years. In 1962 there was really a group of countries here that was industrialized countries and they had small families and long lives. And these were the developing countries. They had large families and they had relatively short lives. Now what has happened since 1962? We want to see the change. Are the students right? It's still two types of countries? Or have these developing countries got smaller families and they live here? Or have they got longer lives and live up there? Let's see. We start the world and this is all UN statistic that has been available. Here we go. Can you see there? It's China. They are moving against better health. They are improving there. All the green Latin American countries, they are moving towards smaller families. The yellow ones here are the Arabic countries and they get larger families but they no longer lives but not larger families. The Africans are the green down here. They still remain here. This is India. Indonesia is moving on pretty fast and in the 80s here you have Bangladesh still among the African countries there but not Bangladesh. It's a miracle that happens in the 80s. The imams start to promote family planning and they move up into that corner and in 90s we have the terrible HIV epidemic that takes down the life expectancy of the African countries and all the rest of the world moves up into the corner where we have long lives and small family and we have a completely new world. The late Hans Rosling gave that talk and it I think inspired probably a generation of data scientists. There's a lot of hope behind data science as well especially for things like data driven decision making and data informed policy making, things like that. I think that's sort of the hope behind data science is really that we can use data to do good and to make good decisions and make good policy. So I guess for my question to you folks is I've talked about maybe a fearful statement. I've shown you some inspiration around data science. What does data science in the term make you think of? Feel free to add any comments you want in the chat there. Some context for me when oftentimes we hear about data science in the media, we hear about companies like Amazon and their efforts with their cloud services and prime and anticipatory shipping so being able to maybe predict what you want to buy and maybe ship you something before you even know you need it kind of thing. So that's one way to look at how you can use data science in the world today. Google analytics, everything we do basically on the web, Google analytics is really tracking us in some way and using that information to again maybe show us ads or selling that data to third parties and using that again to try to sell us something. Facebook for example like it started off being something where you can share a lot of information with friends, share how you're feeling, share photos and it turned into a lot of other things as well now. And again associated with Facebook is sort of that Cambridge Analytical Analytica scandal that happened a few years ago. So there's a lot of things that when you think about data science and you hear about it in the media. A lot of these things are tied to big companies. A lot of other things maybe to think about when you think about data science is careers and jobs so when we think about school and students graduating from, you know, high school and undergraduate like the data science field is actually a pretty decent job so this is from glassdoor.com of the 50 best jobs in America for 2020 data scientist is number three. Pretty decent median based salary job satisfaction is pretty high, large number of job openings that's great and if you look closely sort of at this top six list. You'll see a couple other things like data engineer, product manager and DevOps engineer on that list as well and that can also be closely tied to the data science field as well. The data science position was actually the top job in America I think from 2016 or 2015 all the way to 2019 as well so it's been up there and there's still quite a bit of demand for data science jobs in the world so I think there's a lot of, you know, career opportunities as well in this area as well. Maybe before we dive into the ethics side of it. So what is data science. So really data science is quite interdisciplinary by nature. It's a means of gaining insights from data using a mix of math, some math and stats knowledge, computing, computing skills like programming skills. In this Venn diagram we consider that sort of like hacking skills, but it really means like computer programming skills and subject matter knowledge or substantive expertise and data science kind of fits right in the middle there. We talked about a danger zone so if you have great computing skills and you have maybe subject matter expertise in some area, but you don't have those math and stats knowledge. A lot of the results you can get from, you know, applying data science tools and techniques can lead to a stray without sort of those, a sound sort of math and stats grounding as well so we think in order to have a well rounded sort of data scientists you need to be right smack dab in the middle or at least be surrounded by a team with complimentary skills to yourself that could allow you to really thrive as a data scientist. Okay. And so, when I talk about data science really I've described what a data scientist might look like having those mass skills as programming skills and maybe subject matter knowledge. How does data science fit in with artificial intelligence and machine learning all buzzwords we hear about today. So, generally speaking AI is really the big field of study. So it's a blanket term describing our efforts to make computers think like humans more or less within artificial intelligence we have things like machine learning techniques and tools or algorithms. That allows us to improve or learn through exposure to data or experience, depending on what you want to do, and data science isn't necessarily part of the research area but in order to do data science. You want to take and apply those techniques developed by the machine learning researchers and use those with real world data to derive maybe knowledge about business outcomes or policy making things like that. So, that's really where data science kind of fits into the overall picture as well. If we wanted to dive in a little bit more like what are the components of an AI system. So, again, understanding how an AI system generally works and how it can be used with real life examples so the three main components of an AI system and this is just generally speaking are, you know, you have to have data. So you have to have data that you can process and then you need to have some type of learning algorithm usually typically a machine learning algorithm in the middle there. And then the output of the AI system is some type of prediction and diving into these components a little bit more a data set is a collection of curated data and that's really important the curated word is really the important part here. It includes images, measurements, text, video recordings pretty much anything can be turned into a digital format and turned into data these days so video anything you can think of really can be turned into a data set that can be used to train an AI system. And now talking about that middle chunk the machine learning algorithm. So what happens with an algorithm or you know what's the definition of the algorithm. So like, you have some original state. So like maybe the input states, you add some new data. There are some steps in the middle that will change that state and then you have your new state which is sort of the output state so like those are the general sort of steps within that middle learning for machine learning algorithms step. And finally, you have your prediction so like there's various types of predictions you may be trying to predict the weather for the next day the results of the election. You may be trying to predict how groups can be separated into distinct clusters. You might be trying to predict text. So you might be trying to predict your next word in a sentence, you might be trying to mimic how someone writes for example, with natural language or something. So you might be trying to predict what that might look like if they were to write a text about maybe a science fiction topic or something like that. So there's a lot of different things. And so like, you know, all those things sound really great. What could go wrong really with all these things well, there's there's a few things. And we'll talk about a few examples today, where that gets highlighted. The headlines that you might, you might have come across online, for example, looking at media. So online ads for high paying jobs are targeting men more than women, women. When it comes to policing data is not benign. There's books about this topic the ethical algorithm. Obviously Cambridge Analytica which I mentioned before. So all these companies that are applying data science and AI in our world today, they face a lot of conundrums as well associated with the application of, of what they're doing with these AI systems. And one book I really enjoyed reading was called weapons of math destruction by Kathy O'Neill and this is one of the quotes that she has in there basically you can sum it up by saying algorithms are not neutral they encode our biases. In this case, these models are constructed not just from data but the choices we make about which data to pay attention to, and which to leave out. So again, the concept of curation. Those choices are fundamentally moral. If we back away from them and treat mathematical models as neutral and inevitable forces like the weather or the tides we abdicate our responsibilities so she really is talking about not just accepting the results of an AI algorithm but actually questioning and, and making sure that you know we're in a lot of ways. You know pushing back on these AI algorithms a little bit and questioning them or policing them or making sure there are rules in place to keep them from just making decisions without our direct input. So I'm going to show you sort of another short clip here that about a project which I thought was really interesting and talks about some of these issues that we commonly see in the media as related to sort of ethics and AI and data science. I'm Joy and I research how computers detect, recognize and classify people's faces. In my Ted featured talk I spoke about my experience with the coded gaze, my term for algorithmic bias. The system I was using worked well on my lighter skinned friend's face but when it came to detecting my face it didn't do so well until I put on a white mask. After my talk was posted I tested my speaker image profile across different facial analysis demos. Two of the demos didn't detect my face. The other two? Well, they misgendered me. The demos didn't even distinguish between gender identity and biological sex. They just provided two labels, male and female. Now I wanted to see if these results were just because of my unique facial features or if this was something that was more of a pattern across other faces too. So I began a project that became my MIT thesis, Gender Shades. Or the long title, Gender Shades Intersectional Phenotypic and Demographic Evaluation of Face Data Sets and Gender Classifiers. Or just Gender Shades. I wanted to see how well different gender classification systems worked across different people's faces and if the results changed based on somebody's gender or their skin type. I created a data set of over a thousand images of parliament members ranked among the top ten in the world based on their representation of women in power. To get at a range of skin types I chose three African countries and three European countries so I could see how the system performed on lighter skin and darker skin. Then I chose three companies to evaluate, IBM, Microsoft and Face++ which has access to one of the largest data sets of Chinese faces. So now with the data set and the companies I decided to run a test, the companies appear to have relatively high accuracy overall. Microsoft performed best achieving 94% accuracy on the whole data set. All companies performed better on males than females and all companies also performed better on lighter subjects than on darker subjects. When we analyzed the results by four subgroups, we saw that all companies performed worst on darker females. IBM and Microsoft performed best on lighter males and Face++ performed best on darker males compared to the others. Okay, so as you heard in the video, Joy created a new type of data set. So it's a new facial image data set which is highly balanced across skin phenotype and gender. So that's the gender shades data set and as the results of this research. So IBM for example is developing algorithms for detecting, rating and correcting bias and discrimination across modalities both for data and for models. So I think IBM was part of the sort of things that Joy tested her data set against. So it's really interesting that these companies even though they're really large and potentially intimidating, they'd actually respond to something like this where they recognize their algorithms are still kind of a work in progress and they want to be able to correct and do better with that. So that's kind of interesting I think. But really maybe something to think about as well is like how will these better algorithms be used? Yes, facial recognition programs should improve but there are also going to be commercial applications that come out of this that you might question as well. So a number of companies are considering using race classification as a way for businesses and to target buying habits of different gender groups or of different groups. So you may or may not question whether or not that's fair or not. So that's something to think about at least. And so there's more than one example of this sort of happening as well. So we're creating better algorithms or we're trying to create better algorithms to remove some of that bias, but it also opens up other sort of questions as well. Another sort of example I'd like to talk about today is proctorio example, which is around sort of exam proctorings and this has really come up as a result of COVID-19 as well where a lot of universities and are really relying on this exam pro... like this remote exam proctoring software to proctor their exams and it relies on facial identification and tracking. So it runs into the same sort of gender and ethnicity issues that the gender shades example kind of identified as well. So it has some problems recognizing people of different skin tones. It also has some other issues as well. So when you're set up using one of these exams and I haven't actually done this myself so I don't have first hand knowledge, but from what I've read it takes sort of like a widescreen angle of the room you're working in. So there might be some privacy concerns of what's going on. And there's also concerns associated with what it actually does to the individual taking the exam. So in addition to the stress associated with taking the exams, it can also cause additional stress because you don't know what behaviors you're doing may cause a system or the AI system to flag you as acting suspiciously or something. And so maybe you try not to act suspicious while you're writing an exam on top of the additional stress or the regular stress of actually writing the exam itself. So these remote proctoring exam systems are kind of interesting and for one example of maybe some of the concerns associated with it, there's a proctorial University of British Columbia lawsuit going on currently I believe. And in that case, one UBC instructor I think really these are provided links to some of the training videos that proctorial provides and it shows some information about how proctorial actually works. So some of their intellectual property in terms of like how their system works and they didn't want that information to be revealed. So there's there's a lawsuit going on currently with that. So if you want more information about that you can you can grab a copy of the side that can click on those links but again. Sorry. Can you hear me? Sorry. Yeah. I was just. No, it's from me. Okay. So, yeah, there's ethical concerns with sort of like how these exam proctoring systems are being used today. Another example I kind of wanted to talk about as well was sort of these a level exam results from the United Kingdom this past summer. So here's some of the headlines associated with it. You know nearly 40% of a level result positions to be downgraded in England. The UK exam debacle reminds us that algorithms can't fix broken systems. A level downgrades hit pupils from disadvantaged areas of hardest and maybe a contrasting headline don't blame AI for the British A level test scandal. So there's a whole slew of conversations going around what happened with the level results and I'll dive into that a little bit more. So really what what happened here so really COVID-19 impacted UK students who were sort of in their last year going into university. They couldn't take their final exams, but these are I guess high school kids couldn't take their final exams but government really wanted the results for university admissions decisions so you know very critical steps or junctures in these students lives. So in the office of qualifications off call decided to use a machine learning machine learning algorithm to predict a level exam results. And and this is I think where it becomes interesting and standardized grade inflation so there's this concept of certain schools certain teachers, perhaps, you know, increasing grades for students, a little bit above what would be considered normal relative to other teachers in schools in the in the United Kingdom so how could we potentially standardize as well so not only are they trying to predict, you know exam results but they're also trying to standardize them as well. So that's I think where they got into a little bit of trouble. And then the end result of what actually initially happened was approximately 40% of students had their A level exam scores reduced versus what a teacher would have given them so they also had teachers you know what they would would have given the students in terms of grade or predicted what they would with the students have received on the exam and then the algorithm actually reduced approximately those scores for 40% of the students. Okay, so maybe if we we look back at sort of the components of our AI system and plug in sort of a level example, what does that kind of look like. So the main components the main data sets that we're talking about here are schools past academic performance sort of in that first box, the students past academic performance so not just the individual students but the students passed all the students at that grade at the school, their past performance as well sort of how they ranked. And then you also have a third kind of component for teacher grade and rank predictions. So that information that data is then fed into a machine learning machine learning algorithm. And then that would be used to then predict standardized student exam grades. Now it sounds relatively simple but then you realize as well there are some nuances to this. In terms of when they use machine learning algorithm and how they used it really varied so one of the things they did was very depending on class sizes so if you were in a class was less than five people. You know they would completely just use the teacher kind of grade and rank predictions. If you were in a class size of five to 15 people. They would actually use a mix. So they use a mix of the machine learning predicted exam grade, and also the teacher predicted exam grade. And then if you're in a class size of 15 and above, which is something you typically see in public schools, it would just use the machine learning predicted exam grade. So that's, that's kind of where they got into a bit of trouble as well. So class sizes had a big impact on the predicted standard student exam grades. So if you wanted to take a deeper dive into this, I provided some links in here but this is an example of how students typically achieve that the grade students typically achieve under a level exams at a school. So this is one example of an input they would use for the exam for the exam prediction scores. So yeah, I provided a couple links that kind of dive into this a bit deeper. But basically, and I think a lot of it has to do with the class size issue. So if you were in a class with five to 15 people, typically speaking that would more or less come from a private school or those in those who lived in areas with the higher socio economic sort of grounding so students and lower socio economic backgrounds, with the higher classes were more likely to have their grades decreased, because they're using just purely the algorithm in this case to predict their exam grades, and they're also because they're trying to standardize those exam grades. While those in what their areas were less likely to be increased, because, or less likely to be decreased sorry, because they were mixed using a mix of the predicted and also the teacher grade so I think in some cases they would actually end up using the teacher grade. If it was higher than the predicted grade kind of thing as well a higher proportion of students from independent or private schools receive top grade so a star and a increases compared with those that comprehensive public schools. And so one of the main takeaways was that the inclusion of past school performance and class size data benefited students from private schools so you know it wasn't that it wasn't necessarily that they didn't have intentions in doing what they did. But the fact that they started to introduce additional factors. And depending on the data sets they use as well they encoded biases that tended to favor or benefited students from private schools. So I think after math, it worked as intended but seemed unfairly published unfairly punished students of lower socioeconomic status. There was quite a few protests but the fairness of the results so students definitely pushed back on these results. And you can see examples in social media where you can see students who wanted to get into medical school for example they were, or thought they were going to get into medical school ended up not getting into it because some of the results. And the level results. They basically decided that have been changed the higher for everyone changed the higher the teachers or the algorithms prediction so try to be more fair in the end that's what they ended up doing. And also another sort of side note but also related was that in the European Union they have the data protection policy framework the GDPR, and that actually places strict restrictions on organization. It's making solely automated decisions that have an effect on individuals. So that the GDPR law requires that processing to be fair even when decisions are not automated so it's interesting to see that, you know, they already have a framework in place to help, you know, keep these decisions from being completely automated. And so there are some protections in place for individuals to protect your protect themselves against algorithms and in combination with, you know, public outcry you can actually make quite a difference that you don't necessarily have to accept the results, like sort of what Kathy and Neil was saying you need to kind of push back and sort of push back against the results of these algorithms. Have I heard of instances. There's a question here about hearing instances like this happening in other countries, not specifically with respect to exam grades. I would say there are some examples like if you've read Kathy and Neil's book with respect to like K to 12 teachers for example, they do use sort of like a ranking systems on teachers so there's evaluation systems on teachers that have sort of an AI algorithm or machine system behind the scenes that help, you know, decide on whether or not teachers should be hired or like, you know, given sort of advancements and stuff like that so there are examples of that but I don't know about students directly being impacted by a algorithms. And I guess, generally speaking, you know, what's causing the majority of these AI problems that we hear about so a lot of it has to do with the data itself so there's, like I was saying there's three main components. There's the data, the machine learning algorithm and the prediction, the data makes up a huge component of this so like the machine learning algorithm is in itself neutral. But once you feed it data that is biased, it's going to pick up on those biases and you can't, or it's very hard to determine what biases are existing in your training data sets without actually going through the process of trying to, you know, predict something and seeing what the results are so it requires a lot of work to actually figure out that there are biases in your training data sets as well. In the most cases for bias, we're talking about data being unrepresentative or and or it reflects existing prejudices that, you know, we may not even be aware like we may not consciously think of but are actually encoded in in the images or the texts that are included as part of the data that's being fed into these algorithms, and the resulting predictions. There's a lot of work being done to look at those, the data sets itself, and definitely like that that gender shades example is a great example of this. But there's also an interesting example where one of the first, I guess, first data sets used for image classification like we think of image classification it all started with like, you know identifying whether it's a dog or not type of thing or a cat or cat or dog or something like that. And so one of the the first image training sets that everyone's who's done data science or machine learning has probably tried out at some point. It's called image net. And that recently was found to be quite biased, even though everyone to everyone has come across this data set at some point. And so it's really interesting to kind of go back and realize that even though we've been aware of these problems. We've still used some of these data sets without even really understanding that there's a problem with it. And that's probably the concept of you know, framing the problem like what do you want the AI to achieve. So in the case of the A level exams, you know, predicting students grades is probably quite a fair thing to do with the with the AI algorithm or could be argued to be but also standardizing the student grades across schools across teachers, things like that like that's, that's where you get into a little bit of trouble I think when you start to combine sort of your, your what you want out of the algorithm so really you have to frame the problem with a very sharp lens I would say. And so the decisions are made for often made for various business reasons or other other than fairness or discrimination so we really when you ask questions for the of the AI. So you have to be really sharp questions and other problems, just generally speaking transparency and how the AI is applied so for like how it's applied for example like you don't necessarily want a completely automated decision coming from a you want to have it sort of augment a human to provide sort of like an AI, you know, driven or you know instead of a data driven decision maybe it's like an AI driven decision or something, but it's still the human in the end that has to decide whether or not there's a problem, and whether or not to use the AI to make a decision so I think it's important to have like the human in the loop concept to be at the front of the mind and actually like use the AI to help augment sort of the humans decisions. Okay, I guess, one thing we should talk about as well as the potential of data science and artificial intelligence so AI algorithms and their applications are really ubiquitous these days and have the potential to improve lives so like they're just they're really everywhere. They interact with technology. I think you, there's also the concept of actually being able to use this in the classroom yourself as well. So locally so like in your classroom for example grading and assessments can likely be made quite a bit more efficient through automation and use of algorithms. One example we like to talk about is we've run student hackathon so at the Clista project we've run student hackathons before, and it's relatively straightforward to review like a handful of submissions. But for example if you had something where you had to review hundreds of submissions, you would probably like some way to automate that and make it a little bit easier than having to to go through each and every one because it would take a long time. It would just take a lot of effort for you to do that so if you're an individual teacher for example, it might be like, it might be impossible for you to get through all these things so there's our ways to make your lives a little bit easier with AI and data science. Another example from the curriculum in Alberta is career and life management. In that sort of part of the Alberta curriculum, one of the things they ask you to do is ask the students to do is actually write a resume, and the teacher is supposed to look at that resume and provide feedback on that resume. Even providing feedback on it for a class of 30 on their resumes, you can imagine being quite difficult. Now if you imagine a class of 100 or 200 or 300, like it becomes quite overwhelming. And so it's quite difficult and there's probably some consistencies that you can automate and provide feedback on. And I guess another aspect of that as well is when you actually get out into the workforce these days when you're submitting your resumes as well. You're oftentimes going to be submitting to a system that a company uses that probably has AI incorporated in it, which will go through a checklist of things that it's looking for and if you're not if you don't necessarily have that thing in your resume that it's looking for or you wrote it in a funny way. So that the AI doesn't recognize you might get filtered out as a potential candidate for a job without even realizing it, even though you're, you think you're fully qualified for it so. There's some concept of being familiar with interacting with the AI systems as well that we want to be able to teach our students. And as well I think what's also interesting about data science and artificial intelligence is the open source technology behind a lot of the AI and data, data science tools are actually available to the public and we're actually made available to the public through research through just individual and group efforts to make these tools available to people so a lot of those tools that companies are using or actually based off of open source or public technology and they've, you know they've taken that in house like in these big companies and turn it into their, to their intellectual property so it's kind of interesting that there's a lot of, I guess origins roots of this in sort of the public sector as well through open source technologies. So, you know what can we do so how can we sort of improve our trust in these AI and data science algorithm so what concrete steps can we, we take to improve our trust in the algorithms that run our lives. So really I think, taking part and getting involved and getting your hands dirty with with these, these tools and techniques. So you can talk to data scientists AI researchers. You can try out some of the groups you can try out some of the tools and techniques like there's lots of tools and techniques out there that you can just try and they're open available to the public. They're freely available. It's pretty amazing actually. There's different social meetup groups around data science and AI so like in Calgary there's, you know, quite a few of these groups and similarly throughout the world there's different cities have large social meetup groups and it's just a nice place to have a conversation about data. You can also directly participate in the data hackathon. These social meetup groups will oftentimes organize data hackathons and you don't necessarily have to have technical skills to participate. You just need to be, you know, keen on participating and getting involved. So it's kind of interesting that, you know, you can get more comfortable with these technologies and concepts by participating in it. So we have run student hackathons for example we have other groups that we've worked with like STEM fellowship and let's make it count out of the US. So there's a lot of like opportunities to get involved and actually participate. What you can do directly in the classroom if you're a teacher, you can introduce computational thinking and data science concepts and skills. The Cluster program has a bunch of resources so I've listed some out here. So we have like getting started materials, curriculum modules, lesson plans and we have an online, we have a couple online courses as well that are free for teachers to use and we kind of target grades five to 12 teachers. So feel free to use those or share those with any of your colleagues. You can also do different activities that don't necessarily require sort of a deep dive into the technical side of things. So there's we have a data visualization of the week activity. There's also different curriculum made, for example by MIT, the AI FX education curriculum, which I really think is fantastic and I've provided a link there as well. I want to background about some of the tools that we use at the Cluster project so we use the Jupiter ecosystem so a Jupiter notebook is really what we use to showcase some of these data science techniques and concepts. It's an online document that includes both text and live Python code that you can run. And these documents, you know, they run on our service that we've made available for students and teachers. But you can also run them on commercial services as well, such as Google and IBM and all those things. So yeah, we have all that stuff available through our Cluster.ca website. Yeah, and just maybe another comment about what the data science community is doing. So the data science community is it recognizes a lot of this problems and ethics associated with, you know, AI data science as well. So the Association for Computing Machinery, the ACM, they have a quote saying that five years from now all students majoring in computing will have an experience using computing to better the world. So this almost like data for good concept or AI for good concept. UC Berkeley now has this human context and ethics curriculum as well. So if you're an undergrad there, you can, you can, you can get involved with that. And for algorithms themselves, there's been a big huge push around interpretability. So oftentimes we can, we look at these algorithms as kind of like black boxes, where we throw some data in there, we get a prediction. But like I was always, as I was saying, to understand what biases are actually built into the algorithms, we need to be able to really interpret and understand how these algorithms are working. So there's been a huge effort by big companies like Facebook and Microsoft, Facebook, Microsoft and Google and all those big companies around interpreting machine learning. So interpret ML, fair learn, open differential privacy. So these are all sort of machine learning algorithms that are actually quite interpretable and have tools built in to allow you to interpret what's going on. So that's really exciting. There's also algorithms and concepts like human in the loop where before you make your prediction or before prediction is implemented, you want to have a human or a set of humans there to really verify what's going on. You don't want everything to be sort of automated. There's also the concept of auditing these algorithms. So before you release something to while you want to have a third party come in and audit them to make sure that what's going on is fair and again constant and again like monitoring of what these algorithms are doing is really, really important. There's also a project called open DS for all and it has a lot of interesting resources for people to take a look at and it's, it's creative commons license as well I believe so you can reuse and remix those contents as well. Yeah, I just wanted to wrap up I think I'm almost at the end of my time now so I have a feedback form feel free to fill that out. I have a link here I'll just slide it into the chat. Feel free to let me know how I did today. If you have any information about the project I'm involved in close so you can email us at contact at closer.ca or you can check out our website, and that's just some of the information on how to contact us. So, I think I'll stop there and if you have any questions feel free to let me know. Thank you so much for this. We will open this up to any questions that folks may have in the room. Feel free to drop them in the chat or turn your microphone on and ask him directly. This is all incredibly fascinating. I was a part of the public interest technology team at New America, which is a civic and think tank civic enterprise and think tank in the US. I started with the idea of getting people interested in technology but not to be working at Microsoft and Google, but to be to be public servants, and to work in government and and bring their technical skill set in and so it's a very new field and they're a second year as far as the university network where they're providing grants to universities here in the US to actually build programs. And that involves the curriculum so it has to be multidisciplinary like you were showing that diagram event diagram of how it is very multidisciplinary and bringing the different colleges together can be incredibly challenging and so it's just really interesting to drop the link to this year's grantees. There's a YouTube video that kind of introduces the folks that'll be doing things this year. It includes a lot of your Ivy League schools here in the US but it also includes like Miami date college, because they have a large, you know population and they want to get folks interested in this so this is all really, really interesting and very helpful so thank you so much. Hi, you're welcome. Yeah, there's it is a pretty new and even like the concept of data science and K to 12 is pretty new. So, yeah, there's a lot of ways, but what's really cool is like, like I was saying a lot of this comes from like the open source side of the technology fields and all of it is actually available for you to poke around with and play around with for free with your computer so like it's really interesting how it's kind of evolved and how big companies are taking this but now they're starting to recognize that they need to also understand what's going on with these algorithms and also give back to the community as well so they're making a lot of effort to a lot of efforts in these areas as well. Byron, would you mind dropping the link to your slides in one more time just because we had folks join after and they won't have access to that. Oh yeah, for sure. Okay. Thank you so much. Yeah, you're welcome. Are there any other questions or comments for Byron. All right, everyone give Byron a round of applause. Thank you so much. Thank you. We will start to transition to our next presenter now, Connie, I believe you're