 Let me welcome you welcome everybody to the Future Trends Forum. I'm delighted to see so many of you here today My name is Brian Alexander. I'm the forms creator. I'm your host I'm your chief cat herder for the next hour, and I'm absolutely delighted to have this week's guest who was doing a fantastic Project I'm absolutely delighted to welcome Greg Heilman Greg is among other things a professor of electrical and computer engineering at the University of Arizona He's also a vice provost for undergraduate education at the U of A and on top of this He has assembled a fascinating project for data analytics, which is what he's going to be explaining to us and working with us today So without any further ado, let me bring him up on the stage and me welcome Greg Holman. Hello. Hi. How are you doing today? Very good. How about yourself? Oh, I'm just dandy and I'm really really grateful to see you here today. I'm grateful for your time I have so many questions to put to you Greg, but the first one I want to ask is Thinking of the multiple hats you wear as an inventor as a professor as a provost What are you going to be spending most of your time and and energy and thinking on for the next year? What's what are the big projects and issues? Yeah, you know, we spent this last year Just responding and reacting I think to all of the events around the pandemic and I Think we're going to be spending a lot of time over the next year trying to deal with The fallout from that, you know possible learning loss and other Kind of disruptions that occurred as we try to get back to normal or new normal working conditions at the University. I Hear that. How does that? I mean, what are some of the ways that's playing out at you? Are you are you for example, are you anticipating a fully face-to-face fall semester right now? We are planning to be you know back to the normal mode of operation in the fall and we're providing some opportunities over the summer for students to You know basically take courses that allow them to perhaps fill in gaps change their placement in math and English You know try to get a head start So just try to address maybe some of the gaps that might have occurred due to the pandemic Thank you, thank you Friends on the forum I always have tons of questions But but my job is really to get out of the way and let you ask your questions But in this case we have an special project to show and I'd like to give Greg a chance to just really walk you through the project and I actually do have a couple of visuals I want to flash on the screen, but very very quickly Greg Can you just give us a top-level overview of what the curriculum analytics project is? Sure, you know this has been an effort that started about eight years ago I was previously at the University of New Mexico and the president and provost there at the time really had this student success Imperative that they wanted to implement and improve student outcomes on that campus and One of the first things we did was just create a way for students to see their Degree plans online where they could actually look at their degree plans and understand their trajectory towards earning a Bachelor's degree at that institution and so we put all of those degree plans online And as we collected that we realized that those degree plans are themselves data that could be analyzed And when we did that we found that there was another thing happening on that campus and that is that every Programmer it was required to have 128 credit hours, which is eight credit hours above what any of the institutional accreditors require and so we launched an effort to See about removing those eight credit hours if possible and we created some analytical tools that we gave to faculty and And It you know, it takes a few years to realize the fruits of that work but we significantly reduced the number of credit hours associated with all programs at the institution and The remarkable thing is we ended up with over that the time period just this last year a 300% improvement in the four-year graduation rate. So we created much more efficient pathways to the degree We're at how what percentage increase 300%? Yeah, it was very low. It was started it you know It started it something like 12% for four year because we we actually didn't have four-year degrees But by making, you know providing students with roadmaps that actually were four years You know, we get where the university is now up over. I think 35% The six-year graduation rate didn't change much though, which is quite interesting Which means we just created more efficient pathways and that led us to all different types of studies Related to curriculum and how you might use that to inform improvement efforts student success efforts and most recently We've been working in the state of Kentucky with the council on post-secondary education there And I notice I see that Doyle first news on And working with him in fact on this project of using curricular analytics to Provide the ability to compare curricula across the entire state system and their complexity and also to Enhance transfer articulation because you can use these tools to actually compute transfer articulation pathways And that's what we're working to do there. Wow Okay, let me pull up one of these these images here because you've got some fantastic graphs And I just want to make sure that everyone can Everyone can see them here. So let me just pull this and make sure we dig in it so people can see it accurately Okay, if if in the chat if you just say can you see this? Okay, does this does this work for you? I want to see that So this Yes, good to go okay, so this is an electrical engineering program at a Institution that shall not be named or shall be named later perhaps and In and basically what this is is an organization on a term by term basis of the courses A student would need to take to complete this curriculum and receive the electrical engineering bachelor's degree and If you advance one Graphic you'll see that I've highlighted here Calculus one and most people know that that's the most important course in In electrical engineering curriculum and the lines so so the circles themselves are the courses Right and the lines between them are prerequisites. So you'll see that physics one one one two as is prerequisite this math class that's highlighted in gray and What those green? Circle show those are all of the other classes that you cannot take unless you first pass calculus one All right, so in other words calculus one blocks you from taking all of those past classes until you can pass calculus one And that blue Pathway you see well, that's the longest pathway through this curriculum and it involves five classes And so we refer to that as the delay factor because if you have long pathways in your curriculum And and a student is unable to pass some course on that pathway They are delayed in in preceding and so we refer to the others the blocking factor We found that these actually highly correlate to student success metrics, which I can talk about Shortly, but if you sum up those factors throughout this curriculum you end up with an overall curricular complexity of 118 as I show there At the very top of the figure. So is that understandable to everyone? This is a Degree plan where we've represented as a a graph with nodes and edges the nodes of the classes and the edges of the prerequisites between them So if you have if you'd like to Benjamin, thank you Benjamin Cappell If you have any quick questions about what we're looking at right now Please throw them in the chat or better yet Just either raise your hand for video or enter a quick question in the chat box or in the Q&A box rather So so this is kind of what we started with When we created these degree plans when we all all sudden had this realization that hey these plans themselves are data that can be analyzed and We had a couple Good questions Stephanie Frost wants to know what are the numbers inside the circles? Ah Good questions Stephanie. So the numbers inside are the overall complexity of that that particular course Relative to those two factors I mentioned delay plus blocking so if you see in that calculus one course that I have highlighted Delay factor is five because it's on a path of length five and it blocks ten other courses those ten green courses that are shown there Adding those two together. You'll get 15 Summing those over the entire term the term has an overall complexity of 19 and the curriculum has a hole has a complexity of 118 We'll get into more about what these numbers mean But that's how you compute that does that answer your question I think so a Gabriela Weaver and Neil Fung both asked the versions of this Gabriel asked how his curricular complexity Calculated and Neil asked what does it mean exactly or how does it translate? Does this Gabriela and Neil? Does this make sense? Does this does this work? The what does it mean is a deep question and then we'll get to that through that, you know, so Taking off faith that that number has some meaning and I'll I'll try to get to that meaning through through this talk But for the time being delay plus blocking together We found that it highly correlates when you add that up the entire curriculum to a student's ability to actually complete that curriculum We have one more question from Rella who wants to know are all five node paths equivalent and why is this particular five node path considered the longest? It is a a longest path not the so there are there are there can be more than one for sure It's just finding a longest path The longest path The number of long paths in a curriculum is matters So the fact that there are many long paths in some curriculum matters So that's a very very good question And I don't I don't know how to answer whether or not they're equivalent or not I think you know it may depend on the students background And then we had a kind of flip side question of this and this is from Paul Henley Oh Paul who asks is there a level where you stop calculating and what about the 45 courses blocking your Internship course or student teaching? No, we don't stop you know if if those are if those are requirements That would block you from taking a class then all of those do factor into a student's ability to reach that internship class And so no we don't the reason This is not a very complex curriculum And that's why you're not seeing I think you're you're noting that in the junior and senior year There's no pre-rex. These are just a bunch of electives. By the way, this is not an exact science It's highly likely that that elective you see in the seventh term probably does have a prerequisite, but It captures It captures on a grand scale something that I that I want to show you next let's let's advance one more slide This is there's you know in the notion You know on the internet people you've probably heard about this notion of a hub Hub is a some something that's central a lot of things connect to it Well similar things can be found in these curricula and and the remarkable thing Just to point this out is that the central course the most central course The hub let's call it in almost every double e curriculum is the same class It's circuits one And I we found this in numerous In numerous curricula where you you run this analysis by this the same core, you know Principles of financial accounting and accounting tends to be the most central course in that curriculum What that means is there's a lot of courses you have to pass to get to it Right and then once you can pass that central course it tends to open up the discipline to you So the central course is also an important concept that I'm highlighting here One quick question is so is this basically following skill free networks 80 20 rules it is it's following You know everything in network theory people talk about everyone has their own centrality So it's related to those concepts in network theory But basically it's how many paths go through that node is what what this centrality measure is how many? Curricular pathways go through that node and that node has the most pathways going through it We've got a couple other really good questions here And this is from a Jeff Alderson at MathWorks and says given the given the large-grade analytics this represents It's definitely useful, but is the primary user academic advisors or the learners themselves? good question and Jeff I want to say that Cleve Moller was a very good friend of ours at the University of New Mexico the inventor of MATLAB Professor University of Mexico wonderful person had many He was always very gracious in coming to talk to our class about software engineering at the University of Mexico so The audience is is In an unbelievably important question this the audience for this is I think more Curriculum committees and and again, I can touch on this later in the talk But we also have a set of tools that are designed that are more advisor-facing And those tend to be around degree plans. So this is I'm talking to you about a a you know, I'm showing you a degree plan And I'm analyzing the curriculum underneath this I could restructure the order in which you take these courses I'm limited somewhat and how I can do that by the prerequisites They place some constraints on how I can do that but if I shuffle this around I come up with a different degree plan and I could create a degree plan that allows a student to finish in seven years in seven terms rather than eight as I've shown here or in ten terms or whatever might meet that meet that students needs and so we Separate this notion of a curriculum. I'm not changing the curriculum when I do that But I am changing the degree plan. I'm providing to the student we believe that matters and in particular This notion that certain advisors know that certain students probably shouldn't take certain courses together in the same semester And we refer to those as toxic course combinations. They're toxic to one another. So for example, I found at the University of Kentucky if you take Calculus and physics in the first term which every engineering plan had and your math a CT was 25 or less That physics had a toxic impact on passing calculus. You could separate those and still have a four-year plan That's probably better for those students. That's an advisor-facing tool that we've worked to create Well, that's a good question. Thank you. We have a Fritz van Dover has a question that he immediately almost apologizes for He says have you jumping the gun? But how does bringing these insights forward allow institutions to make substantial curricular or policy shifts? Do you want to do that now or do you want to hit that up later? You know Fritz that's that that's the point. That's what I'm trying to get to in this talk So, you know, if I don't you remind me again, so let's let me here's the first kind of prompt I have for you. Let's let's so this is an electrical engineering Program, it's a better credited. Here's another a better credited engineering program And you'll see up top the curricular complexity for this program is about Four times more than four times greater than the other and if you if you hide over that calc one in this If you go to the next graphic I'm highlighting the same course Now the interesting thing about a better credited programs. They have identical learning outcomes So the students in that first curriculum are attaining the same learning outcomes according to ABET as the students in this Curriculum same thing double E program You've got a delay factor of ten So there's a path of length ten through this curriculum and it's there's more than one of them as somebody's correctly I already pointed out and but the one I'm highlighting there you'll see in term four There's three courses you have to pass all of those in that term in order to move forward in this along that longest path there and There are 27 courses in this curriculum that require you to first pass calculus or something else that depends on calculus so you know, we add that up we've got a Delay plus blogging factor of 37 for that course much more critical course in this curriculum than in the prior Curriculum that I shared with you. Wow if we advance one more you'll see Low and behold, it's the same class as most central in this one again. We're talking about circuits Right, but look at how much more critical circuits is here, right all of those green courses You can't take until you've passed that circuits class And you'll see that there's quite a bit of additional courses that you have to take or take at the same time in terms of Co-requisites in order to get to that circuits class And so, you know my first question to you that I'd like to pose to you all is Which program would you send your kid to and why tell me why? Give us the choices again to make sure Institution one or institution two Okay, so your son or daughter. Let's hope it's your daughter because we need more. Yes, electrical engineers Wants to attend institution one or institution two They've been accepted by both you take a look at the curriculum you say well one of them more looks more difficult But maybe you're gonna get a better quality experience Please I don't know if you can pull folks, but I always love to pull people If you could just you type in a question in the Q&A box or type a quick answer in the chat box Or if you want to say something out loud Please feel free just to raise your hand and we'll be happy to beam you up on stages plenty of room I bring this up because this was one of the first things that faculty When we first started doing this work said well, yes, we could reduce complexity of our program But we're gonna reduce quality in doing so so let's be careful about this And I agree we do need to be careful in how we adjust curricula to make sure that students are still attaining learning outcomes Well, we've we've got one quick answer from Genobondo. Hello, who says number one less time less complex Higher percentage of completing happily You nailed it you nailed it You nailed it Yeah, I mean I would So the first The first program I showed you is an Ivy League school with a doubly program Actually well known for their engineering programs and the second one I showed you is in the lower tier of rankings Within US News and World Report Programs that actually led to this really interesting question we asked because faculty were bringing this up You know, we are highly complex because we are so good and So we said well, let's address that question and I did in my own discipline first and so if we advance If we advance I had one other Question is is do higher quality programs have higher curricula complexity? So that's the question I wanted to try and answer and so we did a quick study where we where we Pulled the curricula complexity of All of the electrical engineering programs in the US News and World Report rankings and we let that serve as a proxy for Quality, I know it's not perfect But it's difficult to assess quality of programs. So we said let's let The ranking serve as that and so you'll see Along the x-axis the tiers we created the top tier a middle tier and a bottom tier of complexity scores and you'll see the blue box and the Inventions in those boxes are confidence intervals and what that says is that the top tier programs They tend to have a medium curricula complexity of less than 200 the one I showed you was One the mid-tier programs tend to have one that's closer to 300 and The lowest tier programs in the ranking system tend to be closer to 400 So at least for electrical engineering programs It seems that complexity and quality are inversely related and this is a very intensive study to Create the curricula for all of these programs and do this analysis But we did an ANOVA analysis of your advance There is actually statistical significance to this is basically What I'm saying and then we did it again for computer science and we low and behold found something very similar that the CS CS rankings is what we used in this case of US schools Same thing top tier schools tend to have less complex programs. So One advanced one more you'll see that there was also statistical relevance in this result now There's this folklore out there that says It for a minority student You should go to the highest rated program you get into because you have a better chance of graduating if you go to that this is this is common knowledge that people use in higher education and What I'm conjecturing is perhaps this has something to do with this you're encountering I'm sorry who just gave that very eloquent response that If you could reread it, I think it captures it in a nutshell. They're going to a Program that allows them to complete Actually experiment and take other courses that might meet their interest. It would be easy to double major in that first A Double e program that I showed you which is Cornell It would be much easier to also study music as a part of that it would possible to do that in the other program so You know, I'm wondering if if this if this might have something to do with that notion that You should go to the highest rated program you get into because you're going to confront an easier Not an easier a more efficient curricula Greg before before we proceed I think this is this is fantastic, and this is fascinating stuff And we have a whole bunch of comments that came in and a couple there are some leftover answers to your question before about which Which school and I want to just make sure we get all these here before you go further And you can grab them as you like Darren McNally says the first part the institution Cornell seemed to help set students up for success What would there be any way to know before the students start at the institution? Okay, this another very good question. You're all a very perceptive These are the questions that always come up when we talk about this So let's let's analyze this one for a second The first program is admitting students who are largely calculus one ready So that degree plan I showed you is the one that they're actually following that second school I showed you admit students who are mostly not calculus one ready They they have the same starting math point. So to me, that's an equity issue. Why are you giving a plan? So in reality the students who are admitted to that second program are having to take a pre-calculus class or maybe You know a math class that proceeds that they really don't have a four-year route to a degree For most of the students who are choosing engineering at that second program. So Yeah, taking into account a student's background is critically important and there's ways of restructuring That second curriculum where you could learn a few math topics that would allow you to take some of those other courses The long the way and perhaps still graduating four years I've got a study of that that I could share with you later in the program Your question and when we have a couple of responses that spin off of that Fritz roundover Observes that a more efficient program although is for academic me and during adjacent to their course or study Which you know someone consider a liberal arts approach But one more point to bring up and this may be getting at some strategic issues Neil Fung Neatly said more courses doesn't necessarily mean higher quality. So I would say the more affordable option would be better Yeah, yeah, I mean There's another imperative here is that it's become harder to earn a degree because higher ed is becoming more and more expensive And so I know at our university. We're giving merit aid for four years and it's gone and so You know, that's when students are accumulating debt within six years. So if you have inefficient programs, that's for sure happening We have one more question Which is a video question and I want so I wanted to bring I wanted to bring The speaker this is Paul Henley coming to us from Stephen Austin State. Hello, Paul Hey, Brian Happy birthday Dr. Heileman, it's really nice to have you here I'm at Stephen F. Austin State University and we're about to get accredited next week we hope and part of accreditation in the south is having a QEP a Quality enhancement plan, which is a major initiative and ours relies heavily on your work The questions that we've been asking each other is First of all, how long do you think it takes before you know that any adjustments to curricular complexity are working? Okay, very good question, you know Assessment plan and making notes right now. Yeah, you know it's interesting when I was at the University of New Mexico they hired a new president and They were gonna give him a bonus If he could improve the six-year graduation rate, but in one year Think about that, right? Yeah, it's so you know the impact of changing the six-year graduation rate should be on You know measured in six years So that's an inherent that that is a really important question if you're gonna So many times throughout my career, I've been involved in curricular discussions where faculty says I believe X Right, and I believe that there's no way to Dispute it or right. It's just their belief and that's hard to have you know conversations around that One of the things we're trying to do with these tools is one give you something where you can take that curriculum and compare it to another school And do studies where you're looking at. Okay, what's the grad rate for this curricular complexity at this program versus this one? Let's try to make sure that we're not Doing some self-selection here. Let's try to make sure that we're putting the same types of students through them But in reality, you're gonna have to wait four years to see if that curricular change Changes before your graduation right to address that we have recently added the ability and I'm happy to if you want to contact me I've got my email address on the very first Brian can send it you and you're all Feel free to contact me If you want to simulate this we've created a simulation environment where you could say, okay Here's the old program. I I'm gonna take my pass fail rates that I currently have for all these courses Here's a restructured program and I'm gonna run a simulation We have a money Carlos simulation capability where you just run student populations and see what the grad rate is So in order to get to this very important question You're asking we're trying to build simulation models that allow you to simulate the change and Either how you teach a class or how you restructure your curriculum Okay, great. All right. That was a long-winded response, but it was a good a good question Thank you very much for picking me Brian. Thanks I would love to hear more about how that goes and how you've used it Please do drop me a note with kind of collecting how folks are using this catch it. We'll share that. Thank you Thank you and friends if you're if you're new to the future transform or you're new to shindig That's an example of video question again all Paul had to do was press one button I pressed another button what time is right and you can join us just like that And there was one more observation that I wanted to share from the From the camera in the chat Which was that this is from Jeff Alderson who said that accreditation has a role here to the Curriculum for ECE would need to be a bet accredited for example So the shortest path through the accredited curriculum and that's that's a constraint to Yeah, so that prompted me to write a paper Called a bet won't let us do that So look it up. It's in change magazine because I got tired of hearing this a bet won't let us do that a Lot of people don't know that I'm an a bet evaluator and I'll go to a meeting and I went to one They say well a bet won't let us have this many international students. I'm like tell me which criteria. That's it so in that paper Again, it's called a bet won't let us do that and it's in change magazine we Actually do an analysis where we show that you can create one 20 credit hour degree programs with a bit right unless you have some ungodly number of state-based gen ed requirements or something like that but There are many many 120 credit-hour engineering programs across the country engineers and I'm one have been very effective in lobbying state Governing boards that you can't create a 120 credit They were very effective in doing that until people started to create them and there's your existence proof And so that's argument is starting to crumble But you know a accreditation is a crutch that people use to not change in many cases. I Just shared one link to to that paper and that's a great cleared title. So again Thank you. Thank you for answering that. Oh, thank you for asking it. What a great question. Oh, it is Well, this is a fantastic community. Well speaking of which correct you do you want to show a couple more results and then Yeah, this this next one actually and I was glad was it Paul at Stephen F. Austin That was yeah, we had another something was sharing we just recently and it gets to this question of How does curricular complexity on time impact on time graduation rates and and Paul this might be something You could also point to if you look at this next graph. This was done by Mitchell Colver at Utah State University He's their institutional researcher. I Found this quite fascinating and it's something that we also saw when I was at New Mexico is that The complexity has a huge impact on four-year graduation rate not so much on six-year and and by the way I worked quite a bit with the the vice president of research at Georgia Tech and Georgia Tech was one of the outliers on the high end of you know on the top tier schools They have complex curricula. They have unbelievably good six-year graduation rates and not so good four-year graduation rate and and as a result They're working to try and create more efficient curricula at that institution What this graphic shows is on the x-axis the the curricula complexity scores for all of the programs at Utah State University And then for the lowest complexity program, you'll see that blue line shows that for low complexity programs about 60% of those who graduate from these three cohorts 2011-2013 could do it in four years But if you take the highest complexity program right at the other end of that blue line You'll see that only about 10 or maybe 12 percent of the students are able to complete that program in four years So you'll see that the blue line is much steeper than the green line So it has a much more dramatic impact on the four-year graduation rate less so on the six-year graduation rate so Somebody asked the question up front of what is this curricula complexity score actually mean The way we created it is we did a logistic regression against Grad rates so it doesn't surprise me that we found that predicting grad rates using these curricula complexity metrics the most informative was taking delay plus blocking And just adding them up and it matched this the best so he's just confirming that at Utah State Found it quite interesting Very nice This one I have a I would love to talk with folks if anyone has any comments, but I'd love to bring this one up Can we can we bring this back at the at the end when we're when we're doing more general because I know you've got some more Detailed things you want to show us from another curricula and I want to make sure we get that example You're talking about a math curriculum, right? Yeah, so go go back up a second here. So this is the this this leads so Do you know my question here is are there equity issues? So there's two ways I believe that you can affect faculty In you know if one of the worst ways to try and give faculty to change their curriculum because faculty are in charge of curriculum Is to just tell them you must You know you must do this because we said so and that's usually leads to big fights I've seen that but the other thing is to is to get them dated say compare yourself to these other places Is there some reason you're doing things this way? And do you realize the impact you're having on who gets to study what so who is successful in those highly complex engineering? programs It's the students who went to the best high school who had calculus in high school Who is that? It's largely white Probably white males for the most part and that's what we see in the engineering professions So what role does curriculum this what I believe is unjust design play? Um, I think it plays a significant role in fact and we're trying to launch a study to look at the impact Of who gets to study and who gets to succeed in particular disciplines? Oh And here's an example I'll go through this very quickly unless there are questions Here's here's a here's a this is the same thing. Here's a here's a I'm sorry. I'm picking doubly. It's my It's my background, but you could apply this it tends to work in almost any stem discipline So this was the one at the university kentucky 228 curricula complexity of 228 A calculus one is the very first course in the curriculum. It's got physics one in the same term Uh, that tends to wash a lot of students out if we go to the next one Next slide then Here's the real pathway if a student comes in And is not math ready, which is going to happen for many of the rural areas in that state in any state for that matter They may have to take foundational math College algebra trigonometry and now in their fourth term They can take calculus one When I showed that to one, uh, associate dean of engineering at one school He told me well, yeah, but we don't have success with those types of students So we don't even need to look at that these types of well I looked and they had graduated six students who started in foundational math So there's nothing that it is possible to create engineers who started that level of math if they want to do it If we can accommodate them What is interesting here is look what it did to the curricular complexity one This is an 11 term program now Right and this is primarily going to be targeted at underserved students So they're not going to have the money to pay for 11 terms most likely And the curricular complexity has jumped way up to 357 Wow So if we go to the next slide what we found is that in almost every This is just an analysis of that you can go to the next one What we found is there's this pattern. This pattern occurs in every double e program You take calc one you take calc two the same time you can take physics one You get physics two differential equations and finally we're at that central course I told you about circuits one which is in the bottom of term three you see it there And then you can take circuits two and the rest of your curriculum This little pattern shows up in almost every curriculum you look at and so In in your own discipline you can sometimes look to find these curricular patterns and look at alternative ways of Attaining the learning outcomes in this pattern So what is needed in that you'll see that differential equations course in the third term That's required to take circuits one. It's required because there's one differential equation You've got to solve a second order linear differential equation I could teach you how to solve that in your freshman year in the first course and you could use it You don't need that entire differential equations class So I'll pick up on that in just a moment, but you'll see here that's that was 75 curricular complexity of 75 Right now. Let's add that foundational. Let's add a pre-count course to that. That's the next slide. Okay You have to 98. Let's add another math prerequisite math course to that one called gel. You have to 126 That's more complex than Cornell's entire program But this is what we're asking that student to do in that uh, Kentucky example. I showed you Now let's take this in this knowledge that I just said Hey, I can teach you how to solve that one differential equation You need to take circuits one in your sophomore year and let's restructure this curriculum I'm gonna pull that learning. Well, okay. Let's add foundation on math now. You're at 157 even more complex So let's pull that out and that's this next revised pattern that some Institutions Wright State University was the first one to pattern this they created a new class called engineering 101 They taught the student how to solve that one differential equation Which allowed those students who came in not quite math prepared to still take circuits one in the third term The overall curricular complexity is 79 Versus 75 for the calc ready student They're still taking differential equations in the fourth term You're not but guess what now when they're taking differential equations. It's in context. They've actually used it for some useful purpose The interesting thing is this was initially created for the lower kind of scoring Students when they compared the students who went through this to the ones who came in calc ready These students became better engineers. They were better prepared So this is this is not saying you're doing something that's less than you can just Structure your learning outcomes in a way where you could attain the same learning outcomes even better If you just take the time to restructure things so so for me, this is an equi perspective that Institution I showed you should be doing something like this for their students. Perhaps they are I don't know So that's the equity issue. I wanted to bring up Greg this is this is fascinating and and you have a whole raft of questions Let me let me just quickly pause this for a second so we can we can take some questions I'll make sure everyone gets a gets it gets a chance because these are all over a whole variety of of domains we've got Let's see From Donny Snellbach. We have a question. G Denison What is the reduction of credit hours across the campus university in mexico a result of reducing the complexity of possible paths And or making the paths more simple you know the number of credit hours does impact curricular complexity so um That is a very good question We have never looked at the amount by which we reduced the overall curricular complexity That would be a good thing to do It would be a difficult thing to do though because I've got to be able to find all of the curriculum for You know the hundred or more programs from that point in time and compare it to the hundred or more for this point in time um We reduced significantly the total number of credit hours Many of the engineering programs also restructured in the way that I just showed you which does reduce complexity But I don't have the exact number that that I think you're asking for It's a good question, but it is a good question. Let's proceed with that We have uh, also a question from Gabriella who asks a very precise question, which I was going to ask myself Is there a tool that you use to make these maps? I'm curious how I could make these the programs in our college If you go to the very first, uh Uh page of the what I provided to you Brian I've got a link to a tool that you can use and I can even pull it Or you can pull it up and or you can contact me But if you if you look I've got a link there So if you go to this curricular analytics org website We've created the ability for you to up put your curriculum into a uh A spreadsheet a csv file and upload it and it creates that graph, you know all the graphics I've been showing you it creates them and Um, we're trying to make that more and more user friendly We you know, I've got myself and some Some students who who work to maintain that site and and we're providing these tools as open source um, so Go there and try and upload your curricula and if you have any difficulty let me know now if you are an ir person And you like to program We've also created a toolbox and it's called curricular analytics dot jl. It's a julia programming language Toolbox In that toolbox you can do far more sophisticated analysis you can do the simulations I mentioned You can You can do all sorts of things And we're actually creating a whole bunch of notebooks that show you how to do those types of studies So if you don't want to program use that web app if you are You or your institutional researchers want to really dig in deep and and construct your own studies We've got some tools for you and then I just provided a reference there If you're just interested in learning like more of our thinking behind how we came up with this That's probably the best paper to read that one that we put in archive The physics pre-print archive, which is also housed in Cornell. So I got that Cornell link You know and somebody I think it was fritz said earlier You know the the free, you know Cornell actually advertises as engineering is the new liberal arts Well, it's it's they have some freedom to explore liberal arts in that program quite true quite true and friends we share these links in the email announcement and And I made you a blog post about this with the links as well And this is very rich speaking of rich. We have a great question from Steve airman Who's been a great guest and Steve congratulations on your new book by the way Steve says am I right the curricular complexity does not reflect dfw rates in the various course But that the simulation does correct If you read that paper we we separate curricular complexity into two pieces One we call the instructional complexity and the other we call the structural complexity and that's what I've been showing you today The two together are what right? So how you teach and how you structure your curriculum? And so if you want to look at dfw rates or actual pass rates in a course You'd need to to look at the instructional complexity and you do that through Simulation in our tools It's interesting. We did a study here computer science very The structural complexity of that is not very high aerospace engineering very high Aerospace engineering though they are instructional complexity such that almost every student passes their their aerospace engineering introductory courses Not so in computer science The the end result is that they end up having very similar six-year graduation rates one due to High structural complexity the other high instructional complexity Very good question indeed indeed And here's another question from Gretchen Gibbs Who asks what are your thoughts about higher education institutional adoption of open source it or proprietary it Perhaps another way to phrase this question course analytics versus course tune. What's the diff? Yeah, so so course tune they They help you know my the demo that I've seen for what they're doing is they're helping you kind of figure out how to Attain the learning outcomes within your curriculum They're not looking at complexity and you know how this changes complexity. So That's That's a loaded question. I think somewhat because you know in my position. I'm sure it happens to many of you I get contacted by these it By these ed tech companies on a daily basis trying to sell me something I think higher ed should own Most of what it does. We should own our data We should create tools that allow us to compare ourselves to one another and and share data And so I'm in favor of more Open source tools created by higher ed, but there's been a huge rush You know starting with arnie dunkin And and you know obama administration where they're saying let's use Let's use data to inform our decision-making and that led to this rush of companies getting into this space and selling to us Well, and they sell miracles in many cases. They said, you know, and I get these emails all the time Would you like to change your? graduation, you know your your retention rate next week Okay, you can't do that Well speaking of course tune. Why don't we bring its ceo up on stage because we have maria right here And maria's been a great guest and great for a great part of the program for years. Hello, maria Hey, hi gray. In fact, we're using course tune here. I know I was gonna say This is a great question because I actually think that um, we can probably build something into Course tune so that one of the exports from course to you and actually just sends data right to The format to upload into curricular analytics. I sent you an email about that Um, but I think it's that you brought up an important point. Who owns your data? And I think absolutely university should always own their data What I find interesting is your comment about like the open platforms and institutions sharing data with each other We have a really difficult time getting any institution to buy into the idea that they would even share their learning objectives with another institution Right, despite the fact that those learning objectives are sitting on syllabi and lms Sometimes on public facing websites, right? And and I I would you know, we is our policy that institutions always own their data I wish they were a little bit more willing to share because we could actually move forward together If we could make smarter tools to help people write better objectives and things like that, right? But um, you know, I think the other issue is that it does It does actually take a lot of work to support a decent software, right? You either have to have willing bodies students who are willing to work on projects Grant funds, etc. Or you have to have a team of people who's willing to answer questions in real time all the time and you know support and institutions. So I think there is a place for commercialization of products, but I think you said something and I just wanted to clarify We don't sell miracles developing curriculum But I think a lot of people do try that Right, just use our product and miracles will happen What I like to say is that If you don't actually know what's in the curriculum There's a lot of forward-facing tech coming around AI And if you don't have the base of that wall of of like, I don't even know what's taught in these courses There's no way you're going to be able to build You know good analytics and and good continuous improvement on top of that. And so, you know, we try to do Is help people to design the interior of the curriculum And I think I think what you've got is a great tool for looking at the path between courses Yeah, I mean, I think that's the right way that we can work, you know that industry and and university can work together There are so many abuses. I mean we deal with it all the time on It's happening in the ed tech space But look at what facebook is doing when they collect information about where you browse and then come Work at that to you these types of things the same thing is happening in higher ed with people collecting information About our students and I won't name the companies, but they're keeping that information Uh, and you know, how are they using it? So what you mentioned is is a is a problem in higher education That's been solved in the medical space So who owns your medical records your medical records that wasn't Resolved so clearly 10 years ago. Doctors would argue with you about that, but that's been I think the same thing is going to happen soon in higher education Who owns the record the student owns it? And so we have a project right now with the american council on education Where we're working to create The ability using blockchain for a student to hold their own transcript and be able to share it and then work through transfer Articulation with that and so I think this who owns the data is a very important piece and there are Companies that do this very well. I work with with a number of companies externally as well So I didn't want to you know, but There are also many who who are rushing into this space, you know kind of selling crazy things I think this brings up an interesting point that we see Just between faculty and institutions Who owns the design of the curriculum? Is it the institution or is the faculty member and when I say design of the curriculum? I mean the course objectives and the learning objectives that sit inside a course And this is contentious at some schools I I my personal stance is that Faculty absolutely should own the activities that are designed the learning activities the assessment activities But at some point an institution has to know what's actually in a course, right? It's absurd to imagine an institution is not allowed to know these things, right? Or to say all sections of calculus or all sections of beginning of circuits one should teach the same fundamental principles, right? And you know, so even within our own academic sphere, we run into this problem of ownership You know one one of the things that we're trying to do with our tools is It's kind of hard to compare. You know, I showed you institution one institution two And you know is ma 1112 is calc one at one place and and math You know one 12 one one or one at the other. I don't know what it was They're both calc one and You can find the learning outcomes for calc one, you know, they're Everyone has basically the same learning outcomes for calc one. There's two sets, I think Well, let's go by what the ap has then for ap So so for the most part people accept those credits as They accept ap credits for calc one credits on their campus and those are clearly articulated in ap So we we refer to those as a canonical course. Yes, actually most disciplines Have that In my discipline if someone says circuit, we don't call it circuits one here, but you say circuits one I know what you're talking about And so I can compare in my field. I can compare my curriculum to You know a double e program at some other place and I can see how they're doing things differently But it's hard to do this with analytics because we all have these different namings And so one of the things that we're working on is trying to come up with some kind of alignment for these Courses around these learning outcomes and the problem you brought up Certainly would make that more difficult I need to be the one who's making things more difficult, but we're actually over time And and i'm so delighted to have the two of you talking about this on on on I just shared a great screenshot of this calling a complimentary curricular analytics Um, it's fantastic to have the two of you Meet face to face. Thank you maria Uh for coming up. I I need I need to wrap things up a little bit Go for it. So so let me say first of all again. Thank you maria. Greg. Thank you for showing us all of this This is a fantastic project um, we had the urls where people could find you and I'll I'll put the I'll put the contact information for you. Uh, as well Again on the on top of the screen so you can just find Uh, Greg right there at highlandman at arizona.edu Um, what's the best way to keep up with this project's work? Uh, just to keep following curricular analytics.org Yeah, we're trying to you know, we're right now The simulation capabilities in the toolbox. We're trying we're working to add that to the web application The ability to create degree plans optimized around particular student characteristics is something we're working to put into that Web application too. So I hope over the next six months or so we'll be able to get those tools in there They're in the toolbox not in the web app fantastic um Well, I need to uh with great regret to send you back on your on your way of not only advancing this project But also trying to advance the fall at uh, University of Arizona. Thank you. Thank you so much, Greg Thank you brine for organizing. This is fantastic. I really enjoyed it. Well, these um, the future transform community Has fantastic questions. Um, and I'm delighted to to do this with them. Thank you But don't go away friends. Uh, just need to point you to uh, the next uh, next couple of weeks of the program Uh, so remember we've got data analytics continued next week. We've got uh accreditation We are closing campuses as well as how to improve education equity for black students coming up You can sign up for those or learn more at tenyworld.com slash forum next Uh, if you'd like to keep talking about these issues of how to use analytics, uh, we have all kinds of venues Especially on twitter just use the hashtag f tte and my kudos to geno bondo for being so good about sharing out Her favorite quotes Uh, if you'd like to dive back into the past if you'd like to see just what brilliant guest maria anderson was or If you want to learn more about our other topics head to our archive tiny world.com slash f tf archive we have nearly 250 videos available right now and We have a program coming up next week, but between now and then Please stay safe. Thank you for all your brilliant comments and thoughts I think i'm going to post all this as a blog post including the questions that we didn't get to unfortunately Along with the recording as soon as we can. Um, but in the meantime work hard. Uh, thank you for visioning with us together and stay safe Bye. Bye