 at the NSF and also at the Lawrence McKinley-Gone Professor in the Biology and Cognitive Science Department at Carlton. And she's, of course, been a member of many national-level committees in the United States. And particularly, she's been the chair of the committee that authored the DBER report recently. And she's been specializing in the undergraduate education, particularly specifically biology and environmental education. And even I was stunned to see that even she's interested in other areas like sustainable energy and various other aspects of the STEM education, particularly at the undergraduate level. And she's going to, of course, talk on a subject which all of you know, because the abstract has been already sent to you. She's also been part of the entire effort in the United States in trying to see how STEM education actually moves towards solving national problems of various kinds, which could mean both the social, economic, and other kinds of problems as well. So she's been at the center of STEM education leadership in the United States. And we are very fortunate to have you with us today, Susan. So over to you. That's terrific. Thank you, everyone. I am so proud I can't be there with you in person. Before we jump in, I guess I should add one brief biosketch update. I am no longer at the National Science Foundation. About a year and a half ago, I started as Provost and Vice President for Academic Affairs at Rawlins College. So you could see on my first slide there that reference to Rawlins. So I'm in Florida, which isn't quite as warm as it usually is this time of year. So in terms of my talk with you today, and I'm especially looking forward to the conversation at the end of the presentation, what I would like to do is two parts. One, establish the context and a little bit of the history of the undergraduate STEM education improvement effort in the United States. Because I think a lot of what's happened informs where we've gone and how we've gotten to embracing deeper as a field of research. It's both a field of research and an area where there is great interest in using the research findings to improve the undergraduate STEM experience. So if we can go to the next slide, the motivation to improve STEM education has always been to provide really high quality STEM workers and a well-educated general populace. More recently, the scale of the global challenges we're all facing has really increased. And these are challenges that require a different kind of preparation to address complex multi-dimensional problems. So to exemplify that, what I'm sharing with you here is an effort at the US National Academy of Engineering to prepare students to address grand challenges. So the National Academy's every 10 years plans to introduce 14 grand challenges. The current ones fall into the four categories that you see on the left. You can see we're talking about everything from clean water and energy with sustainability to delivering health care, addressing strife inequalities. And I think it's particularly important that quality of life falls into their concerns also. What's interesting, and I think will be informative as we move through the presentation, are the competencies that the Academy of Engineering thinks are important for graduating engineers to have. So in addition to understanding the content, really being able to conduct research to have a creative approach to problems, to be able to address things in a multidisciplinary way, and to have a sense of business and entrepreneurship because we need to translate the basic science and engineering work into the marketplace, the importance of cross-cultural competencies, and then the importance of social consciousness that problems aren't solved in a vacuum. If I can have the next slide, could we have the next slide? OK, so what we need to do is to think more about educating T-shaped individuals that have deep disciplinary knowledge, but also the ability to work across disciplinary domains and have strong interpersonal and interpersonal skills. So things that fall more in the socio-emotional domain than in the cognitive domain. And all that we're learning from DEEBER and from other fields is really providing guidance as we think about redesigning the undergraduate experience. If I can have the next slide, I want to talk a little bit about how we've gotten to the point where DEEBER has come into being. So the next slide is very historical. I'm a biologist, and so I picked biology as an example. But for those of you that are physicists and chemists and engineers, please map your own discipline onto a timeline. Back in 1887, not so long ago, our biology was trying to make the case that they were actually a discipline that should be taught in higher education. So this has been a domain in higher ed for just a little over a century. In the United States, fast forward to 1958, when the Russians launched Sputnik, there was a huge shift in the US perspective and a deep interest in advancing science and engineering education. So you start to see a focus on an improvement agenda, but you're still not seen at the undergrad level the focus on research on how to maximize the learning. And by the mid-90s, the professional research societies had to recognize that they had a role, an important role in advancing the quality of undergrad education. This particular monograph was a group of there's about 147 life science societies in the United States, created a coalition of educators in life science, proposed a way to improve education, still didn't get a lot of traction. More recently, an effort called Vision and Change and Undergraduate Biology has very much taken the US biology departments by storm in a lot of ways. Next slide. I don't intend for you to read all the details of this timeline, provision and change, but very briefly, this was an initiative that the US National Science Foundation, the Howard Hughes Medical Institute, the US Department of Agriculture, and the US National Institutes of Health supported and nurtured over more than a decade to bring together leaders in biology departments across the country to rethink how we educate our students in the undergraduate biology courses, as well as to provide tools, resources, supports, mechanisms to advance. On the next slide, what you'll see is a little broader context, because I think it's important to realize we didn't just wake up one day and say, wow, we're not doing a very good job preparing our biology students. It's the ethically right thing to do to get better at this. Rather, what happened was there were rapid changes in biology. Again, for those of you in other disciplines, substitute your own discipline and think what was happening there. Biology over a generation moved from a descriptive science to a very data-intensive science. And students that were coming out with undergraduate degrees in biology were not prepared to work in the field of genomics or to do the large-scale systems level statistical thinking that was required in ecological studies. So the field was saying, hey, we've got to do something. It becomes the workforce development argument. And more broadly, this area of convergence, it's a term that was coined in the National Science Foundation. It's one of the major points of emphasis right now at the US National Science Foundation is convergence. We can only solve the modern problems in health and in agriculture and a lot of fields if we bring together engineering, physical science expertise, and life science expertise. And so the vision and change document see a strong push across disciplinary understanding. And simultaneously, not just in genomics, but in other fields, we've been growing the big data, data-intensive research. And that's important not just for the disciplines, but I also think as we begin to imagine where we can go with undergraduate education research. So the third publication up there is a summary of two workshops we organized at the MSF to look at the future of data-intensive research in education. And the one to the far right was an MSF-supported effort at MIT with their online education to get them to think about ways that what was being learned about learning could actually be integrated into this large-scale online delivery. So we've got three things going on here. We've got changing disciplines, driving education. We have changing ways that we're educating students, taking advantage of online and other resources. And then coupled with those two, we have the opportunity to use data-intensive research to more rapidly advance our understanding of student learning. So that's a little bit about the context, a brief overview, but let's now move to the next slide and jump into the world of Deber. So Deber rose out of the improvement agenda, but it's been an undergrad at a fairly late effort. So what is Deber, right? Deber investigates teaching and learning in the context of a discipline. So it's really grounded in disciplinary priorities, the world view of the discipline, the knowledge that's important, and the practices. Now, this is a field that pulls on cognitive science, educational psychology, K-12 science education research, which has been very robust since the 70s and 80s, mathematical educational research goes back to the turn of the last century. It's just that it hadn't really found its way as robustly into the undergrad curriculum. So again, in understanding the motivation for Deber, it's important to understand that disciplinary scientists felt they were more likely to get traction with their disciplinary colleagues if the undergrad research became very specific to their domain and got embedded in their department. This is good research that it could occur in any kind of department, but in terms of the improvement agenda, the science and engineering department home base became critical. So this is not a field that's independent from all of their education research. Indeed, it's very dependent on and should and needs to collaborate across other arenas of educational research. In the next slide, I want to make a case if we can switch slides that Deber itself has actually become a field of research. So looking back at the emergence of science education in very late 60s, 70s, and 80s, what ended up being more of a K-12 effort was actually quite groundbreaking because within the sciences, it was a break from a more generic let's understand all education to the context of the discipline matters. I'll click to the next slide and what you'll see are some of the criteria that emerged from this analysis that Fensham did on what determines a legitimate field of research. And so when the committee looked at the discipline-based education research area, the conclusion was that Deber has emerged. It's in early stages, and it has its own field of research. So it now has a lot of the structural criteria that you see up here in terms of academic recognition. Within the US faculty positions where Deber scholars have been created, even Ivy Leagues like Cornell just hired a physics education researcher in the physics department. So that's a huge step forward. There are research journals in biology, CBE, life science education is actually a very high bar in terms of standards for research that's published there. The physicists have been at this a long time. The professional associations have embraced Deber. There are research conferences. Research centers, I think it's still a bit of a stretch. And there is training. There are individuals that get graduate degrees from postdoctoral training in Deber as a field. Intra-research criteria, the methodologies and all are emerging. The emphasis on having a theory of change and understanding a theory for the field, I think in education research at all as a whole is something we're still working towards. A lot of conversation, work, but progress is needed. And then definitely there are outcome criteria that are emerging. So I think we can claim that while it's fledgling as a field, Deber is indeed a legitimate field of research. If we go to the next slide, I share with you from my colleague Carl Smith at the University of Minnesota, a really first grade engineering education researcher. A bit of a timeline to contextualize when Deber has really gotten legs. So in this timeline, and it only goes back to the 50s, right, the post-Sputnik time point, the solid bars are where there's actual education research and more open bars indicate a commitment, a lot of efforts to improve STEM or aspects of STEM education, but not informed by research. So the most well instantiated field is actually the medical education research, and we often forget about that, but we can learn a lot from our medical education research colleagues. Chemistry and physics have the longest history for more than two decades. There have been degrees, a limited number, but the degrees of PhDs in chemical engineering at education research. Engineering knew the arena, but has grown very rapidly. There are actually a number of departments of engineering education research in the U.S. now. So hopefully that gives you a sense at least of where we are timeline wise. So moving to the next slide, I want to talk about some of the factors that have allowed Deber to advance fairly rapidly in the last five years. So this is a U.S. centric story, but I think it may help you understand why things have moved here and there may be things you can take away that you've translated into the context in India. So really, I want us to think for the next few minutes about the intersection of research, policy, and in this case, federal level policy, and funding, and I'll focus on the federal funding level given my lived experience. So in 2012, two key reports were released, and they were released together fairly intentionally. The former chair of the Board on Science Education at the National Academies, who had been involved in getting the Deber report going and all, was working in the White House at the time and was supporting the President's Council of Advisors on Science and Technology as they were delving into work on the state of undergrad STEM education. So the Deber report and Engage to Excel came out. It also happened to be the gear of the MOOC, right? It's when all this hoopla was arising around MOOCs. We're at a very different place now, I think in our understanding of where online education fits. It's not taking over the world, but blended approaches have been very valuable, and online is reaching populations that may not otherwise have access. If we go to the next slide, and I put a lot of text on these slides because I wasn't sure about the voice quality and all, so I apologize for the overly crowded slides, but I want to compare some of the key takeaways from the Deber report and key takeaways from the Engage to Excel report because the crosstalk between the two and what followed is important in terms of how Deber, I think, got moving in a fairly positive way after the report. So the report shows that, you know, we have pretty good evidence for students grasp of concepts, a little bit less expert right now in how we get conceptual change to occur, although there is research there. A lot of good work on helping students become good problem solvers, good research, good understanding of how we use representations in our different disciplines to help students learn. This is really tough for students, right? It's a bit of a shorthand that we use in our fields, but for a newcomer, not easy, and we actually know a lot about effective instructional strategies where there's room and there's been some work in the past five years, but for those of you that are launching Deber efforts, there's a lot of space is understanding what works for different populations of students. Most Deber studies have been fairly small and the data was not disaggregated amongst different learners. Also not as many long-term longitudinal studies. So when you think about system level change, you want to understand how student experience connects over the entire undergraduate years, prepares them for graduate school or work. Not so much interdisciplinary work or work bringing in the cognitive sciences. Deber grew up within the discipline, so it's been very siloed in terms of disciplinary domains and there's some exciting work to get out of that. There's a group of colleagues in the U.S. that are bringing together Deber and Cognitive Science and another group, the Deber Alliance. We can follow them through the Trellis site at AAAS that are working to look at questions that are important across all of the STEM domains and how we can advance the field more quickly if we all work together. If you go on to the next slide, one of the key conclusions from the report is we actually know a lot about doing a better job in the STEM classrooms, but there has yet to be widespread change and so the recommendations for the report were focused on how we bring about that change and I think the report wisely calls out multiple actors in this process. You don't just go and say, dear faculty member, you need to change the way you're helping your students learn chemical engineering or geology. We all need to work together and provide supports and we also need to change the reward system at least at the major universities in the United States. Tenure, promotion, decisions are based almost entirely on one's research success within the domain and there's very little incentive and often a disincentive to spend the time that it takes to change what's happening in your classroom. If we go on to the next slide, we're shifting to the engage to excel report, the report out of the President's Council of Advisors in Science and Technology and you'll see they are mirrored many of the things that came in the D.B.A. report. You can think about the D.B.A. report as the research document and the engage to excel as the policy call to action document where again, we know what works in classrooms. Let's actually use that in the classrooms versus simply talking at students. The second recommendation focuses specifically on laboratories and getting more discovery based research based experiences into the laboratory learning for students and the third one, my sense in looking across different countries and all this is a unique challenge in the United States but our failure to prepare students well for the mathematics that they need to go on and stem by the time they get to college and this was a point of great consternation for mathematics educators who actually have a very robust body of research on mathematics education research especially in K-12 but it moved quite rapidly into the undergrad arena. There's a new mathematics education research journal that's been launched quite recently post-2012. There's a group that are called themselves transforming post-secondary education mathematics that are working very hard on national level change. There's work at both the calculus courses and with the getting ready for college kind of algebra statistics level courses it's moved quite quickly. So some very, very exciting work in that arena. If we go to the next slide a year later the federal government released the first ever five-year strategic plan to improve STEM education across all education levels formal and informal education and if you go to the next slide you'll see that there were four strategic objectives specifically for the undergrad community and again, you know, as you look through those they map on to what I was showing you in the DEEBUR report in the Engaged Excel report but the push on can we get these evidence-based practices actually used in classrooms. The second one we haven't talked as much about in this particular convening this evening but we have two-year and four-year colleges and improving the two-year experience of a transition between the two years of four years emerged there and then the other two map on to the undergraduate research experience and improving mathematics education. So in the next slide, right, you can have plans you can have policy statements how do you begin to implement them so in the last administration there were cross-agency priority goals STEM education became one of them and if you go to the next slide 14 federal agencies worked together I had the privilege of convening the agencies that representatives that were working on the undergrad strategic objectives in the next slide you know a number of things happened I picked a couple to highlight here one was that across all of the agencies there's all sorts of great research opportunities for students and funded research opportunities elsewhere very difficult for a student to access that so we created a common portal we also worked together undergraduate research playbook that captured the collective learned experience across all of these agencies next slide and as that's coming up this is not for you to read the details but this is to foreshadow where I want to conclude with my comments a little bit later as part of the implementation of this cross-agency priority goal orderly all the working groups posted metrics on their progress on the White House website and the reason I raised this is it's really difficult you know what indicators one can use at what time frame is useful you know in terms of measuring change right it takes four years for a student to progress through an undergraduate curriculum at a minimum and in many cases it's longer than that so we're going to come back to the indicators question at the national level towards the end of this talk if we can move on now I really want to shift gears a little bit so next slide I hope I've given you a little bit of the context of what was happening in the United States about the undergrad improvement agenda and how Diber rose out of an interest to improve undergrad STEM education but is rightfully a field of research and as the field matures that can be a bit of a challenge because there is a difference between being an effective teacher and using the research that's out there and being a discipline-based education researcher so that's something that I think one needs to be very sensitive to and aware of I think we also now are at a point where great Diber is launched but we have to really think hard about the trajectory for where the field can go, what are the intellectually exciting questions and also how are those questions aligned with providing information understanding findings that will advance the quality of STEM education so we've actually known since 1972 that actively engaging students in their own learning rather than having them passively listen to someone has a much greater impact on their learning. We don't need to keep reinventing that we certainly need to figure out how to change that in the classroom but as far as Diber is concerned we need to move forward and there's a lot of ways one can do this to frame it, what I'm going to do is draw on four recent reports that came out of the National Academy of Science Engineering and Medicine through the Board on Science Education these are reports that were commissioned by the National Science Foundation to take a committee of experts really really expert folks that look deeply at the research that's out there synthesize it and there's a very very high bar for the research that is used to draw conclusions and make recommendations so the way I want to do this is first talk about the undergrad research experience then talk about the and that right that was in the area you saw as a priority then talk about this gap that we have in terms of what works well or which group of students and when and this shift that we're seeing in the United States from access for everyone to really ensuring success of all students I then like to get us to think about moving beyond the cognitive domain so there's a very recent report supporting student success in college that looks at the socio-emotional domain what do we know about advancing and measuring student competencies in the intra and interpersonal competency sets and then conclude coming back to this idea of how do we know if we're making progress and we can look at that at all sorts of levels but again at the institutional level how would you assess the improvement agendas impact so let's move on to the next slide and take a look at undergraduate research which for those of us in science it's really sacred right it's the apprentice model for preparing students if we go back and look 200 years ago humbled by this idea focused on unified teaching and research what's surprising is beyond descriptive research on undergraduate research experiences we do not yet have a robust body of research on what works well for those students this is a very, I think very exciting opportunity for deeper scholars and again I'm not going to do any of these reports the service they deserve I'm doing a very brief overview with a few salient points to the deeper field I'd recommend if any of these specific areas capture your interest or align with your own research you download the reports from National Academy Press I've included the URLs the PDFs are freely downloadable and there's tremendous amount of good information references and analysis in there what the report does well is play down that there are so many different ways we can integrate research experiences beyond the traditional summer experience that capstone experiences see your theses internships co-ops which have long been common in engineering cures are the course-based undergraduate research experiences wraparound experiences in the US are for students where there's a whole package of mentoring and other supports in addition to the mentoring for their overall experience not just the undergrad research bridge programs for students moving into undergrad or moving into grad education projects research projects that are coordinated across the nation so there's multiple ones there's an annotation project with the fly genome there's a project where students go out and gather fashions from the soil and sequence them and collectively the data is shared it's a crowdsourcing approach to addressing real problems and meaningful way and then more and more community based research where classes students are working in their local community using research skills to address relevant real world problems within that community if we go on to the next slide one of the very helpful things the report did was try to define what is undergraduate research right given that there's so many different forms you can take and not all research experiences the report states need to have all of these elements but many of them should be there I think it provides good guidance I think it also provides guidance as one tries to frame research questions about undergraduate research learning so getting students engaged in arguing from the evidence generating novel information working on relevant real world problems mastering specific research techniques and reflecting on the problems and again this is something that goes back to much earlier national academies reports the importance of reflecting on your lab learning is about the only way to know students come to understand deeply what the nature of science is communication and then recognizing that these are structured not free for all experiences and the importance of having a mentor as one is becoming a researcher if we can switch to the next slide and then we'll be past I hope for a while such worthy slides my apologies for so much text we do know that undergraduate research increases graduation retention rates especially for groups that have been in the U.S. at least traditionally underrepresented in the STEM fields it may increase a sense of belonging which has been tied to persistence within STEM fields may increase confidence in understanding the content doing data analysis understanding the nature of experiments one of the places that it's quite striking when you think about it it's not at all clear that a very rich body of evidence on how students learn has in any way been applied to the development of most undergraduate research experience group and the published work that's out there has yet to really delve into what are the benefits to students in terms of their learning from engaging in undergrad research and which aspects of the experience are most effective in bringing about that learning we also know that mentoring is very effective but we have a long way to go with developing professional opportunities for faculty to become better mentors there's some great work that's gone on at the University of Wisconsin at the University of Wisconsin at Madison, for example but we need to spread that work to be effective I'm going to move on to the second theme, next slide which is how do we make STEM education inclusive for all students this barriers and opportunities report is replete with data again it's specific to give you one illustration of that the graph on the left is looking over time literally over a generation at the percent of 24 year olds that have completed degrees and this is across all domains not just STEM as a function of family income so for the highest quartile in terms of income in the U.S. from the 70s to the present we almost doubled the degree completion rate right now it's up to about 77% for the lowest economic quartile again moving left to right on the X axis gone from 6% to 9% so we doubled the gap in who is Bernie for your degree there are other groups other demographic challenges that we face and they're called out in the report so this is something that is a key area of interest in the United States and again the report is very rich I'm only touching on a few points that are relevant to this particular conversation but the committee concluded that the culture of many STEM courses and departments actually is undergirded by the belief that natural ability whether you're good or not good at STEM which we know is not true gender and ethnicity determine one's success in a STEM and you can see how that becomes a very negative self-fulfilling prophecy if we don't change that one of the other important things for us in the US is that the pathway to a degree is actually fairly complex the average student is no longer an 18 year old and the majority of students do not go straight through four year full time and complete a degree it's more like six years and as a result they talk about students swirling through degree completion so as a result, STEM degrees often cost more for students and our large scale data sets in this country are just not adequate to capture the movement of students until this past December we tracked through federal data sets only first time full time students that's expanded and there's still work to do there if we go on to the next slide parallel can we shift to the next slide, thank you parallel to the work that's been going on in all the cognitive domain the deeper work in the US there's been a big student success movement pushing on graduation rate driven by data within institutions so the growth in data analytics has fueled a number of for-profit consulting firms I've listed some of them here as well as homegrown efforts within analytics teams at universities to look at all sorts of patterns with students which courses are students most likely to have trouble with there's some courses where introductory courses where the difference between getting a B and a C in the course makes a 50% difference in retention there's a lot of work done with modeling with financial aid and who's accepted to colleges and then how that feeds into which students are most likely to graduate a time really interesting work I worry with any kind of predictive analytics tools that you can in it inadvertently disadvantage students by advising certain groups of students out of STEM versus really figuring out how do we focus on the disciplinary learning not discipline but disciplinary as in chemistry physics and figure out how we make our classrooms welcome and engaging for all students so I think a very important area of research ahead for us is bringing together the predictive analytics related to student success and the deeper informed and more broadly learning science informed work on how to create classrooms that what we do the data analytics will be pleasantly surprised with the outcome so let's move on to supporting student success and looking there now the cognitive domain at these inter and interpersonal competencies this was a report that the board of science education from national academies under to do a number of things but first they looked at which of these competencies there is evidence for correlating with college success and then looking where there's promising interventions the overarching goal was to understand what do we know about the assessments we have for these competencies how when do they work what do they tell us and what do we need to do to get better at the assessment what's quite interesting about the analysis is where the data is right and we need to be very careful when we say the lack of evidence does not does not mean lack of impact we just haven't demonstrated it yet so if there are competencies like teamwork for example that probably you and I both believe are very very important to students like long don't show up on this list that might be a very interesting area to delve into in terms of research so the three competencies really rose to the top in this study and also have very low cost interventions that could be used at scale are a sense of belonging so if you feel that you are part of the future of chemical engineering you belong in the engineering major or you belong in the biology major you're much more likely to be successful in college if you have a growth mindset so let's think about math for example or the faculty you know in one of my previous slides don't have a acknowledge a growth mindset in students if you believe either you're good at math or you're not good at math you're not going to actually do very well long term even if you start out having a fair amount of success if you believe if I work hard I'm going to be fine and I'll get there it makes a huge difference and also utility goals and values if a student comes in to leading that what they're learning and doing has internet value in terms of where they want to go in life they're much more likely to persist now the remaining competencies on the list are quite promising the research base is not yet as robust the committee called out behaviors related to conscientiousness conscientiousness is a personality trait that's not malleable and certainly we need to be very careful about focusing on things where we can't change the outcome in fact conscientiousness is almost as good a predictor of success in college as one's SAT scores for example also academic self-efficacy believing you're going to be successful at academic pursuits having intrinsic motivation or pro-social motivation wanting to do this work because think about my very first slide about grand challenges that you think by doing this you can help solve very important problems in the world maybe water supply or health issues and then having a positive image of your future self okay let's move to the next slide and talk a little bit about the research gap so I've called some of them out but if you think about my list not my list actually the committee's list none of those competencies were interpersonal and again if you go back to the beginning of my talk the importance of individuals being able to work across multiple domains working effectively in teams we know those are very important workforce outcomes why is this not showing up yet in the research maybe we need to ask different questions but we also need to think are we actually valuing this in the undergrad arena more about community colleges the research that I shared with you is discipline agnostic so we need to know more about the stem domain so that's a great opportunity for deeper colleagues and then we need better assessments and the report does a nice job with that if you're interested so what I want to do is wrap up with how would we know if we are successful so the last couple slides are the next slides just released this past month from the Board on Science Education is framework and goals for an indicator system and so this is looking at a whole range of indicators that collectively will let us know if we're making progress there's three goals so focusing on students mastery and the young concepts and skills by making sure they have evidence based stem experiences so the theme I keep harping on we know this from Bieber we got to get better at doing this the indicator system recognizes that we can't go in and assess learning of every student we don't have national tests in the US so it's what they were focused on is how could we deliver the experience the second goal is focused on equity and diversity and inclusion and the third is making sure we have an adequately sized stem workforce if you go to the next slide um whoops did we miss one this is fine um this is that there's one that somehow dropped out no worries if we look at the model they developed there are a whole bunch of specific goals including inputs then looking at the actual learning environment and then looking at the educational practices and then looking at the outcomes if so just having a model is a big step forward next slide um I just wanted to show you what one of these goals would look like when you break it down and again the point that I was making in the earlier slide that if you want to you use evidence-based practices to advance students mastery of concepts we aren't going to we don't have the capacity to go and measure their learning so you'll see on this list that there's a number of indirect measures that can be used so there's forward momentum there there's an interesting framework if that's something of interest to you and again where I want to leave you and open up for questions is Deber is informing much of this work and its research that's tightly tied to practice and the big challenge is the translation from the research to the practice and moving on and asking generation meaningful questions in Deber so with that let's go the next slide has just opportunities ahead it's sums up the kinds of things that I just shared with you and maybe useful in helping you think about research agendas of your own and with that I'd like to close and address questions comments that you might have the final slide does include my email address if we don't get to your question and you want to reach out to me I'd be very very happy to communicate with you via email so thank you much for your time and let's take a little bit of time to see what's on your mind thank you so much so we'll take questions from the audience I'm looking at the suns coming up in Florida it's not just a black drop behind my face you can see there's actually light in windows okay, yes so is it possible for the she can see the audience I can see the audience so if you take the microphone back I can hear even in the back of the room so we should be able to hear everybody okay, yes thank you very much for this very comprehensive review and I think it's very useful the resources that you have pointed us to seem to be very interesting I just wondered you know so you've been talking about dber, discipline based and but all the sort of pointers or you know you talked about evidence based practices you talked about competencies which are related to success you talked about assessment now in all of these I I was wondering where the discipline entered I mean was there you know how is it that you know I mean perhaps these competencies or these evidences are specific to disciplines or are specific to areas of particular disciplines and I found it somewhat contradictory that you were able to draw such general conclusions and so something is missing I felt exactly so this is a wonderful question so let me tell you the story behind the supporting students success in college report when I was working at the NSF we funded the academies to undertake this study and what we were interested in is specifically your questions what do we know within STEM at least about these competencies and what was fascinating to me was as they dug into it we don't have the research yet that STEM specifics so we've got some of these global conclusions across undergrad education the same with students success and we haven't dug in to ask say in biology right we know you're not going to be able to work on any kind of a genomics problem unless you're part of a team right there's all the bioinformatics expertise there's the molecular expertise there's a specific organismal expertise we know the team matters or look at the work with gravitational waves right that huge teams with different expertise working together and the open questions right and the way I framed this presentation was I think to be provocative in the way you know that I clearly have gotten you thinking hard about this that we need to figure out where the disciplinary expertise is most important there are things that are domain agnostic right that we can all work on but I think for those of us that are in the science disciplines we need to think hard about what are the priorities within our discipline where are these pieces important we do know with things like sense of belonging for science students there's been research done more at the K-12 level but for women that come into science for underrepresented groups things that you can do to help them feel like they belong are going to have a huge outcome on their success but what does belonging look like in a physics learning situation and how do we tie that to learning because I don't think any of us would want to spend the time working on exercises that aren't also advancing students learning in the field so we've got a lot of context specific work to do so you are right there's a real dichotomy there and I think a huge huge opportunity for us to delve in and I'll add one more piece I don't want to give an overly long answer but I think it's also very important for us to recognize that if DBER only works in isolation on just the cognitive piece of learning and we don't partner with or educate others that are working on these broader efforts about what our specific challenges are we're going to miss an opportunity to collectively do better work and I don't think that DBER scholars should become scholars in a lot of these other areas but they need to be aware of them they need to partner with them and there needs to be better knowledge exchange I hope that helps a little thank you that's great so the next question here I have two questions if that's okay first question you've touched on it a little bit but I was wondering what is the state of the art regarding gender if you could say a little bit about sort of what we know or more importantly what we don't know what are the pressing questions that you think surround in this field so you know there's many areas where this is really risen to a bit of a frenzy a lot of attention so in cognitive psychology there was a lot of to-do made about differences in the way men and women think and people paid and I think an undue amount of attention to mental rotation the ability to see things in 3 and 4D over time I'm a developmental biologist so you know cells morph into new shapes and everything of all the different ways one visualizes things that's really the only element that there's some small difference and the National Science Foundation funded a number of science learning centers one that focused primarily on visualization Tufts and Santa Barbara and there's lots of ways you can help people learn to do mental rotation and to be distracted by that I think was really a challenge and I think also then I talked about growth mindset talking about positive mindsets is really important see I'm by like who had been at the University of Chicago very good cognitive psychologist she's now president of Barnard College as of this year did a lot of work with math success and math anxiety and it turned out that in elementary schools we were doing disproportionate disservice to girls because most elementary school teachers were women and a disproportionate number had math anxiety and that got passed on and affected math anxiety and success actually for the girls in the classroom more than the boys there was a gender difference that we were seeing so effective practices like pushing for a growth mindset that if we all work hard versus gender bias messages are likely to be very important in computer science there's a huge amount of work that's gone on because we still have so few women in computer science and there the work was showing that classrooms that really favored problems context interests that were more relevant to the men and the women the men in the classroom and the women was a detriment I would say I went to engineering school as an undergrad and I can still remember in my class being the only woman and they were talking about flywheel generators for those of you that didn't take apart carburetors and other things back when they existed in cars flywheel generators were part of that my dad had died quite young and I didn't grow up repairing cars and they're doing problems with flywheel generators and I had no idea what a flywheel generator was so I naively asked raised my hand and the professor found it incredibly funny I hope this does not happen in classrooms anymore but context matters and we don't all have the same prior learning experience so being aware of prior learning matters where I could send you in terms of a really terrific resource is to go to the University of Colorado's website and search for strategic toolkit there's a really great resource that's up there and it's geared more at advancing the careers of women in the professoriate but great lessons learned about climate in general and this comes from a very very good research study by Ian Austin at Michigan State and Andrew Larson at the University of Colorado a whole number of advanced sites in the US so advanced is an NSF program that's specifically focused on women and why we've had so little progress in advancing women in the professoriate and into leadership roles and this toolkit looks at well what's the challenge at your institution what are the research informed approaches that you might want to use to address that situation and I think that's really useful to look at and I think in some fields certainly not in biology anymore but in other fields having more women present as mentors makes a difference and having more individuals of color as well so a sampling there's a lot of research that's going on there I tried to do kind of a sampler that pulled from a range of different areas to get you thinking so that's a mathematician asking you this question it's the issue of gender in mathematics is becoming a bigger issue I suppose that's one of the motivations of having the question from you so we are actually sort of already just about the time we have crossed about one hour that was allocated to this if there is no other question I can pass the second question to Mania the toolkit you want her to repeat what is the name of the toolkit it's called the strategic the strategic toolkit and if you search I'm trying to think of which is verbally the way I find it easily is to use a Google search and do strategic toolkit University of Colorado and if you can't find it please just drop me an email and I will gladly send you the URL thank you we'll facilitate that so there is one question at the back so sorry Mania I'm not taking the second question from you yeah please identify yourself before yeah Susan this is Max Das I've been grappling with a grand challenge since your talk about how do we equip college and university level faculty to adopt those evidence based teaching practices and become good at using them effectively you have thoughts on that I do so one of the ways and in the US we've been pushing hard through learning and teaching centers and STEM learning and teaching centers there's a coalition of STEM learning and teaching centers that are working hard on that so that's one way to provide support to faculty another really beautiful program the center for integration of research on teaching and learning includes 46 research one universities in the US the goal is to prepare the majority of future faculty because most of them go to a limited number the top 100 universities in the US by preparing them as graduate students and postdocs and that's been in place long enough that there's good evidence that when they become faculty they're more likely to use evidence based practices and they actually get their own research launched more rapidly because they're more confident in the classroom one of the things I'm trying at my own institution with my colleagues is a little simpler and more focused and it comes out of work that the American Association of Colleges and Universities has been starting to do and that's a look at assignment design we're actually in the process of hiring a new director for our faculty development center and one of the key things is to work across all the disciplines to help faculty figure out how to design assignments so ultimately right with effective learning you want students to be doing work where they're really engaged, they're challenged the gap in their learning causes growth it's not too big that they're overwhelmed and it's not too small that there's no learning and that's a way we think we can reach all of our faculty in a fairly straightforward way for us the other piece of it is we've been using in our college-wide assessment and we're doing this across the board they're called value rubrics it's an abbreviation from American Association of College and Universities looks at things like problem solving critical thinking and they're rubric based assessments to evaluate assignments so given that we're growing in assessment culture that looks at assignments by working hard on the assignments to work that our faculty give our students we think we're moving pretty far and my current belief right and we're testing this is that when you have limited resources and you want to reach folks by pushing on that you may move things a little faster than some other ways we also have a lot of other wraparound supports new faculty workshops a series regular meetings with new faculty career development gatherings where you share information one on one classroom observations but I think it has to be positive there has to be an incentive we know a lot about conceptual change in adults and I think the other thing that is key and the American Association of Universities which is the very top tier universities is pushing hard on this is changing the currency of the realm how do we change the way we evaluate teaching so that we get a good accurate picture and then how do we use that to inform tenure and promotion decisions because we don't change the reward system all the supports of the world are going to get us far enough it's a big challenge but a good one and I think we're collectively up to the task thank you so much Susan Singer for taking time an early morning day we couldn't offer you coffee or breakfast today but we are all heading towards a banquet the conference banquet right now so people would have asked you more questions if you are joined us at that time so in any case thanks again once more for your excellent talk and for the audience we have the email on the screen so you can continue to interact with Singer at SRSingeratrolling.edu thank you and I can Skype with you too take care, have a great day and enjoy your banquet, I wish I could be there with all of you I'll say goodbye for now thank you any announcements? first of all I would like to thank Professor Nagarjun for sharing the session and just I will present a small token of appreciation and our dinner are about to leave the first bus will leave at 6.30 around 6.40 and there will be a second round if you miss the first bus in about 20 minutes yeah yeah around 7 p.m from our institute gate you gather at our HBCSE main gate okay we gather back at 9.