 Thank you. So, as we're introduced, we're from Carnegie Mellon University. We're on the Open Learning Initiative. And we're going to talk a little bit about how we brought whole life into learning management systems. And this is actually a natural extension, I think, from where we started with the keynotes this morning and the discussion about data exchange and systems interoperability. We'll talk about some of the challenges that we've ran into. Oh, I'm sorry. Is this thing plugged in? Doesn't seem like it's on. Can you hear me any louder now? Yeah, use it. Is that better? Yes. Okay, I'll try to talk louder also. If you can't hear me, just like maybe down or something. So, first we're going to kind of make a case for why OER should be in the LMS. We're going to talk a little bit about some of the standards that are out there now. Our approach, and I'd really like to hopefully move towards a discussion with all of you about some of the opportunities and challenges that are before us in terms of increasing the reach and consumability of OER. Just a little context about who we are. The Open Learning Initiative creates courses which try to enact instruction and support instructors. We're here, so we're trying to provide open access and another piece of our work is to develop communities of use and research that enable ongoing evaluation, ongoing continuous improvement of the OERs that we create. And so for us, the OER is about, I'm sorry, the LMS is about increasing access. We want to make things convenient for students and instructors. We don't want them to have to mess with multiple URLs, multiple websites that go to you, multiple accounts, multiple grade books, multiple navigation schemas. And actually our users request access to OER from the LMS quite frequently. Since access is part of our core mission, we believe that OER needs to be easier to find and that's being addressed with new standards in metadata and search, but we also think it needs to be easier to consume. And that means we consume the context where it makes sense for users to use it. And we think the learning management system as the, whatever you think of LMS is, as the information of the course is really the place that we need to make some improvements. Standards. So why not building blocks, modules, plugins, the old way of doing things? Well, simply there's too many platforms to target otherwise. There's lots of learning management systems. We want to hit all of them. And in the past, you know, we've seen this question we started with was, are these standards in name only? I think we all remember past conferences where we're doing the hack fest after, we have different vendors trying to get systems to work, or anyone who's trying to take content even just a few years ago and move your cross systems standards haven't been perfect. But they're necessary to provide confidence in technologies that can be adopted. So we did a survey and we realized that, you know, we're more than just a simple content package that needs to be encoded. We have consistent state, multiple assignments, learning analytic components that are becoming more sophisticated, which led us to the learning tools to drop our buildings back from IMS. What's really great about basic LTI is it's really simple to implement. It's the idea of your building tools, things like OLI, and you're bringing them into a tool consumer, things like an LMS. And this is the simple trust relationship between the systems, fields to identify the user, the course, the institution. It's simple but powerful. So I kind of put on this, it's the development systems. It's supposed to be easy to develop, supposed to be easy to consume. This pathway for more advanced features did that work in practice. Well, let's take a look at what we actually built. So we now have, this is the screen capture from Blackboard. So we actually went ahead, we implemented basic LTI on the OLI system. And what that gets you is we can now place a link to where we are from within Blackboard. And in two clicks, we can get to the course content. So the first click takes us to our research consent forum. We are a research project. We do collect data on students so we need their consent. And then students can jump right into the course. And so this is not, these two slides are actually, there's not a lot of magic here because it's all happening behind the scenes. What's happening when you click that link is we're actually doing a lot of work. We're figuring out who the user is in that LMS, we're creating their account in the OLI system, we're signing them in, we're creating their profile, we're dropping them into the right course. And all that happens transparently. And that's the power of the basic LTI standards. It provides a path from the LMS to OLI without actually having to deal with things like user registration, user signing, getting students in the right place, getting them in the right course. And it makes it really easy just to jump right to your content. You now have one click and actually as the students are working here, we're tracking their data, we know who they are, and they return their states persistently. So did it work as planned? When we ran to a number of challenges along the way, and I think rather than sort of dwelling on how we actually build things, I think this is really the area we want to spend some time talking about, because I think there's an opportunity here to improve the reach of OLI by addressing some of these challenges. And kind of on three fronts, technology, policy, and user experience. It really comes down to, we think, that standards today are designed primarily as a technology and not necessarily to serve user needs. We feel like current standards lean towards what content publishers are doing, what institutions are doing, but it's not enough focus on the actual instructors and learners who are consuming OER content or consuming content in general. The people that the standards affect, we're not actually getting most of the benefit, and we'll sort of draw out why. Versus sort of the technology issues, some products require an extension. So it's supposed to be easy to use, it has a wide band to reach, but in order to use basic LTA, we have to make sure that you have the right version of your LMS that it has the plug-in enabled that all the right features there. You might have to install some software. It's not, the technology is easy, but it's not automatic. We also ran to some implementations which we found to be buggy. We actually had to write some patches for the Moodle plug-in in order to accommodate the Internet Explorer. So it's hard to argue for adoption of these standards and tools if they don't work perfectly in every context as expected. And along the development side, the standard is actually way more flexible than a lot of the other LMS standards, which is a benefit, but also makes things complicated on the developer side. Our tools need to accommodate having zero information on the user, just a very opaque identifier, all the way up to all sorts of fields and information. We can handle lots of cases where the LMS is identifying to us, the person's name, their course, their context, and cases where the LMS is only identifying just a opaque identifier. And that makes it hard for us to build consistent experiences with the data that we get from different systems without the user is different. The other thing that came up which surprised us a bit were some of the policy and process issues that we hit on. So at any time we're bringing two systems together, institutions take a closer look. And we thought that since a lot of the institutions we're working with are already using OLI, they're used to having students register, they're used to having students' data coming to the system, that this wouldn't be as much of a question. But the fact it was, and there were questions about ownership, does it register our office, own student registration data? How do we find help for institutions to navigate this space about who to talk to in their institution and what policies they have governing what their LMS can connect to? If we're targeting instructors as the end user consumer of OLI who are adopting OER, we need to think about how to support them if their institution has all sorts of policies about data and privacy in terms of bringing things into the LMS. I also like to add that the institutions we have worked with so far, we're piloting this technology at the moment, have been very eager to work with us and we are nonetheless running into these process considerations. So imagine that you are an instructor at an institution that has not yet set up an LTI in their LMS and you have to figure out navigating your own university, who do I ask, how do I get this set up? So even these barriers as big as they are, we're in almost the best case scenario. Which is the other piece, I'll turn up the bill for which is the user experience component. So as John mentioned, one of the things that we feel is a problem with a lot of these standards including basic LTI is it's a technology and it's not really addressing the users and what they need to be doing in their workflow. I'm just going to yell at that, okay? So why is good user experience critical to adoption? Using a computer system is really easy for some but it's really not for a lot of people when you're trying to impose this and you're trying to improve your workflow. So for an instructor, they may not adopt a technology if it appears that it's complicating their lives. If it looks like it's a real difficulty to set up, they might consider it if they're highly motivated because they see the benefits but if they don't see the benefits and how this is going to improve their workflow, they're not going to be interested. Also about students, students who struggle with interfaces all of a sudden have a negative affect about what you're trying to teach them. Every time I go into it, I'm reminded that I don't like this experience. I'm just going to rattle through a few of the issues that came up. What we showed you with the two-click experience is really the best case. This is not reflecting all the possible situations. So roster management. Since we're relying on the LMS to tell us when students are registered until the student initiates an action, we don't know who they are. So an instructor now has two rosters to manage their LMS and their OLI one. Until we can implement grade exchange which requires more than what basic LTI can provide us, you now have two grade books and that's both for the student and the instructor. Logins. So we can automatically log you in from the LMS and at first we didn't even provide an OLI logout because what if we log you out but you're not logged out of your LMS and then you come back but then people were concerned that they don't have a way to log out once they're in our system. So we had to put log in back but now we have students who can be logged out of our system. Bookmarking. Basic LTI does not allow us to refer back to the authenticator to send back to us. So if you hit a bookmark from OLI but you're not currently logged in we have to kick you back to your LMS but your LMS doesn't have a way to kick you back to us. Desktop support. Where do the questions go? Do you have an OLI question or do you have a Blackboard question or a Google question? How do we figure out what that question is and what we want? Where do you have your account? Do you have an OLI account or do you have a Blackboard account? Well, that sounds like it should be an easy question to answer. You've got one or the other. Well, we have existed before we've done the Basic LTI so we have a lot of relationships with faculty who already have OLI accounts. The students already have OLI accounts. So for the case in Carnegie Mellon we have to support both. For existing users we have to support both. Right? Unless, of course, you're not using an LMS at your institution then you want an OLI on the account. Oh, but you just found out that you could hook OLI up to your LMS. Maybe I want to do that. Can you merge my accounts together? Yeah, we could do that but if you already got students who are registered now we're moving their registrations and now you've got to tell your students they need to go log in somewhere else. The point of all of this is that it's not easy to communicate to novice users about how they should be using this bridge between their LMS and OLI. So now suppose, okay, I've even decided I've looked at all this and decided that's okay I don't mind two grade books, I don't mind this monkey logout thing, I don't mind the account confusion I'm going to go to set this up. Well, as I mentioned before even in our best case scenarios you still need to find out who you need to talk to to get things set up at your institution. Then when you have it set up at your institution you have to go through something like this to actually configure the link. There's actually multiple screens to set things up in Blackboard. You have to enable the basic LTI component. Then you have to register a provider domain not that anyone aside from maybe somebody in IT knows what a provider domain is providing credentials and secrets and those sorts of things. And then actually in the course, as an instructor you're thinking about adding, I want to add this to my course well I have to first bring in that tool to the Blackboard course and then if that's not enough I have to add a URL because that might not even be obvious. I know we are experienced as a URL but I'm thinking about it in a piece of courseware so you find that there's a little tiny checkbox that says this URL is a tool and you have to know what a tool means because a tool could mean different things inside the LMS system and to get all of that right is multiple clicks, multiple screens. There's no straightforward way to get set up and configured to consume in Blackboard and other systems are the same. We found similar Moodle, we found Sakai was a little bit easier but still had a lot of technical language. Well the point there is that depending on the LMS it is different. What they call these things what you have to go through is different. So imagine trying to provide instructions from our side, from Oli's side saying okay what LMS do you have, what versions do you have how's it configured, what have they done you know for the reconfiguration and so even though we can achieve these two clicks scenarios through basic LTI from a user experience point of view there are so many rattles that this can just fall apart through and I really believe that the difficulty there is that this technology solution not one that really pays attention to what's the user experience supposed to be like and I really think that's critical for adoption. So what's next? So things that we're working on we wanted to provide some more configurable entry points in dual life so maybe you could jump right to our learning dashboard which is our learning analytics. More other locations that are instructor configurable. We wanted to do grade exchange which we can do which is a larger spec but that's harder on the development side. We actually have to do all sorts of web service stuff to make that happen whereas basic LTI is more web 2.0 it's easier to simple requests. To a consumer we're actually thinking about a lot about bringing other tools into our environment but I think really where we want to focus sort of remainder of the talk which is why I know that we've kind of spent through things a little bit is to to talk a little about the opportunities going forward and so we see sort of two clear opportunities for OER and the LMS. One is enabling learning analytics to provide that data for continuous improvement and evaluation and to really as Bill just talked to us about is building a better user experience so that anyone not someone not an instructor has to find someone in IT to coordinate through your blackboard administrator to coordinate all these things that happen to be able to use a piece of OER in their LMS. So at the Open Learning Initiative you've probably seen this slide one of our core ideas is that we feel one of the most powerful features of web-based instruction is this ability to embed assessment in every instructional activity and then use that data to provide feedback to the learner, to the instructor to the course design and development team and to the science of learning however right now that data and there's others doing this but that data gets trapped in proprietary tools it's not it doesn't flow across system boundaries it doesn't flow across organizational boundaries it's locked out and inside of a lot we try to provide this comprehensive view of learning we call it our instructor learning dashboard it's a way where we take an outcomes focused outcomes centered approach to giving instructors information about their students learning but imagine if that could be extended to all of the resources that happen that happen to be in an LMS why can't I mix two OERs something like my math lab or my stat lab something I created as an instructor and have all that data flow back to one central analytics tool that provides me that comprehensive view and there's a lot of issues around around this standards for data exchange these are all things we talked about this morning licensing issues, platforms and tools policies but we really feel to work through all these things we have to think about LMS interoperability, systems interoperability and learning analytics as being automatic not an afterthought, not something you go back and put into a system that when you're creating an OER as the OER creator you shouldn't have to think about these things the tools that you're using to offer your content should do this for you all this stuff that we talked about all these technical issues should be behind the scenes so building a better user experience how do we get to that component what do we choose a familiar model what do we choose something that I bet everyone in here has in their pocket that they're familiar with which is the OER app store idea why can't we find OER within the LMS why can't we have one click access to take something that's a complex OER not something that's just a web based resource but something that has interactive learning environments of persistent states and logins and grades and sophisticated learning analytics and just add that to a course in a really easy way we also need something that provides a basis for choice so we can choose from the various things that are out there and that should have evaluation data and context of use and a way to bring communities to those resources that are being consumed and we think that an app store would be one way to do this so the question becomes why hasn't this happened and so I updated this slide with Sutter I'm sure everyone here has heard about the Pearson open class enhancement so we see publishers now moving in this direction but there isn't I know that there's some efforts there's lots of portals and aggregators and things here but we don't actually have in this community something that's pervasive in every learning management system that provides a really easy way to grab OER content again why hasn't this happened probably the same issues institutional policies about data issues always about consuming resources licensing data ownership technology standards and the message that we wanted to leave with is that we need platforms which make it easy to create share, find, use and evaluate OER and the user focus needs to be on the educators and learners who actually consume it not on all this sort of technical stuff that happens behind the scenes we're not going to have a lot of time for discussion but I also want to mention if you're interested in sort of the user experience perspective that we are bringing to OERs Renee Fish and I are talking tomorrow about that and we're also going to be touching more on the idea of OER convergence through learning analytics we're talking with my colleague Norma Beer about that also tomorrow we can kind of dig in more on the data standards issues there so we hope to use the remaining time for discussion because I think that the discussion was really stimulating this morning and so these are some questions that we wanted to pose to you we also want to show you questions for us but things like how does OER make greater inroads to the almas do you agree with this OER app store approach does this model extend replace supplement the content package idea that we've been using so far can open versus close free versus commercial exist side by side in such a system and what are the next steps to make this happen because I really believe that we all need to come together to make this type of thing happen to actually have a place to share the great work that we're doing share the evaluation results that were the context that we used, the pedagogical design it's really easy to consume in a place where educators are looking for resources replacing the resources that are working I just wanted to mention that Blackboard has responded to Pearson's open class by stating that they're going to implement some kind of OER resource within their system and yes I think the app store approach is a very exciting one but part of the problem as I see it is there's so many resources out there, something needs to come about that organizes them in a way that makes it easier to access what you're looking for I really like the OER Commons I think it's a step forward from Merleau in that respect although I like Merleau but it can be just overwhelming OER Commons is a little more I think a step in the right direction and so we agree with that view that it's a dynamic that's the idea of providing a basis for choice and one way to do that is to communicate the design intentions behind resources to provide communities where you can talk with folks who provide case studies of their use to share the results, the context of use data all those sorts of things along with the OER and then to try to pull us communities around similar OERs so that we can actually have convergence and to actually take care of advantageable power and analytics and I heard you talk a little bit about I think most of this in respect to higher ed can you speak of any situations where you've had K-12 experience with this or have you been doing anything from a K-12 perspective I mean the question I really have is addressing the first one and the last one I think reaching out to some of your K-12 environments it will help better perpetuate what's happening down the line but have you had any experience with the K-12 environment yet? Very limited there's a few high schools in the Pittsburgh area and elsewhere that we work with they're using our courses and we engage them with this question about do you have a learning management system and we found that actually they're using very different tools that we're not familiar with we didn't find Blackboarder or Sakai Mood would be as common I know there are schools that use those systems so I think from our side we reached out to familiar with the technology that's in place there high schools and it would be interesting to do you have some insight to talk to you about maybe how we can get on those platforms also I think a lot of it's hard to argue with some vision in particular but a lot of the trick part comes down to the issue of standards yes and I having bashed the standards a little bit in the open session I think it's worth taking a moment to defend on the other side so I haven't worked in textbook company and every single one of the issues that you listed as problems that you have with LTI in its current form of education are issues that we have as well and there are a couple more and so on and so forth but what really happened there from my perspective is that a group of folks got around the table to solve a particular problem and that problem was too interoperable it was placing a video conferencing inside the LMS frame and they solved that problem and it turned out that the way they solved that problem brings them hard way to solving the problems that OLI and that set the age and others in the content fields need to solve it but not the whole way so really there are mechanisms with the IMS including a thing called a profile which basically says you want to get certified for content LTI or operability you have to include data in these fields there are ways of dealing with that most important thing is that you get the right people around the table so it's really important right now for educational institutions and for foundations and for OER consortia to step up to this table and to be involved in working groups to articulate the use cases I don't think there's any barrier other than the usual barrier that it's hard to create standards that work for everyone I don't think there's any political barrier to solving the problems that you're having we just need to get the right people around the table I'd like to just push on that just a little bit because I agree with you and I think I agree with you in terms of that basic LTI original startup for a different intent I do see folks who have run with use in Sky 3 to bring in Sky 2 pieces in a very clean way using basic LTI but I also have seen cases where there will be a standard and then a publisher kind of extended in a use case and then they'll kind of tweak it in some certain way and then we'll get contributed back to the community for a while and standard take a long time to come out and I want to just contrast that with what happens every day in Web 2.0 someone will skunkworks project, call up with something, put it online it'll go viral, it'll take off it'll be a simple web-based API and everyone's consuming it instantly and I just wonder why that hasn't happened in our space like why isn't there why aren't there organic standards that emerge why does it have to always be bringing people at the table and I'd love to be involved in those discussions we have use cases, we want to fix this issue but I also just I also question whether the process to date has been really serving us well so I think we get the time signal so maybe you could do one more question and you can couple of things one of the things from your discussion is do we really need to concept of Nell as it was pointed by this point the way to think about it I would just say maybe that was to be thought for just a minute and try to attack some of these problems that we're dealing with and another thing, dealing with standards and the processes historically you're absolutely right that has been the case there's a lot of reasons behind that I will say this however that the process is never, it's like all of them they're always wrong but they try to be as useful as possible and they are ongoing and I think there are faster venues and then people recognize that problem and are working at this point to try to work through this there are things in process that are attacking any type of venues that may or may not be a standard or maybe just a web service definitions, whistles, those types of things that are being worked at just to respond really fast and we can talk later if you have a chance but to push on where I will be usability direction on that I see the installation of LMSs in so many institutions all over the world as an opportunity to take advantage of something that's there rather than trying to develop a new solution that although it might be better will take time to get out there and so in so far as we can take advantage of an install base and what users are getting familiar with whether they initially have some trouble getting familiar with Blackboard lots of them are familiar with Blackboard now and so can I, if you're familiar with Blackboard get as good as I can so I don't have to teach you anything to pull my OER into it thank you