 So you notice there's four names on the here and there's only two people up here Yeah, the joys of being an operation. So we had an operational issue. So Lance and Rudy are gonna be off dealing with making sure the system stay up rather than talking to everybody But we've got all the information and we know all the goodies. So You probably won't even notice. I just wanted to Didn't want to confuse anybody on that Greg is at least three people. I'm pretty sure No Wow Some people will take anything Let's let's go ahead and get started. I don't want to keep you guys waiting in board. Good afternoon Happy Thursday Hope everybody had a good lunch. My name is Greg lunche This is Ken let over here. Hello. I Was up until recently part of the open source lab team And I'm kind of the person who Started this whole project and then handed off to Ken and the rest of the team Lance and Rudy at the OSL. So we're gonna talk a bit about this project and Tell you where we were where we how we got started where we went and then kind of dive into the technology stack a little bit And then talk about where we're going or where they're going. I should say I think I still think we have you know It's my project still I guess So before I really get started How many folks here so I show of hands are involved in education in some way higher ed k-12 Right on I Love it. Okay. My people are here. This is good second question is how many you are Not from Oregon Yeah, good new people All right, it's gonna be fun So just a quick outline So at Lance and Rudy are The two gentlemen that aren't here. They're the operations team Lance manages the open source lab denado is Oregon State University So he and I were the ones that were kind of the leadership team of the lab during my tenure there I handed everything off to him and he's doing a great job with it Rudy is what is one of our system administrators down there Ken Heads up the software development side of the open source lab so I just kind of give you a some scope on that and I was the program manager and founder of the virtual school district project that Came in design the initial system that these poor guys now have to go clean up the mess that I made which we'll talk about later So actually before even get this far Just to guys give you all kind of some scope and scale here Oregon has roughly three million people in it We have 200 198 school districts about 1200 schools in the state 550,000 students, so we're not a big state Heavily rural we have you know very strong concentration of people here in put the Portland metro area and then lots and lots of very very small school districts So just gonna give you an idea. This is so it's an interesting mixture We have some large urban districts, and we have some very rural districts, and so it was kind of an interesting challenge So back in 2005 I keep stepping away from the mic. I like to walk when I'm talking. I wish I had a laugh So back in 2005 a Study was done That made it up way its way up into the legislature that ranked Oregon 46th in the nation for a technology adoption in the classroom Now the study I think was flawed because they did things like they dinged the state for not having a Central online virtual school when in fact there were like nine of them around the state So I think the metrics were flawed, but regardless these made it to the legislature and the legislature You know oh my god. We can't have this happen And completely flipped out So they did what legislators like to do they passed a law And you're welcome to look this up. It's very simple. It's actually really easy to read Basically just said the State Department of Education shall build a virtual learning online school Infrastructure to improve technology adoption in the classrooms Interestingly enough, that's all they did So this was very different. This is a kind of a surprise to us And that they didn't actually dictate to the Department of Education how to do that They just said here's two point two million dollars for the next biennium. Go make some magic happen Which I am eternally grateful for because this could have been a much very very different project if that had been the case So the department and so this created the Oregon virtual school district Now we were given that name by the legislature. It's actually written into the law as I'll talk a little bit later. It is neither a school nor a district, but that's okay so Department of Education Suddenly had this huge lump of money and you know the Department of Education is primarily a regulatory and research agency You know, they generate statistics. They make sure, you know statewide standards are up to date They manage testing systems things like that. They don't do a lot of technology other than some data warehousing, you know And then reporting and things like that So They're kind of scratching their heads a little bit gone. Okay. We need to do this, but we're not quite sure how Unfortunately, there was a visionary in the Department of Education a fellow named Steve Nelson Who unfortunately it's a very uncommon name and good luck finding him But if you want to know why I know how to get a hold of him. I'm happy to put you in touch Was aware of the activities at the open source lab. He was aware of the Strength of open source. He'd actually been working in state government in other groups dealing with open source stuff and realized wait a minute We need open source to do this And I think this is key because he came to the open source lab and said, okay We have this need. Can you guys help build it? Yay geeks to the rescue So he came to me He and I had had a previous relationship. We knew each other. We we'd worked together in the past He said I need your help. Can you help me build this and I've been working with the community colleges and the and the Universities here in Oregon Working on some distance learning applications things like that and I said, oh, this sounds like fun. Yeah, let's build some fun stuff So let's get to a few definitions here first As I said at the Oregon virtual school district is neither a school nor a district Doesn't employ any teachers doesn't grant any credit doesn't have any classrooms anywhere in a fit in the rickton mortar sense And this is one thing I heard a lot when I initially was getting started on the project I was not talking to teachers in schools. I go out there and I say, hey, I'm Greg I'm working with the Oregon virtual school district. I've got all this great stuff for you. What do you need and? They said, oh great another unfunded mandate and I'm like What million dollars what are you talking about? so No, actually it is it's a funded unmandate. It's here One of the things that was actually included in the law the law said you may not charge the schools for the services Provided by these systems Okay, hmm open source great. No problem. We got it So it is actually funded it still is funded. So this it was originally funded in 2006 It's still being funded today. I am about six months out of date on the current budget status of it But I believe it's been funded for the next biennium to yes it has okay. Thank you Ken and so We realized okay, we're geeks throw technology at it's gonna work, right? No, no, no, no, no. No, these are teachers we're dealing with these are human beings as you know We're talking about the keynote this morning, you know These these messy humans, you know, they mess everything up We knew that we couldn't just throw a bunch of technology at it and make it work So we also engaged with the local education service districts to say we need a training team We need a help desk. We need support We need content, you know, we can't just throw a throw a website up there and expect them to just use it That's out of date. That should be six school years But it's grown just a little bit we started out with it's a few and it's kind of Exponentially virally expanded And I can kind of I'll talk a little bit later about how that happened and some of the things that worked and didn't work And actually I think that 180,000 users is about six months old too So I think it's quite a bit higher than that now probably over 200,000 now No, no, these are user accounts There's some squish in those numbers These are both staff and students we didn't eliminate between staff and students there And also there's it's kind of a sliding window because students You know may come in for a quarter and then not be in there for a quarter. So that's the aggregate number of users so Elevator xkcd. Sorry, I have to use it in all my my talks whether you guys like it or not So feel free to throw things at me We did a lot of thinking actually the sandwich shop right over here. I'm turned around I'm not quite sure which direction just a block from here So a friend of mine and I sat down and we actually sketched out on a napkin what we wanted to build So, you know, it got its start, you know 100 yards from here And we started thinking about okay Who are these people who are we going to be talking to and who do we need who are the key stakeholders? We need to get on board with this to make this thing a success So that we can really make a fundamental difference with what's happening in these classrooms out there in the world So we knew we had to get the administrators on board I mean with administrative support the teachers aren't going to be able to do anything because they're they're not gonna you know We had to get the teachers out there and we know teachers are incredibly busy people We know that if I don't hand a teacher a tool that they can use in the classroom tomorrow They're gonna go. Oh, how nice and walk away You know, it needs to be something that is instantaneously usable and intuitive enough that they don't have to spend a lot of time learning how to use it And then we acknowledged that okay district IT, especially in school districts are perennially horribly underfunded and We knew that if we did anything that made more work for them They were gonna come after us with torches and pitchforks. So we knew that these are kind of our key stakeholders And the last bit here is in education, especially six seven years ago Open source was not particularly well-known at the K-12 level So, you know, they had this kind of if they even knew of it at all It was some, you know, long-haired bearded hippie and they're coding way in their basement wait. Oh, yeah, never mind It's fine really So we needed to have a little bit of education about what is this kind of fuzzy wonky a weird hippie open source thing that we're dealing with So this is what we used I Love this slide. It's alphabet soup on the yummier Um So most these tools are pretty standard stuff, you know, Drupal is a major component of it We built the the central portal on Drupal that linked out to everything else Moodle everything was built on originally on Apache Mahara, which is a variant of Moodle that does e-portfolio work You know WordPress PHP Linux That funny little blob in the middle layer is Gennady, which I'll talk about in a little bit and Jenkins So that's kind of the stack we built it started very simple with just Moodle and Drupal It's kind of created other things as requests have come in for this that and the other thing So I'm gonna dive into some of the technology details here First of all, how many folks here are kind of you know technical administrative or you know in the IT department type people Okay, great So I wasn't sure what the audience was going to be so I wanted to have some tech details And we can I have plenty of time for questions and things like that So we can dive down into the guts of it if you want to know how the gory details of how this was all built So I'll start kind of at the high level and then if we if we need to dive down I can I'll be very happy to do so So since the open-source lab at OSU We've specialized in hosting. That's what we do we host virtual machines and physical hardware for open-source projects This is not too different. So we did what we knew, you know, we fired up a virtualization cluster using Gennady If I fired up a few blade servers Started out actually we started out with two physical servers, which Frighteningly enough are still in service and still grinding away. They're happy. I'm not gonna touch them Throw through some blades into the mix Got some back-end database servers going and then just started kind of growing that cluster as the as the project grew Originally we were doing local storage as the system grew we needed to start getting in a network file systems things like that So we rolled out Gluster for that. It's been a mixed blessing. It's worked. Well, it's easy to configure performance not the best but You know with some tweaking it was it was again good enough Which a lot of this a lot of what I did especially early on was good enough and that's kind of key I think Running everything on originally Gen 2 newer systems are now running on CentOS Doesn't really matter. It's agnostic. You can throw whatever the heck we want in there. It's all Linux. It doesn't matter And then last but absolutely not least is as configuration management originally using CF engine Did some stuff in puppet and are now in the process of migrating into chef and Without that we never would have been able to go this far because you know I was literally a once for one person project for the first four and a half maybe five years of the project I was doing everything. I was doing the coding. I was doing the system administration I had assistance from the team at the OSL for supporting the physical hardware But I was doing everything including you know some of the training and things like that So I just did not have time to be messing around with boxes So configuration management I can write the config once and push it out to all the boxes was the only way I could possibly not have more gray hair than I already do So we started small Two boxes there were decent boxes. They weren't huge, you know quad cores nothing huge They're still grinding away Frighteningly enough But it was it was sufficient for what we needed. We just had a few v-hosts We had a you know a couple of databases nothing fancy, but very quickly we started getting some big sites on there We had a large school district our friends from Salem Kaiser over here Came in third largest school district in the state wanted to put their website on the system Fantastic great suddenly we were getting 20,000 30,000 40,000 hits a day where we were had been getting you know a few hundred So immediately we had to start figuring out. Okay, what I mean to do through some caching in there Started going to multiple back ends started going to you know network file system Brought up on ice-cazi appliance, which I'll talk about in a little bit And so we started segmenting things out a little bit because we wanted each district has different needs each school has different needs And so we started wanted to separate things out so that we could customize things per district so that we could match we could sync to their Particular single sign-on environment, you know, they're using that you know active directory or whatever We wanted to be able to be able to hook into that And so thing, you know, we had kind of this infrastructural scope creep happening things kind of started Expanding and we had these one-offs popping up here and there Which really helped us in the beginning it got us the adoption we needed we got they got the users out there We got them using the system, which means the goal the project was Was being met But it became more and more and more of a management problem because everything was a little bit different And it was all stuck here in my head and I was the only one running and I can't forget things You know my brain is Swiss cheese So Recently in the past year Started really looking at consolidating and rebuilding the infrastructure any much more standardized forms so that You know one no one person is a single point of failure It's not all stuck in my head or second a document somewhere But that the rest of the team at the open source lab whether it be the student workers of the other full-time staff Can actually work on the system plus we wanted to build in some more redundancy because we had multiple points single points of failure in the system So we started building out the same infrastructure. It's actually still on the same physical hardware. It's still on those virtualized nodes on Gennady, but you know doing things like HA proxy for load balancing Running multiple redundant web nodes where all of the sites live on all the web nodes So that rather than segmenting them out by school or district We put them all in the same pool because they're all really need a lot of the same resources And we did the customization and at other levels rather than at the per machine level Which saved a lot of headaches that also made things a lot easier for both scalability You know things are starting to slow down through another web node into the mix fired up. We're done not a lot of work It also allowed us to do maintenance and things like that We could do rolling updates take a note out of the rotation update it put it back in take the next one out Update it put it back in Can you guess where a lot of sysadvins here? Expanded Study using more network file systems This is where we really got into Gluster instead of using Gluster sfs for shared files between all those nodes because we had to have all The same files on all the same nodes because we didn't know when a user comes in which node they're gonna end up on And then also expanded to my sequel cluster. So we had multiple mass multiple slave so that we could actually scale out that so then I'm gonna hand this off to Ken here in a second, but I want to kind of give the background here We'd you know about four years in we started to see some real success We started to see some real traction. We had schools all over the state We'd had some contact with every single district all every one of the 200 districts all over the state at some point Somebody in that district had logged into the system and tried the tools out some districts are going in whole hog You know, we had tens of thousands of users at some districts some we had maybe one teacher with a class of five students and We were constantly in need of some metrics, you know, we needed to report back to the legislature at funding time to say hey You know, let's let's show some outcomes here. What's happening? What's working? What's not working? How many users do you have in there? And then we kind of you know, I could scrape through the system and count the number of user counts and things like I could tell them How many you know virtual servers were running and things like that But we really needed a lot more there and so we went to Ken here and said Ken We need help because we've got the system. It's kind of outgrown this manual structure that we had and So this is what I'm gonna hand it off to Ken now This slide probably really ought to start with the last point because this was my entry point into the the project Gathering statistics from all these thousands of sites Some of them we didn't know where they were if they were active We wanted to know things like how many people how many user accounts are on the site How many user accounts have actually been used on the site where they created and never used again So we began writing some plugins for both Drupal and Moodle. By the way, how many people here are familiar with Moodle? Okay, good. So Like Drupal, it's a PHP system like Drupal. You can write plugins. You can create web services in it It's much more difficult to work with than Drupal. It's a much larger more complicated complex system but We found that it was not too terribly difficult to get Drupal and Moodle to talk to each other and To gather those statistics we needed so once we started gathering the statistics and the metrics We realized we needed some way to gather them all in one place so that we could actually look at them and Have some sort of central management for all these thousands and thousands of Moodle and Drupal sites So then we're going to jump back up at the top of the slide with the OR VSD central This was at some point while writing these plugins. I realized we need a central application to manage all this stuff It's actually a Python application, but it relies heavily on REST APIs We create web services type modules in Moodle and in Drupal so that the central control panel can talk to all those systems We install a plug-in in each one that collects Basically everything we can figure out how to collect from each site how many courses are installed how many courses are being used how many users are enrolled I think there's a few other things the system details basically where that server lives What it's URI URIs are and We can gather all that data in one place and generate a report that we can hand to Steve who can walk into the legislature and say look These are all the schools that are using our sites. These are how many users are using how many students how many courses This is what we're actually doing with all the all the resources that we've been given So I mentioned REST APIs a little plug for Drupal working with both Moodle and Drupal Drupal is a pleasure to work with it's been really really nice Writing plugins to talk to other systems is so much simpler and it's It's becoming a real key to putting all this stuff together because Drupal allows you to have that portal site that people can log into Drupal can talk to or VSD central find out what courses are available to install on the Moodle site We can create we're in the progress of doing this right now We can create a block in Drupal where people can go log into their school district See the courses that are available that they can install on their Moodle site Click some some checkboxes and hit a button and everything goes together. It's it's beautiful The other thing we're hoping to do soon is implement Drupal as a user management Platform so each district can go in and add their users to the Drupal site Which has much nicer cleaner user management tools and then those those users are automatically imported into Moodle You can go directly to your Moodle site if you're logged into Drupal your Excuse me automatically logged into the Moodle site. So it's it's been It's been a great tool for us Let's see. I Think I've mentioned the deploying news. Well We're working on deploying courses to Moodle sites using that or a VSD central and the the rest APIs We're hoping to also be able to deploy entire sites. So I don't know on the previous Slide there was a little box off to the side labeled Jenkins. I don't know how many of you might be familiar with Jenkins It's the the butler of the system in world. It takes care of things for you. It does things for you reliably When you want them done and in this much detail as you want them done so Or VSD central can use those same tools the same Jenkins API that the infrastructure is using to keep sites updated To deploy entire sites So a new district decides they want to get their Moodle and Drupal set up. They can come to us we can push a button and Voila their Moodle site and their Drupal site pops up with the correct users whatever courses they wanted all put together for them. I Think that's that that covers most of what I'm doing. Let's see if I should add some more details there So turn it back over So I want to talk a little bit here about what worked and what didn't work And then I can dive more deeply into either the technical side or the educational side Which of you prefer after we go through this real quick What works really really well Was we ended up going out to professional development events with teachers whether they be conferences I've been up on this stage in front of a bunch of teachers multiple times in this particular building Talking directly to the teachers because this is because this isn't doesn't have any constant because It's all hosted remotely. I can hand a user ID and a password to a teacher I can hand the URL of the site to a teacher and say here go at it And within you know five minutes of being registered. They're looking at their course Which they have had at rights on a Moodle server There are they could be looking at a Drupal site if they wanted to And that was key, you know, it's all about though your teachers are busy if you don't give them something can use tomorrow They're not gonna use it. This was our way of okay. I'm going straight to you as a teacher and saying here use this It's free. You get to use it. We it's supported. We've got a help desk to help you with it By the way, I have to point out Jackie here is on the help desk. She is absolutely wonderful. She's brilliant So I have to you know, I have to applaud her because she kept me sane for the last five years The other thing we did was we engaged teachers to teach other teachers As the bottom bullet says success is viral So what would happen is we'd get one of those early adopter teachers And this was one of those weird difficult kind of contradictions because we wanted to get the same in the hands of the teachers But we wanted to make it easy enough that any teacher can just pick it up and use it But in order to do that, we had the attract the power users Which means we also then had to make the tool powerful enough to be attractive to the power users So what we did was we went straight to those power users and we said here we have some tools We you know, try them out. Tell us what you think we want the feedback Please come yell at me because something doesn't work because I need to know so we can make it better For the teachers who are less savvy than you are And once we got those teachers going we got them working in their classroom You know, they're getting their kids excited You know, maybe they're actually seeing was better results in the test scores, which we don't have empirical evidence But definitely anecdotal evidence coming from teachers who say use the tools in one of the classes and another section didn't They actually saw a difference in the test scores. I actually saw an improvement in a blended environment where they were Hmm, I forgot to put this in my slides. This is important These tools were targeted specifically to classroom teachers to enhance what they're already doing in the classroom This is not a replacement for the classroom. This is not a go do your class online This is a you have a classroom We are going to hear we are here to help you enhance what you're already doing in the classroom to bring the internet into your class Not to allow teachers to serve 500 students per class And that was a key thing I think Because the teachers saw this as a tool to do better not a tool to do more and I think that's an important very important fact But we so we got these early adopter teachers we got them in there We got them working on stuff. They started seeing results, you know The kids would maybe drop to the next room over and say you should see what Mrs. Jones is doing over here, man It's great. It's so much fun. It's so exciting or maybe, you know, the teachers run the lunchroom and they're talking about stuff And they're like, wow, this is working really well for me. I'm seeing results with my students, you know And the other teachers are going, huh? What's going on over there? Can I get that and so what we started seeing was we get a teacher really energetic Energized and working with the tool in a school it would spread and Suddenly we'd have the whole school infected and then maybe the district and it started getting attention So when we were trying to get attention when we were talking about those key stakeholders back at the beginning the administrators and the IT departments Talked to them. I got crickets. Nothing didn't hear a word. There was no interest They don't have time to deal with that they didn't understand the benefit got to the teachers got the teachers starting using it They went yeah, right, let's go and then all sudden the teachers and the administrators in the AT departments are going Hey, what's going on over there? What's all this noise? What's this virtual school district stuff? Why am I seeing all this traffic going to an external server? Which as I'll get to is some of the things that didn't work Actually I'll get to it now. Oh wait. No, I won't I'll talk about the technical stuff first So because this is a state funded project and because we're at the whim of the legislature We wanted to make sure That if the legislature tomorrow zero funded the project we didn't completely leave the schools in the large now I know there's you know You can't just forklift something and drop it into school data center and expect it or a school district's data center and expect it to Work I totally understand that but we could at least do what we could to mitigate that and the first thing is Let's not tie them to a large proprietary software licensing bill You know, especially these districts with you know, 40 50 to 60,000 students on the system You know, they're already pressed for for money. That's one of the reasons we rebuilding the system Let's provide them tools without costing them anything So we wanted to make sure that whatever we built we could turn over to their school IT department and say here are the tools This is what we built here is the code You can do this we will help you do this and I said this multiple times the IT departments that I worked with If you want to bring this in-house great I will help you do it, you know, you don't have to wait, you know If you want to do this now you like what you see I will come down to you and sit down with your IT department and help you deploy it internally We love that because that as far as I'm concerned Better serves the goal of the of the project than just having them run on our servers because that's something locally managed locally owned The control is where it needs to be I'm very much in pushing that control as far out to that edge as I can We don't want to control and tell people what to do with the system We want the users to define how that system is getting used Using a cloud like model we were really running a private virtual cloud Worked well because it was a daily enabled us to spin up servers spin down servers reconfigure servers Do things very in a sort of very agile manner when we get feedback from schools or we see traffic increasing We were able to adjust to it quickly Config management I already mentioned We were running so many systems. There is no way we would have been no run that many systems without it It is a complete life saver as crafty as CF engineers. It's got its words and it's arcane as hell It got us past that initial growth spurt It got us to the point where we were big enough that we could start dedicating the resources to actually working more on the infrastructure So config management. It was definitely key here And the Jenkins bit the ability to automate things like server and course recreation Made everything work, you know both with Jenkins and as well I had some crafty PhD scripts that I wrote initially to do that to automatically create sites and automatically create courses We're key because we didn't want those teachers to have to wait Again, there's that not waiting thing teachers should be able to sign into the site say I you know look at the other library Of courses that are available say I want that physics course on my server with control They're able to do that without having to wait for any any human being whether it be me or somebody at their district To approve that they just do it Which again, I'll get to is one of things that didn't work So Some of the things that caused problems one We were a little too aggressive in talking to the teachers and not aggressive enough in talking to the IT staffs So what did we get we got a lot across norms? We still we still do I think It's fair. It's it's a completely valid and Reasonable problem that this that the district IT had with us because we were this unknown outside vendor Coming in from to the statewide level, you know saying here teachers have all this stuff And of course the AT department is doing one I don't understand this stuff because a lot of them are Microsoft shops, you know and It's it's a threat it really is you know, it's like gosh there's all this technology coming into my district I'm responsible for technology in my district, and I don't understand what my teachers are using this is a problem and So at some districts it was pretty severe, you know, we had grievances filed We had all sorts of things happen. It was a very unpleasant negative Interaction which I really regret and I wish you know if I could just go back five years and do it over again I'd try to find a way to do it differently Yes, the program succeeding. Yes, we're making a very strong impact in the classrooms but at the cost of Driving even a further wedge between the teachers and their IT staff locally Because the IT stuff saw saw this as oh my god, this is gonna make more work Even though I argue yeah, we have you know the help desk and things like that to assist I understand the complaint and it's totally valid Also, it was a bit of the non-advented here thing, you know, especially at the larger districts The IT departments at the larger districts, especially were like oh these these outsiders these you know these Wild Harry hippies coming in telling us how you they can do better stuff than we can and we've got a great system here We know exactly what we're doing. We have all these tools. We can do all of what you're giving Why are you coming in and doing this to our teachers? We can do this ourselves true but Clearly something's not Sinking here because the teachers were still coming to us wanting to use the tools rather than going to the local IT department So yeah, there was a definite disconnect there and there was a definite sort of a bit of butting heads that we didn't handle well The other problem we had was just plain enthusiasm and that seems a little odd but As I had this image at one point where one of the districts I worked with had been had rolled out Moodle They were had a bunch of their students. It was an alternative high school. They were really excited things were going great and then they got this grant to get a bunch of iPads and They started doing stuff and I had this image in my head of them running around with iPads banging down nails Because they did you know the iPad was suddenly and these was Three four years ago, you know the iPads were still very much a very new hot thing and people hadn't quite figured out how to use them intelligently in the classroom yet And so they were using them in wholly inappropriate ways and they got really excited about it and they rolled this out And it fell flat on its face Because they had this shiny hammer when what they really needed was, you know, maybe a wrench or a screwdriver and We didn't necessarily help handhold them on that And didn't do a particularly good job of helping them understand when what you know helping them figure out What the appropriate tool is and the appropriate application of the tools that they had So they got overly enthusiastic. They did some great stuff and some stuff that didn't work so well The last thing is when we originally built the portal I Wanted to build a strong social component that connected teachers across district or all over the state to allow them to share content You know, hey, I did this really great unit on you know fractions I want to share this with other elementary school teachers in the state so built this had this whole structure had this way of you know, allowing teachers to upload content and rate it and Crickets as another speaker. I heard earlier this week say tumbleweeds Not a single object was uploaded nothing And I you know, I talked to the teachers and there were some concerns there were things like well You know, if I create this stuff is that the visit my intellectual property is a district's intellectual property, you know I you know, I don't have and and also part of it is again teachers just don't have time They were looking for content. They didn't have the time to generate their own necessarily and contribute it back They like the idea As as one of my friends in public administration likes to stay and then volunteerism Collaboration they love the idea of collaboration. It's kind of like teen sex. Everybody's talking about it But nobody's actually doing it very well Same sort of thing, you know, they really worked hard at it They liked the idea, but it just never really came to anything which is a great disappointment to me But it's the reality of the world So things that we stumbled over technology wise As the site the system started to perforate we got more and more sites out there It become became much much much harder to keep things updated and patched We had so many different configurations so many different versions Primarily Drupal 6 stuff now getting into Drupal 7 We even had Drupal 5 stuff floating around out there I think up until last year. I actually they may still be running for all I know There were still Drupal 5 sites running in production because there was just too much there and the effort to upgrade those and keep them patched because they were all a little bit different Was a problem, which is one of the reasons we started moving to the Jenkins model and the more standardization of things As a real as a related thing way too many one-offs We had wordpress sites up there were the kind of floating out on their own We had you know wordpress multi you know as a wordpress MU or whatever the wordpress network stuff that he worked up at some point we had all these things and they were all configured different and You know we had no standardization because we tried to standardize the structure We had you know, maybe a server at the district level on a server per school Some districts wanted everybody on one server some districts wanted them per school And so things you know even as tries we might we tried to standardize couldn't do it And so it became this kind of mishmash of okay wait is Portland on one server is it on eight? Okay, yeah, and and so it became a very difficult thing to manage The other thing is and this is our problem. We are greatest technical failure was we had a nice busy storage appliance that Failed hard Exactly a year ago this week actually it was Memorial Day weekend I'm completely blew up on us Lost I think we managed to resurrect almost all of the data But we had a significant downtime in the system. It was a bad single point of failure Which is one of the reasons we went to Gloucester Because you know we had then multiple copies spread across multiple machines Even though the storage array was supposed to be redundant it turned out it wasn't we had you know Card go bad and it's speed a bunch of bad data onto the systems lost multiple volumes It was a very bad situation So the one thing we learned the most was yeah, I don't trust that damn appliance so We've got let's see About 15 20 minutes for questions and discussion and I can dive down into any of the technical details Or Ken can talk about some of the software level stuff I'm I wanted to make sure we had plenty of time for talking about this because I wasn't quite sure who was gonna be here today so I encourage anybody with a question come on up to the mic here and Let's chat Actually working in school district now a question about how do you guys handle the aging of everything in the system? You know schools run, you know, we kind of run for a year then we burn everything down and we start up for a new year I mean you guys looking at having high school students finding their kindergarten records still on the system or is there some sort of Method of removing the old to keep everything running We actually had explicit instructions from the Department of Ed not to remove old user accounts They wanted to keep those around and not entirely sure why But what we ended up doing with a lot of districts is there was certainly namespace issues You know especially when you're talking statewide I mean their namespace issues in any large district and when you're talking about statewide We really had naming convention issues and so siloing them off using like graduation year and things like that Tended to help but for the most part also Up until very recently. We hadn't done a lot with single sign-on so You know as a student would move into a new school their account wouldn't necessarily travel with them they'd get a new account at the new school and The schools a lot of the schools we hooked them into this the school active directory And so a lot of that just got aged out anyway because they removed the account from active directory It gets taken off the system. They're done But it hadn't been much of a problem, but that does also account for some of the bloat in the user numbers that we see Yeah, yeah, they're recording. So we'll need to get on the microphone. Thank you So you you mentioned your experiences with iPads as that situation gotten better. Are you Are you able to to integrate? You know storing stuff, you know that they do natively on the iPad and and get it in the moodle smoothly I know that the very recent versions of moodle have gotten better as far as content creation on iPad But yeah, it's it's an issue regardless I think a lot of the schools and the teachers are starting to figure out that iPads are great consumption devices They work really good for looking at things and maybe interacting with things But when they get into the state where they're needing to submit content Generally, they go to the laptop part of the computer lab Right and and get on a real keyboard because it just it was too frustrating for them. It does Does today's moodle interface well? With iPads it's gotten better. Yeah, it does some detection on mobile devices and actually provides the ability to do a mobile theme Which is decent. It's not truly responsive Which is certainly something that you know the Drupal world is done And I know that there's a lot of work especially in the Drupal community to start seeing about Replacing a lot of those functions in moodle and Drupal which I would kill for Because I'm working with moodle has reminded me just how much I adore how Drupal core is set up From a technical standpoint Yeah, dealing with moodle core is a nightmare so Anything you know we could do to try to push stuff in, you know, even using WordPress instead of moodle It's just getting away from the moodle interface because it's heavy bloated and difficult to work with It's still used the teachers like it. They have the mind share So, you know, we did what we could but there isn't a whole lot you can do and One of the questions so in Oregon is this is this open to other schools in terms of private schools or Good question. Yes. It is open to any Student or teacher who is connected with a public school district So private schools unfortunately are not included. They're excluded by law. It wasn't our choice. That's the way it is I would have opened it up to everybody if I had my choice But we were excluded by Private schools were excluded by law a lot of homeschools are included though because most of the homeschooling is done through the local district So a lot of homeschool students can use this use the system that said the system is primarily targeted at teachers So, you know, it's tools for teachers more than tools for students There was a goal of an initial diagrams we drew having a student portal. I had a bunch of content for students That never really materialized there. We just didn't have the demand and we didn't have the resources to build it But any teacher who's working with a student regardless of that students a homeschool student or an in-classroom student is That is employed by a public school entity is able to use it in Oregon Yeah, unfortunately again, it's yeah that said, you know, I What we're doing is all open source software So all the content that we're creating all the plugins that we're creating are open source We're happy to share our expertise in our systems We can't host your site if you're from another state, but we'd be happy to share those Those setups and a lot of stuff on github, which Yeah, ping one of us and we'll get you github access. Yeah, I Work for a county office of ed in California And I too have gotten into trouble by being a crusader rabbit to the teachers saying no I'll just go out and sign up for these things and just go do it Don't wait for your district to do this and I've gotten into some trouble for doing that But I'm kind of sitting back and then cackling now as I'm seeing this Wave of devices just wash over the IT department that I work for, you know, I'm saying all right Even our office tried to ban iPads and that lasted like two weeks You know before the iPads just started just washing in and now we're actually you know that all the all of our divisions Want to do these experimental iPad programs where they're buying a hundred at a time and you know, that's not an experiment folks You're deploying Yeah, yes, exactly, but but I know But is it our IT departments becoming more reasonable with teachers now are they because it's happening in corporations people are bringing in their own technology and Even buying decisions such as whether or not to buy Drupal is moving in corporations It seems more to the to the actual users and has less to do with the IT department Are you perceiving that happening to some extent? Yeah, I think there there are kind of a few things happening One is a truly unfortunate thing which is budgets are getting so tight. There are no IT departments in some districts I remember getting a call not long ago from a from a superintendent in a rural district Said hey, I need help with this. This isn't working and I said, oh gosh It looks like it's something to do with your single sign-on in your active directory Yeah, like get me in touch with your ID department. I'll help them work through it. I Am the IT department. They all got laid off last month What am I gonna do, you know, and we tried to help him as best we could but you know without access to the systems or what You know, so that's one thing is you know just out of sheer attrition The teachers don't have an IT department to work with any longer So they're they just you know, they're grasping at whatever resources they can get just to get their job done And so that is actually unfortunately part of the growth I think also there are districts that because there is so much pressure on them financially especially or They're just getting better educated. They're starting to learn. They're seeing stuff like this going on They're like, okay. Yeah, we can we can do this. We can get our toes in it We can you know and now that there's somebody else that's done it, you know I mean, this is the great public sector thing, right? No, you don't want to be first you always want to see I want to see somebody else has done it first So now that we can start pointing at districts that say look, you know these guys at Salem Kaiser They're doing it right now. They've completely offloaded their website, you know They're saving however many thousand dollars a year on this It's okay. It doesn't hurt and the other districts can go well if those guys are doing it we can do it That's no problem So we're starting to see that and there are a few districts that saw the light from the beginning You know, there are a couple of districts out there that you know Google apps came out that was what part of this that we did talk about in the presentation is actually Oregon was the first state in The country to sign a statewide Google apps contract Google apps for education So every single public school in Oregon has free access to Google apps for education the full suite You know first one to do a statewide agreement on that So, you know, they're they're doing these things. There's districts that just threw it wide open, you know, that I There's one in the Newburgh school district here in Oregon. I love those guys. They're amazing They just said you want to try it. You want to try open source stuff? Great. Let's make it happen. Let's do it And then they totally embrace the sort of disruptive attitude and it's worked great for them So, you know, there are a few out there that's done that but it's yeah, unfortunately attrition seems to be the worst But most common reason Do you guys feel like your system is pretty well stabilized or are there a major kind of user? Facing components you're thinking about at Lance is here. Great Yeah, I think the changes we've made in the you know in the past year have certainly made it a lot more stabilized We still have a ways to go, but we've definitely redesigned it in a way that it's a lot more robust Are there other open source projects or you know features that you get requests for or personally interested in? That you think would make a good addition to it to the or VST infrastructure specifically Not right now. I mean we've already tapped into quite a bit of things like Gloucester FS being the latest development and working that out I certainly helped quite a bit. I'm trying to think of anything else. Yeah I mean, I think Jenkins was another big thing. Yeah, and that's one thing I didn't actually talk very much about is the automated deployment on Jenkins and the fact that we're actually doing entirely automated upgrades of both Moodle and Drupal via Jenkins And even to the point of sharing get reposed with the school district IT for them to upload their own modifications Into and then Jenkins deploys it to staging when they do a commit to the staging branch They get to try it on on staging If it's if it's working, okay, then we merge staging into prod and Jenkins immediately deploys it to prod Which has saved a huge amount of work Now that's not a user-facing then that they're necessarily going to notice But it does allow the local district and us What the OSL teams keep thinking in us To do stuff more quickly, but I think between you know the e portfolios on Mahara, which never really caught on Moodle And Drupal and WordPress I mean those tools seem to be the key ones the teachers just want a website that can put stuff on and have the teachers And the students go look at that's what I really cared about Does your setup communicate with any student information systems? That's number of districts did not want us to do that That was there was very much firewall issues involves a lot of times and they just that and that was where some of the Resistance with the IT staff came in is like no way in hell. Am I letting somebody outside do something? The closest we've come is looking into say, you know their LDAP or active directory But pushing data into and out of the student information systems while we were willing to do it The other half generally wasn't Which is kind of I think a blessing in disguise because that Is a hairy nasty ugly problem and I'm kind of glad we didn't have to address it yet because that would have been a lot of work So does each district have their own information system? Okay. Yes, and it's the dog's breakfast out there everything I mean there are districts that were still doing things with Excel for their SIS We have some really small districts So yeah, thank you Just wondering if there was any ability to track or any metrics for student success especially across years or Is it it's everything expunged? Yeah, it's not necessarily expunged But for the most part and it's kind of ties in with the student information system thing is that teachers while there are grade Book functions and little and things like that teachers were really using their local SIS for all of that. So yeah, we didn't we didn't have the metrics. I wanted to gather them We just didn't have any way to We're starting to be able to gather more metrics on all the sites through the the RVSD central program And that's kind of only limited by what kind of data we can gather from Moodle and Drupal So Moodle tracks a certain amount of you know student success depending on you know, how they've got it set up So in the future, we might be able to do something a lot more detailed and comprehensive you know rates of How many people got A's and certain courses how many people completed which material that kind of stuff so yeah What level of hardware virtualization were you able to achieve do you for instance? Do you have a network fabric? so that Network connections are all virtualized Do you have a lot of ants lands answer that one I could answer but he'll do it better. I Mean we use KVM as the hypervisor and all of our virtualization. I think One of the sites kind of showed the new architecture Basically the database servers are hardware the file servers are hardware and then we have the physical nodes I mean, there's a couple of older machines that are still physical hardware, but those are going away So I would say on average probably 80 to 90 percent of infrastructure is virtualized But networking-wise it's just your file servers are in fact nodes They're their own they're the the cluster FS nodes So we I don't know if you just grabbed how we split up the file systems on some of the stuff But you say that you got rid of your your one file server appliance I assume that this is doing some kind of raid in software. Oh, yeah, all of them has raided. Yeah Yeah, we still have an older NFS server that's by itself But we've had been replicating stuff off of it for like a cold backup But the new system with the Gluster FS kind of does it more automatically And so we've had to split some things up because Gluster FS does not work really well if you have PHP files on it So we have to that's why we have Jenkins to kind of have all the PHP files On the local file system and then like moodle data and triple files It's it's the type of The way Gluster FS works and the type of system calls that it does It's just incredibly slow. It doesn't do deal while well with small files and PHP Used any kind of SSD caching to try to speed things up Well, we've done as much caching as we can do basically, but this is the solution we've had and Basically all of the advice we've gotten from Gluster FS is the same thing, you know, don't host the PHP files on Gluster FS Yeah, I use a lot of memcache heavily used varnish on the front end for caching of static static assets Basically, I mean, you know, just like any other large-scale Drupal or PHP site in general is just you know Do whatever you can to reduce the number of hits on the PHP back end so we don't have to actually ever touch disk So lots of stuff's cached in memcache and RAM and what we can is cached on the front end of varnish Yeah, cool. Thanks. Oh That's the guanetti logo Yeah, that's a fairly new logo. Yeah, we put the slides together and I saw this logo up here And I'm like what the heck is that, you know, I remember the guanetti for like six years and I don't know what it is So yeah, that's that's a new one. You you had some numbers up there about the number of sites that you have 500 or 750 websites are those a combination of Drupal and yeah Yeah, it's preponderance is a moodle It's probably 90 percent Moodle sites mostly because The in the original iteration of the site the Drupal portal When a teacher signs in and they go browse in the content I had I created a Moodle module that Ken hates me for don't he don't let him tell you otherwise He hates me for it Awful crafty PHP thing that basically would when the teacher signs up they say what school district and what school they're with When they go to say, oh, I like this piece of content I want to create a course based on it They hit create a course the system would then check the back end see if a server exists a Virtual v host for that particular school if it doesn't it would spawn one it would load Moodle into it It would then load the course and do it add the user and then it would give them the link back to the course But what that meant was every curious teacher that goes oh, that's kind of fun. Click that button new server So, you know, it was good because then they had it and we know we never have to create it again And it was fine But there are you know probably a couple of hundred if not more servers sitting there that have one user on them That is never locked in so the Moodle. I'm not familiar with the learning system. Okay, but the Drupal part That's for everything else, right? Yes course wears that software and everything else Although we do have teachers doing things in Moodle and in WordPress both because more often than not Moodle is way overkill for what the teachers actually wanted to do they just wanted a place to like maybe have a discussion and Have students, you know download assignments or you know, put the syllabus up there or something like that So it depends on the teacher's needs and that was one of the kind of the weird things is we had this Huge variation on what the teachers were using it for some teachers did the entire class on there They use the quiz they use the you know the assignment module for their assignment submissions They had tons of stuff on there in Moodle. Some of them are like yeah, I post my lecture notes up there So, yeah, it varied wildly so my question about Drupal is is Since you probably have a lot of those sites as well How do you I guess you have to have a standard list of modules that you yeah So that people don't go off the rails here and create crazy websites that you have to maintain So you you set Exactly what they get and so that you can scale up to a lot of websites the initial set. Yes, however They they do have the ability of uploading Site specific modules and using if they want to yeah, we give them plenty of opportunities to shoot themselves on the foot You know partly because we just didn't have the staff to you know manage at that level and to we wanted to give them the control You know and you know with you know I did the you know with great power comes great responsibility speech to them You know it's like okay, you know I'm gonna give you this access what I want you know if you do the wrong thing you're gonna blow up your site And we'll try to help you but You know don't be stupid right? I guess my final thought is a question rather is So how are you is this a multi-site environment is a single sites That's really what I was getting at the answer to that is yes Unfortunately, yeah, and Ken and Lance can talk more because it's changed significantly Yeah, I want to say originally it was multi-site, and I think the new architecture is not Simply because the multi-site problem was well if I upgrade this one source Repository that everything has to get upgraded all at once and we didn't want to work we did want to work around that So that's where Jenkins kind of tied in we're kind of did that more automatically at least that's how I recall it being set up Yeah, well what we discovered really was there's kind of a sweet spot for where multi-site really made sense in the number of sites You know and so for a very small number of sites You know single sites is probably fine because you're only dealing with you know four or five web roots You get into the 8 or 10 or 12 site maybe 15 sites Multi-site But then as that number grows it begin then it becomes more unwieldy again because Do you really want to pull the trigger and up late upgrade 150 sites all at once? You know something is going to blow up and so it's much nicer to be able to batch those And I think I don't see the Oregon State folks here right now OSU on Tuesday did a talk on yeah Same problem exactly the same issue. Yeah, and so we ended up doing something. We did it in a slightly different way But it's exactly the same problem. Yeah So yeah, I would go if you have large numbers of sites I would strongly recommend using some sort of continuous integration to elect Jenkins and deploying single site Unless you have a compelling reason why you would want to do multi-site. Oh, we are. I'm sorry. Yes. We are out of time. Okay, I Was just wondering I'm not a teacher, but where do curious teachers go? Is there a central website that I missed the URL for? No, and I'm silly and forgot to put it in there but it's or vsd.org and The site has this was this originally been online five years ago. It has not changed since so I apologize But it works. So, you know, yeah or a vsd.org Thanks, everybody. Sorry for going long