 My name is Michael Spall. How many people are self-hosting right now? So fair number. How many people are hosted with a Moodle partner? A few people. And then how many people are using just kind of a simple web host? No one. And then how many people are already cloud gurus that just came here to heckle me and see the mistakes. OK, I figured there are a few people like that. My background is in math, science, and education. And so I'm not necessarily the typical IT guy. I'm a problem solver type guy. I've been at ISU since the day after Martin was the King Day, 2005. And I've been a Moodle administrator since 2006. And then I've been a member of the Moodle Gradebook working group for just over a year. So I'll give you a little background on me. So why is ISU investigating the cloud? One is cost. Is this cloud going to be a better cost option? And specifically having to do with hardware churn, especially our SAN. In fact, if it wasn't for the fact that we had to keep buying new SAN hardware, it seems constantly, we would probably be very happy just staying with self-hosting. It could also offload some of our server maintenance. So I maintain and our co-worker maintains everything except the actual physical hardware and the network connections. But in any institution, there's a limit. And so when they have big projects, maybe we're not first in line for some things. And then virtualization. ISU does not have tons of experience in virtualization. And so let someone else handle virtualization. This is the reference architecture that we were looking at. And that's the URL about the blog architecture. And so we happen to be investigating Amazon web services. But we've looked at a few other ones as well, not as in depth. And I think a lot of what I would say applies to any of them. So this is the general architecture. Load balancer, multiple web servers, and auto scaling, and the DB, and S3 storage, and that kind of stuff. So what I have some worries about using the cloud, and that's part of why we're investigating and hoping to get answers to some of these questions, we're self-hosted. We can control costs because we know how much I'm going to make this year in salary. And we know how much we have to pay for our internet because ISU already has internet. We don't pay anything for our internet. Storage costs. Again, you buy a sand, you've already paid for it. You know how much it's going to cost. And then bandwidth issues. And then I have real worries about performance, and S3 performance specifically. And then there's a general concern from our networking people about the campus internet border being a bottleneck. And that right now, we're self-hosted, we're on campus. 40% of our usage is on campus. And a lot of that is instructors doing things that are maybe some of the even more bandwidth things. When they do a backup and restore, they often do it on campus. When they call in and say, oh, my backup and restore is not working, we often ask, are you off campus? And it turns out it's their internet service provider that's causing the problem. So if we go to the cloud, that 40% that was completely inside essentially our infinite bandwidth is going to be outside now. One of the benefits is having that reference architecture was the multiple web servers. So you get redundancy and reliability. But there is a little bit of a performance hit that I'll cover. Amazon provides a load balancer. So that's handy, because even talking to someone yesterday from another school, they had a tough time getting their load balancer working right. Though you do have to take care of Moodle sessions, if you have multiple web servers, you cannot have your Moodle session data in the file structure on each server, because as soon as you get bounced, then you're logged out. Unless somehow you set up sticky sessions. You also need a Moodle data solution that can be shared. S3 was what was mentioned in that reference architecture. And you need a solution for the Moodle universal cache. Again, you can't use files on a particular server. There has to be something that's shared between everyone. So again, coming back to here, the web servers are these yellow guys. And you have the S3 buckets. This reference architecture has two S3 buckets. One is for the more static stuff, and one is for the uploaded files and everything. So the problem with using S3 is S3 is not a file system. And even when I'm working with the Amazon engineer and the sales person, I make this comment and talking about our testing and our setup. And the sales person is kind of like, of course, it's not a file system. But I'm like, yeah, but I don't think you get it. Moodle expects a file system. And one of the keys, and I just copied the whole thing here, is that the Amazon S3 buckets in most regions provide, well, I guess all regions provide read after write consistency for puts of new objects. So that means when you create something new, an eventual consistency for overwrites. So you really can't use S3 if you're editing files and expect any kind of speed and anything like a file. New files, yes. And then the US standard region only provides read after write consistency. So you need to use something that makes it seem like a file system. And there's a few different ones. We're using this one. I have a link to their GitHub page, the S3FS fuse. And almost all of them are fuse-based. So you have to install fuse on your web server and do some configuration. But it's all to make it seem like it's a file system when it's not really a file system. So this leads to a worry, because that's going to be a performance hit. And then there's also just with S3 a 64 gigabit file size limit. So there are alternatives to using S3. It's in preview right now, but there's the Amazon Elastic File System. There's Gluster File System. It's a distributed file system. It's one of a few. But it's used by Moodle Cloud for non-static files. So we know this one can work. They're being successful so far with it. And so I think we maybe need to check in with the person or the people at Moodle.org that have set this up and get some advice on whether it would meet our needs. And the other thing is use repositories that link to files and don't copy files. And there's tons of resources out there on the internet that you can use. You can have your videos don't have to be loaded in Moodle. They can be in YouTube and linked to. But that's an education thing. And tons of faculty, for whatever reason, don't want their learning objects shared, even though you can clamp down with. They just prefer the easy way of just loading things up. And then thinking also that file size limit of 64 gigabytes, that seems huge in some ways. But if you're backing up a course that has uploaded a ton of videos, I could see a course backup. And I know course backups that have done this that it can be mind blowing, but it exceeded the 64 gigabyte file limit. So I'm not sold on S3 as being the right solution for files, especially for all files in Moodle. So, this reference architecture, I kind of found it and also the Amazon engineer found it in this article. And looking at it and it seemed really great and I'm like all positive. And the more I look at it, the more I realize I'm not convinced that the person that wrote this up and drew this little diagram actually ever installed Moodle in the cloud. Because where is the Moodle universal cache? I don't see it here. They're using DynamoDB to deal with sessions and I think that's probably a great way to do it. But that's not the Moodle universal cache. So that's one of the things that we're still figuring out how to make work really well and to have good performance. So, if you use the file cache, which is the standard, you need fast disks and it cannot be on S3. Moodle universal cache will not work on S3. I know it won't, I've proved it. You can use Memcache, DA or MongoDB. Those are the two things besides the file that Moodle already has. And if you know enough about doing this thing, this stuff you could use other stuff. So that's what you kind of need to use if you're using multiple web servers. Memcache D purges everything. So one thing you need to do is have different stores for all your key value pairs because some caches get purged a lot. And Jeff sitting over there from Minnesota, I was talking to him a lot and he has a panel that he's gonna discuss some of this with at 230 today. And then there's also Martin Langhoff's presentation where he talks about the Moodle universal cache. And I'm definitely gonna be there because I'm not the Moodle universal cache guru yet. And I need to learn more to figure out how to make it more efficient and see if I'm doing everything correctly with that. So auto scaling in Amazon and I'm gonna guess most of the cloud computing, that's fairly easy. You make a snapshot of your web server and you make sure it's hooked up to any shared storage and all that stuff. And in Amazon it's using cloud watch to monitor your system but other ones would have something else. And you can just spin up new web servers on demand. So if you're getting overloaded just keep adding new web servers so you don't have enough load. Now of course the devil's in the details there and everything but the one thing is on demand costs are more than dedicated instances. So you'd have to see if having a bunch of small web servers in the middle of the night and then scaling that up at your peak load and scaling down was less expensive than just having more dedicated and so that you don't have to add on demand. There's cost calculators and knowing your traffic flow and analyzing your current traffic will be very important for figuring out the best way to go there. So again the orange stuff in the middle here and the session store and the invisible little universal cache. Here's our load balancer here. Now we're gonna talk about the database. And so using Amazon, their RDS service you can choose to have a synchronous standby a replicant in a different zone that will do automatic failover. So that's kind of nice. I mean that's kind of automatic and you can spend a lot of time trying to set that stuff up on your own. So again, that's offloading a lot of work right there. In our testing we're using the MySQL community 5.6. You have a lot of different choices you could choose but we're just using the most recent. But people have been taught and so I've been on a vacation and I went to the mountain moon and then another vacation. And so in that month Aurora's come out and I already feel like I'm a year or two years behind an internet time. But so we need to investigate it. They're making all kinds of claims about it that it's five times the performance and that it clusters real well. And it's compatible with MySQL 5.6. So that would be handy for us but I have not investigated it. But I heard it mentioned a lot in the Moodle Mood already. So people are very excited about that. Performance testing. If you're not already doing it, I recommend that you get a good performance testing scheme that works for you. There's a blog post by Tim Hunt that really goes into good detail. I use the core. In core there's make a JMeter test plan. I would make that. We use Bad Boy which is open free software if you're non-commercial. To make more JMeter tests manually and to run those. And then I also like to have real people during some of the testing when we're pushing upload, do things, like take a quiz, upload a file, do backup and restore to get a feel like is the system real usable? Because you can generate a lot of data and JMeter is very easy to break a system because it's very easy to just put way more users than your system could ever handle. But making a realistic plan is, there's a little bit of an art to it. There's science for sure, but there's a little bit of an art. And so if you're thinking about moving here and wondering whether, how much it'll handle, but it'll also show you the costs too because you run these tests and then you're gonna get a bill from Amazon. And can you survive that bill from Amazon? And Amazon has a pretty good cost calculator. It takes a little while to wrap your head around it and everything. So, and then monitoring. I think monitoring is very important. Amazon has CloudWatch and it can do a lot. We use Moonen at ISU for our current monitoring and everything. And Cacti is like a little bit like Moonen on steroids. So if it was me, I might install Cacti to monitor. We also just use Google Analytics for an easy interface for people to see. And then Pwik is something that Gavin turned me on to at the Mountain Moot. Woohoo! And it's kind of like the open source version of Google Analytics. And there's even a plugin that works well with Google Analytics or Pwik because of the way Moodle URLs don't really mean anything that can help out with that. So some of the decisions that ISU needs to make and this is the way I'm leaning until ISU gets better knowledge of the Moodle universal cache. We use one web server right now and the advantage to going to the cloud would be to virtualize everything. But until we can get really good performance and make sure the Muck is really working well, I'd almost say just use one web server in the cloud and rely on that if that zone has problems, you have a clone and bring down the one and spin up the other in a different availability zone and set stuff up to there. We have to really decide what file system to use for Moodle data because that one that's in preview from Amazon I think is about 10 times the cost of S3 storage. And 10 is a lot as a multiplier. But I'm still not sold on S3. I just can see it giving us heartache. And I'm actually wondering if we might even continue to host in-house and try to run Gluster FS and make our own SAN somehow. So, and then the question is what is acceptable performance? We have certain characteristics now. It's not gonna be exactly the same. I think we could get very good database. I think we could get web server. What's acceptable file server performance? What's acceptable uptime? Maybe uptime is even higher on Amazon because they handle database updates and everything. So, I'm definitely on time today. So, we are open for questions. Hey, have you thought about how you might tie this into your authentication and enrollment systems? Yes, we have in a document kind of like a detailed testing plan and everything. So, at ISU right now, we have one web server, one database server, and we tie into LDAP for authentication. And then we use Banner as our SIS in an aluminum portal. And so, we have that single sign-on from Ludamanus. And then the other thing is we also use Eric Merrill. I don't know if he's here, but we use his LMB to bring student information. So, those are the big three that we tie into and then also we're tied into Gmail. And I didn't talk about it, but Amazon web services has a simple email server. And so, I think we'll use that to pass email onto Gmail. And then, in our tests, we have a test SIS that we're testing against, test LDAP, and a test Luminus portal to, we have everything, a test everything, except we don't have a test Gmail. Well, our tests are being run with all the systems that we're integrated with and all the Moodle, we don't have that many Moodle plugins but with all our Moodle plugins. So, our testing environment in the cloud right now is, except for integration with Gmail is 100% like live. So, you have an LDAP server in the cloud? No, no, we're testing against using our, I see where you're going with your question. That is an interesting idea and I don't know how our security people would feel about having that LDAP server in the cloud, but we're running our tests with going across our boundary, our LDAP local and going across. So, you have a tunnel or something? Yeah, yeah. So, actually, not so much a tunnel, but yeah, we've opened a thing through our firewall and we use the encrypted LDAP, you know, LDAP-S. And so, yeah, so those kind of security concerns. So, on your monitoring slide, you talked about some really interesting tools. Have you thought about using anything like Uptime Robot or Pingdom to monitor external accessibility? From our help desk, our help desk has access to one of the external providers, web providers locally, and they do have something that does pings from off to simulate off-site traffic. And I think what we would be more concerned about then, because if it would be all off, we would be probably just setting that up to run from on campus to constantly check to make sure we could reach our Amazon from, because if it was reachable from our campus, it's probably reachable anywhere. But yeah, that is a good point. Hi, Mike. So, one comment and a question. So, the Elastic File system, I think, is gonna be a really potential great thing for the, you know, not using S3. The cost on it is a lot more, it's 30 cents per gigabyte per month, but that's a terabyte for 3,600 a year. And if you think about sand costs, that's not too much. Right. But anyway, one of the concerns I have is what my network costs are gonna be. And have you done any exploration of like how many gigabytes in and out your servers do right now and how that maps to the network costs in Amazon? Yeah, so right now we've been running with the Moon in and everything. We have 24 seven monitoring of our current in and out on the Moodle web interface. And so we know what our current load is and we've been testing. And when you plug in the numbers for the internet, when you plug in the numbers for the type of server, I think we need to ensure success. And then the internet, you're getting up to about, and we're getting to be about costs of about 2,000 a month, somewhere between 1,500 and 2,400 a month for various designs that I've come up with when you include the internet, when you include the hardware cost and the networking cost to serve us. So yeah, it's not cheap. And that's the other thing is like, so it's not cheap to buy a $12,000 server, but you know you've bought it and you've got it. What if our monthly costs start to escalate? What do we do then? That's a very, very big concern. And I think the more we could share stories about what to expect when moving to the cloud, the better. And Amazon's not the only one, but it's the one that we investigated the deepest. So how many students do you need before you start considering cloud type services? When does it start to make sense? Oh, I don't know, I haven't thought of it in that way. I can tell you that ISU is a school of 14,000 FTE. We have 90,000 users loaded in Moodle. They're not all active, but the way the LMB communicates when they make a potential student, those all come across. We have about 15,000 active users in a semester. And we get upwards of periods of time in the two, our peaks are between 200 and 400 active users in doing stuff in Moodle at the same time. And we definitely have in a week thousands of you, at least somewhere between seven and 10,000 users in a week. And have you looked at other configurations where maybe you're putting certain Moodle tables onto one database server, like Gradebook goes into? No, because our current hardware is Adele R710 and we have 80 gigs of memory. And even though now our Moodle database is bigger than 80 gigs, most of it is loaded in memory. And we have very fast disks. So we are not having to jump through a lot of hoops to make our database run well. In fact, we're very happy with performance. The only thing that could happen with us moving to the cloud is things get worse. It couldn't get better for us right now. It has to do with costs and sand and then whether we're getting enough support from the university for continuing on. So yeah, we're not in the situation where the database server is not a bottleneck for us. Related to your concurrency testing, I'm just curious if you have any conclusions about for the architecture you've been trying out, what kind of concurrency you're able to get and still deliver good performance? So again, the highest concurrency I've done is with the one web server, one database server. And if you get the big enough machine, we're able to run hundreds of concurrent users. So it seems like from that perspective, the performance won't suffer. It's more a question in the real world situation, the file storage issues. So I think, yeah, the web server can handle everything just fine if you get a big enough machine. And so we're not looking at small web servers, we're looking at one, again, trying to have like on the database machine having between 80 and 120 gigabytes for memory and fairly good memory on the web server. So that's tuned a little better. And then we're picking the Amazon image and then running my SQL 5.6, but then the latest PHP from the Amazon image.