 started. Welcome. I'm Cliff Lynch. I'm the director of the Coalition for Networked Information and I will be introducing the session today. The session you've joined us for is one of the project briefings in week four of the fall 2020 virtual member meeting of CNI. Week four, just to remind you, is focused on emerging issues and responses to the various crises that we're all collectively trying to navigate and help others to navigate. A few logistic things. We are recording the session and the recording will be available after the session. Cloth captioning is available. Please use it if it's helpful for you. We have got a chat box running. Feel free to use that and there is also a Q&A tool at the bottom of your screen. You can use that to pose questions to the presenter at any time. We'll deal with all the questions in a Q&A session, which Diane Goldenberg of CNI will moderate at the end of the presentation. With that, let me move on to introduce our speaker, Fred Gilmore of the University of Texas at Austin Library. The topic today is one that I believe many of our institutions are facing in various settings. We have specialized computer lab spaces around our campuses in our libraries and elsewhere. Often these computers are specially configured. They come with, in some cases, expensive specialized software that is really important either for instructional purposes in certain classes or for students and even faculty who need to do certain kinds of tasks. All of a sudden, we had to close down and people couldn't come and physically use those computer labs. This created quite a set of challenges and Fred is going to walk us through what they learned in trying to deal with them. I will also be very interested to hear any speculations he might share about how much of these adjustments is permanent and how much of it is temporary as we emerge from the pandemic. Thank you for joining us today, Fred, and over to you. It's terrific. Thank you for the introduction. I want to thank the organizers for giving me this opportunity and thank the rest of you for making time in your schedule. Like most academic libraries or many academic libraries, UT spent a good part of the last half of the decade transforming the areas especially in our buildings for more collaborative work, new and different kinds of communication and interaction. One of the constants in those spaces, many of those spaces, remained the networked devices, primarily desktops that the students and the faculty and public continued to use. For the longest time, I think it was the device itself that was sort of the commodity we were offering people, but anymore as the key piece of curriculum support, it's the licensed software that's on those devices. We really started to focus on what we needed to do to get that software into the hands of those people who needed it and certainly this year has put that service to the test. In broad outline, the conditions under which we were operating are similar, if not identical to, again, most libraries over the last nine to ten months. We've got devices in 12 different locations across campus. 11 of those were closed in the spring and remain closed. The devices, roughly half of the total that you see on the slide, actually reside in the one location that was reopened in the summer of the Perry-Caston Native Library. But even there, a decision was made to remove or disable roughly two-thirds of the devices that would have been available just to promote social distancing. The rough calculations would be, we had about 40 Macs and about 100 PCs to begin with, but then pulling those down even further. We were dealing with a handful of devices that were going to be offering the software we had for in-person use. So we had known for some time that a significant portion of the use on these devices had become sort of secondary in nature. The users were in our collaborative spaces for group meetings or they were in the building for another reason. And sometimes the contact with our devices was more a convenience or a drive-by kind of thing. They were using software that they could obtain in their homes in other spaces off campus. Browser, the licensed Microsoft Office Suite that we had on the computers, but we'll touch upon how we intended to act on that information in a slide or two. But there's also, even for the tremendous amount of traffic that those pieces of software were getting, the graph still shows that there was a persistent use of the licensed software that we had, video editing software, titles like Matlab or Wolfram, as well as the Adobe Suite also was getting a good amount of use. One of the areas that is not on the slide that had just started to gain traction here with them last year were mapping in GIS tools that we were offering in conjunction with the rollout of GIS services, which we were relatively new to. So it was kind of critical to find a way to maintain access to these pieces of software, even though people couldn't necessarily meet us in the building. I mean, it's true that some people probably did go ahead and obtain personal licensed copies of what they needed, particularly early in the spring, as we were trying to close out the semester. But at that, how were we going to resume the curriculum support that this kind of software was providing to people? And again, in broad outline, we were operating under the same kind of conditions with regard to employment that many of the rest of you are. We were almost immediately under a hiring freeze, additionally a salary freeze, which probably contributed to us losing several key members of the systems administration unit. Purchase requests were subject to additional scrutiny or justification. So finding a way to do this inexpensively and quickly as well as securely was going to be a challenge. Speed was obviously important as well. We weren't going to be in a position to do what I would consider the normal vetting that we do for a project like this. So realistically, given the time we had, I was hoping to at least get partial check marks in each of these boxes. I wasn't really thinking we were going to manage to hit every one of these sort of requirements for what we wanted to roll out, but at least if we could get halfway there on all of them, that would get us moving. So given that broad set of requirements, it actually left a fairly narrow series of options, again driven primarily by the lack of quality time that I probably would have liked for both me and the squad to have to work through these things. The options as we saw them split roughly into two buckets. One of those was looking at virtual desktops via some sort of third party like Azure or AWS. We also could have attempted to do something like this in the campus cloud. There was also a possibility that we could just deal with an on-prem RDP gateway or remote gateway. So that's actually what we worked through are both sets of options. So circling back to the bar chart that I showed a couple of minutes ago, we were well aware that the use the devices we're getting in the building didn't necessarily justify a full replacement of the fleet the next time the replacement cycle came up. We needed some devices on hand to deal with the specialized software, but for this reason we started to look at what would it be for us to just the next time renewal came up. We go with a series of thin clients and we start to deliver the software in a more remote fashion. So we started to look via proof of concept in the fall of 2019 at AppStream and various other options that we could get from again either Amazon or Microsoft. There was a lot of appeal there in terms of capital expenditure savings. We could focus again on the thing that we felt people were really coming to us for anymore, which was not the device but the software. It could help us in other ways with staffing as well. But on the other hand, once we got the proof of concept going, it became clear that there was going to be a fairly radical shift in the interface users would encounter. And there was a learning curve that would be attached to that that we were going to have to take some time to plan for. The licensing was going to be a bit trickier trying to make a quick pivot to something like that, especially if we were considering this to deal with the situation that we found ourselves in in spring. And actually we found a fair amount of latency. A lot of these data centers were far enough away that I wasn't immediately impressed with how responsive the system was going to be. There were going to be ways to work around that. But again, there weren't going to be ways to work around it with the amount of time that we had to address the current situation. On top of all that, the virtual desktops really didn't address our Macintosh situation. And a significant portion of the licensed software that we have actually resides on the Macs more than the Windows side. And so we were going to still have to deal with that situation independent of trying to make a move like this. So in the end, we decided that it was just a little too early. You know, we had done a lot of the homework to make a situation like that work, but the timing just didn't work line up really well. What we did have on hand clearly was two thirds of a managed fleet that was going to be offline by necessity to promote social distancing. And we also had a staff with approximately 50 years of systems administration experience. And between those two pieces, it seemed like there should be a way to make a remote gateway that we operated ourselves work. Most of the machines that had been disabled had not been powered down. We had either pulled them from the floor for security reasons and powered them back up in an office area. Or we had removed monitors, mice, peripherals, and kept the machine if we could keep it cabled in place. So those machines were still available to us to remote into. So obviously the other thing that would have made it easier to start with is to go with a piece of software we're already familiar with. And we did already license key server for metrics reasons, collecting use data on how the workstations were used, how the software was used, and Sassafras, which is Sassafras all site, which is what the software is now being referred to as a formerly key server, came out with an Apache guacamole wrapper that would have worked for us. And so we started trialing that. It also allowed software searching by title. It allowed device mapping, other features, which would have appealed, especially if someone was looking for a lab with less traffic at that time, things of that nature. The other product that we looked at was called lab stats, which actually was the first one to come to market this spring. It tended to be more Windows or RDP centric. So it wasn't also necessarily going to be a total solution for us. We liked the interface. We also did not really have time given the very sort of compressed runway for this to devote a whole lot of UI or UX developer time into coming up with something. So eventually a presentation like this comes to the point where the local conditions sort of totally influence the decision you have to go with. And this slide pretty much represents that. On the plus side, we were able to sit here and consider options for the labs that we managed because the labs are not centrally administered on campus. We did that ourselves. On the other hand, it was made clear to us by the security office that the remote gateways that would be allowed on campus for this type of work were going to be centrally administered. And we were going to have to sort of vet or seek approval for whatever we came up with with the security office. The gateway would have to have two factor authentication attached to it. And for all of these reasons, it sort of started to eliminate many of our options much more quickly. If we were going to be placed in a position where if our request to manage a gateway was successful, we were told it was going to be quite possible that other units on campus would be steered towards us for that service, regardless of which option we picked, whether it had been lab stats or guacamole or key server. And that pretty much meant client-based options. In other words, options where you were required to install a piece of software on the endpoint that you intended to make available remotely, like both Sassafras and lab stats, require a client install, that pretty much eliminated us being able to accommodate other units on campus since we didn't manage their devices. So this pretty much meant we were looking much more closely at Apache guacamole. And there's plenty of reasons that it was an attractive option. It was open source. It was a very established code base, which we couldn't necessarily say about some of the other options we considered. There was absolutely no client involved, which not only meant we could accommodate other units on campus, it also meant we weren't imposing any kind of demands on the user from wherever they were to get to these desktops. It was also very function-focused. The guacamole developers tend to establish very bright lines around what they consider their responsibility. And they focus pretty intently on making sure that the gateway itself works. And many of the other pieces are possible, but they're essentially the responsibility of the user to deal with via an API. Cut and paste options are relatively clear for how a user would interact with the system and the output options, while not optimal, you know, you don't have access to local printers, you end up having to deal with PDF as your output and finding a way to print it wherever you are. It still meant that those functions were available to the user in some form. So given the concessions that we were seeing in going this route, we tried to just get comfortable with the notion that rolling this out didn't necessarily mean it was going to be a solution for a permanent solution or a solution for all time, but it was certainly going to help us hit the mark that we had set for ourselves, which was getting people back in touch with the license software that we had. So this is what, this is a screenshot basically of the menu that is presented to the user. The Windows users basically get auto provisioned a machine by the guacamole server. The Macintosh users need to select a machine. This is a function of the fact that RDP sessions clear fairly quickly and the VNC sessions that we have, we've experimented with various timeouts on the Macintoshes, but the sessions don't clear quite as quickly and so it was going to be more necessary for the user to be able to see which machines were available and to pick and open one themselves. The easy parts of this were the guacamole server itself has a fairly small footprint and the scaling is very clear and coherent. You basically need two gigabytes of RAM for every 25 concurrent users per core. So it was going to be relatively easy for us to both meet that initial demand and to expand it quickly if we needed to. It was also the case that it was easily put into a container. Most of this, all of it operates through a single port which meant it was possible for us to orchestrate this in Kubernetes without much difficulty. The SQL backend that it uses for maintaining sessions also allowed us to push metrics data out to Splunk so that we could keep track of what was going on in the system and address bottlenecks. And unlike the latency I was encountering when we looked at things like virtual desktops, if we were going to be the data center, we could meet what is still largely a regional demand, meaning most of our users, even though they weren't in the building, they were somewhere in Texas, they were somewhere in Kansas, Nebraska, they weren't really that far flung. So on the downside, the less easy pieces of this implementation have been the fact that the Guacamole server really hasn't, the service itself doesn't spend a whole lot of time on the user experience. It maximizes real estate to basically present as large and complete a display of the endpoint as it can and that comes at the cost of knowing what to do when you want the session to end. The way we're trying to deal with that right now is basically we watermark the desktops that we're using in Guacamole with the key combination that's necessary to pull up the menu for cutting and pasting or ending the session and things like that. SAML integration was introduced over the summer by the developers. It hasn't been terribly automatic. We attempted various manual supplying metadata manually to get a connection to the IDP. In the end, what we ended up using is simple SAML at PHP in order to meet the two factor requirements that our campus has given us. The MAC endpoints, RDP tends to be faster because it can cache. The Macintosh endpoints require VNC. We're looking into ways to speed up the user experience for MAC users, but right now it's it can be a bit slow at times. And the thing about being the data center for ourselves sort of cuts both ways. I mean, eventually the fleet that we're using right now is going to age out. And so then the question is do we do we persist with this or not? The enhancements that I see us attempting to circle back and offer hopefully for the spring semester would be better branding and better navigation going back to the compromises we were making with trying to get people to know how to pull up the navigation menu when they need it. We'll experiment with real VNC for the Macintosh in hopes of boosting the performance there. To this point, the Macs also do not allow remote audio, which we will look into. I would like it if we're going to pull in other units on campus to use the squacamole gateway to be able to delegate permissions to them. And that's going to require some developer time. Same thing with getting native SAML to work as opposed to the stopgap measure that we've put in place. And ultimately, as Clifford referred to at the beginning of the presentation, this isn't going to be a sort of a binary pivot out of this. And so what does the interregnum look like for this? When we're halfway between inviting people back on campus, when that happens, we will have to start pulling down workstations that we've put into guacamole to serve that need. And so at what point do we decide to revisit the notion of virtual desktops? I would say right now we will, I'm sure we will persist with guacamole. I think we're going to take the time to make these other enhancements and see where it takes us. I don't, I still don't think it's a solution for all time, but I certainly see it carrying us through till we can make a better go at what either AWS or Azure would offer us. That is what I have on this topic today. I want to thank you for your time and I'm interested in what questions you might have. Thank you, Fred. Really interesting overview of your process there and the trade-offs given the requirements and limitations. Thanks for bringing that C&I. It's really interesting to hear about it. The floor is now open for questions. Please share your questions in the Q&A box and we do have a question that has come in from Michael Seidel who wants to know why the university felt that a two-factor authentication was necessary and he asked how great was the risk of outside people breaking in? And he notes that in Berlin, where Michael is, this is not currently required. Well, I kind of get to bail out on that question. On the basis of campus policy, it would be, guacamole would have been the outlier on our campus had it not been two-factor required. Basically, everything else, easy proxy, every other service we offer at this stage, has now been run behind two-factor. So that's easy. That's a bit of a cop out, but I mean, it's, yeah. Okay. All right. Thank you. Thanks, Michael, for asking that question. We have another question now. If you were not trying to offer remote access to Max, would you have gone with virtual desktops? It's possible. It's actually more possible that I probably would have gone with one of the cliented local gateways, really. Just because, again, there was still the latency issues which we just didn't, I think there's ways to work around that. I know that, you know, Azure offers us the possibility of data centers just up the road in San Antonio, which probably would take care of some of that. I didn't, I wanted a fairly clean interface, too, which I know I could have, we could have gotten out of either lab stats or key server. So realistically, no, I don't think we would have made the immediate hop. I think there's also a return on investment piece to this. I mean, we were at a good point with the fleet. We weren't at the fourth year. So there was no need to sort of immediately dash off and maybe do a less than optimal run at the virtual desktops. Right. Okay. Well, that was an interesting question. Thanks for asking that. And thanks for your reply, Fred. I do see that we are a little bit past time here. So I'm going to just invite any attendees who still have time to hang out with us. Please do so. I'm going to turn off the recording. And anyone who has more questions, Fred has agreed to stick around and chat a little bit with us. Please just raise your hand, let me know, and I will turn on your microphone. But I will say thank you to everyone who came today and thank you to Fred. And we hope to see you back at CNI soon. Bye-bye.