 I think it's about time to get started. Let me welcome everybody to this project briefing which is part of CNI's virtual spring 2020 membership meeting. We'll have a presentation today with two speakers, Mary Beth Snap and Jennifer Vinapal, both from Ohio State. After we hear from them, we will take questions and answers. We'll be using the Q&A tool which is at the bottom of your screen and feel free to queue up questions as they occur to you at any point during the presentation and we'll come and go through all of those at the end of the presentation. Diane Goldenberg Hart from CNI will moderate the Q&A at the end. Our topic today is DevOps and I was very pleased to see this one. We've had a few sessions over the over the years covering aspects of DevOps and DevOps really as the title here suggests is indeed bigger than IT. It's a very different way of thinking about IT related activities and so I'm very much looking forward to this talk. With that, let me thank both of our presenters for being here and I'll turn it over to Mary Beth. Welcome. Thank you, Cliff. Thank you all for being with us today. I know it's a difficult time in the world and so we appreciate you taking some time out of your day to listen to our presentation and hope you will enjoy it. Today Jennifer and I will focus on two topics primarily. One, what is DevOps in particular? What does the DevOps culture look like? And secondly, we'll take a case study of digital preservation and the Ohio State libraries to illustrate what the organization can learn in particular from the DevOps movement. So what is DevOps, the mysterious and elusive DevOps? There's quite a bit of misunderstanding around it and if you Google DevOps, you might have difficulty finding a useful concrete or even concise definition. So I wanted to take a second to look at these common associations with DevOps, which I'm sure you're familiar with. You could be deploying code multiple times per week or even a day and for sure that's a homework of DevOps, frequent and fast delivery. You might have someone on your team with a title of DevOps engineer. I did a search recently on Indeed. I found quite a few posted in Columbus with that title, DevOps engineer. Your systems could all be in the cloud. You could be using a DevOps tool chain such as CI, CD, Puppet, Ansible, Kubernetes, Docker, many, many technologies out there these days. And your developers could be practicing an agile framework like Scrum or Kanban. So the question is, are you DevOps? I think perhaps, but not necessarily because practices, so these practices are all extremely valuable, don't automatically lead to a DevOps culture, which is what I believe is especially important. I wanted to drill down a bit on agile software development, which I'm sure you're all familiar with. Here's a typical diagram of Scrum, one of agile's frameworks, actually the most popular agile framework. Our team practices a sort of unique modified version of Scrum plus Kanban in two weeks sprints. I don't want to spend a lot of time on Scrum. There are plenty of resources out there, but we'd rather like you to notice on the screen the output of this iterative cycle that's at the bottom right in the circle. And that's a potentially shippable product increment. In Scrum, that increment should meet your definition of done or the DOD. And the definition of done could include things like there are unit tests, there is sufficient test coverage, the code integrate successfully, the product owner has approved for release, could include many different things. But the choice of language is interesting. The idea of a shippable product and done, definition of done, it suggests and what I've experienced as a manager of a development team for many years, and also I've actually encouraged, unfortunately, is that developers tend to consider their jobs to be finished when their code lands in production. So our code is deployed, let's celebrate, let's go home. And so the attitude there can be problematic and it can be a source of angst for some infrastructure and operations teams. It's a major reason, in fact, why DevOps came about. And it's because Agile simply didn't go far enough to include all members of the delivery team. There's a phrase in DevOps, the wall of confusion between those who write the code and those who support the systems the code lives on. And if you think about it, this wall would be expected if you think about how the different functions are incentivized and rewarded. So developers are rewarded for their creativity, innovation, taking risks, and visible output. Whereas infrastructure and operations people are rewarded for keeping the systems up, stable and secure, and their work is not visible. If it is visible, then there probably is a problem. So the natural tendency, of course, would be to avoid any kind of risk. And what we can end up with is tension and even competing goals between developers and sysadmins. If you add the wall of confusion to this diagram of scrum, it might look like this, with the handoff between the product to the developers, to the sysadmins and operations staff. And the operations people may think there's some type of mystery happening on the other side of the wall. And vice versa, the developers are not quite sure what's happening with operations. So what if we eliminated that handoff? Well, that's exactly what DevOps intends to do. Intends to break down the wall of confusion by treating software delivery as a continuous iterative pipeline through collaboration and feedback loops. And if you're familiar with lean, this sounds very much like lean thinking, and it definitely is. DevOps has been heavily influenced by lean, which I'll talk about in a second here. So in our particular organization, our two systems administrators moved to my team to join developers. So now we have the functions and expertise for end-to-end delivery of our digital library software in one team of nine people. This includes UX expertise and application security. So we're extending agile to include everyone, and we're intentionally putting the user's experience back into the equation. We're now two years into our DevOps transformation, and I've learned the hard way that it is a journey, and it doesn't happen overnight by simply reorganizing teams. And this will come as no surprise to anyone. It's because changing culture is the hard part. When it comes to culture, I've found the comms framework to be especially useful, and comms is an acronym for culture, meaning cross-functional collaboration, automation, reducing the work that doesn't add value to the user. Lean, of course, which I mentioned earlier, continuous improvement, visualizing flow of work, and eliminating waste like handoffs. Measurement, data-driven feedback loops, and sharing. Sharing to decrease blame and increase trust. To expand a bit on DevOps culture, it looks something like this. In a DevOps team, we have shared objectives. We all know why we are doing what we're doing, and we're all committed to that purpose. We use technology and processes for what they're best at, which is automating and streamlining, so that we can spend our time on creating value, whatever is meaningful to our patrons and our users. We're all in this together, and it feels like we're all in this together. We respect each other's expertise and contributions toward these shared goals, and we measure to learn and use introspection, retrospection, and data to improve. We appreciate that our own actions affect those around us, that we are interdependent in achieving our objectives. This sounds like a tall order, so you may be asking yourself, yeah, I would really like to go DevOps with our teams. Where do I even begin? The three ways described as essentially the basis of the principles of DevOps, and they're described in the book, The Phoenix Project, which I'd highly recommend if you haven't read. The three ways are flow, feedback, and learning. I think starting with the first of the three ways is important, and that's visualizing the flow of work throughout the entire organization, not just in IT. This quote is an excellent quote from Gene Kim, one of the say grandfathers of DevOps, but definitely a thought leader in the field. The first way is never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system. I'm going to turn this over to Jennifer now, and she'll talk more about. Thanks, Beth. All right, DevOps in the organization. We'll go back to the delivery pipeline. We have, as you have just heard, a development team that's structured around and following DevOps practices, and also developing a DevOps culture. As Beth said, there are organizational growth opportunities within her team, and all teams migrating to DevOps practices that they're working through. But applications development, so Beth's team is called applications development, is embedded in a larger organization, and that is our university libraries. So I'm going to talk now about the rest of the library. So Beth, yeah, thank you. So how do the applications development teams DevOps culture, the comms framework function within a larger organization that, while the organization as a whole appreciates many of the same values, doesn't really know about and hasn't adopted the DevOps principles and methodology? And how can these same DevOps principles and practices help us to partner better as IT with others across the organization, and also advance organizational learning? So as a quick case study, I'm going to talk about how we're rethinking our digital preservation workflow. So without going into all the details about our preservation environment, I'm just going to share here that we do not have a common definition across our organization of exactly what digital preservation is, what our expected outcomes are or should be from digital preservation, when we should or should not do it, and how we may all contribute to doing digital preservation effectively. There are lots of assumptions across the organization that are built into and structure our work. But if we don't know what those assumptions are, we can't know if they're actually the right assumptions on which to be building a digital preservation program. So for the rest of this presentation, I'm going to highlight a couple concepts and tools in DevOps methods that we can use and we are using to make our organizational work in this initiative effective. I'm specifically going to focus on three concepts that are going to start to move us toward higher levels of communication and trust across our teams, and also that will reveal assumptions that are frustrating our work. And those happen to align perfectly with the three ways. So I'm going to talk about the value stream, feedback and feedforward loops, and then organizational learning. So Beth's team is leading the organization in an exercise to visualize the flow of digital preservation work throughout the entire organization. A key concept in Lean and in DevOps is the value stream, which is the sequence of activities an organization undertakes to design, produce, and deliver a good or service to a customer. This language of customer goods and services comes from the manufacturing sector, but here you can see how it maps well enough onto our work in higher ed. Beth's team, along with the digital preservation librarian, are working with all organizational partners to map our current digital preservation workflow from start to finish. That is called a value stream mapping. It's going to allow us to know all the inputs and outputs of our current system. In order to design a value stream that gets you the results you want, you first need to know who is the customer or customers you're serving, and what are the goods or services that you should be offering. And it turns out that those questions are actually harder to answer than I had originally thought when I came to Ohio State three and a half years ago. Everyone contributing to the present value stream seems to have slightly different ideas about what the systems we've developed should actually be doing and delivering and for whom. So this, from my perspective, this is a people challenge, not a technology challenge. In addition to understanding our current value value stream, we also need to identify who we're building our systems for and why. And when I say systems, I don't just mean technology systems. I mean the systems that include everything we do and the people who do it. That requires a different kind of investigation. So if you're curious about how we're doing it, I'll just say that I think we might have lost Jennifer Cliff. Yep, I think we may have lost Jennifer. Let's see. What do you think, Beth? Do you think you can talk through some of Jennifer's slides while she reboots? Yeah, let's give her a second though. Okay. All right. It looks like she's going to be rejoining us here. Sorry for that, folks. While we wait for Jennifer to rejoin us, just want to remind you that this webinar is part of CNI's Spring 2020 virtual membership meeting, sharing their direct link to the schedule of our meeting. So you can take a look at what we have yet to come between now and the end of May when we will be having plenty of offerings. Diane, I just noticed here we have a question that came up during Mary Beth's part of the presentation and I wonder what while we're waiting for Jennifer to come back, whether we might talk about that one for a moment. I'd be happy to. Yeah, the question is about whether you have user interface, user experience designers as part of your team of nine and do you also develop and manage the public website? That's a good question. So we've been fortunate to be able to hire a graphic designer front-end web developer who has deep UX user experience. And we also have the Discovery Services librarian who also has a human factors background. So what we've been doing rather than because obviously our resources are limited, our teams are small, rather than hiring a full-time person, what we're doing is collaborating around, combining our various levels of experience and expertise to form a user experience interest group. So it's more of a grassroots. Unfortunately, you know, it'd be great if we had a dedicated user experience staff person, but that's not the case. The second question was what Cliff? It was about whether the public facing website was also within the scope of. Yes. Yes. So all digital products that are developed in the library, for example, our Fedora high rack stack for digital collections and our website, essentially anything that would require an in-house developer is managed as a product among my team members. Thank you. I see Jennifer is back and perhaps we just want to pick up where we left off. Yeah. Thank you. And sorry about that. I don't know what happened. Next slide, Beth. Okay. Next slide. So while we're mapping our current workflow, mapping our current workflow is important, but in and of itself, it's not going to get us to ideal state. Rather, we need to increase the amount of information flowing through and about the system. We need to know more about the system and our relationship to it so we can understand what is and isn't working. We need to know what assumptions are underlying our systems so we can see if they're really the right drivers for the development and ongoing improvement of our workflow. So feedback and feed forward loops allow us to iteratively test, design and operating assumptions. In the DevOps handbook, the authors state that the more assumptions we can invalidate the faster we can find and fix problems, increasing our resilience agility and ability to learn and innovate. I'll also add that this is going to reduce human frustration with the system. The idea of feedback loops comes from Peter Senge's work. And Senge explains in his book, The Fifth Discipline, that feedback is learning to recognize types of structures that recur again and again. And he says that everyone shares responsibility for the problems generated by a system. So our ultimate goal is to have shared understandings and shared investment and to build trust among people and also in the systems that we use. Unless the system itself is built on eliciting this feedback, we're not going to get there. Next slide. Oh, there you are. This kind of systems thinking about the entirety of the system sees the collection of people and tools and workflows and assumptions and processes and documentation and all of that as all interconnected and all continually influencing and reinforcing and also frustrating each other. So as Senge says, our own actions create the problems that we experience. So we have to recognize that we're part of the system in order to recognize that we are part of the problems that we're experiencing. If we are doing DevOps right within IT and then also bringing these practices out into the organization, we're going to be creating a learning organization that is adept at creating new knowledge and integrating it into improved work practices. And I'm actually going to be speaking more about this on May 26th at a CNI presentation if you want to hear more about the learning organization. So we have to learn from our mistakes. We must not blame but instead investigate problems and learn from them. We do midstream reviews and retrospectives. We elicit and test assumptions. We understand that complex systems are complex and so are people who are also part of the systems and all the parts of the system influence each other. And we need to get to a point where we can interrogate the assumptions underlying our work without people feeling attacked and getting defensive. If we do this right with DevOps principles, some good tools and practices and goodwill to learn from our experiences together, we won't just be able to develop a better digital preservation workflow. We will also have developed more trust in our work, in our systems, and also in our mistakes and our own learning. So in many ways, the technical work, from my perspective, is the easy part. And I believe that it's Beth's perspective as well, that the people work is actually the more challenging and the more rewarding work, assuming that we can get it right. Are there any questions? Oh, that was great. You recovered magnificently, Jennifer. And thanks to both of you for that great talk and really interesting speaking of learning from mistakes. We do have a question for you that's come in from Jamie Wittenberg. And Jamie asks, how did you get buy-in from the organization, especially middle managers, during your transition to DevOps? Great question. Beth, you want to take that or you want me to take that? Yeah, I can take that one. So when I first started with libraries, which is about seven years ago, I believe, I came in from an agile development shop. So one of my first goals was to move our development operation to an agile way of thinking. And that's, thankfully, to the support of my managers. That's been an easy sell with the organization. And I've appreciated that we've had a chance to experiment. So I think the organizational culture has been such that the idea of experimenting with new ideas is acceptable. And so it hasn't been as difficult as you might think to try out new ideas. We have very strong partnerships with our colleagues on our projects. And so people are generally and most usually very accepting and very patient with the way we're doing things. And when we try something new, would you agree with that, Jennifer? Yes. At the same time, we've been on a kind of the middle managers, especially of which the executive team is part of a middle managers group, a management committee. We've been on a learning curve and have been learning from each other. And we also all got sort of base level lean training. So we're used to some vocabulary and some practices, some basic practices around efficiency, the idea of efficiency for not for saving money, but for actually doing your work more efficiently. So I think that does create a culture where this kind of experimentation kind of slots in and makes sense. I would like to add to that. Thanks, Jennifer. Because we've been practicing Scrum, which is actually pretty rigorous with repeatable and expected events, and we've included our partners in the Sprint events, people have gotten used to over time our way of working because they know what to expect. So I think introducing DevOps has not been the kind of challenge in this particular organization that I might have experienced earlier in my career. Interesting. Go ahead, Jennifer. Did you have something to add? No, but I see the next two questions, and I think they're in some ways related to each other. So I could talk about both of them, Monica's and Emily's questions together. Sure. Shall I, let me just read them so folks can hear them. Monica McCormick asks, could you give examples of some of the people work, people work that was required, interested in hearing more about how you expressed frustration and perhaps also hidden assumptions. And Emily Gore asks, can you give some examples of some of the people work that you have had to do? It definitely feels like this would be the biggest challenge, and some examples tactics you've used to bring people along would be great. Yes, please. Yeah, and Monica does clarify that this is not just about Beth's team, but in the whole organization. So I recently, just coincidental to everything else, I recently have taken on responsibility for Special Collections and Area Studies. So I was hired to be the Associate Director now and Associate Dean for IT and have now brought Special Collections and Area Studies into my responsibilities. So I'm now able to kind of bridge as one person the IT work and much of the digital preservation work, although there's another unit that's also involved in a great deal of the flow. And also the, to talk to the staff on Special, the Special Collections and Area Studies side, especially Special Collections, about their expectations and frustrations with our current digital preservation workflow. So I was in a kind of a unique position to wearing both hats, to be able to talk to them, and also based on my kind of decades working in digital preservation, to listen carefully to what they were saying and to start to understand that what they were looking for was kind of solutions and technologies that didn't necessarily have anything to do with digital preservation. And yet they were kind of mapping those expectations onto the one large system and solution that we have, which is a digital preservation workflow and a digital preservation repository. So because I've spent a good deal of time just talking to them about their needs and building trust with them, I was able to have a very open conversation about what they felt like they needed and help them to understand that not all, although this is the system that we've been focusing on for any number of years now, that maybe some of their expectations are somewhat misplaced, and that let's tease out what they, some of their needs away from this digital preservation system at the same time that I was kind of educating them to understand that digital preservation is hard, it's a heavy lift, and that maybe some of their needs don't need to be met, they don't need to place their frustration on the system because the system isn't, is not the right system for what they need for some of those needs. So I hope that helped to, the work is talking work, it's a lot of listening. So you know, any of the tools that are available for listening are going to be useful only in as much as we are already building trust with the people that we're talking to and listening carefully to what they're saying and helping them to be really clear about what it is that they need, independent of any existing systems. All right, thank you, Jennifer. Definitely a challenging problem, and we have a comment actually that speaks pretty much directly to that, this comes from Charles Blair, and he says, this is not so much a question, but an observation. I like the principle, but I'm afraid it might be unworkable in our environment. We too have nine staff, UI, UX, graphic designer, programmer slash analysts, and two system administrators. We support almost 100 systems, including the library catalog. In addition, all campus IT units are mandated by the university's board of trustees to remediate IT risk. We make all our systems available to central IT scanning tools, and we get scored. The score is discussed with the library director, so our fundamental pressure is to mitigate risk. With minimal staffing and a hiring freeze into the foreseeable future, I'm afraid that for us this is an ideal which we cannot achieve given our current operating, i.e., organizational environment. It's not a question of culture, but outside pressure. I'd like to, if I might, because that sounds like a description of our environment. That's what I was going to say. Yeah, I think this is sort of where we are in the world right now in higher ed, and it will be for quite some time. I didn't, in my career, have a lot of experience working in infrastructure and particular information security before our two CIS admins joined the team. Our two CIS admins have also been doing the kind of monitoring and compliance and risk mitigation work that you're referring to. It is a lot of work. So I think, I would argue that, in fact, in this particular situation, agile is, in agile and DevOps, you really can't afford not to consider approaching those, because those frameworks are going to give you a way to have a conversation with your administration in a way that might be different than you're used to. So there's a lot of, I could talk all day about this. So if you want to talk to me offline, I would, especially agile frameworks starting there, will help you to think about more about value. What is the value that you're delivering to the organization? So less about the plan, but what is the value that you're delivering? And that's really been helpful for me in the situation where there's simply too much work to do and not enough staff, is start to have conversations about prioritizing based on value. Hopefully that's helpful. Jennifer, did you want to add to that? Oh, that was perfect. Okay. Yeah, that was nice. We also have another question, and Charles, thanks you for that comment, Beth. We have another question coming in from Nicholas Taylor, who first off, thanks you for your presentation. I appreciate your framing of DevOps as requiring an organization or team level change of approach. Do you think that DevOps is ideally also located in dedicated roles, I'm sorry, e.g. DevOps engineer that are distinct from sys admins and app developers? Or is it really more about onboarding all roles to doing DevOps? Yes. That's a very good question. So based on my own experience, which admittedly is limited, I feel the most value of the source of value with DevOps is the respect of everyone's relative expertise and also the willingness to help out and step outside of your own expertise. So really the cross functional nature of the work and the sense that it is about culture, I think really everybody in the team, if you want to be affected, needs to be on board with this idea that we're all in this together. And when you start to look at specializations like the DevOps engineer, or the database administrator, or I only do windows kind of thing, then you're really, I think, defeating the goal of DevOps, which is more the idea that we're all in this together means that sometimes we have to step out of our own specializations and work towards the same objective. So I'm not against the idea of having a dedicated DevOps engineer. I would be concerned about that because you want your whole team's culture to adapt to the DevOps way of doing things and not just have a person there who's responsible for DevOps and expect the entire team to operate around that person, if that makes sense. Thank you, Beth. That was a great answer and a really good question too. So thank you, Nicholas, for asking that question. At this time, I think we're kind of past time. We had a bunch of really wonderful questions. So thank you so much to our panelists and our attendees. Thank you so much for coming to CNI to talk about DevOps and thank you to our attendees for coming. I'm going to turn off the recording now, but Beth and Jennifer and I will stick around a little bit. And if you would like to come up to the podium and ask some questions or make some comments, just raise your virtual hand. I'll turn on your mic and we would love to hear what you have to say about this. So with that, I will thank our panelists one last time and thank you all. Be well. Bye-bye.