 Okay, let's get started. Hello, everyone. Thank you very much for joining us today. Welcome to today's CNCF webinar, Effective Kubernetes Onboarding. I'm Jerry Fallon and I'll be moderating today's webinar. We'd like to welcome our presenter today, Jayman Camiso, developer editor at Digital Ocean. Just a few housekeeping items before we get started. During the webinar, you are not able to talk as an attendee. There's a Q&A box at the bottom of your screen. Please feel free to drop your questions in there and we'll get to as many as we can at the end. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that would be in violation of the Code of Conduct. Please be respectful of your fellow participants and presenters. Please note that the recording insides will also be posted later today to the CNCF webinar page at cncf.io slash webinars. And with that, I'll hand it over to Jayman for today's presentation. Hey, great. Well, welcome everybody. Thank you, Jerry, for that introduction. I'm really pleased today to be here to talk about effective Kubernetes onboarding. Not the most exciting topic at first glance, but hopefully by the time we get through some of today's material, you'll be more interested and it'll give you a new perspective on something that maybe we've all gone through if we've used Kubernetes. So today's talk will be based on my Kubernetes experience as somebody who's been involved in an operations role, as well as somebody who writes tutorials on the Digital Ocean community site, as well as part of the team that was part of migrating Digital Ocean community site onto Kubernetes. A little bit about me, Jerry covered this, but I'm a developer educator at Digital Ocean. I teach course in system administration at York University in Toronto. In the past I've worked as a system administrator SRE in places like the Canadian federal government, economical supporting Ubuntu. But I'm really happy to be part of the community team now at Digital Ocean. One other note before I get started, these slides were done by another Digital Ocean community team member, Kathleen Jewell, so big shout out to her for doing this. She and I and the rest of the team worked really closely on everything that we're going to cover today from the content tutorials curriculum, continuous integration pipelines, Kubernetes services, deployments and everything that goes into the community website. So to get started, this web page, what I'm involved in and what the team is involved in is tutorials, curriculums, ebooks, cheat sheets, quick starts, meetup kits, all sorts of content that you may have encountered on the community platform. And the idea behind everything here is that it's designed to spread awareness knowledge of open source technologies to anyone in the world who wants to come and visit it. We've been doing translations of tutorials as well to try to increase, you know, the awareness of open source tools and some of the procedural tutorials and how to is that you can actually use to accomplish your goals. One last thing before I get going, I'd like to draw your attention to our community writing program called Write for Donations. So if anybody out there wants to share some of the things you've been working on with the rest of the community, please feel free to get in touch with me at the end, reach out, and I'll get you in touch with the right people. So to get started, why am I here today? I'm here with you some different perspectives from a content development perspective and a development perspective on getting up to speed with Kubernetes. So from the content side actually writing tutorials. Our team has written a bunch of tutorials on website. We've also worked with community contributors to work with them and edit their tutorials and design for various audiences. Also been involved in, like I said, migrating the Digital Ocean Community website from standalone virtual machines on to Kubernetes. So I'm in an interesting position where I have both of those perspectives. So the idea here is I can I can share this with you and, you know, maybe based on my or our experience and some of our tutorials. The next time you need to get yourself or someone else up to speed, you need to advocate for Kubernetes in your organization. You can use this experience as a way to get started. So, first off, how did we even get familiar with Kubernetes at Digital Ocean. We started with our Digital Ocean managed Kubernetes product that was made generally available last May. But the community team itself was was given a specific task, which was this teach the world Kubernetes. That was in effect what we were tasked with. That's a pretty tall order but you know we said challenge accepted. So, let's do it let's figure out how to teach people Kubernetes. So we started off planning, you know, what are we going to do in terms of content, we have existing content on the community site already. We thought about framing things in terms of the curriculum. So, the result I'll talk about the curriculum a little bit more later, but in in designing that in collecting existing material and planning new material. The team sort of identified the need that there needs to be a way to onboard people to Kubernetes you can't just give them documentation and say go learn Kubernetes, there needs to be a way to grant people's knowledge up. And it needs to be specific to their, their goals, depending on what their role is in an organization. So that was the general task from the community side on the content team. So really where do you start, you know we've identified the need but how do you, how do you break this down. Kubernetes, I think, if you're familiar with it is fairly complex, the ecosystems complex it's becoming more and more complex every day. Running applications on Kubernetes your applications themselves are probably fairly complex the organizations are complex people are everything that's involved in Kubernetes is complex. So where to start, how do you begin thinking about Kubernetes. So one option, maybe use the illustrated children's guide to Kubernetes spread from the CNCF, you know, Fippy tells a great story about going from a single PHP application running in a big shared web hosting environment to being safe and happy on on the Kubernetes cluster. And it does the guide actually covers containers pods labels replicas services volumes there's quite a bit of foundational knowledge in a storybook form. So, you know, maybe, maybe that's enough to teach anyone to convince them. Now I'm joking here obviously but my first experience with Kubernetes this probably would have actually helped me even if I had just produced the children's guide. To go and rotate secrets in a production cluster for the default image registry that the Kubernetes cluster was using and, as you can well imagine, that didn't end well. So I would definitely have benefited from just reviewing Fippy learning about namespaces and knowing which secret to go and edit. So really back to this topic here though effective Kubernetes onboarding I think Fippy is great. Fippy is a really fun way to think about Kubernetes but maybe not the most effective way to onboard people. So the content team in in building this curriculum and trying to get people going needed to have some kinds of operating principles, otherwise we just be creating random tutorials about things that were interesting to us or about pieces of the documentation we've been reading or our experiences. So we came up with some principles to start teaching world Kubernetes. So the first principle, and this goes for pretty much anything if you're teaching writing speaking any kind of communication is just know your audience and their objectives. So, asking the questions. Who is this for what what is this curriculum for what do they want to do what do they want from this curriculum what are they trying to achieve. So that's the first thing and that's not specific to Kubernetes that's just a good way to frame thinking about transmitting knowledge to people. Second thing is determining the best approach to meet their objectives their goals so what what knowledge do you think that the user is going to need to achieve their goals. What are the things they have to know to be successful. The third thing is scope your approach appropriately. So your audience is going to determine your approach and the scope. For example, a CIO chief information officer, maybe isn't going to care so much about Linux control groups or namespaces, but if you're talking about hardware costs and scaling over a short medium and long term that may get their attention. Your operations team may not even know what your application is on the kubernetes cluster, but I can guarantee they'll definitely know when it's down. So you need to tailor the onboarding approach that you're going to use for those different roles. In the case of a CIO maybe use a white paper for your business audience or in the case of an operations team have an SLA have documentation on instrumentation and monitoring so that you know when they're troubleshooting at three in the morning they know what to go and look at. And then the last point I put there. Excuse me. I hope I can get to this at the end is to plan for day two after you have onboarded to kubernetes. I'll try to get to this at the end. So this is what the content team is thinking about this is the way we've started thinking about you know framing. Who we're writing these tutorials for and we're using the existing tutorials to do they target. How we can slot all this into a curriculum. Meanwhile, the community platform itself. As the community content team is developing content, the developers themselves are working on the Rails app that runs a community. And a set of questions and issues around migration to kubernetes was starting to come up as we're you know we've launched a kubernetes as a service product. But the community platform is still running on virtual machines. So this is a picture of Katie's car. She said it's a 1997 Toyota Corolla. And the idea here is this is like the community website before it was migrated. It's absolutely great. It gets you from A to B. It's trustee, probably mostly reliable, you know, do some preventative maintenance and that thing probably go to 300,000 miles. But you know, sometimes newer the kubernetes way here. It's safer. It's faster. It's more reliable. It's more comfortable. There's more automation. The list goes on and on and on. So as we're talking about this content push, we're also talking about moving our staging production continuous integration pipelines, everything to do with the community website on to our internal kubernetes clusters. Before we've been running on virtual machines and outdated hardware. I think I remember somebody saying when the migration was happening that the database was running on a bunch of precise or maybe trustee. In any case, when we decommissioned it, I know it had at least 1000 days of uptime. So, you know, it had been there trustee and reliable like I said, getting pretty old but you know we're still serving millions of users a month on this platform. So the principles. The principles also come from the experience of moving to kubernetes, not just writing content but the actual team involved in migrating the application onto kubernetes. So, like I said, rails is the community applications of rails application. And it's a great case study. So it's a group of web developers, they're used to deploying in a traditional virtualized environment. And for them, moving to kubernetes meant things like what wears Capistrano or Ansible or puppet or all of the shell scripts that I've used. It meant involving pretty much the entire team. So there was conversations around changes to the application changes to the environment. How do we make changes? What are the architectural consequences? What are the risks, you know, the different failure modes and tradeoffs to using a distributed kubernetes architecture as opposed to the old, you know, multiple virtual machines and a shared database. So the idea here is, again, with all of these points, knowing the audience, determining how to meet their objectives, scope and approach and planning for day two is to combine insights from both of those, the content side and the development side. I helped contribute to the kubernetes curriculum, like I said, and did a bit of a review and helped out with the development team that were migrating to the internal kubernetes clusters. So first thing, audience, let's talk about that. Who are you dealing with? Let's say we're dealing with application developers. You know, these are the people who are just day to day involved in working on your application. They probably have some goals. They probably want to do things like learn how to develop with a distributed architecture in mind. They also have their day to day. They have to deploy new features, things like that. But when you're onboarding them, they need to know how they're developing environment and their day to day, their processes are going to change in a distributed environment. Otherwise, you'll get the invariable result of, well, it worked on local host. So we probably all experienced that. Another example of developing with a distributed architecture in mind would be cron jobs. So a cron task is a task that runs at, you know, a set interval on a system. And they work a little bit differently in a distributed kubernetes environment than they do on a virtual machine. On a virtual machine, you can go and you can edit the cron tab for a user or for a system process. In a distributed environment, well, you need to think about, okay, do I need to have a separate application image or can I use the same image as production to run this task? Where do logs from crongo in a distributed environment? In a virtual machine or a hardware machine, it's really easy to go and check the system log to see what the output from a task was. But in a stateless and distributed environment, you might have to go look through cabana or a gray log or something like that. So if a server is working on the features that rely on a cron task, a task running at some specified interval, they need to know about that architectural change. They need to be involved in that decision-making. Another goal that they might have, deploy successfully on kubernetes. If they're not doing that deployment, they need to be able to at least figure out, how do I get to the point with my code where I can make a deployment happen? Or somebody else can go and press the button for me to make a deployment happen to the cluster. So again, for the community application itself, deploying to kubernetes would mean completely different processes from what used with virtual machines. They're no more Capistrano. There's no more SSHing into a jump box or anything like that. Instead, we came up with a continuous integration, continuous delivery pipeline with automated deploys that are triggered by developers and using Git hooks or a make file to be able to push to different environments, instead of having a dedicated operations person like we'd had before. So we had to bring everybody up to speed on how they could actually do a deploy, whereas before they just had to make sure their code was able to be merged into the main bridge. Another group that we had to bring up to speed after getting developers on board with things. Operations people. So maybe, you know, you are actually onboarding a separate operations person or a team, or maybe people in your development team will be functioning as DevOps. They'll just wear the operations hat on occasion. So maybe you've got a pager rotation, let's say, so everybody gets a turn being on call so you can share responsibility. This is different than when there's just one person who was used to deploying virtual machines. So goals for operators. Likewise with developers want to make sure develop want to make sure that deployments are successful. And to do that they still need to know the processes procedures and tools that are being used for deployments, you know, they may not be involved in actually pressing the button. They may not be just involved in running the cluster itself, making sure your applications healthy on the cluster, or they could actually be the ones who are triggering the CI pipeline, making sure that your images get pushed to a private repository and they're the ones who are doing the deployment. Regardless of, you know, what level they're working at in the stack they need to be able to deploy successfully, including rolling back. So part of deploying is being able to roll back. Whether it's using cube CTL or continuous integration pipeline to trigger a redeploy or a read rollback. Another goal that they might have similar to deploying is to automate deployments. So, whereas before again we had in the traditional virtual machine environment, Capistrano, you know, you go you run your scripts do a deploy operations people may be in charge of setting up continuous integration pipelines for developers so developers never have to go and access that they don't have to know what a concourse job or an artifact or anything like that is they just have to know how to trigger CI whereas operations people need to be able to automate that deployment. Likewise, you'll probably be tasked with scheduling testing backups and other operations tasks. So those are goals for operators. So the last one I forgot about yeah implementing logging monitoring and alerting as well wouldn't be an effective operations person role team group whatever it is without visibility into applications running on Kubernetes so the information for this group when you're designing content onboarding them is going to be different than than application developers. The ops people probably don't really need to know about you know your rails initializers or gem files or things like that, but they are going to care about readiness and liveness probes that Kubernetes use to make sure that your pods are healthy. So they want to be able to know what are those endpoints in the application so they can go and troubleshoot and inspect a misbehaving application on Kubernetes. Another audience that we identified were business decision makers or other stakeholders at a different level than the business who aren't involved in the day to day technical work. So these could be people who are approving the headcount that you need because you know you've deployed to Kubernetes and things are going really well you need more people so you can get more things done. So you need to ask for more headcount or maybe it's just you need more hardware maybe your cluster is starting to get overloaded maybe it was like a prototype where you launched some products and they're being they're really successful and you need to scale them horizontally you need to add some hardware so business decision maker is somebody who'd be involved in being on boarded to Kubernetes in that capacity. So their goals probably make responsible financial decisions as one. So they're going to need to understand the technology the broad high level architecture as it's related to the top level of the business let's say, you know, less downtime that's good, faster development, more features for users. Like I said, earlier they probably don't really need to know about core Kubernetes objects, they don't need to know what a pod is or a replica set. Maybe they do maybe they're a technical CIO or something like that, and they want to know it, but that comes down to knowing your audience. That's understanding who it is that you're you're communicating this information to another goal that they might have, again is scoping for long term growth so short medium long term. Maybe they need to do some capacity planning to be able to say hey we're we're going to run out of our, our cloud budget account for for the month, because this, this product is just so successful we've added so many nodes, you know, our Kubernetes cluster is at capacity. So as you're growing, you need to be able to understand that those are the goals of, of people higher up in the business. So back to the Kubernetes curriculum then. These groups that we identified were were central to the way that we organized content itself so new tutorials again, we wrote them for the curriculum specifically. In existing content, we already had lots of tutorials on Kubernetes that we organized and tried to present in a meaningful way with all these audiences in mind. So at first we considered having tracks where we targeted those specific audiences that we'd identified, but instead we went with a kind of choose your own adventure or self guided approach where people could take what was most interesting from the curriculum to them, or what was most relevant, and focus on that. Again, if you were more technical business decision maker, there's something there for you, if you're more high level, but we also came up with a white paper as well we didn't put this in the curriculum. But we do have a white paper for those those business stakeholders who maybe are more used to reading white paper format. I still think for that maybe 50 is best but that's me, and bias there. So the second part, once we've identified audience is approach. So once you know who you're addressing what their goals are. So what should you approach onboarding them. So let's take application developers as an example audience. Their goals, as we said earlier, need to develop with a distributed distributed architecture in mind. And they have a goal of deploying successfully, they wouldn't have a goal of not deploying successfully, but it's good to just articulate that. And at that point when you talk about doing deployment and doing development, you can start to break things out into theoretical topics that you can use to get them sort of familiar with some of the ideas behind a distributed architecture. And then some of the specific tasks some of the things that they'll need to do to be able to accomplish their goals. So an example. These are the 12 factor application principles. So, if they're not familiar, you know, learn about things like version control dependency management, stateless and stateful configuration environment variables. All sorts of different things in the 12 factor method that you can, you can use to get your developers familiar with the idea. So environment variables are a great example here. You know, you're just hard coding credentials into a configuration file and using Capistrano or Ansible or puppet to deploy that config file into your environment. So they never needed to be any environment variables and 12 factor application and a distributed environment environment variables are kind of like a first class citizen they're really important. When you're onboarding developers, they need to know what needs to be refactored to take environment variables into account. Also need to know about roles and responsibilities. So who has the keys for updating these variables, you probably don't want to give you know a contractor or junior developer credentials to your production database, let's say. You know, you're just involved and that's only accessed by CI and only certain people have access to the CI pipeline. So the idea here again is is the application needs to be made environment aware. That's one of the 12 factor principles. So another approach. Another thing that, you know, might need to do, you know, I talked about the Rails application for community what was Rails for running on a virtual machine system. It's not a modernization that could be involved or engaged in there. So things like logging monitoring and adding instrumentation, so that you can look at your code and see, you know, which database queries are taking the longest when we're browsing these pages, learning how to add liveness and readiness checks for your pods, so that your operations team has some insight into the health of your application. And even when logging and monitoring are handled by other teams completely, your developers are still probably responsible for defining those health readiness endpoints and whatever hooks or tools that you're using in your code base to do that instrumentation and logging. So the idea here is your developers are in charge of making your pods happy so that everybody else can monitor them. So containerization. Another thing that was new, you know, what do they mean for your application when it's just, okay, an Ubuntu Bionic server, you can run app get install and use whatever application version of a library is available from Ubuntu. One thing when it's a containerized application, and you can define everything yourself, it's a little bit different. So have your developers set up a local environment using Docker compose or LXD so that they can get familiar with images and containerized stacks and learn about the interactions between components, the data flow between different servers in your system network overlays. All those different bits of Kubernetes. So a good example is the community app itself sits behind Nginx. And so having a local setup where developers can replicate that deployed stack gets them more familiar. So they can understand what a change that they're making locally will look like, mostly when it's deployed into production so that they're comfortable and able to just work with the stack as it's mostly deployed in production. So deployment itself. Whoever's doing deployments, what what's required to make sure that that's successful. What does a deployment look like? Are they zero downtime deployments? Are there maintenance windows? Who's doing the work? What needs to be done? And what happens when there's a failure? Failures are kind of inevitable in any large scale complex system. So what do you do when there's a failure? How do you roll back? So as you're doing this, as you're, you know, identifying all these different approaches, there are going to be some blockers. There are going to be some questions that everybody who's involved is going to be asking questions and rightly so, you know, they're going to ask, well, why do I need to design a 12 factor application? You know, I had a perfectly good working site that was serving millions of users a month. What's the point of splitting out environment variables? Why can't we just, you know, bake credentials into our Docker images, let's say. Likewise, modernizing, you know, what is a distributed architecture? You know, it's great when I know, okay, I've got these three different machines living in this data center and they each have a specific task and I know exactly what each one of those machines does. It's a little bit different when you say, well, it's just it's running in the cluster. So your team is going to start thinking about these things. They're going to spend a lot of time, you know, getting up to speed and you want to help avoid these kinds of questions. One of them that comes up all the time when talking about Kubernetes is containers themselves. So, you know, what fundamentally are containers? How are they different from virtual machines? What are the advantages that they actually bring? I thought virtual machines were supposed to isolate my application from other applications running on machines. So what are they, why are they better or different from virtual machines? And then, again, with deployments, you know, you're not using Capistrano, you're using CI and get hooks and make files, let's say. So there's a lot of changes there. So it's really important to be able to engage your developers in this case with thinking about these ideas up front and these these metal level topics as a part of your process and that way, people have lots of expertise in all of these areas and it lets them preempt problems. They can start thinking about these problems before you need to address them as a group or before they become a problem in production. So in terms of breaking things out into those metal level topics I talked about, you know, a good example would be, okay, well, maybe if you're not familiar with Kubernetes, let's talk about what is Kubernetes as a first step. And then talk about, again, what are containers, because this question comes up all the time, what's a control group and what's a namespace and what are all those abstractions on top of those that make Kubernetes all the way up to deployments and replicas, let's say. In terms of operational tasks like the day to day, you know, how do I design a 12 factor application modernize containerize and deploy, you know, another example would be, how do I make sure that my application, how do I make sure that my application is stateless at this point so that, you know, if my pod gets evicted or, you know, there's a failure there. That's okay. I just a new pod comes and takes its place and a user doesn't lose their data or a transaction doesn't get lost. So again, in scoping things for your audience, it's really important to understand the needs of the group. What are they trying to accomplish. So, in this case, in the case of the community platform, you know, we're talking about web developers from a traditional virtual machine background, maybe not a DevOps team with with years and years of Kubernetes experience who've used all of the different network overlays over the years and knows the ins and outs of each. So back to the Kubernetes curriculum that again. We decided to address that that meta level idea by having just an introductory topic section. So readers could opt in, you could pick any one of the topics inside there just to get familiar with things like what, what's the Kubernetes scheduler what's at CD do, what is the QV BI server, and those other core objects that make up Kubernetes. The audience could be everyone this this made the most sense where we didn't have to split things out into each individual audience business decision makers, operations people developers we could make it, you know, relevant enough to anyone who seemed interested in one of those topics, made it optional, use it as a reference make it a refresher for people. So back on the community platform team now. There's happening testing production cut over getting ready to actually move the application. You know, there's a pretty big team involved so some of the decision making was happening in smaller groups, you know, infrastructure specific things application specific didn't need to convene the entire group to be able to talk about, you know, environment variables in staging or something like that. We're also creating internal documentation so we had machine manifests for doing deployment in our CI CD concourse pipeline guides on how to deploy into staging production how to work locally in your local development environment, how to do rollbacks how to do redeploy so the focus there was really on just those operational making sure that everybody knew what are the things you'll have to do when you're in the role of somebody who's developing or operating the application on Kubernetes. So we need to make sure again some of these topics that we had we knew where secrets were talking about that a little bit earlier. How do we actually define our application server images. How we're troubleshooting concourse like our build pipeline when it fails. Is it a failure in production for Kubernetes or is it a failure in the CI pipeline itself. And how do we safely kick off a deployment or rollback in, you know, any of those scenarios. There's a big but here. This is specific to our experience again note the developers on community are the specific audience for this slide. So the big but here is, we didn't really talk about some of those metal level topics early on. We didn't ask the question what does it mean to deploy in a distributed environment at the application level. So a good example is we didn't really think about schedule jobs where we're using Rails sidekick redis. And as we, you know, started discussing things and looking at things we realized, well, the way that we've architected things we're going to have multiple clusters. We have a redis cache in each cluster that stand alone. We have sidekick workers that talk to those redis caches so we could maybe have, you know, some consistency issues with their data across clusters since there's a shared database. We're not doing redis replication. So that was something we hadn't really thought about we hadn't talked about that metal level. What does it mean to have a distributed application across multiple clusters. And so once everybody was on the same page with that architecture, we figured out okay well what's the best solution hey let's just run sidekick and read us on one cluster. But we couldn't make that decision before everybody understood the architecture and understood stateful and stateless development models. It just wasn't something we could decide because we didn't know about it. So the key takeaway here again is make make time for those metal level discussions. You need to have some kind of theoretical foundation before you start thinking about the operation of the day to day the mechanics of deploying to Kubernetes. It's absolutely fine to have those operational things and have documentation about them but everybody needs to have at least some kind of high level understanding so that you all have a common, you know, language and ability to talk about your application and talk about those operational tasks. So absolutely fine to break things out in terms of developer operations business decision makers. You still need to have some kind of metal level way that all those groups can communicate together. And that should probably happen based on our experience at the beginning of this planning process. So that leads to the next part of the defining your onboarding process which is scoping. So, once you know who you're dealing with. You know what their objectives are their goals, you know how you're going to get them knowledge so they can achieve their goals. Meta approach or operational. It's time to think about the communication process itself. So different formats you can use one would be documentation. We're probably all quite familiar with reading documentation they're, they're a great way to communicate information asynchronously kind of at your own pace. Documenting precise specific technical steps. For example, our tutorials they're designed to be copy and paste you can copy a command, you can paste it, we test them all so that they all they work they get you from zero to whatever it is you're trying to accomplish. Documentation is also a great way depending on how it's structured to have again a self selecting pathway through a bunch of different material. So, you've got a lot to cover. It's really good to let people kind of find their own way find the things that are relevant and interesting to them. So that brings me back to the Kubernetes curriculum I talked about. It's again topical you can select your own path through there's 25 tutorials in it but the last time I checked our site there were at least 60 tutorials on Kubernetes on the community site. So, again, talking about those audiences. We targeted or plan these these contents and topic headings around those audiences developers operators business decision makers and the color coding arrows on the left that you see were different combinations so I forget what colors we chose but you know we have maybe green for an operator and blue for a developer, and then yellow could be a combination of operator or developer. And it was ways of just segmenting content so that we could label it and tag it and surface it depending on whatever pathway that people chose. Another good example, Kubernetes documentation so depending on your hardware, your cloud provider, whatever it is. There are tons and tons of different self selecting pathways that you can choose to learn about Kubernetes if you just want to do local development you could use mini cube or microkates or whatever else it is that you want where users can just self guide themselves they can pick the concepts that are interesting to them or focus on the operational tasks that they are trying to solve. Another example with documentation in your internal documentation. This is an example slide from one of our tutorials is add definitions. This is a really useful way of coming up with that metal level understanding. So, this example here highlights a passage about services services which will ensure that the pods running our containers remain accessible. Now, if you know Kubernetes, you know services are a lot more than that. There's no mention here of, you know, the difference between node ports or ingress controllers or anything like that but as a business decision maker as an operations person as a developer as a high level idea of what a service is. This is a really great definition it gives everybody a common language that they can use to talk about services so if there's an outage and you've got an SLA and customers are complaining. You can all talk about okay well our service was not available or something like that and everybody knows what you're talking about there. So the idea here again, use small phrases explain particular tools topics technologies concepts. If you're in doubt, good rule here, define it because if you know what the definition already is, you probably just skip reading it. But if you don't somebody's going to learn something and know more about the topic. So this tutorial again highlight services deployments persistent volume claims config maps and secrets, all with small little individual self contained sentences. So another great format for onboarding would be a tech talk like what we're doing today. It helps you learn from other people's experiences, you can watch people learn, watch people do technical tasks, let's say you could go on Twitch or YouTube and figure out how to deploy an ingress controller from somebody else's experience or you could learn how to automate your TLS certificates in Kubernetes based on somebody else's YouTube video. So invaluable to be able to watch something like that. Digital ocean. We also have a munchin learn on Wednesdays. So we have different topics from different experts. We have lots of in house expertise on things like staff for Kubernetes controllers, concourse building CI pipelines. Great ways in, you know, a 30 minute little window to scope a topic, make it approachable to people who may be not involved in the day to day technical details of all these tools, it's to be able to communicate out to a larger, less technical audience across the company. So it's a really great way to have company wide best practices made visible to the rest of the company. Another format meetings, you know, okay, maybe we all have too many meetings these days but when it comes to doing something like migrating an application from virtual machines to Kubernetes. We're probably not going to do that effectively as a team without some meetings. So they're a great time saver, absolutely necessary with these big scope changes. I think it's really good to have an agenda, especially if you're trying to get people acquainted with Kubernetes for the first time. What we would do is have a just high level quick high level architectural overview at the beginning of whatever it was we were talking about that day. We could go back to the streaming term things in terms of those those meta level topics so we could always come back to those if somebody got lost or wasn't, you know, didn't quite understand what we were talking about and maybe the technical operational area. We could go back to those meta level topics and come up with a way to explain things. So you can use any or all of these doesn't matter you don't have to pick one or the other. So you'll move through these formats, they'll be very specific scope by the time you get down to a meeting docs, you know, Kubernetes documents are designed for everybody. The curriculum was designed to teach the world Kubernetes, whereas our internal meetings on the community team whether it was designing content and tutorials was, you know, very specific to maybe a particular tutorial. In terms of talks, I think, keep them short, you know, a quick half an hour somebody can find the time to be able to review that if it's relevant to them. And again with meetings keep them specific because with a specific meeting with an agenda you can make assumptions about the shared knowledge that you have with your team. And you can be more effective there. So, in terms of formats. Again, we we focused with meetings internal documentation and some videos. We also designed a local Docker environment to look and behave kind of like how the production Kubernetes environment worked. We didn't change anybody's workflows yet. Like it's still just have a virtual machine or, you know, use our VM on their, their Mac or something like that. But in the end everybody after doing this got, you know, pretty familiar with what things look like on Kubernetes. We helped the team help the developers on the community, which is the audience here develop faster, we have got more resiliency, keep on scaling as we continue increasing visitors to the site. Nobody on the team is really a Kubernetes expert, you know, we're not going and upgrading clusters ourselves that wasn't what we were intending to do. Most of the time nobody's even interacting with Kubernetes directly, you know, we're just using our CI pipeline and our get hooks. We want to make sure that people could do their best work with enough knowledge about architecture to make decisions that were relevant to them. So they know about shared storage or the lack there of shared state or statelessness scaling. So the last thing I know we're almost out of time here is day two considerations. So I want to talk about this a little bit because it gets overlooked quite quite often. You know, most, most applications don't spend their, their application lifetime being developed they spend their, their life in production with fixes and maintenance and new features being added to them. So it's really important to think about well what happens after you've made the transition to a Kubernetes cluster. So, again, thinking about these audiences developers operations people decision makers. So how you're going to make changes to your application or your Kubernetes cluster itself. Those different changes architecture infrastructure are going to affect each role differently developers care about, you know, making quick changes to the application faster development operations care about reducing work less downtime less risk. Same thing with business decision makers they want to care, you know, about cost keep costs down less risk. How do we scale things, you know to meet our budgetary constraints or our user growth goals. So the idea here is plan for iterative change. All of the things I've talked about today, do all of those again iterate so define your audience and goals determine the best approach scope appropriately. Just on and on and on. So, pick an audience. Figure out what's the approach that you're going to use scope to documentation, have a smaller meeting about, you know, deploying an ingress controller let's say instead of moving from virtual machines to Kubernetes. So good example, I'll wrap it up here shortly is let's say you've got an operations team working on a new Kubernetes cluster. What are the goals they want to have an up to date cluster that you don't want to get left behind and have a cluster where you just don't have an upgrade path from like Kubernetes 1.05 to 1.19. So there's a goal there there is another goal of ensuring there's no cluster downtime when doing upgrades. So, maybe it's a new team. You want to cover those meta level topics again. So talk about what are services and ingress controllers how do we make sure that our applications available. You know, you're going to be upgrading those. Why do pods get evicted sometimes from nodes. It's really important that the team understand that. Whereas, you know, in terms of operations when you're actually doing an upgrade. Well, what what are the mechanics of draining a node so that you can go and upgrade cubelet on it. How do you use cube admin upgrade plan. How do you actually apply the upgrade. You link up with a private image registry, let's say, these are all things that an operations team would want to think about. So you can scope it again with meetings have an internal upgrade playbook look at the Kubernetes documentation. Really important one as well is as your team is working through these things and doing the work, keeping track of a route of things that that maybe didn't work properly or manual steps failures that happened and any troubleshooting keep track of that because that'll be a great way to iterate and improve things that you don't run into those problems in the future. Another example would be a storage architect. Let's say, you know, you're moving from a big sand array and storage architect has been tasked with building up deploying Rook inside Kubernetes cluster. So again, they've got some goals want to deploy Rook have no data loss be able to scale storage. So they need to know things met a level again like what are custom resource definitions. How does Rook manage stuff, you know, they're probably used to looking at stuff and running stuff command line tools to look at stuff monitors or to monitor your object storage team and how to manage your placement groups and how to manage new disks, add new disks in the cluster. Now that's different in a Kubernetes environment with Rook. So again, you know, as a storage architect, probably going to look up some videos to see how it's been done Rook support so many different storage back ends you probably want if you're doing stuff to find that specific documentation or videos, then you want to look at documentation and I put that down three times because you want to look at the documentation for Rook, Seth and Kubernetes lots and lots of readings. And then you know, storage if you're migrating from a big sand environment, you have storage administrators, it's probably going to be a pretty far reaching change so you're going to want to have some meetings work with developers work with operations and if you had maybe a change advisory board group, you're going to want to work with them to look at risks and things like that. So I think that's time for me. I wanted to leave some time at the end for questions. Anybody wants to get in touch, you can use on Jamination on Twitter, you can find me there. So I'm not sure if any questions have come in, but I'll hand it over to you Jerry. Thank you very much for a wonderful presentation. We do have one question here. I'm wondering more about how to govern teams while still providing them with flexibility. Should we allow the creation of namespaces and resource quotas on their behalf? Are there any tools that can help with onboarding? Yeah, so namespaces are a tricky one because there's really nothing fundamentally about a namespace that enforces any kind of isolation or constraint between teams. So that's where you'd want to look into something like like role based access control or RBAC. So you can define a team, you can define contacts and namespaces that only those teams would have access to in the cluster. Hopefully that answers a question. Okay, do we have any more questions at all from anyone? Well, that's great. It means I covered everything. So plenty of time, folks. Five more minutes. So if anyone has any questions, please feel free to ask. We have another question here. Is entry point for assist ops difficult to start with Kubernetes? Right, I see the question there. I think if you spend the time, excuse me, and think about those those meta level topics again, you know, what fundamentally is a container? What do you mean to work in a distributed environment? Review those 12 factor principles. I would say it's no different than all the work that it took you to learn to submit in the beginning, coming from a virtualization or bare metal environment. You know, in many cases, having a heterogeneous bare metal environment is more complex because you have to support so many different kinds of pieces of hardware and equipment. I wouldn't necessarily say it's more or less difficult. It's just a matter of, you know, again, what it is you're trying to accomplish. And your understanding that that higher level of what it is that Kubernetes can do for you in your environment. Anyone else. Well, Jim, and I would like to thank you again for a wonderful presentation. As I said before, today's presentation and slides will be available on the CNCF website. Thank you again everyone for attending. Have a wonderful rest of your day. Thanks everyone.