 Hello So this is a talk about developers demanding UX for Kubernetes with a bang at the end because it's very important My name is Mo Duffy. Oh Man, is it not gonna work? Is it because I'm standing over here? Let me see Okay, my fancy thing doesn't work so we'll do it the old school way Really? Oh, no. Well, okay. I got something and is it gonna go back and forth? Okay, cool. All right. It's working again Okay, anyway, my name is Maureen Duffy. I'm a UX engineer at Red Hat and Work on the Padman desktop project And here's Connor. Yeah, my name is Connor I was a student of software development at the time of working on this Report here that we're going to talk to you about so I interned with Red Hat This is one of the projects I took on while I was interning Since then I've been gone the full-time position with Red Hat. I've been employed so but yes at the time Yeah, I was just an intern And we are cube con newbies so so be gentle. Okay We also feel a little bit like outsiders because we're UX, but we're bringing kind of a fresh outside perspective So also be kind to okay. Not that you wouldn't be but okay So what did we do and how did we do it? So for three months we did a user research study. It completed this past December We talked to 11 Participants was a combination of developers and platform engineers who support developers who work with kubernetes as a deployment target We recruited through the Padman desktop community and full disclosure The reason we did this research was to inform a new kubernetes feature that we just launched in Padman desktop Yeah, and most of the interviews were video calls. They averaged about 45 minutes apiece We actually scheduled them for 30 minutes But you know because we put the bang in the top top title like they actually literally did demand better UX for kubernetes And they went long so the participants were very enthusiastic This is a list of the participants we talked to you'll see there actually are a lot of red headers What ended up happening is we reached out to the palm and desktop community and Just based on scheduling issues and time zones. We wanted to make sure we had at least 10 participants so we kind of reached into our internal community as well and This is sort of we had a range of experiences that we talked to the average of The average participant had two to three years experience with kubernetes on the low end We talked to a platform engineer He he worked at an ML company that supported data scientists that not only had no kubernetes experience But they had no infrastructure experience either So that's sort of like the low end of experience that what we talked about captured and then on the high end These are red headers Somebody from since 2015 and somebody since 2016 had kubernetes experience. So we had a pretty good range in the participants Okay, so now Connor is going to talk you through our research protocol Yes, so we had 16 questions in total for the developers that we spoke to and the platform engineers as well So these 16 questions they related to different topic areas. So their environment that they work with their workflow general workflow their approach to debugging and Also any recurring issues or pain points that they experienced if anyone would like a copy of the full research questions You can click the link there or the QR code, but I'll just go into them a little more detail the moment Please please reproduce our research talk to your own users and let's share the results Yes, so There was a couple of questions. I won't go through all of these but just to give you a quick Insight for example, we asked them in terms of the environments. They work with we asked how many kubernetes environments Their organization maintains and what kind of environments they are we also asked how how these environments are created and And Questions to do that effect so Then in terms of their workflow one of the main questions we asked was what primary tools CLIs and components they use how they utilize them in their standard workflow to walk us through it and Just also whether they prefer a CLI or GUI to us We also asked if there are any particular challenges or difficulties at the encounter when writing kubernetes YAML files most of the answers to this question began with yes and Then just for general issues that they would encounter or pain points So we asked them to describe their general approach to debugging issues including the tools that they use to aid them in the process Where they go when they have questions and also how they approach debugging issues that are only reproduced within production very carefully We also had an additional questions section you may notice there's just one question there But that was because this was the final question that we ended on so this gave Gave them an opportunity. It was a bit of an open question and gave an opportunity to Talk about a couple of things that maybe they felt weren't mentioned along the way and Allow them to kind of continue and say whatever they were reaching to say along the way so Yeah, so then the methodology. So this is how we analyzed The interviews basically once we had completed them So we performed an affinity mapping exercise and categorized findings from the interviews So basically you went through the transcripts of the interviews and we found we extracted the main points that the developers mentioned and we then categorized them into separate categories such as Workflow and debugging in the overall category of the workflow that we mentioned So Yes, and so with that then we're going on to the next section. So this is how can kubernetes provide a better experience for developers How'd you over the moe? where we can we we provide some Suggestions as to what we think could be a good way to address Maybe the community has already addressed it and neither also the developers we spoke to were aware That's an opportunity to maybe better promote the projects because I know you guys are the experts, right? So if we point something out as an issue the developers surface to us, and you're like, oh, no project X will solve that Well promote project X to developers. Okay, that's like a takeaway I want you guys to get maybe we don't know about the technical solution to something that was raised to us if you do Help help promote it to developers to help our ecosystem. All right, so yes this quote I hope you guys were reading it while I was blabbering This was a kind of an interesting quote that one of the participants said We have a range of Experiences that we talked to with kubernetes. So some of the more newbie to kubernetes developers. I mean generally They expressed a lot of intimidation And I don't know if that resonates with anybody in here when you're first starting out trying to learn kubernetes as a developer There was a lot of intimidation This was a quote from somebody who was more of an expert user, but he said, you know, it's beautiful when it works I totally get it. But when it's broken It's so hard. So that's sort of like the even an experienced user is kind of coming from this perspective Yeah, so the first issue then this was debugging networking issues So I'll just talk a bit about this. So this emerge is like a large pain point among the developers They've all had some kind of experience with this and especially networking issues that occurred between in the gray area between kubernetes and infrastructure layers So the main thing that they mentioned is that there's limited visibility into networking layers particularly in cloud environments like AWS and there's also a lack of proper metrics So that makes debugging networking issues quite difficult for them And they mentioned that this is especially the case among developers around familiar with container network interface configurations So those who are a little bit less experienced find this in particular extra daunting and The general approach that they take In situations like that is just to deploy to a different zone To assess reproducibility. They feel like there isn't really any other approach that they can take Another point that was mentioned is that there is a complex relationship between different networking options such as the kubernetes default option kubernetes and the private azure CNI and a lot of developers said that they Are unaware of the intricate relationships between these sometimes and it can also Cause unexpected issues So one particular example that was raised is that if you use the azure CNI overlay That it may restrict the use of the application gateway for ingress And then on the note of network topology, so While the while these issues are rare while network topology issues are rare such as how the VLANs are trunked That was one thing that was mentioned in particular when they do occur troubleshooting in production is often necessary and They mentioned that in this case all you can do again is just to set up a mimic environment and hope the problem is easily reproducible And a note on this nice quotation here if you hear hoof beats behind you assume It's a horse, but if it's a zebra proceed accordingly That's kind of the reaction that the developers will have when they realize This is a network topology issue and this is something that I'm going to have to address in production We listed this as number one because I think pretty much everybody we talked to cited network issues So it's number one for a reason. It seems to have the biggest impact Okay, so number two now, we're gonna have a theme going on YAML I hear laughter, so I think you get where we're going at This one's pretty easy Basically YAML is human readable, but the space isn't a indentation matter It's very hard with the human eye to review a file Especially when you know, you're on the hook to fix an issue like oh my god We need to figure out what's going on here, and you're looking at it. It looks fine. Looks fine passes the lint check Something is indented wrong. It's misconfigured That that was something that people surfaced as this is a continuous problem And it's something I think it's been a problem with kubernetes configuration for a while It seems to impact developers a lot We had some Questions, you know could we develop a tool has one to been developed before I don't know That would maybe give you some sort of preview before you commit the new YAML Could it give you a preview of what it's going to do? So before you push it You would get an idea of the impact. So if you did have some indent that was broken It would let you know You know like reliable previews of helm and customized Output that was something a couple people said like, you know in a dream world If you could have what you wanted, what would you ask for? That's what they asked for Also the various ways values can be stacked and added to objects before the application of the template. That was another thing that came up So That's that's the thought here. So now on to the next YAML theme Yes, so the next one then this is a lack of proper YAML validation This is something that the developers raise as well So they they feel that this is a significant issue that they feel like the tools for YAML validation They're not integrated with for example the custom resource definitions that are finding clusters So if you have a crd for tomcat, this is one that was specifically raised The ammo validation tool it won't be able to validate the ammo unless you import the tomcat crd into the tool first They also mentioned that many this is something that the platform engineers we spoke to Mentioned but they said that many of the organizations they have resources created without the proper GitOps mindset in mind while they're creating them and They mentioned this is a problem too because if automation is part of their continuous development pipeline There isn't a viable tool to generate GitOps YAML files That they can run Git version on and then reapply to other clusters as a baseline So some of the recommendations that the developers themselves said they would like to see They would like to see a tool a reliable tool for family validation, which accounts for custom resource definitions and it's Takes into consideration as well GitOps principles So this is a tool that could potentially be incorporated with the previous tool we mentioned for indentation But what they would like to see is the tool that will can validate the syntax and structure of YAML files to ensure They're valid be integrated with custom resource definitions That are defined in clusters and also generate GitOps YAML files that can be version controlled and reapply to other clusters as a baseline quite easily Number four and this is the third and final YAML series Developers complaining about kubernetes feature and by the way, there's eight of these we're halfway through I know these are dense slides, but there's a lot that the developers told us So this one is when developers are exporting YAML kind of has a lot of tags and labels and annotations and time stamps But otherwise it may be the same as another YAML But because of those little minute differences two YAMLs can seem like they're different or they're applying to different types of things Um Something that multiple the developers we spoke with said it would be so nice If I could just export a clean copy of the YAML that doesn't have all these little bits in it So that I can have a better clearer view of what the differences and similarities are in the environment And again, it's one of those things where I think by default kubernetes is very chatty. I I get it. I'm very chatty too But when you're new to a platform like this and you're just getting started It is a fire hose of information and I think that's why on the whole developers kind of express this intimidation So I think this is one maybe easy way if we provided a tool to export clean YAML It would make it so that only the differences that really mattered They would stand up because there's less there's less. I mean, this is a general UX principle, right? The less information you have When some you make an alert for something or you highlight something It stands out so much more because there's less than it that it's competing with right So if we had some sort of export clean YAML feature Then those little bits like the time stamps and the tags and the annotations that don't really matter Won't add to the noise and it'll help the developer hone in on the bits that they should be paying attention to um, and and that is Exactly what we suggested here with a little bit more details And again, like inadequate export functionality hampers the development of a comprehensive single pane of glass That's what i'm talking about some way that you can just get an overview And what surfaces to your attention is only the stuff that really is different Yes, and then another issue raised by the developers was a feeling of inadequate security analysis tools, so They mentioned that security analysis tools. They often generate warnings for non issues As they tend to check the library versions on base images So this was raised as like a really big problem because it causes desensitization to these warnings. The developers will sometimes Anticipate that a warning is a non-issue and then that makes it easier for the real serious issues to Be missed basically the kubernetes that cried wolf. Yeah, exactly. Um So Despite efforts to address security vulnerabilities. Um, this was another thing that was mentioned as well Many bills on artifact.io Have these vulnerabilities in them as and this is partially as a result of that desensitization to the warnings Where and only the critical alerts get the attention that they deserve in cases like this That's a feeling among the platform engineers that we spoke to Um another Issue this was also mentioned by the platform engineers. They said that A lot of the developers that are unfamiliar with differences in application security at runtime and at rest and then this leads to unexpected vulnerabilities appearing at runtime too And One thing in particular they mentioned as well This is something that the developers said that they feel uncomfortable with They said that certain security analysis tools like rocks ctl They require you to push the image to the registry before you can scan it and they feel that this risks Inadvertent pulls of those insecure images and it's a security risk in itself So the recommendations that they had for that was to basically just have a tool in place that's capable of displaying information on the fly Even during development that can provide timely insights into issues that are as they're occurring or They would also like to see the security analysis tools just refined to improve accuracy reduce false positives They would like a developer awareness to be increased around security at rest versus at runtime And uh also to have uh enhanced tool integration and workflow to minimize the need for uh registry images before scanning We got three more. Are you guys okay with the demands? Is it too much? Okay Come on now. Come on to be happy be responsive. Please. We're very nervous. Um, so number six was crash back loops Um, this probably won't be a surprise But it's a little bit hard to to bug a thing when it crashes so fast that you can't actually take a look at it um Accessing logs during crash back loops Is one of the biggest things that hinders troubleshooting and actually in one case Inconsistent log visibility across platforms that one The developer actually said you know what open shift actually has this really nice I'm not saying this because I work at red hat by the way It has this really nice thing that lets you view the log when you're in this situation But it's not on a screen the developer anticipated So the entire time he was in this trying to figure out what was going on He figured out that he could scroll down on this one page and access this log But it wasn't where he anticipated he anticipated it being on the events page So that's another where it's we're kind of almost there, but it needs a little bit of refinement The other thing that we actually talked to our friends at sneak at the expo hall and and verified this with them If you're like me you prefer to use container files And um some some of when when you're using sneak in certain configurations It will only check docker files and if the file extension is container file it won't check it. Oops Um, so that was something that one of the developers surfaced to us Um Is there anything else that I missed connor that was worth surfacing? And no, I think I think you covered it. Yeah, so um our recommendation again is a tool I mean designers got a design right and the idea would be something that lets you that's specifically designed To resolve crashback loops that are triggered at this specific problem When a pod is in it's stuck in a crashback loop Maybe stop it from doing that instead of letting it do that over and over again um Have a user friendly interface for troubleshooting maybe surface those tools like for example The log file tool that the developer didn't even know was there, you know, maybe when you detect there's a crashback loop happening Surface that to you know, so just in time contextual Assistance for the user when they're in that situation you detect they're there in that situation You provide them the tooling you've made to help them out of it And then um, yeah options for creating debug containers within pods letting the users ssh into the pod because they just they feel more comfortable Having an ssh prompt to tool around I certainly do when things are going wrong So all right next one Yeah, yeah, so we're on to the cli versus a goi debate So the participants had some things to say about this as well most of them preferred The cli of course, but they still feel that there's a lot of room for improvement with the gui So especially for inexperienced users as well having refined gui Can make the everything seem a lot less daunting and it can avoid scaring away new talent basically as well so Um One of the interesting things that came along for the platform engineers we spoke to they said they will often have to provide remote support to developers And those developers they will not be able to share their screen so Trying to guide Especially if the person you're trying to guide is unfamiliar with kubernetes is unfamiliar with infrastructure in general Trying to guide them through a gui when you can't see what they can see that's Very difficult thing to do so that was a A massive pain point raised among the platform engineers that we spoke to So the requests on that part from the developers and the platform engineers was to basically just refine the gui interfaces to accommodate various experience levels um enhance interactive troubleshooting features within gui tools as well And uh also ensure that gui tools they offer clear status indicators and alerts for better visibility into cluster health and performance because that was something that The developers mentioned that was the strength of the gui tools being able to see at a glance what the health is like of a cluster So they would like More more of that basically Okay, so the last one are you ready? Are you bored? Is this helpful? Okay, good. Thank you So the last one is basically just our grab bag bucket of just general issues that kept surfacing up over and over And i'll just go through them The lack of direct access to a file system was something that made the developers feel a little bit like i'm not at home I'm not safe here. What's going on? It was just an impression that they surfaced to us Working with multiple kube config files that are constantly changing and merging is cumbersome This is something that we're actually working on with podman desktop when you start I don't mean to pitch podman desktop But when you started it looks for your kube config file and we're looking at you know On the the yaml complaint front a way to just provide a nice gui for working with your kube config rather than having to do with the yaml Despite sass vendors offering applications exclusively through kubernetes or containers Vendor supported customers are deterred by its perceived complexity. So this is the case where you know what? The the company needs some tool They don't know anything about kubernetes and the vendor tells them well, you want to run our tool We're happy to help you You have to set up kubernetes and then we deploy it on top of that So they're sort of like dragged into kubernetes just by the sake of this tool they want to run So that's sort of you have to think about this is the perspective. They're coming into it, right? Um troubleshooting kubernetes can feel overwhelming. That's the whole quote that I showed up front about When it works, it's beautiful, but when something goes wrong the pieces are all over the floor And I don't even know where to start Um any kind of guidance we could give users when we can clearly detect something happens You know at the point that it happens Would be very reassuring and helpful to them Um and then just fragility and kubernetes component interoperability Especially during upgrades deploying platform level components like service message or service meshes Provides challenges and risks. Um, you have dependencies. You have charts referencing older versions of the api It's kind of difficult to navigate Um, so our recommendations here intuitive tools for editing kubernetes configurations Um at enhanced debugging tools to accommodate pods with multiple containers avoid altering internal dns config on windows Reducing the need for admin access and then simplifying kubernetes pipeline for debugging Would be nice okay, so We made a lot of complaints to you guys we what we didn't make the complaints We we are advocating for the developers and platform engineers we talked about okay And it's great to come with a list of things this needs fixing this needs fixing this needs fixing But what about actually fixing it? So we'll just give you a quick little Talk about pop in desktop and how we took some of this research and put it into action What I would love everybody in this room to do Wherever you you integrate with the kubernetes ecosystem Think about the findings we had think about the things that we surface to you and think about is this something that I can change About what i'm doing. Can I make this better? Is there some small patch? I can write or can I change the way I think about a thing to make this better and then as a community I think that we can tackle this especially the yaml stuff. I mean that seems pretty doable, right? but Yes So my lovely seal friends here would like to introduce you to popman desktop 1.8 This just came out yesterday, I believe And we have new kubernetes objects support. We have deployments We have services and we have ingress routes. So you can view these from the popman desktop ui popman desktop the thing about popman desktop is it's very developer centric So it kind of gives developers a view that's tooled to their interest and laid out in a way that Is built for their workflows. So we're trying to expand that from local pods on podman to to showing kubernetes stuff and um You can also this is a feature that's been around for a while You don't need 1.8 to do it, but you can create a kind cluster You can create a mini cube cluster from within popman desktop as well and the workflow would be you set up pods locally You have your application going you're ready to go to kubernetes. You create a kind cluster You create a mini cube cluster you push to that You test you iterate and then when you're ready, you can push to an external cluster So, um, yeah try it out and that's what we got um any questions Please somebody ask a question even if it's silly What Do we have a demo? The seals are demonstrating different kubernetes objects I don't know if anybody is familiar with the original kubernetes meme that this came from but it has been recreated with seals No check, okay Uh Is there a way to connect to the api through open shift? Through podman desktop, that's an interesting question um I mean We have support for open shifts. So if you wanted to push your pod to open shift, that's possible Um, stefan are you in the audience? Can you take that one? Can somebody get that man a mic? I'm phoning a friend guys So, uh, the answer is yes So you can effectively connect to an open shift environment directly from uh from podman desktop But you can also connect to any kind of kubernetes environment through the apis So you can deploy there and you can see the different resources That are running there as well. I saw a hand up there before. Do you still have a question? Oh really? I think I think one thing about the logs would be interesting is in docker compose You had this where it showed you all the containers when you do the logs Whereas in kubernetes, you have the issue that well it defaults to one container, but it's not inherent or which one it defaults to That's one thing and the other thing is that I don't work for uber But I know they have a tool where if a pod crashes it generates a zip file of the logs before the crash And uploads that to their compute platform if kubernetes has a single a simple like a similar concept that would be super cool Those are great ideas. Yes, please work on them I think we had one over there. Yeah Right there Thank you for the presentation. I think most of us are not in the ux knowledge department If we were to reproduce your experience in your um process in our companies, how would we go between gathering information into Some results do you have some process I can share with us? Sure. Do you want to hit on it a little bit connor or do you want me to take it? I think Okay, I'll go back to I just want to say that like we we are open source zealots at red hat Which shouldn't be a shocker to anyone. This is a screenshot of a tool called pen pot It is an open source design tool that we can use as like a sticky note app. So A fancy pants ux designer will tell you this is an affinity mapping process I'm just going to tell you you basically you interview You can take the questions that we we sent out. There's the qr code and the slides I'm sure the slides will be available after the talk And then you you find some developers to talk to you go through the questions What we did is we did um we use google meet and it does the auto transcribe function And we went through the transcriptions after the fact and it's it's sort of a technique. It's called ethnography It comes from the anthropology field and I know I'm going way into too much detail here But you go through the transcripts and you just pick out those nuggets of information that were really insightful So what me and connor did is we kind of split the transcripts havesies And we each kind of reviewed the transcripts in detail and anytime that we saw something. Oh, yeah, that was insightful Oh, wow, that was a good point. Oh, maybe we should look more into that We just made a sticky note And the sticky notes were sort of all over the place to start And then once we had enough of them we we had I know with the two two hours three hours Took a while about tree. I think this was a big study and like like I said in the beginning The developers were so Into it that the meetings went much longer than we had planned so we had a lot of data to work with But we basically had a couple calls where we went through all the sticky notes and we just clustered them And it's not like it's not like a scientific Well, it is literally a scientific process But you're basically reading through them and the ones that seem similar you just physically Place them next to each other. So we did that we ended up having different clusters And then we went through the clusters and came up with labels for them And then connor very helpfully Organized them into columns And you'll see like in this like you'll see like we have cloud native mindset Workflow networking. So we ended up with like a lot of subcategories And what we ended up doing is clustering the subcategories into larger categories And that is what we ended up with the seven categories of of information that we came out with So that that's how we went through the process and then once we went through each category We kind of thought through and it's interesting because These are listed not in any order YAML Was one of the ones that really popped up to the top and it's reflected in the results where we had three actual YAML results that we felt were actionable. So does that help? Cool Hello Thank you for the talk For a better context I work at a company that delivers application platforms and we are Yeah, I can attest to the challenges for delivering these application platforms for developers And I think some of the problems that we see as well is that even though we can try to make Some of the problems observable through open telemetry and curated observability stacks and stuff like this and It's even it's hard for developers at least for some of them to to actually Prevision these local setups so that the visibility or observability is the same on the local setup as in Kubernetes, so I have you Any experiences with sort of challenging that or any recommendations for bridging that I mean, it's it's basically shifting left Right You're trying to get the environment that developer is working in as close to what the unproduction environment is going to be And having the ability to have a monitoring observability Locally on their local cluster that is comparable to the actual production cluster It's just going to make it so much easier. They're going to understand how it operates in in the production easier Um, it's going to be more intuitive for them when they're trying to debug something So 100% I mean, I would say yes. That's a great recommendation is Anything you can do to get their local cluster as close to production as possible. It will minimize a lot of those those impedance issues Thank you. That was very insightful Hello Thank you for the awesome talk. Um, quite a few of these points really hit home The one part that I wasn't really clear about is you were talking about user experience But specifically are you talking about user interfaces? Are you talking about the cli? and At least for the cli like I feel like some of these issues like are not well known But there are solutions like, you know, the plugins for kubectl like kubectl need that will take out all the Crap out of your yaml and just give you like the The specific things that you care about Um, the other thing that I find is really useful for kubectl is doing a server side apply with a dry run And that will run you through all the validation checks and give you a lot of information about what's wrong Um, but yeah, some of those things like, you know networking and trying to trace every packet that goes through and you know What ip table rules touch it and you know, why is it, you know crashing? I'm totally with you on that So I wasn't sure whether, you know, it makes sense to differentiate like, you know our Is this on the cli or is this on the ui and when it comes to ui, what kind of ui is and what kind of cli is people Yeah, when we when we say user experience, we mean the entire user experience whether it's the cli whether it's some gooey Whether it's just going to the a web page of a kubernetes platform and starting from there or going through the docs It's just the entire experience And I I would say hey if the cli has tooling to address some of the issues we surface the developers aren't finding it So that could be does that mean maybe this feature in the cli? We should look at surfacing it and some of the ui tools for developers Does it mean we need to advertise it better? Does it mean maybe we should examine the layout of the cli's verbs and and commands and see Maybe maybe we should have a better way to make it more obvious that this functionality exists Or maybe if we detect an error state, we suggest a specific verb for the command that could help and then that's how they learn Right, so there's a lot of ways you could take it, but definitely the user experience is complete end to end Thanks for the question So as Looking up to see if there was a special interest group for this kind of stuff and apparently there was a sig usability at some point and it was stopped But it seems like the room was full. So there's definitely demand for this. Is that something that might get started up again or maybe you would Be interested in Uh, if people want to talk about that i'm around all week conor's here too. Yeah come talk to us. We'll see Thank you for pointing that out We're good Thank you so much. We really appreciate your time and your attention. Thank you