 You are here for software delivery and the Rube Goldberg machine. Just a little bit of background about me. I'm Melissa McKay. First and foremost, I'm a developer. I've been a developer for years and years, all the way from lowly intern to a principal. There was a point in time when I started to speak a little bit more in conferences. This is really difficult if you're holding an engineering position. I mean, maybe you can do it maybe once a year, especially if your team has travel budget. That's great. But it's more and more difficult to travel, especially if you have other obligations like family and a full-time job that you need to keep down. So I started looking around for positions that would support me doing this type of thing because I love talking to other developers. And I don't want to be limited to just my individual team that I'm working with. I want the broad spectrum. So conferences like this are perfect. That's when I started working at JFrog for a few years ago as a developer advocate. So that's why I'm here today. Thank you, JFrog. During that period of time, I became a Java champion. Java is my primary language, certainly not my only language. I don't know a Java developer these days that aren't playing with something else. I'm also a Docker captain. I really enjoy containers. I like getting into the details, the ins and outs of building them, deploying them, that kind of stuff. I love how easy it is now, getting everything out into a cloud environment with these web apps. I'm also on the CDF TOC. That's been an interesting education for me. It's not something I've done before. So getting into the governance and behind the scenes of how open source projects are developed and how they're managed through a foundation. That's been really interesting to me and I still have a lot more to learn. So looking forward to serving the rest of my membership on that committee. Here is my Twitter and LinkedIn info. Please reach out to me if you have any questions. I'm concerned I won't have a ton of time for questions because I'm gonna be doing a panel after this, which I also encourage you to go to. It's about being on the TOC. I will be able to answer questions, what that's about, what we do, why we exist. So definitely look for that after. Otherwise, I'll be around in the main area there if you have questions. Lastly, I do live in Denver. I enjoy that area. Not very many remote employees of JFrog in Denver, so I do love being able to travel and meet colleagues when I come to these conferences. All right, Rube Goldberg Machines. How many of you built one? How many of you write software? You've all built Rube Goldberg Machines. This is something we do every day, it feels like. And obviously I'm gonna turn this conversation towards software, but I had to go out and do a little bit of research on Rube Goldberg Machines. My favorites are ones I built myself, of course, just for fun, or the ones, anything that includes dominoes, I love those, those are my favorite. It's pretty easy internet search, obviously. They're very popular. There's actually a website out there that you can go to. I've got a slide on that. But Rube Goldberg himself was a Pulitzer Prize-winning inventor, innovator, and cartoonist. So the original Rube Goldberg machine was actually drawing, so they weren't real physical things that we're playing with building today. I thought that was interesting. So as a result, some of them are kind of unrealistic because they're cartoonish, but pretty fun. This one here is called the self-operating napkin. Pretty cute. Another interesting fact about Rube Goldberg is he is the only person whose name is in the Merriam-Webster dictionary as an adjective. So if you have a goal in life, this might be something to strive for, get your name in the dictionary for something positive, of course. So I did go out and look. There is a website, RubeGoldberg.org, and there's actually contests held every year for kids. Anyone can submit to create these machines. Each year has a different theme. Last year, the focus was on literacy. This year, the challenge was to make a RubeGoldberg machine that will create a lunchable. So pretty interesting. That contest recently took place. They have videos of the top 10 submissions. They're very, very cute, so very clever kids out there. This year, they did a couple of other things, too. So if you don't have access to a bunch of materials, I suppose you could get onto Minecraft and build your RubeGoldberg machine in there. There's a video on that one as well. It's very, very cute. So I'm not gonna go any more into that. You can go look those up yourself or build one yourself. But there are three characteristics of RubeGoldberg machines that help me make this comparison between software delivery pipelines and RubeGoldberg machines. One, there's a lot of moving parts. We all know this. Anyone who's been involved in this process knows there's a lot of stuff you need to have in your toolbox just to get from A to B. They're also inefficient. Characteristic of RubeGoldberg, it's on purpose. For us, we don't like them to be efficient, but they end up being that way. If we're not careful, it requires us to change our outlook or make some different decisions. Third, they have a tendency to be unreliable and unsafe. That self-operating napkin, for example, might kick you in the face. It's definitely unsafe. I know that there's been plenty of pipelines I built when I was first starting out as a junior that were definitely unsafe. We have this tendency to build them in a happy pathway and when they break, it can be disastrous. Plus, there's a lot that has come out along the lines of security, some pretty severe problems that have happened in the last few years with supply chain security, things like that. These are all things that need to be considered. So I don't wanna make a broad sweeping judgment about software practitioners that we all behave like this on purpose. We make these inefficient things that don't work very well. Sometimes that's just a consequence of our jobs, but it is not an easy problem to solve. So we need some help. I had a lot of insight writing this book. There was a couple of colleagues of mine, a few colleagues at JFrog actually. We got together and wrote this book on DevOps tools for Java developers. We go through all of the main points, especially source control, containers, microservices, things like that, continuous integration, of course. And package management, that one was a huge one for me. So it gives you a good idea of what you're getting into. Obviously, most of you here at a continuous delivery conference, you're going to have a lot of exposure to these different components. This is the largest Rube Goldberg machine that made it into the Guinness Book of World Records. It has 427 steps. The whole thing takes four minutes and 26 seconds, and look how happy they are at the end when it succeeds and they're all done. This is exactly what my team looked like when we got our first pipeline up and running and everything was working. So pretty cool. This was built or finished on December 10th, 2021, so there's some opportunity to hear to try and beat that record. All of the ingredients that we need for software delivery. You know, occasionally it feels good to sit down and update your resume, right? All these things that you know, all these things that you've learned over the past years, and when I did mine a couple of years ago, it was pretty got a little carried away because after many years in your career, you're going to know a lot of stuff. But think about everything that goes into developing software. And the very first thing you need is your developer resources. This is going to be choosing your OS. This is going to be your IDEs for your team, building developer environments, even all of your planning tools like JIRA. You need to choose your coding language for your team or for your project, and you might have different teams working in different code bases. Source control, deciding which one to use, which one to establish among your team. I remember I did have to move from SVN to Git years and years and years ago. That wasn't so easy for me after being in SVN world for years. It's not a direct translation, you know? So all of these things need to be considered. Your build servers. I was in an organization, I think every team had a different build server. So if you happen to be a developer that likes to jump between teams to get experience in different areas, you're going to be learning what feels like the same thing over and over, just using a different tool. Security scanning. That's more and more important now than ever. I will admit that I believe I've been on teams in the past we didn't do any security scanning. There's a lot of trust in the software world. We use libraries that are well-known that may be developed by larger organizations or contributed that way. So now we know that that's not such an easy thing to do. So consider your security scanning. There's, you know, for containers, you've got your Docker desktop plugins. You have IDE plugins now that help you with that as a developer. These are new things for me in the past few years that I've never seen before that were really, really useful. Artifact management. Of course, I'm going to shout out for JFrog there. This is, you know, something you need to consider, especially if you're in, you know, cloud environments and you have containers you need to deploy. Maybe you want to choose, you know, a container registry that's near your environments that you're deploying to. All kinds of performance and efficiency concerns there. Your deployment infrastructure. Yeah, choose your cloud provider. Maybe choose the tools carefully if you don't want to be locked in or figure out, you know, your plan B. If your cloud provider goes down, maybe spread your eggs to multiple baskets and use multiple clouds, all of these things. Automation, Ansible, Puppet. I remember being on a team once where we were just implementing automation. This was a brand new forage into DevOps in general. So we were learning the tips and tricks. We had a situation where just to deploy a particular version, there was a bunch of shell scripts out there living on different machines. And part of your Rube Goldberg machine was to find the person that did the last deploy so that you could find the configuration that you needed. And then heaven forbid something go wrong with that because then you need to remember who deployed before so you could roll everything back the right way. So we learned very quickly automation, super important because we're human beings, we make mistakes. Observability and monitoring tools. Getting your logging out there so that it doesn't get lost. Making sure you're using tools as Splunk is a good example. Cystig, Datadog, all of these tools are useful to add to the toolbox. All of these interesting to me especially because I was always on the developer side of things. So these are concerns I didn't have at the very beginning. I was in the typical silo of my generation where we did our coding, we focused on the prettiest code and then we threw it over the wall. So lots of learning there. I'm sure you're all familiar with this. The CNCF landscape. Just to highlight one of the smaller sections, the orchestration and management section, there's now 20 items categorized in there just for remote procedure calls, service proxies, API gateways. There's 17 items under service mesh. That one circled there I think is 20 items under scheduling and orchestration. So pretty expansive world we live in and sometimes it can be really hard to decide what to use. Best advice ever. You can do research but limit your research and then just pick away and go because you're going to get feedback, positive and negative no matter what you choose. Just pick away and go otherwise you get that decision fatigue and you don't get anything done. I love jigsaw puzzles. I really do. And a few years ago this came out as a meme and I absolutely wanted one so I went out to see if any were left and 42 were sold. They're sold out. Maybe I'll make my own one of these days but I thought that was pretty cute. All right so there's the argument for a lot of moving parts. We got that characteristic out of the way. Whatever you use for your existing use case, just make sure you're documenting whatever tools you choose to put into your project and make sure that if you make a choice for a different tool for whatever reason, something didn't work out, something didn't work as well as you wanted, some feature was missing that you were looking for. Make sure that that's documented somewhere because I tell you what, I have definitely been on projects long enough where I see one tool put in, bunch of people leave, another tool gets put in, bunch of people leave, and then the same original tool comes back. It's ridiculous. We don't wanna be doing that, repeating the same nonsense over and over just because we didn't know the background and the history of the choices that we made for the project. All right, inefficiency. Obviously if you're new to this landscape it's gonna take you a while to find your place that's totally reasonable to expect especially if you're just out of school, you're not gonna know a lot. It's going to take the expertise of the entire team from development all the way to operations. Some may be a part of your team from beginning to end. I would assume those might be more senior folks, maybe folks that have stayed on the team the longest that have a lot of institutional knowledge, but something that's really, really useful for teams is to have a mix. You don't want everyone on the team to be experts. You need to have new people in because you need that fresh perspective, and there's also new technologies out there that frankly there's just not enough time to stop and learn. So it's kind of nice to pull from new folks as they come in. So if you start seeing that your team is a little bit too experienced might be time to start shifting some people around. Also one of the biggest misconceptions about the DevOps methodology is that one person should know in depth everything, and that's absolutely not true. It is the composition of the team that matters. So you need all of the different skill sets on your team and collaborate with each other and communicate with each other. It should never be on one person to know everything. Typewriters, I'm gonna age myself a little bit here. Anyone used a typewriter? Oh good, I'm in a good group. All right, I was worried. I was worried when I say I've actually used typewriters, especially my kids. I don't think they've even ever seen one. So interesting. This is a group Goldberg machine and it kind of demonstrates a specific phenomenon that I see sometimes in projects. There's so many times I've encountered tools being used in ways they were not intended to be used. I'm just thinking of all of the major, you know, very long shell scripts, lots of glue code maybe between different tools that you're using. It's really interesting when you're working on a project that's being used by others and you get all kinds of feedback of bugs or this didn't work as I expected and you find that no matter how well you plan your project, you do not anticipate the ways that the users of your product are going to use it. So it's very valuable to keep up with that feedback and get ahead of that. But another reason this might happen, you may have lack of due diligence when you're evaluating a tool in the first place or maybe you just simply are attached to a specific tool and we kind of tend to want to make that tool work for everything, make it be a one-stop shop. I don't want to pick on Jarrett because it's wonderful. I've used it in many projects. I think there are ways to use it, ways not to use it, but have any of you experienced the attempt at trying to add a bug into the system and you have like this 12 page report you have to fill out first before you can get anything out or a feature request or anything. I've had that experience. I've gotten to the point where as a developer I'd be working a specific bug and I can't even figure out how to close the ticket because there's too many things to fill in that have business value but they have nothing to do with me. So really interesting how we can make a tool try to work for everything. Development of your pipeline ad hoc. This is a lack of planning in the beginning of a project figuring out where you want to end up. Always approach from the goal and then take the steps backward to figure out how you want to get there. This way you can avoid a lot of the meandering paths we sometimes take to get to the end step which is deployment. Now some of that has to do with communication and understanding the different groups, what the responsibilities are, especially if you're still in the situation when you have developers that are separated from the deployment team. So very important to have those communication channels open so you understand what's happening. I know I was in this situation once where we were developing containerized applications, they were deployed out there but the containers kept dying. We didn't know that, our development team never knew that because that never came back to us. Instead operations built a script to just relaunch every time they were killed. Well obviously that's not the best user experience because they didn't understand what the consequences of doing that was and then we never made an effort to fix it. So these kinds of communication problems are very important in the initial stages. Make sure everyone understands what your build pipeline does, how it operates. Sometimes you have a specific feature that you're missing. Maybe the team doesn't really know where to look or maybe you're not allowed to use a particular tool because of licensing or other corporate restrictions. So that happens, that can cause some problems with using a tool in a way it shouldn't be used. When you start putting all these together, I talked a little bit ago about glue code, huge problem. This is a big subject on interoperability that the CDF focuses on. We have a special interest group for that purpose. This is just the idea that instead of building integrations, which are hard to maintain and keep up with, maybe there is a way that these tools can interoperate better using a standard language that everyone agrees upon, things like that. Lots of project focus. Sometimes a project, you start out, you have this wonderful idea, but eventually it becomes spaghetti code, it becomes unmanageable, some of that is due to trying to respond to the community and all of the requests that come in, all of the features that they want to add. Sometimes it's best to just kind of step back and say what was the initial purpose of this project and curb all of the requests that are not falling within those specifications. So one way to protect yourself there is to really make clear what this project does, what its intentions are, and have a good team that you can count on to review these requests as they come in. There's a lot to unpack there, but what you don't want is a project that does too much, too fast, because then it doesn't do anything very well at all. All right, inefficiency. One left, tendency to be unreliable and unsafe. This is a slide that often appears in JFrogs corporate meetings and decks sometimes. It's really busy, but I just love how it shows all of the interconnections and all of the different tools out there that you might be using to complete your pipelines. This actually doesn't even address what happens after deployment, so it doesn't include observability tools, that whole realm, but each place, each of those green lines is an opportunity for failure. So when you're building your pipelines, you need to not only consider when it works perfectly, how am I gonna behave and how am I gonna react when one of these connections breaks? That goes along with the happy path coding. Sometimes we are guilty of that just in our normal development as developers. You can certainly do that as well in your pipelines. Lots to consider putting your pipelines together. This is Salsa. If you go to their site, salsa.dev, it's described as a checklist of standards and controls to prevent tampering and prove integrity and secure packages and infrastructure in your projects. So each of these red triangles is another opportunity for someone to get in there and disturb your progress. I focus on dependencies a lot. That's just something that's near and dear to me as a developer. The more and more complex our projects are, the more we're gonna bring into our projects because we don't wanna start over from scratch, of course. Developers are notoriously lazy. We don't want to reinvent the wheel if we don't have to. This was something that was interesting that happened and this is not a dig at GitHub because everyone is allowed to make decisions for their own projects and tools, but something that was unexpected and you may have noticed this at the end of January, they decided to change the default compression that was used for the source archives at which made the checksum for those archives change. So if you're using a package manager that relies on that, all of a sudden you have issues. You're gonna have to clear your cache to solve these build issues. Anything that you don't own, you can't secure and anything you don't own, if you don't own your own assets and your own artifacts, including your release artifacts, which I think are super important for your project, your reliability's at risk. So make sure you account for that when you're measuring the risk assessment of your project. If you don't have everything internal to your organization, there's a risk you're not gonna be able to get to it later. I said I was a container person. I do enjoy them, I like fiddling with them. But there's a lot more to do with dependencies when you're working with containers. This not only applies to every other software project you're on, you've got the code you wrote and then you have everything else you're pulling in. By the time you're using containers, this is exploded exponentially. So not only are you worried about your libraries that you're pulling in, you're also worried about things like your base images. When you're first starting learning something like Docker, I learned the hard way. I mean, you just take a little bite off and do the best you can and then you make a mistake and then you go back and review. And that's what I did. This is a very contrived Docker file, of course, but several things wrong here or that can be improved anyway. Use trusted barrent images. Even if you're using a well-known image that's available on Docker Hub, you're probably best to bring it in internally so that you can repeatedly scan it. It's available to you when you need it. You're able to update it if you need to. In line two, we often need to make some adjustments to the base image. We need to install additional packages. Put in versions. When you go and rebuild this image later for some requirement or maybe your build machine is building everything fresh, you're gonna land in a troubleshooting nightmare because all of a sudden, your stuff isn't going to work and it might be due to something that has updated on you way early in the process. So infuriating for developers, when they're trying to figure out why their code broke and it has nothing to do with their code, it has to do with something in the background that they weren't responsible for. I've spent hours and hours and days on stuff like that. Another one here. Oh, line seven, that's a good one. I see this all the time. I see WGet, I see Curl. Anything that accesses some external resource, you're running the risk of that resource running away. I had a situation once where we had some proprietary software that we needed to install within the container and there's a trust relationship there with a third party, so they provide an install script. Well, reaching out to their install script is obviously only going to work for so long. One day the build broke and it was because they moved repositories. Simple enough, but the fear is, what if it just goes away entirely? Now you don't have it. You have to rebuild it yourself. So avoid stuff like that. Pretty simple things but can cause a lot of problems if you don't pay attention to them. This one, letting containers run as root. There was a report that Cystig came out with last year and still 76% of the containers that they observed which came out to three million containers were running as root. Obviously there's some legitimate reasons you might do this but this is something that we don't take seriously enough. All right, so what can be done? There's a lot of complaining here. There are also some solutions. This one is near and dear to me. Organizations start to come together. Sometimes they have partnership deals where they have an agreement on things that are wrong in the industry. We're trying to fix it. This is something that a project that I'm working on right now between JFrog and Nginx. Just trying to make things a little bit better as far as defining an infrastructure to deploy containerized apps too. So that's interesting. Foundations of course have a, they find this really important, the CD Foundation. This is the landscape for the CD Foundation, not as intimidating as the CNCF one. These projects are more focused on continuous delivery aspects. And one of the more interesting projects that came out of that work is CD Advance. Just yesterday afternoon, I don't know if any of you were in there. I don't recognize, oh, Lori was there. Brett was there. Andrea Fratulli, he hosted the CD event summit yesterday. And that was the opportunity for people in various positions to come in and collaborate on the CD event spec. For those of you who don't know what that is, it's a specification for events that can be consumed and emitted from various different tools in order to solve this interoperability issue that a lot of us experience. There is a white paper that describes it in detail out at the CD events booth. Check that out. But I mean, there were folks from Apple there, from Fidelity, J-Frog of course, from Seth. I know I'm gonna forget some people. We had a Jenkins representative in there. We were all discussing like how we might want this to work between all of us. So pretty cool, something to pay attention to. So we are working on this problem. We're getting better and better at it. I'm really interested to see what comes around the corner, especially when we start introducing AI and all the newest technologies out there, how this is going to improve for all of us, especially those of us who have to build these things. Kinda nice to have a little bit better tools to choose from and some recipes on how to do it the right way. All right, if you have any questions for me, I think we're kind of a little bit over time. But join the panel for the CDF TOC, that's coming right up. And I'll definitely be around in the foyer for the rest of the day, so feel free to reach out.