 Good local time, everybody. My name is Justin Rackliffe, and we're gonna be doing this presentation around kind of avoiding your own jungle of the commercial software supply chain. It's a riff, essentially, on Upton Sinclair's book, The Jungle. We're not gonna be talking about meatpacking plants. That's not the discussion we're gonna be having. We're gonna apply the same kind of concepts and the metaphor to our own supply chain, the ones that we produce and the ones that we consume through commercial services. So a little bit about who I am. So I work for Fidelity Investments. I kind of handle all their open source stuff. I lead up their open source program office. I'm the kind of main point of contact for Linux Foundation and the to-do group, which is all the open source program office leads. Internally, it's really just being that the advocate for doing cool things in open source at Fidelity. And that could be, we wanna use it, we wanna patch it, we wanna contribute it. We want to work with a supplier who depends on it, which is pretty much all of them. So I tend to be fairly busy, but it's fun stuff. And just because this is a global event, I am in Cary, North Carolina, which is part of the Research Triangle Park. Just as a whole geolocation. So why does this matter to me? We all know that open source is in everything. We've all seen all the surveys, all the information, all the stuff is gathered. I did a little video. I didn't write this app to somebody else's, but it's going to kind of go through React Native. Just an NPM that many of us may have used, we may use internally, we may kind of watch. And yeah, I just want you to kind of see why. And I think of visual really just kind of helps. So I'm selecting out React Native. And what it's doing is it's walking through every single module that MPM is pulling out. So this is the kind of normal speed of the thing. I realized though that this was going a little slow. So I actually had to accelerate time. It's going to pop in right around now. And what's happening is again, every single link, every single module is being connected. And potentially creating another edge. And you can see, oh, so I just included one React Native. And all of a sudden my software supply chain looks like this. That's a little complicated. So that's it. So almost 600 packages were needed. They're linked a thousand times. And that's all the people who are responsible in contributing to making React Native a reality so that I can use it. Yikes, it's not necessarily bad. They're all doing really, really cool stuff. But it's pretty complicated. And that's both what we absorb when we write our own software. But also what also contributes to some areas of risk when we source somebody else's software. So the kind of the narrative is really figuring out what are people doing? Do they care about their build materials? Do they have concerns around this? Are all those modules really needed? And what does it mean to, quote unquote, operate locally, eat locally, and really understand what my supply chain means to me? So this is a quote from the jungle. Not gonna read it out loud. But again, this idea that we're getting a product. And it's a fine product. Probably was put in the can. May have a nice label on it. That's awesome. Now that if you kind of apply this beyond the concept of food, this could be an endpoint from a cloud service provider. This could be a commercial distribution. This could be a commercial of open source. It could be a commercial product that's both proprietary and open source. But if you don't know what's inside it, well, how do you know it's good for you? How do you know it's not been doctored? And again, this was a problem that the book was written to kind of resolve and just to wrestle with. And it caused a whole lot of changes when we look at food quality for the United States. And it's kind of a turning point when we look at what's the responsibility for the social good and keeping people healthy and what's the responsibility for the producer because they didn't have an infinite budget. People are only willing to spend so much. But it's important to kind of wrestle with these kind of questions. So kind of like I mentioned before, people use open source. No shocker. So again, this comes from the to-do group. I'll clarify, I need to refresh it to the 2020. But when we look at the ratios, and this is only so that people were willing to admit. For those who have already been taking the survey, they felt comfortable talking about it. Honestly, I feel like most products, commercial or commercial distributions of open source, they include open source. It's a big part of what we do today. And so those numbers, I actually probably think are a little bit low. And when we have to take this as kind of an end-to-end thing, well, if that amount of open source is sitting in our commercial products or is sitting at the other side of an endpoint for a software as a service product and we're giving them our data for them to protect, well, other people's supply chain matters as well. And life comes back to fast. There's always something new. Again, before we used to source things from some big commercial supplier, that could be any name brand that you know, just go Microsoft, whatever the case may be. It's not that those are bad, but the world is very different. And oftentimes those same suppliers are contributing to open source projects. And it's not that those closed source projects were without bugs or errors, but oftentimes we kind of trust it. The brand came with some credibility. Fidelity investments is very brand-centric. You come to us as a broker, as someone who will make a trade or manage your retirement. Those are really important things. Those are long-lasting things. And it's that trust that's there in the brand. With an open source project, it's harder to draw that bright line and there could be a number of reasons why that project could have problems. And again, companies can have problems. It happens all the time. But how do we make ourselves aware of those things? So what can be done? I think one of the basic things is basic education. Talk to your developers about it. Make it kind of part of their curriculum. Oftentimes they're required to take some basic security. Well, tie in some license, tie in some open source lifecycle. If we know that they're gonna be 60, 70, 80, 90% of their code base is gonna be open source. Well, let's make sure they understand what they're getting and how they're getting it. They don't need to be an expert. We have application security teams for a reason. We have open source program offices for a reason. This allows, again, some of those policy type questions, things that are really down in the weeds. What does a FARO GPL mean to me kind of things? Well, that's not the responsibility of every developer. Be really expensive. It would take a lot of their time. But there's some basic operations of dealing in an open source world that actually can be educated. They can be part of that portfolio so that as they're developing that application, they can go in it with eyes open. If they see a license and they know that this is for a mobile app, well, that's a distribution. I have to keep an eye out for the GPL. It doesn't mean you can't use it, but there might be other things that we as an organization need to do if we do. So let's try to figure that out. You can be an advocate for tools and practices that make sense for stuff that generates the level of ownership and accountability, giving developers options, telling a developer, hey, you can only choose this color or this front-end web dev framework, probably is not gonna end up well. It's gonna be a little bit of a demotivator and, well, they're probably gonna get a bit of shadow IT signal because this is a fast-moving environment. We're always being incented to take a look at alternatives and different technology. And there's some good to that and we wanna be able to foster that as organizations, but there's also some risk to it. So we have to be able to measure that. So tools and processes, especially like with a React Native example, you can't be policing 600 dependencies that probably are changing every day. That's not a scalable thing for a person to do. Tools can do it really, really well though. And we need to be the advocate to take that, put tools in the appropriate locations and then get that data to developers along with some education to help them kind of distill that and say, what does this mean to me? Not just gates, not just hard policies, but really understanding what those motivations are. Why does this matter in this case, but maybe not in this case? Inviting the developers to the conversation can be really, really important. And this is also part of engaging with commercial suppliers. So when you ask them for a bill of materials or what is their practice around open source governance, they don't have an answer. Well, it's not a bad thing. Some people have just not thought about it, but being curious can actually really help them. They may actually have something, but you may be talking with a salesperson or a legal person and they have to bring in a subject matter expert. Those are important conversations to have. And again, being that advocate for tools and practices, even in our commercial supply chains, very valuable because oftentimes, everybody's looking at the bottom line, whether it's products you're writing or products you're purchasing. So if customers are not asking for a capability or an artifact or something along those lines, you're probably not gonna get it. In this case, if more people ask for it, it'll become normalized, kind of like wearing masks, where we're figuring this thing out together. And then engaging in the sustainability integrity of our own supply chain. So caring about it. Investing your talent, investing your dollars. Not just asking the questions to our commercial suppliers, but potentially writing upstream patches, helping to, again, move those ecosystems forward. It's really, really interesting and fun stuff. So basically, we need to expect more. We need to look at it and say, hey, we can't just trust what it's on the shelf. Something that's bad. We can give it a little bit of trust, but it's still a little bit wild west there. There is no USDA of software. There's no one making sure this stuff is good. Even with all the flaws of the USDA checks and being able to time them and the overhead. It's not a perfect system, but it is a system. And it helps most of the time, us in the States, to eat food safely. We can rely on it. We don't have that for software, especially when we get beyond legal and copyright related stuff. Security best practices or maintenance best practices. There's a lot of best practices, not a lot of policies or regulations. So being able to say what is good is a lot more complex. So here's some specifics. I'm gonna kind of run through them. Kind of four points, but I think it make a big difference in avoiding essentially bad stuff making into our own supply chains, whether from things that we buy or things we pull in from GitHub or NPM. So knowing your bill of materials seems to be kind of an obvious one, but oftentimes teams overlook it. Maybe they just look at their direct dependency. Even if they have it, they may avoid doing transitive work. And again, maybe back in the day when you had five or 10 or 15 dependencies and they were kind of big, it's called Java. Maybe that was a feasible way of approaching this. But now we have these heavily interrelated ecosystems like Node and NPM that understanding your real bill of materials, not just the surface stuff, get complicated. Again, as we saw in React Native. Quality ingredients. So a little bit of the food theme again, but buying things that you can trust. And then how do you trust things? What are characteristics of the things that you do trust? Those can be hard questions. Is it just because you find it on a store shelf that you should apply some trust to it? Maybe, but there's some other options there. Organizational preferences, I think is one that small organizations can get by with. But as you get into bigger organizations, Fidelity has a wide breadth of developers. We have those who are still responsible for our mainframe. Those who are still writing COBOL. And then we have those who are responsible for our mobile apps. Both have equal responsibilities to security and compliance and all that kind of good stuff. Their needs are really different though, based around the ecosystems that they deal with and the velocity of change in those ecosystems. So being able to apply some context into understanding your supply chain can really make a difference to your developers. Because what works for a mobile developer probably won't work for a mainframe developer. And recognizing that, make sure the policies and practices reflect that. They're not overly biased in one direction. But just making sure that organizationally, we can figure it out and then mine that data. To help the next set of developers. And then again, investing upstream. So it's all for bill of materials. This is a little bit complicated. There's a couple of specifications out there. I put a couple of them on there. So you have SPDX. So that's the Linux Foundation's primary one. There's also Cyclone DX, a little lighter weight. I'm finding some organizations are adopting that as opposed to SPDX. Not gonna get into specification wars. The goal is that it's machine readable though. You don't want the back of the cocktail napkin kind of ingredients list. You want something that it should be somewhat reliable. And it's been created by an automated tool. And if you ever looked at either of those schemas you're not gonna write those by hand. So being able to adopt a bill of material schema that is machine readable but can still be converted to something human readable. It's super important. And it becomes what they're intended to be in exchange. I could have a supplier give me that and say, hey, yeah, these are all my ingredients. And I should be able to trust it a little bit. One thing to always remember though is these bills of materials are generated by sometimes open source, sometimes proprietary code bases. They're all built around heuristics. And it could be natural language processing or kind of fragment analysis and tokenization and all this kind of fun parsing stuff that somewhere in the back of my head from the days of writing software but they're in some ways proprietary. And everybody might come up with something a little bit different which is why you need that kind of normalize or neutral schema to hand it off to somebody else. There's a couple of plenty of open source options. I put a couple there. So pathology, Linux foundation project, next piece scan code, very popular project out there. Certain organizations will use something like the OOF dependency check and dependency track projects kind of core in the Java world but doing a lot more than that lately. And then there's also any number of commercial entities getting into this space. They all realize that static analysis for security and dynamic analysis for security when it was 100% proprietary code, maybe that made sense. But now with open source composition analysis is kind of a key part of your pipeline. Because what you do with those findings is a little bit different. They may become a CVE. It may be someone that is publicly registered with the government and possibly has an answer. It says, hey, just update. Those are important parts that we can take part of. We also have platform capabilities. So a couple GitHub and GitLab. Both have kind of accepted that dependency analysis should be a platform capability. You can use that dependency graph to feed any number of other types of systems. A security system, do you have any CVEs for this dependency? It could be a licensing system. Or if this is a distributable artifact, do you have GPL code in it? And does that create a concern? Or it could be a maintenance type of concern. Is this project being maintained? And being able to surface those kind of indicators can be very valuable to a developer. But they're all based around that core bill of material. So when we're thinking about the bill of materials, a lot of concerns you have to bring in. One is declared versus observed. And then we get into licensing in that way. So a lot of projects will say, hey, this is a license that I'm under. But then if you look at its dependencies, so that transitive dependency chain, well, really? If you're including GPL, but you're declaring that you're Apache, that doesn't work. So what can we do about that? Well, you may not be able to resolve that yourself. That may take some upstream work. But you have to be able to surface those things. And that's a great role of a tool. And then when we think about end of life, technology, life cycle management, how do we get this type of data available to our developers? Because if it isn't broke, oftentimes we don't fix it. And then if it does break, and well, you haven't fixed it for, we haven't maintained it for a decade, that could be a lot more complex as a fix. We like to get our oil changes done regularly, even if the engine doesn't seize. So we have to be able to kind of get ahead of that stuff and keep to a regular cadence. And a lot of that will all be built around a bill of materials. And again, this may sound like it only matters to the software that we're writing, but feel free to talk to your commercial suppliers. Some of them will put their third party notices or attribution notice or some form of disclosure on their website to say, hey, these are the open source work that we depend on, possibly even the version that they depend on. So you can use those, but ask, can you provide me a software bill of material? Because if they can, and they can do it on a pretty rapid pace, probably they have a good process and practice in place. They know what's inside their code base. They may have policies built off that bill of materials. So it becomes a strong indicator that they're aware of what's going into their products. If they look at you and kind of go, no clue. Well, it doesn't mean there's a problem. Again, you could be talking to the wrong person, but it is something that's not necessarily intuitive to them. And it may be a growth area. An area of risk that they weren't aware of before and you're helping make them more aware of the criticality because you're gonna be giving them some sort of data. That could be very minor, a query for maybe public data. We'll say a stock figure or something like that. Or it could be personally identifiable information, TII, so being in the regulated space, that matters a lot. So being able to surface that is really important. And helping organizations know you're not out there to nitpick their software development lifecycle. What you're there is to say, hey, I'm giving you potentially the keys to the kingdom, something that represents my brand. And I wanna make sure you'll care for it, that you'll protect it. And that you understand all the things that go into it. Cause oftentimes they're very comfortable with doing this for their subcontractors, maybe a geography type of check for any kind of procurement. We don't think of it for software, but it does matter. And you're also seeing a lot of work in the industry standard space. So open chain, again, another Linux foundation project. They're gonna be an ISO standard shortly, if not done by the time this air. And that's really important. It gives us all a place to look and say, okay, that's consistent. We can all kind of rotate towards that from our practices. And while it's really focused around the legal compliance, it provides us a framework for discussion. Cause again, ingredients are what's in the mix. Speaking of ingredients. So I'm gonna call it to a friend. Everybody likes Link. I know I like Link. Maybe I'm dating myself a little bit. But for a more recent game. Yeah, we all remember if you played Breath of the Wild, you couldn't just slam any ingredients together and get a good product. And unfortunately this didn't poison you. But yeah, it wasn't what you were looking for. You hopefully were looking for this one. And we all remember where you had to go in and get the ingredients for that. But in general, we have to care about where the ingredients come from. We see this a lot in the eating local. So you'll have now a restaurant saying, hey, we're a local based restaurant. That means they're just sourcing ingredients from the places around them. And why would you do that? And a lot of it comes down to the freshness. But it's again, that trust concern again. When you can go and see and you can visit a farm and you can look at the practices or you can look at the animals, you can look at the produce, you're gonna feel more confident that what you're getting is good. It isn't something that you're way separated from. So being able to apply those things to software is complicated. Because especially when we look at open source, one of its strengths is that it's borderless. That we have so many individual users, organizations, foundations, all contributing to create something common and awesome. So throwing the easy one up there on Kubernetes, we should feel proud there's 20 almost, 2,500 plus people that have contributed to this. And those are the only ones that are recorded on GitHub. All those behind the shadows are not listed. And that's awesome. We should all be very, very proud of that. So when we think of it in the context of an ingredient, is the community active? Well, obviously we have so many people playing on it. We have so many pull requests coming in to saying, hey, this is a fix or this is something new. And that engagement is very high. Well, there's a lot of issues. How many were closed though? What's that mean time to close? All of those you can look at and you can start kind of pulling out and saying, okay, well, this is a community that's vibrant that I can in some ways depend on in my supply chain. I can't name a specific developer and say, because of them, this project is good. It's not a brand name type of thing, but we can look at the characteristics of the community and decide that, hey, this looks awesome. Double projects that are kind of in this space. And again, they're related. So chaos, the health and, characters in health and open source and attributes of open source software. Sorry, I can believe what you're that. The sub foundation that we look at the health of communities. And this isn't just looking at, hey, I have an open source work and I need to maintain this work and I need to build a community around it. But it can also be around, hey, I'm gonna use this open source work. Take those same characteristics and the same attributes and apply them almost like a risk lens and say, is this a vibrant thing? Can I depend on this community? And auger is one of the projects that's partially maintained by the chaos foundation. To help kind of pull that data out. So it'll go out and it'll look at GitHub, it'll aggregate this data and help you make some decisions. This is also an emerging area in the commercial space where you have maintenance of open source works becoming far more critical because the first generation of open source passed a long time ago. So we may have adopted some Apache project from early 2000s and that's great because it saved us from buying a commercial product. But because that project is not broken, we may still be using it and it may be in the apache attic. Well, how can we surface that? How can we make that available to the developers and help nudge them potentially to an alternative? So again, this is the organizational preferences. I tend to kind of call this the Amazon shopping cart experience. A lot of people are very familiar with Amazon so it makes for a good metaphor. But it's a little bit of an abstract concept. What you're looking at is metadata. You have this giant storefront, this giant warehouse full of stuff. We don't know what's good. And a lot of the stuff looks exactly the same. I throw a spatula on here just because it's fun. And that spatula is awesome by itself but there's thousands of spatulas in Amazon. Well, which one's good? Which one's gonna meet my needs? And some of that becomes from what we see and we can see things like star ratings. We can see that there's people answering questions. We can see there's a badge that says Amazon's choice. And those are all indicators to us that, hey, this is probably a pretty good spatula. It probably will meet my needs. I don't know what the cost is but maybe this is the one that I should get. And some of this is organic but some of it is definitely not. Amazon has a lot of smart people and they're trying to steer you and that's okay. And we can do that along the lines with what comes up in our software builds and materials. We wanna help our organizations adopt similar products, generate a little bit of consistency because if, hey, we're gonna work upstream on a project, well, it'd be great to have more people work on fewer projects because we'll be able to materially influence them rather than spreading it out and not really being able to move the needle on any project. But we also need to be able to help people not buy what's directly on the menu. So you might be an Angular shop. And that's great. It's still an awesome project, very well maintained. But your developers may come to you and say, hey, I wanna try React or View or Svelte. Okay, well, we're an Angular shop. All of our knowledge, all of our experience may be there but we shouldn't necessarily be kind of single-minded. How do we allow our developers a little bit of freedom? Have them take some accountability because there's some risk. If you now have a defect in React and all of a sudden it goes back to an Angular developer that's probably not gonna work out well. But if we aren't aware of these things we'll probably miss emerging concepts. We'll not be able to stay up. We'll increase our technical debt. So you have to find this balance and exposing this type of data around who's using what, how effective was it, how supported do they feel and can they help support use? Can be really, really powerful. So a lot of times it's building a taxonomy, building topics and allowing for organic communities to grow. Centers of excellence can be let's say a formal way of creating this but oftentimes those are almost a lagging. They can't really catch those emerging concepts whereas a tagging taxonomy can't. If you are subscribed to front-end web dev and all of a sudden they start seeing topics on view well, okay, I should pay attention to this. And again, that goal is really to reduce the amount of variety and focus that allows us as businesses as well as from our suppliers to maximize those relationships. It's a basic economies of scale. Investing upstream. So this is one that I think a lot of organizations are still trying to figure out, mine included. This is maybe the most comfortable, at least a fidelity, budget dollars. We can allocate those things to foundations. We can do it on events, through and all things open. Just happened last week for me and here in Raleigh or ideally in Raleigh. We can sponsor events. We can work with partners like I put in Tidelift and get up sponsors which really start getting at hey, how do we help support a specific project or a specific domain of projects? All of those things take money. So this just helps you direct it. There's some benefits there because when we think about investment in some of these kind of core platforms, it generates talent. It gets our brand out there. Hopefully folks will be interested in coming and talking to us about a job. As well as it helps to stabilize potentially commercial distributions for cloud service provider endpoints that we source. That's important because we're all depending on those at some point. Time. So I think this was actually probably from the Kubernetes project as well. But we can just have our development associates spend their time fixing bugs, handling issues, just being generally engaged. They can do that in something like GitHub issues. They can do it in Stack Overflow. They can be focused on documentation. That's a challenge for every group out there. And it isn't just your read me. It could be a quick start. It could be an in-depth motivational thing. It could be an architectural decision point. Any of those will document artifacts become super valuable to the next generation as well as those who wanna learn from the project. So taking our resources and investing that time can really make a difference. And again, this bubbles up. It flows from these open source projects back into our commercial supply chain. So if we open a patch and fix a bug or a public CVE on an open source work, well, we're probably not the only ones using that open source work. We probably have suppliers that are using that open source work and they didn't have that time. So that bubbles up into their code base and helps to protect their code base which is protecting our data. So again, there's a lot of these cascading benefits that we can talk about and wrestle with. And then code. So again, most obvious one for source code is open those pull requests. Those could be bug fixes, it could be enhancements but working upstream rather than almost creating something new and sometimes it could be hard for organizations because they want that control and that brand awareness but having all the organizations work on fewer projects can be beneficial. Now again, it doesn't come without any risk. Always gonna be looking and making sure that project is the right one. It's going in a good direction. So neutral governance and all that other kind of fun stuff. Forks are a way to handle that but being able to create freedom in your development talent to work upstream on the actual source code can make a big difference. So here's some kind of next steps. The stuff sounds all great and everything like that but how does it fit? Where does it fit in my world? So when you're coming up with your product review documents or maybe you're working with your procurement team make sure open source supply chain is a question you ask and be willing to listen and to learn and to adapt how to answer and respond to that question. And also when you're creating products within reason be able to talk about what your supply chain is. If you wanna put a third party notices out there that's great. In some ways, radical transparency that's an effective light to shine light and say, okay, yeah, we don't want these bugs. But on request is good as well. You're not legally required in many circumstances especially as software as a service to providing this type of notice but community, it's a good thing. And it may help you and people may make you aware of a concern that you were maybe not seeing. Engaging communities of interest. So being involved in the things that matter to you. So if you're a big Kubernetes shop and honestly who doesn't have it on their radar right now making sure your talent is also attending in special interest groups or working groups or user groups. Maybe not leading them, day one, but at least listening. Getting exposed to alternative opinions. How are people solving a problem? Is this a good problem to have? Or potentially is it an anti-pattern? One of my favorites is always kind of multi-cluster management for Kubernetes. Everybody's got an opinion there. And you wanna be able to hear those. You're gonna land somewhere in that decision spectrum. So having a number of people be able to kind of talk about that problem. Potentially even writing code or writing documentation. Those who have tried to do it and maybe not have success, super valuable. Make your developer experience first class. This isn't just about tools. It's not just about GitHub or IDs or anything along those lines. But when developers are there and they're trying to do their day to day do they feel supported? Do they feel respected? Are they given information so that they can make decisions themselves? That they can take on risk and document that motivation? Or are they just putting in lines of code? So really thinking through that developer experience and trying to empower them, it's not anarchy but empower them to make good decisions. Build that accountability and that trust inside your organization. And when we get to questions I'm always up for suggestions on this stuff. I think it's an interesting topic for us all to be really wrestling with when we think about supply chains. So just talk about it. Those could be on medium posts. It could be wherever, LinkedIn, whatever your platform of choice is. But really thinking through all the ingredients that go into the software that we write and the software that we use. In trying to generate something that we can all trust. Now I'll leave you with a quote. So this was again, this is PBS's poison squad talking about the situation that spawned up in Sinclair's book. Well, we all look at things and we see this is awesome. We can't go to GitHub and not be in awe of what so many people have created in a common place. And it's not perfect, but seeing just such great stuff coming out of it in this generation. But we need to care. We still need to wonder what's inside because it isn't necessarily good for us. So feel free to ask those questions. Stay curious and you'll make it better for everyone. Thanks. Have a great day.