 This is Mary Vang and Nithya Ruff, and we both run open source offices at two very different companies. And so we wanted to talk to you today about two case studies. One is about Walwell Car. How many of you know Walwell Car? Yes. And how many of you know Amazon? So the second case study will be Amazon. The first will be Walwell. And we wanted to talk, a lot of you are familiar with Ospo's. We wanted to talk about the fact that there's more to an Ospo than managing risk, managing compliance. It's all about also making our developers' lives easier. So that is the angle that we're going to take today, because all of you know compliance, so we'll really focus on engineering effectiveness. And we'll start with Walwell and Mary Vang. Thank you, Nithya. Okay, great. My name is Mary Wang, and I'm a director of open source ecosystem for Walwell cars. My background is quite complicated. At the beginning, I was a DevOps engineer, I think almost 10 years ago. They are transferred to these CIACV areas, continuous integration, continuous deployment. Of course, open source is my side area in Walwell cars. That's also why I picked this up kind of quickly. Yes, time flies. I still remember last year exactly the same time almost September. I joined the open source summit in Dublin. That is my first time to join open source summit conference as a participant. So I'm so lucky and happy. This time, the second time I can join this event as a speaker, especially with Amazon together. So this is a picture I found which is very mapping the situation. Walwell car, Walwell Amazon car. How many of you have driven this car before? Oh, really? Okay, I uploaded this car to the AWS. Nithya, you can download and drive it. Let me know what is your drive experience for this car. Okay, as we know, Amazon Osborn, I think you have created for maybe 15 years ago. I think it's a big brother or big sister for us. So what was Osborn, how old are we? Nine months. In fact, this is an edge in the paper. In the reality, I think it can be like two years, one and a half years old from the data I initiated this activity. So even though it's a little bit young, but I think in this past year, we still accomplish a lot based on the creative and fundamental stuff. As you know, open source today is like flocking into many areas, many industries. But can you find any open source here in our antique car? For example, the first car drove out from the factory in 1927. Probably no open source, no Osborn, no open chain. Nothing. But can you find anything related to open source here? Not the latest car, but focus on this three point safety seat. I think that is the open source conception, because one of our greatest engineers in Volvo cars have invented this three point seat belt. We didn't take this as a commercial product, because it saves the life all in the whole world. So we make it as an open source. This is the greatest contribution in the world, I think. So time flies. Any open source software here with our new model car. So we have some new car here, EX90, EX30, EX90, et cetera. Of course, we focus on the safety, sustainability and the new technology as before, but with a lot of open sourcing it. That's why Osborn was born. So we bring a lot of value here also to reduce the risk to try to be compliant and focus on the security part also. But let me know if you want to buy one of this car after this meeting. I'm not sure how much discount you can get, but I think I can be changing my career from the IT to the salesman. Okay. Here is our open source fundamental approach here from we built the open source. This happens, I forgot to mention one thing is important here, it's like before we formed Osborn. I think there are some information I would like to share to you. Different companies have different approach for how you form your Osborn in your organization. I can only take my case, it doesn't mean it's a perfect or bad or good, but this is just a real case I have. It depends on your where the initiative located, if you are the high manager or the middle manager or engineers, et cetera. In my case, it's where from the bottom to up. So engineers does driven this and go to up to the R&D manager. I think we did use some risk reduction as an excuse for the compliance part. But I realize that is not enough because reduced risk compared to product itself, product usually has the higher priority. But in this reality, if the developer needed to have a copy left license for example, it takes a long time to get it, that will affect the production speed. Also, if you don't have the good open source contribution process in place, the developer cannot upstream the bug fix. So that also affects this productivity. In that point, I think we get the buy-in easily. So our fundamental approach here is we still use this ISO standard from OpenChem project in Linux Foundation as a high level governance here. According to it, we create our open source directive. Oh, sorry. As an internal law to govern the whole company in enterprise level. That is owned by our IP people, but it's still in our awesome team. Of course, with this in place, nothing will happen if we don't train all the employee. So open source training is very important here too. Based on the ISO directive, then you build your process method and tools and instructions for implementation. This is the key, I think. And also when you do the training, it's very important with the practical instruction in place, not just to train the employee in the high level. They are not interested. They are just telling me what to do, how to do, what is the result you want. Of course, we also developed our open source status dashboard, not to embarrass people just as transparent information to everyone. So if every part is red, red, red, then no one cares in this big company. If someone becomes green, all the people want to be green. Yes. And we continue to send out to the open source newsletter as a broadcasting and sharing information, conference, etc. That's great. Here, rules and responsibility. As a result-oriented person, I usually like the first thing is define who, who do what. This is very important. So I put here three circle here. I didn't put the Ospawn very big because from the member, Ospawn in a big company is not that big compared to the developers. It's put it in the side and it's not the central. All the company, I think, almost all the company will focus on the product itself. That's the key. Without product in place, Ospawn is not needed. So product is always high priority, but the side area is also very important. So Ospawn's role, I think, is very clearly described in the to-do groups mind map. You can find a lot of this Ospawn definition here. What here I want to share with you is about the open source champion role. I know Amazon have 200 open source champion role, which has a connection between the Ospawn and the development teams. So this is what we learned from them. And also it depends how many open source champions you need. It depends on how many developers you have. So that's kind of what usually I think one of our colleagues said, like 500 developers need one Ospawn in place. And the open source champions acting the intermediate role to connect the Ospawn and the development teams, which is helping them, guiding them, training them and finish the compliance work. And you can see the overlap part. I put Ospawn is connected to both open source champions and the development teams. I always feel like if Ospawn is just standing in the cloud, don't touch the detailed things. It's very difficult to have the result very efficiently. So our Ospawn tried to kind of connect both OSS champions and play with the teams in practical to help them to build the CI chain, to help them to automation, to help them to guide what the output, where to store all of these criterias, et cetera. So open source training model, this is quite challenging areas. It's not difficult to create these materials, but how to draw out in a global enterprise level, how you reach out to all the employees. Just think about the work hours, like if one employee takes two hours for this course, how many hours a company needed. So it's not easy, but we divided this open source training model into like focus on engineers or leaders, managers on open source champions, et cetera, for new employees, so just cover as much as possible. This is more like forcing the open source culture in your company. Compliance, usually companies start with the compliance work, the same we do it also. The end-to-end compliance, in practice, from the beginning, when people initiate the open source request, until the end, you have the notice file, you have this copy-left archive stuff in place in your system, and when you need it, for example, when I need the information from this vehicle, I would like to know all the open source component with the corresponding copy-left archive software. Can you have it? You do not need to have it all the time, you should be able to fetch it or extract from your central system at any time. So without the automation here in place, I think it's very difficult to have the result. It will make the engineer mad. In the CI chain, I think it's different, companies have different tools. Of course, in Volvo cars, even in one big department, you can have different variants of these different toolings, different scanning to different version control tools. So how to, kind of, as an awesome to give them a centralized function or library for them to use is very difficult, but still you can find some kind of general ways for this to help them to achieve. It's not easy. We are trying to do so, but it's not finished yet. So many of this is in our pilot project today. We are not 100% done yet. Always deliver compliance artifacts in the binaries, I think everyone knows it. But in reality, Osborne, I don't know, maybe this task is not in other companies Osborne, but we Osborne have been involved in many detailed work here, including the name convention for your result, for the car related or non-car related or commercial other product, et cetera. So we covered many areas with detail here. Otherwise, you will get a lot of questions all the time. It's good to do yourself with a clear instruction and you get less bothered. So this is one of our pilot projects. It's the same as your iPhone, right? You have all of this information in your system setting, and we do similar. So if you drive the world car in the future, this will be in your system setting in your car. You get all of this open source information here. This is our fancy internal portal here. I'm proud of it because we got the first three as the best internal open source portal here. Because this is quite from engineers' needs. Because as an engineer, which information you need? I think as the entrance, you need to give the engineers a very easy way to find out what they need. Basically, what is the license list? If I include the open source in our car, what should I do? If I just use the open source in internal development, what should I do? If I just want to install this open source, if it's due license, what should I do? Just give them a clear instruction, one, two, three, four. And try to build all of these instructions in a transparent and automation way. So don't like hiding some information, like email communication, that is not transparent. So it's like the CI to chain visualization. In which phase, what is the status now? It's very useful. Open source contribution. Compared to the open source compliance, I think this part is quite common between companies with companies, because GitHub is probably the mainly used tool for today. It's like usually you divide all the open source contribution projects into the license part, security part, project part, and IP patent part. So all of these have different people responsible for these questions. And in this case, I read some blogs. I still like the SAP's way of working for this process. We are trying to build our own process now with this transparent. I think the CI conception is really great for anything. It's the same for, I take an example for the license part. For example, developers do not need to bother you all the time. They only need to bother you when they are not concerned or when it's not in the green list. And when they do the scanning, they have this result in place. Then this kind of the CI visualization, mapping, if it's in the list or not, if it is, just continue. If it's not, you get a notification, either by email or by SMS or by whatever. So that is very good, I think. Try to reduce the three steps. So based on our current, I don't think it's the totally automation way, but we are working on this to improve this open source contribution part. So far we have released nine projects. This is one of the fundamental also, like building your open source external portal. We have nine open source projects, which is focused on some, is focused on the product, some is focused on the tooling points. So, sorry. So open source part, contribution and consumption, all this kind of fundamental stuff, the two chain end-to-end, using a pilot project to make it in place is the first step for the OSPON to build it. So in these nine months, I think we have almost finished this fundamental stuff. Meanwhile, I will roll out all these trainings to find the open source champions, to think with each department's higher managers. Like nowadays, we have more and more questions. We have opensource.ovocast.com, email which used for both internal and external communication. Previously, it's very quiet. Oh, I said, oh, no one contacted me. And when it's rolled out, then now we are so busy, we have so many cases now. So it's good not to create it. Great. But we still have a lot of challenges nowadays, even though we have made a lot of progress, but still, for example, finding the experienced open source champions across the company is not easy. Especially the people with the passion or interest, even though some people are good at open source, but they don't like this role, they don't want to kind of continue working in this area, then it's not a good candidate for this role. So this, like, usually we focus on the developer or system architect to take this role because they know how the code interacts with the open source more than the corporation people. But still, the time, usually these people are quite busy. The second part is the priority, as I mentioned. You know, the product always has a high priority compared to all other side areas. This is a big change. I think it's similar to all products or all companies. Like if you belong to R&D, you are focused on the product, the budget is usually easier to get than other areas. And also different teams have different open source materiality level. For example, if it's in the infotainment system, 90% of the source code is open source. So it's quite easier for them to understand this. But some teams like Propulsion Area, for example, it's quite low level open source materiality. So to align this to kind of education, education is the only channel here to influence people step by step. I did, I do a summary here. All of these parts for the open source area request the process integrated to CI and help the team to resolve the how and to end the compliance process, contribution process is automation. Without automation in place, that is waste, any nearest life with the company's life, and time and money. So I think also there's quite a key here. Don't throw out anything if you are not ready. I mean, you can have a workaround based on the sample like if you don't have, you cannot build an automation in one month, let's say. You can have some workaround, but don't let that to be too long. So try to avoid any work itself to engineers. And if your legal people don't know how to use GitHub or how to use your JIRA, just train them. Because you can get, you can only have one or two legal, right? But you have thousands and thousands of engineers. So don't make them to suffering from this. The future, I think all of your car in the future is based on more than 99% of open source. So I will continue to drive open source, maybe not drive here, but drive. This is one of our speakers. Great, thank you very much. Any questions? Oh, yes, that's great, yeah. Next, welcome, Nithya. Are you okay with that? And so she was kind enough to do that. And this would be, okay, thank you. Fantastic, thank you. Thank you, Mary. And Mary is so thorough and so complete in her approach. And by the way, she started the OSPO only in 2022. So this is, in one year, there's a huge transformation that Mary and her team have made in Volvo. So I'll do it at a slightly higher level. And one of the things I wanted to talk about first is, you know, I get asked this question all the time, you know, why does Amazon support open source? Why do we do open source? And there really are three main reasons. The first one is we use open source all over the company. From the beginning of the history of Amazon, we have built our infrastructure, our products, our services, all on open source. And I think Black Duck's study shows that companies have almost 70 to 90% open source in their open source stacks, in their product stacks, that is. So that's one of the reasons we do it. Because we use it, we need to support it, we need to get involved in it, we need to manage it from a compliance perspective. The second is, and this is more true on the AWS side, my team supports all of Amazon's stores, devices, Prime and AWS. On the AWS side in particular, our customers want us to support open source. They use our platform to do open source development themselves. So it's important for us to host services like managed Kubernetes or managed Postgres or Airflow or Kafka and things like that. And also to be experts at open source when we provide services. And the third reason is we feel that the world depends upon open source and we want to be a part of that global movement and be a good citizen in supporting open source. So building upon open source, I mentioned that it's foundational to our work. We have approximately, I don't know, between 70 and 80,000 developers across the company who build our products, who build services for everybody. And so imagine, as Mary said, imagine if every developer had to find the answers themselves about what is the license I should use, how should I contribute, how should I do this? It would take up so much of their time. So from an efficiency perspective, we wanted to create the OSPO so we can remove the barriers and delay from their work. So it should be really smooth from the time they intake a new open source project. It automatically verifies whether the license is acceptable. If it's not, it opens a ticket for the open source program office to verify. Otherwise, it goes through the build process, goes through the release process and goes through the entire process. If it's a distributed product, it actually comes to a point where in their program management process it says you need to do compliance, you need to do a distribution notice for this product. So they open a ticket and they come and talk to us and we provide guidance on the tools that they can use to scan and do the distribution and attribution notice. We approve and allow them to then go off and do an attribution notice. And as Mary was saying, we've also started doing champions because when you have 80,000 developers and you have like five or six open source program office people who help them, you cannot scale. So you really need to have a champion practically in every single product division who also know the identities of their own business. So the distribution challenges, say with an Alexa, is different than distribution challenges with a store or with something that we did on Prime Music. So we want to make sure that we have champions in those groups that work with us and help us do compliance faster. And the other area that I wanted to touch upon is risk management is good, but as Mary very well indicated, the lifeblood of the company, of any company's innovation, it's getting new products out to market. And you talk to any software developer, he or she, their happy moment is when their pull request gets merged or when they're able to get their code out or release out. It's not when they're doing a scanning to do a compliance document. So we want to balance the speed and risk. So sometimes it's automation. Sometimes it is tooling. Sometimes it is we centralize, we do work centrally on behalf of developers so that they don't have to do it. So lots of different techniques to remove some of the speed bumps and the delays from them. And frankly, open source is a core, core part of modern development today. So you have to build it into the CI CD process, into the development workflow. You cannot have it as an afterthought or an add-on or a bolt-on somewhere else. It has to be part of the development workflow. With regards to customers choosing open source, many of our customers do development using open source on our cloud platform. And they want us in every stage of the process to help them with choosing the right open source, knowing how to use our platform to do open source development. So we need to provide a platform that enables open source development. And a lot of them also want to customize our services so they'll send us a pull request. So things like user interface or tooling we provide on GitHub so that customers can then send us change requests on GitHub to use our services more effectively or to integrate it with their in-house services. And they want the cloud to be as easy for development as they want, you know, as they do on site. Being a good citizen in open source and for many, many years, I think the world was saying Amazon consumes but does not contribute and does not participate in open source, is not a good citizen. And what we have tried to do is to listen to that feedback and realize that we need to be part of the sustaining of open source. So you'll find that since the last, you know, 10 years we have been very, very visible in projects, in contributions, in supporting financially, in giving cloud credits, in really knowing that we need to be part of the solution. We cannot be just consuming and not doing anything. We contribute to thousands and thousands of projects and just like Mary was saying we have an external GitHub on which we release projects and host projects as well. And I will say all of us are on a journey to be a good citizen, an open source citizen because there is no kind of finish line. You're constantly learning how to, you know, improve. There is also no number which says you have to, you know, contribute so many times or you have to do so much work or so much money. So we are just on a journey on that. This is a pretty well-known picture in the Osbro community. It talks about the maturity stages of open source. Most companies will start with consumption. Then when they consume they realize, oh, oh, I've got to deal with licenses. I've got to deal with compliance if I distribute, et cetera. And so it's a mandatory thing so they end up doing that. And then they realize if I do a pull request or if I'm dependent upon a community I need to be engaged with that community so they get into community engagement, community education. But leadership truly, truly is about, I think, making open source kind of a seamless part of your company so that the culture is just embedded inside how you think, how you work, how you reach out. And so to me engineering effectiveness and being a strategic partner to the business so that when the CEO is making a decision he says, I want to talk to my open source people. I want to see how I can use that in this innovation strategy. So we want to be at the table of those important decisions and also build open source into the rest of the company. A couple of last thoughts. Today we burden developers too much. A study says that only 35% of a developer's time is spent on innovation. Almost 65% of the time is administration, paperwork, security, accessibility, compliance, administration, meetings. And so we really rob them of the opportunity to innovate on behalf of the company. And so we really, really need to make it easy. I remember in my previous company we would say, we need to take compliance all the way to the left and we need to make sure that developers are doing it from day one. But without giving them tools, without giving them education, without giving them support, you can't just throw one more task at them. And that's what a lot of us are trying very hard to do is to enable them to do it easily. And so really the nirvana is to go beyond compliance, making it delightful for customers to do their work every day, make it easy to comply, make it easy to use open source, and then automate, operationalize, centralize wherever it makes sense, reduce friction and delay, even if they have to cut a ticket. My dream would be, today we get almost 17,000 tickets a year, my dream would be zero tickets because just automatically things are resolved in the system or we have champions or people know how to do it. You know, with AI and with knowledge systems, you have got to be able to kind of self-resolve these things, right? And it should be a normal part of the development process so that we increase time for innovation, increase time for productivity, and help developers grow, and rather than having to do another checklist item. So with that, I'd love to invite Mary up here and take questions. I think we have time for about five to ten minutes. You take it. So the question is, who are these champions, what is their competency, and do they have dedicated time to do this? You want to take it? Yeah. As I mentioned, this in our case is not the full-time job like open source champion. So we start like, let's say, 10% kind of. We focus on the people who have the software-developed skills or system architectures this role so they know how the interaction with the internal code and the open source code. So sometimes you change it, remove it, or add on. So they know, because they're training, when we do the training, you know what kind of open source compliance you have. It depends on how you use it also, for this, let's say, LGPL and Apache, et cetera. When you modify it or not, it's different compliance work. So it's easier for them to understand it and easier for the compliance work. And it's also, usually, these people are fond of automation. So that's either part two. I think in our case, we particularly target teams that submit a lot of tickets. And we say, I think you need a champion in your team who can be a local help to you. And it could be someone who, in that software-development team, is responsible for doing compliance for that team. And we provide them a lot of training. And it's not dedicated. So one year, it may be 5% of their time. Another year, it could be 10%, 15%, depending on how many releases they do. We're trying to get to a point where we can keep them active and we can identify them in the system, in the outlook system, so that teams know who is a champion, has a star or something against their name and know who to reach out to. And also, we have an open source champion and an Osborne weekly meeting. So all of these champions can join this meeting regularly, weekly. Bring up your issues and questions to Osborne and discuss it together. This is a very good kind of environment for people to share to resolve the practical problems. Good idea. Shay. So the question or comment was, is champions a good way to create resilience in the business and also scale Osborne work across the company because Osborne budgets often get cut since it's a cost center. So I will say yes, absolutely. And I think Osborne should not be huge. They should not. They should be small groups of experts and their job should be to build it so much into the rest of the company that they're out of a job. But thanks to the complexity of open source, we'll always have a job. However, it should be. I'm not the manager. I don't have money, so I don't know. We really need these folks. The one challenge I have with champions is that they change jobs. And so constantly we have to keep training new people because they keep moving and they don't have relationship to open source anymore or whatever. Do you have another hour to talk? So the question is, and we both will answer, how did you get buy-in from leadership to build in open source compliance into CI CD pipelines across the company or even a central pipeline? How did you justify doing one central pipeline? And then how do you assist teams where they have contributors and they have to contribute to open source? How do you help those teams justify that developers need to spend 20% of the time doing open source when managers may not support that? I'll just take a quick stab at it. We are very distributed. We have teams who do so many different pipelines in the company. But there is one strong central pipeline for AWS in particular because the engineering teams for services want one central team to take care of security, SBOM, tooling and everything and not have to worry about the tooling themselves. So they have resourced and invested in this one central team that I belong to. I'm part of an Amazon software builder team. So they've given us money to kind of do a central BI for the whole company. So about 60, 70% of the company uses it, 40% use their own teams. And on the second question, we try to work with managers and business units to justify spending time to work with upstream communities. We basically say this is a big dependency for you. It's a business continuity issue. It's a risk issue you've got and more because that language they understand as opposed to saying it's a good thing. And you've got to work with your upstream community or else one day it will be gone and you will be holding a service with no upstream community. I really like your how because how is the problem result. I'll answer your two questions here. The first one is how you kind of make this into your CI chain. I think it's good that I have the CI experience. If the team, let's say, I just take one of this pilot project in place, what is blocked here? Because each company has your CI CD chain already in your team, right? How to insert this open source compliance part into it. Like integrate the tool, like scan into whatever, scan into to your CI chain, configure it and call your API to generate this result, where to store it in your existing system. So just connect all of this to them, help them to build their local teams. But let's say you have a different variant between teams, two teams. Take some example like you have the Garrett, you have your Zoo, you have the, let's say, scan also, BlackDuck, just take this as an example. So others can take this as a reference or build some common library just to write clearly what is the input and output, what is the parameter. So people can reuse it. For the contribution part, I forgot your question. The managers justify spending time, you know, their engineers spending time. Yes, if the engineer knows the value of the contribution, let's say upstream of their bug fix stuff, that need to be like support them positively. One important thing for us is like, how often they involved this, because this is all the open source contribution project that should have the record in your company or your offspring somewhere. Because you have this data in place, that is the next step you analyze which open source project is heavily involved in your company. Let's say you can take like 10% of this 100 open source projects and then focus on this 10%, maybe team A, team B, team H, use the same one. So you can gather this together to focus on this project. I agree with Mary. Some companies, including ours, will do a ranking of the top dependencies in the company. And at least for the top 50, you should have a good strategy for how you're working with that project. And then you've got to decide for the next thousands what you do, right? Whether you will have someone contributing or whether you will keep a tab on what's happening. Are we out of time or do we have time? Please. I think the Linux Foundation has done kind of an assessment of if you were to do the Linux kernel yourself, what it would cost you and how many years it would take. So to his point, you know, it's worth it. And from a technical debt perspective and all that stuff. I think this is a very hot topic. Yeah. I think this is quite a hot topic nowadays. It's, let's say, Sbom is not currently the Sbom 2.3 version. Sbdx version cannot fulfill the license compliance part totally. Cannot fulfill the security compliance totally. But we need it in place, right? For what reason? Maybe traceability stuff. So how to generate? I think many tools can generate this Sbom. But what information is there? We need some standard. Like each company can define your own, which data you need to put it. But with the Sbdx 3.0, now it's RC version, but hopefully we'll get it soon. Then it will integrate the license compliance part, security compliance part and AI data in it. So, and also this is very important. I think the developers probably need to get confused. Okay. All Sbom require this. Notice file and the archive, copy left archive. Security team comes here. You need to have this vulnerability report and blah, blah. Then the Sbom maybe from the product, or we need Sbom. But how you align these three different teams requirement into one channel, or in your development process, your CI CD, don't do it three times to get the same thing. Not the same thing, but you can use one or two tools to generate together. So that's very challenging part. I think it's a very interesting part. Thank you.