 Hello and welcome everybody to our presentation on scaling open source software in a large tech R&D environment. Unfortunately, we're not able to join you in person at Ospoke on Europe, but thanks to support from Linux Foundation, we at least able to contribute to the event by means of our recorded presentation. So whenever you have a question or would like to discuss one of the topics we bring up here in more detail, feel free to reach out to us using the email addresses that you can see on the slide here. And we'll also show them at the end of our presentation. Who are we? My name is Geo Kunz. I recently joined the newly created Ericsson open source programs office and I'm a contributor to multiple open source communities and I'm on the technical steering committee of the Linux Foundation Networking Project, NUCHET. And I'm Godan Ilson. I've been working at the CTO office of Ericsson with open source related questions for at least 15 years. And primarily with compliance and how we actually engage in the sense of consuming open source, but also with going out in different strategic initiatives and contributing to different projects. So I'm pretty happy to see that we actually are now going into an Ospo setup in the CTO office going forward. So let's take a little quick look on the different paradigms that we have seen at Ericsson. And of course, this picture is rather simplified, but nevertheless, I mean, if you go back a number of decades, Ericsson took use of different open source bits and pieces, but this was very much on technology supply activity. We saw these things as smaller components going into our proprietary offerings. And roughly at the millennium shift, we started to take use of Linux as the operating system and swapping out proprietary operating systems and for that sake, other commercial operating systems. And by the use of Linux, we also got the full Linux ecosystem, which basically meant that we saw more and more open source coming into Ericsson that wasn't just these bits and pieces that we previously had seen. And 10 years ago or 12 years ago, perhaps even we saw virtualization coming in to the industry, we see containers, we see cloud native, we started with SDN controllers. We have seen OpenStack, we see cloud native computing today. And of course, these things even more put the efforts on that open source is so much more than just these small technology supplies. It's actually full technology elements that can almost stand by themselves and execute and do different bits and pieces. So our thinking then when it come to technology supply was very much that we had to find their keeper's mentality basically we took a copy, we kept that in house and we were happy with that. We didn't really consider so much what happened in the community itself. When we started to use the Linux, we realized very much that this was not a free lunch. There was a lot of these components that actually needed some sort of caring. There was also connected to costs of using open source in the sense of partly that we acquired Linux distributions from commercial offerings or commercial vendors, but also that there was all these bits and pieces that needed to come together to form the bigger thing. And when we now see the virtualization or things like cloud native, it's rather obvious that we can't just take it and keep it in house. We actually have to join up in these different community projects and we need to be there actively. And as I see, this is something that we will continue with going forward as well within Ericsson. Good, let's take a look at the driving factors behind that particularly last paradigm change. And when you look at the typical telco technology stack before virtualization came along, it very much looked like the three examples you see on the left-hand side of the slide. So technology stack is basically composed of a collection of very tightly, vertically integrated, specialized pieces of hard and software that basically made up the boxes that we shipped to our customers to make up the customer's network. Now with virtualization and later on cloud native coming along, what actually happened is that those proprietary and tightly integrated layers got broken up, disaggregated and more or less all of the layers got replaced by open and industry standard hard and software components. And these components or these layers typically expose also open APIs that get naturally defined by the very open source projects that develop and provide the software that make up these platforms. So this has basically resulted in very much tight chain change in how we build our products. Now it's very much disaggregated and based on open APIs. And as a result of that, we of course became active in all or let's say in a lot of projects covering all of this technology stack. And even though on the slide, I just show a subset of all the projects, I also don't want to go through all of them but highlight just a few of those. For instance, at the bottom with regard to the industry standard networking layer, we have been very active in open daylight which has been the core of our SDN controller for virtualized cloud environments. Now we also investing heavily in network service mesh which we're using to realize complex telco use cases in containerized environments. And obviously we are active in Kubernetes, we are active in OpenStack holding, for instance, PTL positions and key projects such as Nova or Neutron. And then of course the very telco specific thing is OWNAP which builds and defines an open source orchestrator for telco networks. In addition to the technology stack, we also active in terms of the tooling part around this. So for instance, supporting development environments by contributing to thea or eclipse or also contributing to other initiatives and projects such as Eiffel or the Continuous Delivery Foundation which built and defined processes for building COACD pipelines. So as you can see, this technology change has resulted in us becoming very much reliant on open source software and becoming very active working towards it with those open source projects and communities. So before we start to jump into open source and all of these bits and pieces, let's take a quick look of what the telco industry was before open source or rather in parallel with open source all over the place on these other bits and pieces. So telco is very much standardization activity in the sense that we have proprietary development and we unite that proprietary development and we get that to interrupt it between each other through standardization. And these proprietary implementations that are resting on these specifications and specifications are the thing that we very much relate to when it comes to how do we specify things or how do we get things to work towards another vendor or towards a customer or towards a supplier and so on. And also a big building block of all of these things is of course that these things happen to be brand based i.e. you initiate inventions and patents on these things and then you of course also can have cross licensing of that which has somehow formed this industry as well. The horizon for all of these things are of course very long-term in the sense that you start to do a specification and that specification can take X number of years and when that specification is ready you start to make an implementation of that and eventually you reach a market with these things then of course these things can happen in parallel but nevertheless it is a long-term activity that is going on in this thing and compared to open source there's pretty large difference in that sense. And then of course it's still an open thing to do standardization in SDOs but it's still a limited group of participants it's not open in the sense of open source that anyone can go to a community and start to hack away on code but this is more in a limited set and the challenges for an OSPO then in all of this or rather the responsibilities of an OSPO would of course be then to accept and see that these things still exist i.e. we need to perform a risk management of the patent portfolio we have to understand that we still have a portfolio of that and that portfolio needs to be able to live in parallel with an open source activity and thus we also then carefully need to do these evaluations and clearance of contributions that we are pushing to open source this can sometimes be cumbersome and this can sometimes be very easy that all depends on what sort of licenses is there at hand what sort of ecosystem does this all spread to etc. But the culture here is very much also that when you then suddenly come into open source and you bring open source into this scheme of SDO thinking and specification writing of course the initial thing is that there's a limited understanding of how do actually open source work what are the mechanisms that do make open source tick in the sense of that communities attract different developers developers work together towards the best possible development of functional software and instead we have had a very much an inverse facing approach to technical challenges in the sense that it's better to solve those technical challenges on a proprietary system and thus take use of it as a proprietary feature rather than actually go out and in a collaborative way join forces to find the best way for that technical problem or technical challenge and then of course another thing which we I think we can see with many open source companies or companies that are more have over longer time embraced open source there's a huge marketing potential when it comes to open source in the sense that you can actually make a difference by showing that this is an open code you can actually enter yourself anyone can come in and improve this code and we are happy to share this code with whoever do come to that community and from an Ospo perspective then this is very much that we need to define open source strategies so that we can push people to open source we need to provide training and advocacy of these things I we need to get people to understand that okay I can actually be an open source there's nothing dangerous with that like we can train management to also understand that this is a good way to do things we can actually accomplish things going forward in all of these bits and pieces and all in all this is very much for us to act as a cultural change agent when it comes to changing the culture changing the perception of open source and getting open source into the heart basically of Ericsson in the sense of that it's on equal terms as any other SDO activities that we already today are doing because if you look on these two things standardization and open source they are not really opposite to each other in the sense that we have to choose either or it's more the opposite in the sense that they complement each other we will do things in standardization for sure but we will also do things in open source and the important piece here is to actually find the connecting points here when it comes to how can SDOs work together with open source and vice versa and for sure I mean the more of these connecting points that we actually find the more both these parts would remain relevant going forward and here from an OSPO perspective we very much are the ones that then and this is a bit of a cultural thing but we are then bridging the gap here between standardization and open source activities being the talking point when it comes to telling telling the organization that open source plays a vital complement to SDO and vice versa So moving from the Teiku industry in general to Ericsson specifically I'd like to first put Ericsson a little bit into perspective here by giving you some numbers about the company So certainly Ericsson is a global company we're active in more than 180 countries worldwide We have about 100,000 employees and a quarter of those more than 26,000 of those are working in R&D We hold almost 60,000 patents and those posts a particular challenge that's just explained by Gunnar and across all of the R&D organizations and groups that exist in Ericsson like on a yearly basis we have around about 20,000 fossil registration events happening per year So what does that actually mean? Of course we're tracking each and every version of an open source software that we use in any of our products and registration is either by bringing in a new component and registering it in our database or by registering a new version of it whenever we bump up a version the new release gets consumed by one of the product organizations So you can see 20,000 such updates is a lot and that basically and logically results in another set of responsibilities for the OSPO Obviously we need to define and create processes that are highly scalable and that we can give to our development organizations to guide them, to enable them to still move with the speed and we need to also ensure that we create synergies and somewhat align potentially very different strategies that exist or can exist across the different business areas that we have in Ericsson So that we basically have a consistent way of working with the key open source projects that are important for our products and hence we need to guide and help all of the people basically across different development organizations and business units Now I said that we have roughly 26,000 people working in R&D and of course R&D is not a homogeneous group Obviously there are different types of organizations within R&D focusing on different aspects of the technology work they're doing And for the sake of this presentation I'd like to categorize those organizations into three different classes They are classic product organizations that build and ship products nowadays with a lot of open source software in those products But they are also research and technology organizations Research doing things which is like a couple of years ahead of the product organizations And the technology organizations somewhat in between product development and research maybe one or two years ahead of the product However, the common combining factor is that all of them are kind of working with the same piece of open source software But instead of productifying that and shipping it as a product as the product organizations are doing it Research, technology and solution organizations they build prototypes, they run trials they come up with experimental features And as part of all of these activities the open source software they use is often modified to make stuff work But typically, and that's a cultural thing those modifications are typically done internally and also kept internally So there's not that much outward facing in that regard And now from an Osbro perspective we think that there is a lot of potential in changing that and creating awareness for cross-organizational synergies that we can leverage because they are enabled by using roughly the same pieces of open source software So it's one of our responsibilities to foster a cultural change and to make those other organizations actively engage in open source software What would the benefit look like? So I'd like to explain that in a bit more detail So here let's first start by looking at classic requirement flow So let's assume we have all various organizations here and they do all do their thing So research comes up with some study results the technology organizations come up with some improvement proposals Solution organizations have additional requirements and all of them basically feed their requirements into the poor product organizations and they need to take this in they need to prioritize these and then act upon this and the product organizations at the end of the day obviously need to build a product which in our case here includes a significant portion of open source software Now since resources are always limited some of these things just fall into the cracks and just get lost Now the important observation that we can make and that we'd like to convey to all of these organizations is that due to open source software being part of a product this open source software naturally extends the surface basically where all of those organizations can work towards So if research actively engages with an open source project and drives them to implement new features or to adopt whatever new experimental features they have come up with At the end of the day whenever a product organization consumes a new version of that particular open source software we naturally inherit the new features that have been developed jointly by research for instance and the corresponding open source community and the same is true for all the other organizations as well So the key observation is that open source software allows us to parallelize this process and thereby the key benefit is that we can actually reduce time to market by leveraging all of the organizations we have in R&D if they jointly or independently but anyway working towards and with the same pieces of open source software that's out there Now looking at the current state of this in the company to be honest with you we have successfully tested that in practice in a small subset of the organization and it worked very nicely and now we need to scale this basically up to to avoid R&D organization and that's part of the cultural change activities that we need to drive Yes and let's have a little look on some of these current challenges that we are facing today Let's start with looking a bit on the outside pieces I mean if you look on compliance like going back 15 years for instance it was very much a legal and licensed compliance activity I mean it was a matter of understanding which licenses were included and then we fulfilled this when we distributed something to a customer Usually a programming language was C or C++ etc or Java for that sake Today I would say that we are shifting more and more into the security and traceability that doesn't mean that the legal and license is not important anymore on the contrary that's equally important but there's a lot of focus on these security parts and also the traceability and this comes a bit from that complexity increases I mean we have container images we have languages providing large dependency trees we have an uncertainty of what is included in these bits and pieces and what is making them up basically is understanding if we have a full insight to traceability of the source code etc Cyber security itself is growing as an issue in the industry I mean there's more and more scare that we will see things that pop up and harm us both from a build pipeline but also from the actual products that we distribute and of course all of these things together means also that our customers expect much more today and they expect that transparency when it comes to what sort of open source do they actually get with the packages that is delivered from us and thus this customer transparency becomes more and more important as well going forward so let's see how these compliance challenges map to the development flow overall so to abstract that a little bit obviously the the intern development flow consists of three separate domains there is the upstream community there is Ericsson and its internal processes and R&D organizations and there are our customers and then across those three domains there are a set of separate processes or steps along the development flow and those steps interact obviously with each other in upstream obviously there is upstream development happening then we take in that source code into Ericsson we stage it we package it such that then the product or development organizations can compose the products out of both the packaged open source components and the proprietary components and then finally we ship the readily composed product or we feed it into the product delivery process to ship it to our customers and then as Gunnar already mentioned the challenge the key challenge today is to really understand how can we ensure and understand integrity and traceability across this entire end-to-end development flow looking at traceability and integrity specifically we're starting with integrity rather we need to make sure that we always have a clear understanding of what's in the open source software that we consume where do we starting at basically at where do we get it from what's the right repository when we fetching it getting it into the company we need to understand how we handle it throughout the development flow how does it get packaged potentially changed along the way so that we basically at every step know exactly what's in it so that whatever we ship to a customer in terms of a product we have a good understanding of what's inside actually I was not only talking about open source software but we rather need to differentiate between binaries and source code so obviously integrity handling integrity is a little bit easier if you just basically consume the source code and then build everything from scratch package it and shipping it to their product composition step however that is obviously a huge burden in terms of time and resource consumption on the development organizations so since they want to act with speed they have a desire to also consume binaries and as going to mention before container images have become one of the prime resources of delivering software and they are mostly naturally composed of binary so what we need to come up with is the right tools and the right processes and methodologies to trade off basically or find the right balance between consuming binaries to give flexibility and speed to organizations while at the same time ensuring integrity so that we know what's actually inside a product that we ship in addition to integrity we also need to ensure that whatever we do along this development flow at each and every step we have traceability so that for every product that we ship to a customer we can actually trace back to the very specific version of an open source software component basically what's inside of the products and in case of vulnerability showing up we basically know which products are affected and where it came from and which version of open source software we need to to fix in addition to ensuring integrity and traceability we also need to streamline the false compliance handling process I mentioned the 20 000 false registrations a little bit earlier in this talk and obviously here it means we need to balance against speed allowing development organizations to for instance easily move to a new release of an open source software component they consume versus being very stringent in terms of the compliance handling and basically doing all our due diligence checks whenever we move to or register a new version of an open source software component and the only solution to that is basically automation automating this as much as possible to minimize the the manual interaction here and from this basically we deduce again further responsibilities for us in the hospital we need to again define scalable processes and policies methodologies and develop tools that leverage automation as much as possible and we need to adopt and understand industry best practices and bring them into the entire organization now talking about best practices what are those right so the the challenges that I just mentioned it's not that we kind of told you how to solve this it's really to some degree open challenges for the entire industry but still even though this is a challenge created by open source software we believe that the best approach to solving this is using an open source approach and an open source mindset so as it is an industry challenge the industry needs to come together to jointly solve these issues in particular since there's no single silver bullet to it that would solve all of the problems at once but instead in good open source fashion there are various open source communities out there addressing very specific subsets of those challenges and and again that's the final responsibility for us in the hospital what we need to do is we need to ensure that we engage in the right open source initiatives such as open SSF open chain continuous delivery foundation spdx to just name a few to make sure that we evolve the approaches developed there so that they meet the requirements of the industry and that they actually complement each other and solve the industry's needs overall so that takes us to the end of this talk maybe worthwhile to just recap some parts of what we have been talking about as we said I mean technology evolution is something that drives open source culture and what we mean with that very much is that that basically it takes time for open source culture to grow into a company that maybe have a proprietary portfolio initially but but it helps that open source actually drives technology evolution in different bits and pieces and we see that there is a complement between open source and standardization thus they are not like either or as we talked about previously they will be there together just a matter of finding the best connecting points and bridging that gap that we talked about of course a very important part and maybe a key thing for us as an Ospo people in Ericsson is to to leverage the mindset across the full r&d organization and and actually beyond the r&d also I mean we there's a lot of people that will come into contact with with open source also outside of that and then coming back to the previous slide that Georg was talking about I mean super important that we we see I jointly addressing these key challenges of security and traceability I I don't see that we can solve all these problems by ourselves and and neither can anyone else probably so thus this is definitely something where a community can step in and add and help and actually produce results out of so so with that I just want to state what Georg already said on the first slide I mean feel free to reach out to us if you want I mean we are there via mail or via Twitter for Georg then and please please stay in touch with us or reach out to us in whatever way you find appropriate and we will try to answer you as soon as possible thank you very much thank you