 Hello, everyone. My name is Tim Bird. I'm a Principal Software Engineer at Sony Electronics, and welcome to this session of Embedded Linux Conference. I'm going to be talking about issues with open source license compliance in consumer electronics. So I usually like to put the abstract inside the slides in case people come along afterwards, but you can see that online. And this is the agenda that I'm going to be talking about today. I'm going to go through a little bit of the legal disclaimer. It's important for a talk like this. I'll talk about aspects of compliance and give a little bit of the license history. Talk about some of the gotchas that you may encounter as you're dealing with compliance and then give some of my own personal recommendations. So let's start with this disclaimer. So first, it's important to recognize that the law is way different than code. And there are a lot of armchair lawyers out in open source. But it's important to remember, well, so one, I'm a software developer. I'm not a lawyer, and I'm specifically not your lawyer. So there are a lot of unexpected things in law that unless you have legal experience, you're not going to know about. So don't trust what you read on the internet or even what you hear from me. People will make strong statements about what's required by licenses, but it's just their opinion. The same same goes for this presentation. There are actually very few legal court decisions that have been made about GPL. And the language of the GPL in a couple of places, it's a little bit ambiguous. And I don't know of any open source license that includes choice of law, which means that it might be interpreted differently in different legal venues. And I know for a fact that German law in terms of licenses and contracts is a little bit different than US law. And so there's unexpected things that can occur in different jurisdictions. So you should always check with your own company legal counsel when you're making decisions that you think might have some legal risk. And it's really important for me to let you know, these are my views. I am not speaking on behalf of Sony Corporation, even though I've got the logo on my slides. These are my viewpoints. And so let me get started. Part of the reason for this talk is because I do have some pet peeves. I've been doing Linux for over 25 years now and I hear this a lot on forums and stuff. It's trivial to comply with the license. Just post the source code. What's the big deal? Well, it's actually not as trivial as it sounds. So that works when you have an in-house product and you're in complete control of all the source top to bottom. And that's true for some products. But there's a lot of other products where you're getting components from other places and it just doesn't, in particular with large products like a mobile phone. I think the last time we, there's one product at Sony that I measured the code on. We had 58 million lines of code. It came from multiple vendors. And in case of a mobile phone, you're talking about code from Google, MediaTek or Qualcomm and Sony. And there are a bunch of others in the supply chain providing software for drivers and sometimes black box components that go inside the product. So it's not as simple as having all of the code just in-house. And there are some business units that do black box or what we call white label products. I'm aware of some old security cameras that were just relabeled by the vendor and they'd come from a completely separate vendor. And some business units are not in the software business per se. So Sony B2B is a business unit that does IP and AV suites for stadium packages for stadium suites. And so they do not have the same level of expertise as system level developers inside some of the other Sony product teams. So it's much more complicated than people assert. But having said that, let's talk about what is required. So these are the aspects of license compliance. You have to know what's in your product. I'm going to talk about what types of things you have to provide and some of the requirements around those. So let's get into that. The first thing, and this is really important is you got to know what's in your product. If you're dealing with a large product, by large, I mean lots of lines of code, lots of components, then you can expect to maintain a spreadsheet possibly with thousands of lines of rows. And most people will look at a package, the list of packages on a website and say, well, there's not that many packages. It's like, well, but there are that many files and the license granularity is not necessarily at the granularity of the package. So you have to know what's inside your product. There are tools to assist with this, both commercial and open source. And you can go look at those. There's lots of resources to find out about those and how to use those. I have found in my experience that most often the problem comes from suppliers. So there are some suppliers out there who do not take their responsibility seriously. And there is a working group in the Linux Foundation called Open Chain, and they focus on education and certification. So there's a standard, and it's actually an ISO standard. It's either it's already been approved or it's on track to be approved, but for open source processes within a company. And this is really important because as products become more complex, you're getting software and hardware for more and more suppliers. And all it takes is one supplier to kind of screw it up in your supply chain. And so as Sony, we're a platinum member of Open Chain. We're very interested in seeing that this education and these processes and methods that improve open source compliance are used throughout the industry. I think it helps not just us, but it helps the entire industry. So the first thing is you got to know what's in your product. For in order to do this, it's very helpful if you can actually build what you get from your supplier. So that is interesting. Sometimes the software is embedded and so deeply embedded. And so it's not you can't do this 100% of the time, but it's useful if you can validate the source that's provided to you and that that matches the binaries on your component. So some of the one of the things that you have to do is you have to provide what we call attribution. And most licenses, even, even permissive ones like MIT or BSD require that you give someone notice about the presence of that software and possibly the copyright or the authors somewhere in the product text. And some licenses require this to only accompany the source and others require that accompany the product. What most, well, and so it used to be that we just put stuff on a paper manual, but as we've been used more and more open source, it's gotten way too long for that. So here's some of the requirements for different licenses. Different licenses like MIT or BSD require stuff either to accompany the source or they require that it accompany the binaries, which means the product. BSD4 actually required that you give notice in the advertising materials. That is a super obnoxious clause that actually the whole industry kind of worked hard to eliminate that. I don't think you'll find much BSD4 licensed software out there, but that's an example of, you know, there's a big variety of places where this, these requirements required you to put things. And so in order to simplify things, most companies will just gather everything. The licenses, the attributions, the copyrights licenses, and they'll put it on a single page in the device. And that's accessible by a menu item. And so if you look on your own phone, I have an Xperia phone here. So if you look under about and legal information on your phone, you'll see this huge long screen. In fact, the scroll bar I think is only one pixel big or two pixels. It's hard actually to get to the scroll bar, but you have all this information. It's included with the product. Now, so what happens if you have a product that has no display? And there are some products that have no paper docs either. And there are even some products that have no display, no paper docs and no packaging. So what people do, and I'm not, I'm not going to make any judgment about whether this is sufficient or not. But what people have seen people do is they put a web link either on the product packaging, if it has packaging, or if there's no product packaging, I've heard of people describing a web link on the product itself. So I think in that case, that's about as well as you can do. And then of course, at that link, whatever the website is, you have to keep that website up and you have to have the information all the relevant information there. The other thing that you have to provide is you have to provide source code or an offer for source. So the GPL gives two options for this. So let's let's talk about those two options. So one of the things you can do is provide the actual source code with the product itself. So the GPL says accompany it with the complete corresponding machine readable source code on a medium customarily used for software interchange. So some of your options here, you could include physical media like a USB stick or a CD-ROM. And I've actually seen examples of this and and those are fine. Some some products that you can actually put it on the internal storage in the product itself. And this is an interesting option. It does solve some issues. It makes certain other things easier. So you don't have to create a download site. You don't have to keep it populated for a period of time. You don't have to do physical fulfillment after the fact. Once you've done it in the product, you don't have to you don't have to keep doing it in the in the future. And you don't have to give the software to anyone who asks for it. You only have to give it to your product recipients. It's kind of baked into your product delivery. The reason that this is not used very often, there's a couple of reasons. One, there are products that don't have sufficient internal storage, right? We are talking about embedded products here, deeply embedded. But even even then, if you're going to do firmware updates, you're going to do those over the air. You can't be shipping a USB stick or a new CD-ROM with every firmware update. So there are size issues, even if you wanted to kind of include the source with the CBD update. So ultimately, this is a solution that is not used very often in the industry. So the other solution is to provide an offer for source. Instead of the source itself, you provide an offer. And the wording of the GPL here is accompany it with a written offer valid for at least three years to give any third party for a charge of no more than your cost of physically performing source submission, a complete machine-readable copy of the corresponding source code on a medium customarily used for software interchange. And that last part is kind of key. So most community members feel like an internet download site is sufficient. However, I've talked to a fair number of lawyers and most lawyers believe that this should be interpreted, that language should be interpreted as the medium used for software interchange in 1999 when the license was written. So offering only on the internet is not sufficient. So an offer of a CD or something, maybe a mag tape, I don't know, that'd be rude, is required. And then I want to take a little segue here. But it varies. So different venues may interpret this differently. So I already said the licenses don't include a choice of law. So something that may be made a decision in one country may not apply in another country. But if you're dealing with a large multinational corporation, which I am with Sony, you have to look at all the different countries. The license can be interpreted in different countries. Actually, in Germany, there is case law. There was a court case that said you must provide the option to obtain physical media. So there's a legal precedent there. And German courts have taken the GPL seriously, including details about the source offer. So you have to be careful. I just want to take just a tiny bit of a segue here or a detour. Certain effects or features of German law were used by one litigant to gain a lot of money with a particular legal strategy. So and I'm not going to go into the details of that. I have a resource page at the end of these slides that has information about that case. It was the litigant was his name was Patrick McCarty. And the bottom line there is don't sign any consent agreements or compliance agreements in Germany without consulting a lawyer and reading up on the strategy there. Anyway, I'll just leave it at that. So there are places where you need to be careful with your interpretation and your fulfillment of the license requirements. And then even within a single country, you can have different contributors to a project who interpret the license differently. So it's not as clear cut. Fulfilling all the requirements is not as clear cut as a lot of people say. In terms of duration, so how long do you have to offer the source code? Well, it's not three. So you have to offer it from three years for the last shipment of the binary. So you're distributing a binary to someone and it's three years from that date. But that's not the last shipment of the product. If you do over there updates, you're going to be providing updated firmware and that's going to be another distribution of the binary. And it doesn't mean three years from the date that you last put the firmware on your website. It's three years from the last from the time someone last downloaded the firmware. So the firmware could be up there a really long time, even after the lifetime of the product. You have to keep providing software so that people could obtain it within three years after. So this is usually, however, after having said all that, it's usually not a big problem. There's virtually no incentive for companies to take software off their sites and stuff just hangs around a long time. So technically, you could be removing stuff, but most people don't do it. So the duration is not that big of a deal. And then I'll just make one observation. If someone is asking you for physical media, they are most likely a person who has more interest in your compliance and how you're doing with that than they do with the source because if your source is out on the internet site, that's easier for people to obtain. So I just a red flag handle requests for source promptly and carefully. So now to the source itself, what must the source include? So for the GPL license, which is the main license that requires people to distribute source says, for an executable work complete source code means all of the source code for all modules it contains plus any associated interface definition files plus the scripts used to control compilation and installation of the executable. So there's a term used elsewhere in the license. And by most lawyers, there's an acronym that's used complete and corresponding source code or CCS. You'll hear that for short. So the GPL also says the source code for work means the preferred form of the work for making modifications for it. So that means you cannot deliver obfuscated code. And it needs to be something that actually can be worked with by human in terms of modifying it and rebuilding it. And it's unfortunate, but I actually have seen examples of a supplier providing obfuscated code. I'm not going to name names because they came clean and provided unobfuscated code later. So we treat that as an anomaly. But you should not do it. It should be something that a developer can actually use. Now about rebuilding. So does the definition of CCS require that a recipient can reproduce binaries exactly? And I would argue no. So it's very difficult to reproduce a binary in exact form. In fact, there's a whole not industry, but a whole movement to try to do reproduce builds. And this has been a source subject for decades. So I think that's too high of a bar to require of a vendor. Does the definition of CCS require distribution of compilers, linkers, IDEs, or proprietary tools? No, it does not. You don't have to provide the compiler. You don't have to provide any proprietary tools. It does raise an interesting question in my opinion. And I'm not going to talk about this a lot. But is it compliant to use proprietary tools to build open source software? So can I use a commercial tool chain? Yes, you can. You can use a commercial tool chain. But you can keep doing this thought experiment for longer and say, well, is it compliant to use tools that are unavailable to the public? Can I make my own tool and use that? I'm not even going to get into that. I don't know. It's never been tested to my knowledge. And people don't actually even talk about it. But having said all that, my view is that what is required is the scripts used to control compilation and installation. And this means auto tools, configure scripts, config files, and makes files. And that's sufficient for most packages. Now, another issue that comes up sometimes is reinstalling. Because the license requires scripts used to control installation, it has that phrase in it. Some people interpret that to mean that you need to be able to install the software on a particular device. So license does not actually say that. It doesn't mention the act of installation or the target of installation. It just says the script used for installation. And I'll talk a little bit about that. So industry practice, in fact, fully supports the use of GPL software in read-only memory or ROM. So you can put GPL software in a ROM where it cannot be reinstalled in place on the same device that it was delivered on. And that's been a practice that's been supported. And there are even quotes about GPL reinstallation not applying in that case. So I don't think the license requires reinstalling. So if you go back and look at the license history, it's pretty interesting. So BSD licenses kind of came into effect around 1988. GPL actually, version one came out in 89. And it was a generalization of the Emacs license that Richard Solomon had been applying to some of the software in the GNU sweet or GNU project. And the history is pretty interesting. GPL v2 was released in 91. And it was adopted by Linus Torvalds. And then GPL v3, there were some perceived flaws in GPL v2. And Richard Solomon and others worked on a new release of GPL that came out in 2007. There was a long discussion period on that several years. And for a variety of reasons, including not really agreeing with some of the intent behind the license, GPL v3 was not ever adopted by the Linux kernel. And so I want to talk here about interpreting the license. I think it's important because at this point, people will start talking about, well, what is the intent of the license? And first, the overriding legal principle is that the intent only matters if the license is unclear. So a license does not have its own tent. It is not a sentient being. License is an expression of the intent of the copyright owners of the software. And it's not, it may reflect the intent of the author of the license. But that's not the person who is making the agreement. It's the copyright owners of the software who are making the agreement. So the problem comes up with Linux and with other projects that have multiple authors, as well, whose intent matters. Is it the leaders of the project? Is it anyone? The Linux kernel now has about 14,000 contributors. So I'm sure that there's probably a bell curve of different intents for how the license should be interpreted. And, but I will, I will say this. So Linus Torvald's intent for the kernel is very clear. He has said on numerous times that the intent is tit for tat. And that expression means that I give you something, you give me the same thing back. So if I give you source code, you give me source code back. And again, that's been, Linus has said that a number of times and a number of different venues. And I think that's, so I believe that that is his intent. I've seen other leading kernel developers agree with that interpretation. So what are the license requirements? And again, this is my view. I'm not speaking on behalf of Sony Corporation, but my view is you give people the source, all of the source, including make files, configure strips, config files, customers should be able to build the software. But there's no requirement to exactly reproduce what you've built. And there's no requirement to reinstall on a particular device. So if we have, if, if a vendor has made fixes to bash and you think those fixes are good, you can apply them to your version of bash and use it on your desktop or anywhere else. If we've put a driver in for, for a chipset or a feature on a chip, and then you can take that feature and use it on the phone of your own making. So vendors are not required, in my opinion, to produce hardware that can run binaries that are produced by third parties. So bottom line, I believe secure product lockdown is allowed by GPL v2. And I don't think I'm alone in that requirement, interpretation of the requirement. So GPL v2, GPL v3 came about because of, because of that interpretation, I believe. So if you look at GPL v3, really the requirements there are sought to impede lockdown and change the terms. And GPL v3 does actually require keys to install the software on certain products. And most CE vendors, including Sony, strongly avoid GPL v3 for this reason. So there are a couple of different views about that. One is that the GPL v2 had that intent all along. And the changes in GPL v3 are just changing the wording to strengthen that. But another view is, well, if the GPL v2 incorporated that requirement, you wouldn't need to change the wording. So GPL v3 is a different license. And there are some people, very clearly some people who have been vocal that that was not their intent to disallow lockdown. So take that for what it's worth. And the final thing here is that many projects adopted the Linux kernel and GPL software based on the wording of GPL v2 and not of GPL v3. So you can't hold them. I don't think it's fair to hold them to to the standard of GPL v3 or to the requirements of GPL v3. And even Richard Stallman has said it's evil to change the terms of the deal and expect people to abide by a different license than what they agreed to. So now I want to talk a little bit about gotchas. There are some nuances that can come in and bite you. So what happens is languages change over time and projects change over time. And these are two sources of things that you really have to watch out for. So MPL stands for Mozilla public license. And over time it changed its wording slightly. And so as a vendor, if you're using MPL software, you need to pay attention to the license that's on the version of the code that you're using, the version of the license on the version of the code you're using. MPL 2.0 has a complete and corresponding source code wording that's more similar to the GPL, an MPL 1.0 or not. Also, well, a similar issue with GPL v2 to GPL v3. So there are packages out there that are widely used packages. And this particular one is owned by the Free Software Foundation that was originally written under GPL v2, but then switched license to GPL v3. And it's not the packages themselves. You can go back and find those old versions of the packages. But the interesting issue comes up with what about patches to that project? So for instance, there's a very interesting and useful patch to plug a security vulnerability and bash called shell shock. And the patch for that was submitted simultaneously by the same person or same organization to version 4.3 of bash, which was GPL v3, and version 3.2 of bash, which was GPL v2. So what is the license of that patch? Well, I would say it's the license of the project it was contributed to, but it gets more complicated than that. So you can envision places where people are extracting patches from versions of the software that have a different license and can they be accepted in backports. So there's a lot of factors to that. That's something that you really need to look out for. So just be really, really careful in handling of any packages that are GPL v2 that you know have switched to a different license. And then there's another example. There are other examples where it's not trivial to figure out the license for all of the outputs from a package. So for instance, in Util Linux NG, it's mostly GPL v2. And if you look at the copying file that's at the top of the source code directory that says GPL v2, I think it may now say GPL v2 and some other stuff. But you have to go look at the headers that are on individual files because there are a couple of files that are actually GPL v3. So you have to be paying attention and looking closely at this stuff to make sure you don't get caught. So this is the part of the presentation. Well, unlockability in my opinion is not required by GPL v2, but I think it's a topic that a lot of companies, CE companies are interested in and there's a lot of discussion in the industry about it. So I'm going to talk about it anyway at my own legal and reputational peril. So you can have, there's a spectrum of unlockability, right? So you can have a fully unlocked product, a not unlockable product, which is a lockdown product put in more positive terms with a double negative. Or you can have something that you can unlock and then reinstall the OSS only. So let me talk about those. A fully unlocked product is something where, okay, I'm talking about kind of a specific class of products that have the ability to stream content and or and or the ability to connect to a backend service. So I'm not talking about some little IoT device that doesn't have a whole bunch of other considerations that's like I got 100% open source stack. That's that's not what I'm talking about. I'm talking about a product with a mixed stack consisting of proprietary and open source software. So a fully unlocked product would mean that that you you can reinstall all of the software and retain all all the keys and connect the backend services. Okay, so first, this is not required by any open source license, including GPL v3. So open source licenses only cover software that is open source, it doesn't cover the proprietary software or the or the streaming keys or the backend keys. Those are not those are not part of the open source. And for some products, not for all of them, but for some products, it's simply not supportable as a business proposition. And so the example I'll give out here is video game consoles and network play. So if you can unlock the product and you can alter the operating system, then trust me, you can figure out a way to cheat in online gameplay. And that's kind of the reason that people do this. And so that is an example of a product where is the the ecosystem relies on there being a protection of the capabilities of the individual nodes on the network, or to make it fair for other players. And so that's part of the reason people buy the software is when they do network play, they don't want people to be playing unfairly. So it's actually kind of baked into the whole business model and and what customers want. So a fully unlocked product, I think is unrealistic. It only works in a world of perfect and honest actors. And that is unfortunately not the world we live in. So let's talk about the opposite end of the spectrum, which is most C companies locked down the product and just don't look back. And so you can argue whether that's good or bad. But the security for this type of thing is often done in multiple layers. You've got encryption for the update images themselves, the things that are transferred from the back end over the air to the product for updating. You've got on the device, you've got trusted execution rings and trusted protection modules that are holding the secret keys and stuff. You've got encrypted file systems. And then you've got kernel module application signing. So everything's verified before it starts. So these products are really locked down hard because the industry has shown itself, well, certain sectors of the consumers of these products have shown themselves to be pretty agile at breaking in and doing stuff. But there are counter examples, right? So I know that there are some experienced phones that are unlockable and some Google Pixel phones, not all of the phones, but some of them. So it is possible to do some level of unlocking. You don't have to lock everything up. And so this is a product I'm going to describe here that allows you to unlock the product and reinstall only the open source software. And a couple of notes about this. So this does give consumers the ability to install modified open source software while at the same time protects the interests of the content industry and consumers who want to stream content. If you, anyway, so there's no obligation to make proprietary software work with open source software. That's not in any license. And so what that means is it may require the removal or disabling of proprietary software. What that ends up doing is, so the device, once you've unlocked it like this, you may lose quite a bit of functionality, depending on the stack. It's heavily dependent on what items are in your stack. So the customer, the bottom line is customer demand for this is very low. This is not your average consumer that's trying to break into their TV set or their phone or whatever or their streaming device. Because the customer demand is low, it's a very hard sell to your business and so there's just not enough, you know, customers to make it interesting to do the difficult engineering work involved. This can be a one-way action. That's most often how it's done is that once you unlock the product, you can't relock it again and get back those proprietary software, those content keys. That is actually technically very, very difficult once you've removed the content keys. There are some places in those layers of security where right once operations are performed. So you can only, you can't actually rewrite the encryption keys for some flash devices, for example. So, okay, so having said all that, let me just dive into what I recommend. So this is kind of the summary wrap up portion here. These are the kind of main things that you need to think about and what I recommend you do. So first in terms of communication, please, please, please communicate with your upstream suppliers and do it in advance. Don't do it as an afterthought. Get your suppliers involved with OpenChain, the OpenChain project. Make sure they have processes in place. It is sad, but I'm aware of several major vendors who have had what I call OCRUD moments where they found out a supplier was not giving them the right stuff or was not giving them any stuff and that's a really awkward and difficult situation for a vendor to get into. So some of the ideas I have are you can put penalties in place in your procurement contracts. Again, this is for kind of large suppliers who have procurement contracts. You can put penalties for failure to disclose open-source software or you can require permission in advance to deliver components with open-source software. There's a couple different ways to deal with that. I think it's good if you can get a license manifest that talks about all the files on your device and try and get the license for every single file. Again, don't do this as an afterthought. Don't have this be during your QA of your product. You're assembling all this information. Try to get it up front. Another thing that you can try, I don't think very many companies do this and I'm not going to assert that Sony does it in all cases, but you can try to build what you get from your supplier. Ask that your source be accompanied with a script that doesn't start to finish build. And one of the things you find is that products have a lifetime. If product has life in five to ten years, the build environment changes substantially over that time or what's the kind of the current distro and so it becomes obsolete and very hard to reproduce. I recommend Dockerfile to capture things like the host OS, the tool chains, and other programs. If you have an internal team, if you could have another team besides that team kind of test out the build, this is highly aspirational. I'm not saying anybody does this now, but I think this would be a good thing for the industry to move towards. In source availability, this is pretty obvious. Make an internet download site where people can obtain the source for your products and there are some people who sometimes play games with only trying to allow just customers to get to it, but I don't see a reason for that. Just make the source code available to all and then you do need an offer for physical media. You won't get much traffic. You won't have to find yourself burning a lot of CDs, but you should make sure you have one and that it works. In terms of unlockability, in my view, unlocking your product is not required by GPL v2, but it does make community members and the customer is happy. You may decide that you want to do that. There are good reasons to make an unlockable product and so independent of what your view is of the license requirements. If you support unlocking, then if you handle backend services or streaming content, anything protected, you'll have to figure out a way to deal with that and it's more difficult than it sounds. If you don't support reinstallation, you must avoid GPL v3 soccer. That's my opinion. Final statements. First, I hope I haven't scared you with all this talk. It sounds like a lot of work and there are some eyes to dot and teased across, but it is possible. People have been doing this for many years. Compliance with the license is required. It's not like it's an optional thing. It's part of the payment that you're making for the open source that you received. You've gotten an awful lot of value as a vendor from open source software, from the developers to put time and effort into it and giving back to them what they ask for is appropriate and legal. It's not trivial, but it is achievable. Again, I'm going to make my disclaimer at the end here. Due to legal uncertainty, make sure you make decisions based on kind of knowledge of the risk and with your own company's legal team. Then my last thing is a plug for the open chain product. You really should check out the open chain project. I have a little four leaf clover here on here. I hope that the information in this presentation has been useful to you and that you'll benefit from it. Good luck in your compliance activities and I hope to see you around the industry. With that, I will go ahead and take some questions.