 dylai'r rhaglaeth yw'r cyffredin? A dylai'r rhaglaeth yw'r rhaglaeth? Hyddi'r dros yng Nghymru Johnathan? Ac mae gweld Cytrix. mae'r ysgolwyr ar gyfer yn gyfan yw'r ysgolwyr yn werthon. Mae'r ysgolwyr cyffredin Cytrix. Mae'r ysgolwyr nawr ymlaen o'r lluniau'r ysgolwyr. Yna, yw'r hefyd... Ychydig? Ychydig. ystod o'r rhannu gyda'r project yw'r ystod o'r hynny o'r hynny. Rwy'n credu bod wedi'i gael ei ddwylo'n hynny o'r rhannu. Ystod o'r rhannu yn 2004, mae'r Ystod Cymru yn ystod y bwrdd Cymru yw'r ystod y gyfnod y Zenn. Mae'r ystod y bêr metal hefyd. Yn ystod, mae'r hefyd, y Zenn, Acedemics had produced it, Ian Pratt, a Keir Fraser. They created a company to monetise and to commercialise the hypervisor. That company was called Zensauce and their commercial version was called Zen Enterprise. As you can imagine, it contained the features which would be useful for an enterprise. The kind of features like resilience, high availability, failover, that weren't available in the basic bare-metal hypervisor. That company must have been successful because in 2007, Citrix, an American software company, bought Zensauce and renamed the product to Zenserver. Zenserver is the name that the product was known for for many years. The open source Zen project continued to grow and in 2013, it became one of the Linux Foundation-sponsored projects. The latest historical mark on this chart is that in 2019, Citrix decided to rebrand Zenserver as Citrix Hypervisor. This will explain why you won't be able to buy Zenserver anymore. We're now in 2022. You can see from 2007 Citrix has had a 15-year history of building and selling a commercial product based on an open source upstream. I joined Citrix around two years ago and I was curious about our relationship with the upstream community because it seemed that the relationship with upstream was slightly uneasy. There were traces of past strains, if you like, the scars of something that had happened. I was intrigued what had gone wrong. In fact, it was the call for proposals for this conference which gave me a chance to investigate because I submitted a proposal to talk about what had gone wrong. Once that was accepted, I had to find out. I needed some data for this talk. What I did was interview people who had been there at the time and listened to their stories. I interviewed six veterans of the Citrix-Zen relationship. Each one of them had 10, between 10 and 20 years, experience of working in Citrix on products derived from ZEN. I talked to both current and former employees. Some had left, moved to Amazon EC2, which has been another user of ZEN project. What I've got for you is a synthesis of learned experience. This isn't an academic study, it's purely subjective and it's based on people's memories. Some of the stories that people told me were very technical and specific to ZEN and I'm not going to be talking about those things today because it's not relevant for this context. However, listening to the stories in the past, it struck me that there were four general lessons that might be useful to other commercial products based on upstreams which open sourced. As a baseline, I better explain the original open source business relationship. As we've seen, ZEN source and then Citrix commercialised the open source ZEN project. What they used originally was what's called the freemia model, sometimes called open core, where there's a technical core which is open and then enterprise-level features have to be paid for. Maintenance was only offered to paying customers and on the basis of the revenue generated, a team was funded to make upstream contributions. In fact, at the peak we had eight full-time engineers and a community manager funded by Citrix supporting the upstream project full-time. Citrix's first significant decision about the relationship happened in 2011, four years after the acquisition. It was a time there was great enthusiasm for open source within Citrix. It had become fashionable. Someone, I think in product management, but it's difficult to know who was really responsible, maybe engineering leadership, decided it would be a good idea to open source the enterprise-level features, as well as the base product. Revenue was going to be backed by a charge for maintenance. So this is a very puritanical open source model where all the software is available, open source, and all you charge for is the service of fixing it. The goal presumably was to get more community buy-in and all the benefits of open source projects for maintaining and developing the enterprise-level features. Well, we learned a tough lesson from that decision. We had a very poor understanding of our customers and what they would actually pay for because customers gladly took the new features. It turned out they were not so keen to pay for maintenance. As a result, our revenue crashed. And if that happens in a commercial organisation, people's reputation suffer. Beyond that, not just the reputation of the people who made the decision, but it tainted the reputation of open source within the company as well. Because we had less revenue, we had less funding from the company to pay for the engineers who supported upstream. And our contribution slowed. And one important point is if you open source your software, there's no way back. It's a warmway trip. You'd better be sure before you do it. Another consequence followed from our decision to make our code free. As I've said, there was a keen commitment to open sourcing the software. And perhaps we had a naive enthusiasm. And one of the things we forgot to do was to provide the tooling. It turns out that just releasing the source code isn't enough to get people engaged. Upstream engineers will need your tooling for building. They'll need the tooling for testing and the tooling around documenting the system. Even if we'd have remembered or thought to make all these tools available, we'd have still needed to teach people in the upstream community how to use these. It's all very well giving your source code away, but for a complex software system, upstream is going to struggle to build, test and document without some level of automating tools. You have to give the tooling to help. I'm not saying you have to make the tools open source, but you have to make them available. What happened as a result of just providing the source code was that we disappointed the enthusiast who had been waiting for it. Rather than increasing contributions, we inhibited contributions and we lost enthusiasm. Some participants wandered away to other projects. That was the story in 2011. Now, revolutions often result in counter revolutions and the pendulum swings back. By 2017, product management decided to take a new decision. In an atmosphere of mistrust of open source projects, suspicion that many customers were just free riding and not making any contribution, product management decided to reintroduce limits, both on new features and on features which had previously been available. In fact, some of the limits imposed on the free version of the Citrix hypervisor were lower than the original free version back in 2007. Now, this may have been a desperate attempt to claw back some revenue. We did cut the limits. For example, the number of hosts that could be pulled together clustered to provide resilience, load balancing and scalability. As I've said, those limits were cut severely and what had been a limit of 64 hosts in the free edition was reduced to three. It turned out this particular change hit a really significant sub-community in the upstream project. One of the people I interviewed, one of the veteran software engineers, used this phrase, weird systems, to describe a hobbyist community who ran hypervisor-based systems to provide virtual machines for not-for-profit projects, for other open sources, maybe for charitable foundations, often using a strange mix of machines which have been inherited, cast off by other projects, no longer used commercially. These hobbyists are non-commercial. They have no funds to buy licences. But because of the nature of the things they did, as enthusiasts, they were very important bug finders and fixers. They had the enthusiasm, the technical capability and the weird systems to find bugs which wouldn't normally show up. These important contributors were now unable to use the free version. Of course, they were forced to move to other products. And you might be thinking, well, how can you reintroduce limits in an open source project? And the answer is, well, you can't. We also got forked and a whole new open source version of Zen server was produced, XCPNG, Zen community project next generation. And that's perhaps where these hobbyist enthusiasts moved to. So we did regain revenue from our paying customers, the big ones, the enterprise ones who were never going to make contributions. But we lost those enthusiasts who were keeping the upstream project alive. So why did we make these mistakes? In retrospect, it's clear that we misunderstood and underestimated the breadth and richness of the community. Not only did we underestimate the number of types of stakeholders in the community, but even the ones we'd identified, we hadn't properly analyzed what their needs and desires might be. For example, in particular, we didn't do enough to support the explainers, the bloggers, the influencers who draw people into open source projects. We never thought about doing something with the other commercial peer contributors to Zen. And that would be, for example, significant contributions through the years have been made by Red Hat, Oracle, Suzer and Amazon to name a few. We misunderstood the goals of paying customers. Well, the goals of paying customers are usually to get the same thing for less money. That's quite important when you're a commercial organisation. Indeed, you could say we didn't even understand what our own commercial needs were. So this is the fourth lesson. And if we'd learnt this lesson, we might not have made the first three mistakes. We needed to make the effort to find out what makes the community, identify the contributors and then find ways to support their contributions, however they're made. One of the things that my interviewees particularly mentioned was that we'd been poor at engaging with developer conferences and explaining the new functionality that we'd been releasing into the project. It's so important to devote funding and budget to supporting this. So when you look back over 15 years, things, of course, appear much more clearly in the rearview mirror. We never intended to cause any harm, but it's true we did cause harm to the upstream project and to people's ability to contribute to it. We infuriated and frustrated several important communities. Not just the upstream community, but as a commercial organisation, also our investors and executives which damaged and tarnished the reputation of open source within the company. And interestingly, we infuriated and frustrated ourselves because we wanted to be good citizens, make a contribution and get the benefits of open source. And in some ways, it really didn't work out. Well, we can't change the past. All we can do is try to learn and share the lessons from this hard-won experience. It's not every day that you get the chance to look back 15 years in a software project which has had these characteristics, the balance between an open source upstream and an enterprise level product used to run critical workloads all over the world. Let me summarise the four lessons. If you're a commercial organisation, you must think about protecting and defending your revenue because if you don't, you'll lose the ability to contribute to open source projects. You'll be shut down. Remember to share the tools you've developed when you open source your code because if you don't, you're not enabling contributions. You're just causing confusion. In your free edition, make sure there's enough value to engage and entice enthusiasts who are the ones who are perhaps the most valuable contributors outside your organisation. And finally, the one that perhaps is most important, understand and support the whole community, the whole upstream community and your other stakeholders. You know, I really hope we can do better with the open source community in the future. As I said, we can't change the past. And really, we've come to the point where in a traditional presentation, I might say, are there any questions? But I'm not really an expert. I'm just an observer and reporter. I'd be really happy to hear any comments or suggestions. And if you do have any questions, I'll try to answer them. There's a mic coming. When was it discovered, basically, that no-wearing from 64 to 3 made the intimacy as go away? Even then, anyone realise at that time that maybe we should give the non-profit organisation some special non-profit licence or to use more think of that at the time? Yeah, that's a good question. I know there was historically a conflict between the engineers engaged with the community and product management. And product management won that conflict, perhaps because they didn't listen. I'm not sure anyone had the idea that you've had. I mean, of course, you can solve limit problems by issuing special licences. And that's quite a neat solution. I don't know. But thanks for the suggestion. I'll bear that in mind. We have another question? Two more questions. Let's do it that way. Yeah, okay. So the comment was, this has been a very useful in educational presentation. And an acknowledgement that this is perhaps a hard story for any organisation to tell and to admit blunders and mistakes. I hope we do better. Okay. So for the recording, the question was, is there some kind of data that should have been analysed to identify influential parts of the community? For example, these users of weird systems. That's a good question. I talked about the importance of doing the analysis of the community if you want to engage with the open community. I don't know if that's necessarily data driven. That might come from, better community engagement and things like going to developer workshops and outreach. That would be one way. I really don't know about the data. And actually, there might be something specific about XEN in that context. Because it's a bare metal, it's an operating system. We can't even know whether it's connected to the internet. So it's very difficult for us to do things like telemetry, which are much more common in cloud-based open-source projects. So we've always struggled even to know who our customers are, let alone the non-fee-pay ones. So I think there's probably no data story there. It's much more doing the legwork of getting engaged and listening and talking to people. Maybe we lost that and maybe that caused all this mess. Yes, this is a question at the back. We respect each other. At the engineering level, there's a healthy level of respect and both are contributing to upstream. I can't really speak about commercial aspects, not because it's secret, but because I don't know any more. I don't work in product management and I don't know much about whether we even compete for the same market niche. Because although they forked us, in the meantime, we've developed new technologies. For example, things like live migration of GPUs which they don't have access to. So that has tended to fragment the market and we're probably aiming at different levels. I mean, we do provide the support that large enterprises want and what their market niche is probably not quite at the same top level of enterprise size. But I'm not, you know, I think usually we see our main competitors, you know, the commercial VMware and we don't worry about XCPNG much on a commercial basis. Yeah, that's a good and a hard question, which but no, I don't think I even thought about that, partly because the people I talk to don't know and partly because this is an open source conference rather than yeah. Okay, you got me. That's an angle I hadn't even considered and I don't know how to repair it in a, you know, in a commercial context. I guess being successful in making open source work so that we put in and everyone gets something out. Everyone puts in and everyone gets a slice of all the benefits that everyone's put in. That's what the dream was always supposed to be. And if we could get that working, then we'd have something to show. Look at this, you know, look at this sustainable development model that we're using. Yeah. At the bank. Okay, so when I talk about developer workshops, Zen, the Zen project does run its own series of summits. It was, so for example, next week in Cambridge, England, the Zen developer summit is taking place. And historically, we have been bad at engaging with those developer summits and going out to the community and explaining. So things are maybe getting better. Two of the engineers that I'm responsible for will be presenting talks at Zen Summit next week, which I'm very happy about. And we're trying to do that to repair the relationship. So it wasn't us that needed to set up those developer workshops. So maybe that is a good idea, but that Zen project does have its own series of engagement activities for engineers. And we are not the only commercial organisation which contributes to Zen. Our product, for example, is based on x86 instructions that architecture, Intel and AMD processes mostly, whereas ARM also contributes, and there are many contributors to running Zen on ARM, which is not something we're into. It's not part of our product mix. We're specialised in a different area. So, you know, there are other commercial organisations which contribute to the Zen project, both specific architectures and, in the general case, the resilience facilities which would be common to any type of clustering, no matter what the basic CPU architecture would be. Anyone else? Okay. Well, thank you all very much.