 Well, thank you very much for coming. This talk is going to be about comparing the communities of OpenStack and the kernel. I'm CTO of server virtualization at Parallels, which is actually my day job. So it means I'm in charge of a lot of technology at Parallels and OpenStack is a very important project at Parallels. But my hobby job is that I'm actually a kernel developer. So effectively I'm coming to I'm representing the kernel point of view to an OpenStack audience. One of the key things about the kernel is that the skill it tends to value most is C programming. Everything in the purity of the C language and obviously OpenStack is Python. And as an actual kernel maintainer, especially a device driver maintainer, it would be expected that I would be a C purist as well, C purist to people who don't really like any language other than C. But I thought I'd give you at least a little resume from me. Not only am I a kernel programmer, I'm also a sort of serial drive-by programmer and a lot of open source projects. I actually have a lot of Java and Python experience. I did actually write a whole graphical user interface in Java once, but my most recent Java experience was actually coding for Android, the Android platform. If any of you run Sanogen mod on your Android phones, you'll notice it has open VPN support. I actually wrote that. And the reason I wrote it is because I had an open VPN server that I wanted to connect to, like all open source projects. There's also another Android application called CitDroid that I actually wrote fairly extensive set of patches for, one of which was to do the multiple codec support. Again, for me, because talking at conferences where the wireless network is as bad as this one, if you can use a very low bandwidth codec like the spx codec, you can actually get to your home asterisk server reasonably as opposed to everybody else using GSM who has a lot of stutter on their calls. This presentation, as you probably noticed with some of the CSS3 trickery is actually written in JavaScript, HTML and CSS. Being a kernel developer, I can just about get away with programming a fourth generation language like Java. It's much more difficult for me to get away with being an actual web developer. So if this were a kernel audience, I'd apologize for the presentation and say I'm a recovering web developer. Since you're actually a completely different audience, I can confess to the fact that I have actually written several websites. I count PHP and Ruby and other website programming languages among my proficiencies. But this talk of this actually never has to get out to people in the kernel, so I'm trusting you. I'm only coming out to this audience, not globally. And I suppose the final thing I should say is my marketing department insisted that I put a Twitter handle on this and told you to use the right hashtag. And of course, I'm not really very social media friendly. I actually, social media for me is right mostly. I tend not to read it just because the fire hose is too big. But with that, let's get into the community comparisons. One of the most difficult things is actually how do you compare communities that are so disparate like OpenStack and the kernel? Well, one of the things that I've actually seen a lot of the community comparisons that have been done before this do is compare the headline commit rate. How many commits went into each project in the last month, year, whatever. OLO actually tracks them. If you actually look at the commits on OLO, OpenStack already passed the kernel on active contributions. These are the active contributions. OpenStack, blue line at the bottom, kernel, red line at the top. In February 2014, you actually crossed. So OpenStack in that month had more commits per month than the Linux kernel did. So on that metric, you already beat the kernel. Well done, by the way. You can congratulate yourselves. However, what I'm really more interested in is other than the headline commit rate, which is useful for competition, getting grants, convincing your management, is that actually anything more interesting we could say about the comparison between the two communities. One of the really interesting things is what could we actually learn from each other? What does OpenStack do well? The kernel doesn't do very well at all. And what does the kernel do well that OpenStack doesn't do very well? The first thing you obviously like to consider is process. I mean, we all have it, right? Both OpenStack and the kernel have an extensive process for getting commits upstream. The kernel process, it might surprise you to know, actually exists. It is formally documented. We have a big documentation directory. It has files like submitting patches that tell you in excruciating detail exactly how to do it. But the key point is that nothing in the kernel process for commits enforces this. A maintainer, if they choose, and for the kernel, maintainers are people who actually accept patches on behalf of Linux, build a Git tree, and then pass that Git tree up to Linus. Our process is much more informal in the OpenStack one. We have no concept of Garrett reviews or anything like that. Reviews are all done on the mailing list of patches. So the maintainers choose exactly how much of that process they follow. If they don't want to follow any of it, they can actually legally do that. OpenStack has this extensive process. I think I saw someone document the 27 steps you need to do to get a patch into OpenStack. But the key difference between us is that your process is enforced and ours is not. Ours is mostly enforced. If you look at some of the flame wars on the mailing list, you see active enforcement going on all the time. But it's not a requirement of the submit process that it be enforced. And one of the interesting things here is that process enforcement does actually create barriers to contribution. Sometimes some people trying to submit a patch go away just because you make them jump through too many hoops. And we can have arguments over how many hoops is too many. But I've got to say that 27 hoops is actually rather a lot for an OpenStack commit. Deciding where you draw the line in terms of process is really key to actually what goes on. And sometimes that line is not a hard line. It should be a variable line. For some people doing something really simple, it's obviously correct. Perhaps you can short circuit, you know, 20 of the 27 steps of the process. Perhaps you just don't need it. For somebody who's trying to get a full-fledged feature into a system of OpenStack that's already moving fast, perhaps you really do need it. Deciding where to draw the line is key. One of the things, the kernel values in this that you don't have the freedom for an OpenStack is flexibility. Once a process is set in stone and enforced by all the mechanisms as it is an OpenStack, you can't really waive it for anybody. OK, so testing. For those of you keeping score, this is the real killer one. This would be kernel zero, OpenStack one. Because one of the things I actually do as a kernel maintainer is I'm on a lot of sort of our informal committees that actually represent the kernel to bodies like the Linux Foundation and even to a lot of our funders. And a lot of my conversations in the last six months have been going, why isn't your testing process more like OpenStack? So, one of the things I can tell you today is that the kernel is now developing more of an extensive testing infrastructure. The guys who are beating me up about it six months ago are now giving us money to do it. So it will be actually an official project funded by the Linux Foundation. It can't quite be modeled on the OpenStack one using Jenkins because that has to be tied to the commit process. And as I told you, our commit process is pretty informal. So our model will be a forgiveness based one. We actually already have a guy called Feng Wang Wu who looks at all of the git trees that are sitting waiting for Linux to pull them and runs all these tests over the commits to see what's going on. We're actually going to add a more formal testing structure to that. So the tests won't happen at commit time as they do in OpenStack, but they'll happen in between the commit goes into my tree to push on to Linux and Linux actually getting it. And usually that time period is pretty long. It's usually the order of a month. So we actually have time to do the testing and for me to throw these patches out if they fail. But this I'm proud to say is one of the things that we sort of learned from OpenStack. I mean, learned is I learned it with lots of people beating me up with crowbars, but we still got it from OpenStack. Reviews. This is a mixed bag. In theory, you have a much more formal review process than we do because our review process is just over the mailing list. But we both have the same fundamental problem at the base, which is neither of us can find sufficient good people to actually do the reviews. And in that case, it doesn't matter whether your process is completely informal or fully formal. If you can't find the people to do it, commit stack up behind reviews, which is what's happening in both OpenStack and in the kernel. And encouraging more people to actually do reviews is a problem for both of our projects. And if I'm lucky, it's a problem we'll be discussing at the Kernel Summit. But I gather it's a problem that lots of people in OpenStack are already thinking about how do they fix. One of the particular problems that both of us have is this thing called bike shedding in reviews. That's why when a reviewer actually looks at something and says, oh, but what if you implemented this feature in a completely different way? And you go away three months later, you come back with it in a completely different way. And the reviewer looks it again and says, no, I didn't like that anymore. What about if you did it this way? And unfortunately, that is a feature of both of our projects as well. We do get a lot of people who participate in the review process, who don't seem to add a lot of value. Part of my job as a maintainer in the kernel is actually recognizing these people and either shutting them down on the mailing list or ignoring the reviews or telling the committer privately to pay attention to these five reviews, but not this one. Part of the actual law in the kernel is trying to learn whose reviews you should listen to and whose you shouldn't. The social system that Garrett actually has does have the ability to do this, but I don't think you make use of it. So an open stack, getting more people, some people's reviews to count more than others is done, but it's not actually apparently done in the open. It's done behind the scenes. But the base problem is neither of us has enough reviewers and neither of us has a process which actually encourages people to become reviewers. Regression analysis is a sort of mixed bag. It's inconclusive. Both projects have regressions. I could give you graphs up the wazoo that say what the regressions are. A regression, by the way, is a fault in a feature that was in a previous release, but didn't have the fault and is now in the next release. So it's something that was a feature that now has a bug in it that wasn't in the previous release. For the kernel, we actually used to have a formal process that tracked and concentrated on regressions. But if you look at the number of regressions we have with and without that process, they're exactly the same. So the formal process did actually nothing to help us prevent regressions. As far as I can tell, open stack doesn't even track regressions formally. So like I said, the data is pretty inconclusive. The release process question is actually much more interesting. The kernel, as you probably know, has two to three month release cycles. We have a two week merge window right at the beginning of the release cycle and a six to ten week stabilization period right at the end. Open stack does almost the same thing, except you have six month release cycles. You have about a four week planning cycle, which the design summits apart. You have what we could call roughly a 14 week merge window, because you could say it starts when the planning cycle ends and it goes up to the feature freeze. And then you have a six week stabilization period where you start releasing release candidates. The kernel, when you compare these processes, we don't have a designated design period. We concentrate on the code only. We don't really concentrate on the design. That means that code at the merge window has to appear fully formed. If it's not ready by the merge window, it moves over to the next merge window. So we do actually do design stuff. It all goes on on all of our different mailing lists, but it's not done effectively formally. That means if you look at the open stack release processes, they look something like this. There's your, well actually, the color bars are slightly wrong. The purple, which would be the merge window is slightly bigger. This would be your design merge window release stabilization. You go on like that. The kernel, if you look at it, actually does this. Effectively, if you add in the actual design period, which is what we're doing between one merge window and the next is all our design time. Our release cycles are almost the same length as yours. Theoretically, they vary somewhere between four months and six months. But for the sake of argument, we could say they're six months. So we do have this formal design period. We have this tiny merge window and we have this long stabilization. But the key takeaway from this graph is that if you look at what you do and what we do, we have an overlapping release process. Effectively, this means that we do the same as you, but we release twice as fast. And if you also look at it, we do spend more time on stabilization than OpenStack does. Now, the question you could ask yourself is, who benefits from this? It is hard to actually get any conclusive data on this. Regressions, we could say are the conclusive data, but I've just told you they don't really say very much. So I present this as a difference between the projects that you might like to consider. But I'm not actually going to draw any formal conclusions from it. I mean, the obvious conclusion is the kernel does go twice as fast because we release twice as often. We're a code base of equivalent size and we have the same cycle length that you do. So overlapping releases definitely work for us. I would have to say that most open source projects do not have this really tight merge window followed by a long stabilization. The kernel is fairly unique in open source projects doing this. And because most open source projects don't do it, what I can tell you is it works very well for us. I'm not entirely sure from the data if it would work very well for you, but it's something you could actually consider. OK, so other comparison methods. What are the other ways that we could say different things about the projects? One interesting metric is prolificness, which means that you take any project and you use get statistics to tell you who are the top five committers to this project and you look over a period of a year and say how many commits do each of these top committers make over the period of the year. That's a very good way of actually measuring the prolificness of the project, how much change is actually physically going into this project. So that's basically what I did for these statistics and this is what they actually come out at. OpenStack is actually an interesting project to try and do this for, because I can take what would be considered core, and I know what's considered core is a bit controversial, so I took an arbitrary cut. But most actual open stack committers are project monogamous. That means they usually have commits that peak in one project and not very many in another. So the statistics look virtually the same, whether I take a single project or whether I take the entirety of OpenStack. So I actually did both. This is roughly what it looks like. That's the kernel. This is OpenStack as a whole for all of what you consider core projects. This is quantum and this is neutron. As you can see, the top committers for the kernel peak over the last year at around 1250 patches for the top committer. For OpenStack it was somewhere around 350. Now one of the things I could tell you as a kernel developer is this means that we're about four times more prolific than you are. But there's a dirty secret hiding in all of these figures. You probably don't know it, but a lot of commits to the kernel are actually done mechanically by people who run source code checker over the sea that says, you know, you have spaces instead of tabs here. You have duplicate semicolons here. This include is done twice. And in fact, the top three committers here all make those type of changes. They're all people who basically have the Linux kernel mailing list feeding into a checker for the commits. They put it on a work queue. Then they produce patches that correct all of the mistakes and have them put upstream later, which basically means that the statistics I've showed you here are completely worthless because our top committers are done mechanically, not actually by substantive useful commits. So how could I give you statistics that are actually substantive useful commits? What it means is I actually have to go through our top committers. So I went through our top 15 committers and it helps being a developer. I know personally which ones are actually doing decent work and which ones are doing trivial work. So when I take out all of the people doing trivial work, this is actually what the statistics look like. So this is our top committer who actually does real work. And as you can see, OpenStack is still sitting at just shy of 350, but the top committer in the kernel is only around 650. So the prolificness of the kernel over OpenStack on this measure alone is only a factor of two. But then you get into process problems. In the kernel, because we have a mailing list process, we're very good at forcing people to break changes up into easily reviewable components. In OpenStack, in theory, you are supposed to do this. It's actually written down that you should do this because it makes reviews easier. But in practice, getting commits through Garrett is a time consuming process. And so most people would rather do one review up to Garrett than actually try and work out how to divide it into two. So some of these commits here are multiple patch sets that wouldn't actually be present in OpenStack. But in theory, the figures show that our prolific committers are about twice as prolific as yours. In practice, if you actually analyze the patch series and work out how many they do, you might get a 50 percent, well, sort of 25 percent reduction in the kernel. It might come down to somewhere like 400 instead. Perhaps we're not that much more prolific at the top end than you are. But again, I can give you the figures that Garrett says, and I can explain why it's hard to compare them. I think, I mean, being a kernel guy, I'd obviously say that we go faster than you just because we have a release cycle that's half as long. We do the same type of process, so we release twice as fast. These figures seem to say that our top committers are twice as prolific. The data does seem to match in some form. The other thing you can look at is called the long tail. That's how easy it is for someone like me when I did Android OpenVPN or CitDroid or some of my other websites. How easy it is for someone who's an outsider in the community to actually make a contribution to it. So it's basically look at the total number of committers who submitted n or fewer patches and ask how many are there in each project. And it looks like this. That's the kernel in dark blue and open stack in light blue. In theory, the long tail of the kernel is a lot bigger than the long tail of open stack. That means that the kernel is, in theory, a much more accepting process of patches from drive by committers than open stack is. Now, in practice, this statistic is absolute and therefore not correct because it's showing the absolute number of committers. If you think that the kernel has about 60,000 committers and open stack, I think in the last release cycle has about 25,000 to 30,000. Obviously, since it's a bigger project, the kernel would actually expect to have more long tail committers than open stack would. How do you actually account for that? One of the things I can do is actually scale this down to a percentage. So here, open stack actually beats the kernel on long tail because I have divided the statistics by the total number of committers over the year. So in theory, that says open stack has a bigger long tail in the kernel. The problem with doing it this way is that what I've actually done is I've overcompensated for the prolificness because we have many more contributors right at the top end who do more patches than open stack does. And I've just divided by all of those contributors. So again, I've given you a metric that isn't actually properly accurate. So if I do it in a way where I scale the prolificness figure so that it's the same for both open stack and the kernel and then do the division again, this is what it looks like. And this time it says that the long tail and open stack in the kernel for each of the patch things is about the same. Perhaps there's a slight advantage to the kernel, but that advantages of the order of 5%. Perhaps it's not even significant. But if you want to look like the kernel does in its incarnation over here, you obviously have an interest in making open stack more accepting to drive by committers. So like I said, the anecdotal signs are that the kernel's long tail might be longer, but there's nothing really conclusive in this because I can slice statistics either way. In fact, what I think I've just proved with all of this is Disraeli's contention that there are lies, damned lies and statistics. However, one of the things you can do to actually increase your long tail and actually even make your prolific committers be much more prolific is to reduce the barriers to contribution. If the hops they have to go to to get a patch in are much lower, they'll tend to get more patches in and people who would have been put off by it and not submitted the patch will be more actually eager to do it. One of the barriers to contribution you definitely have an open stack that we don't have in the kernel is contribution agreements. Everybody who makes a commit to any accepted core project of open stack is supposed to have signed your CLA. Now Mark McLaughlin actually has a session on this immediately after me in BO through to in the design summit. So I'm not going to give you the long version of this talk that I was going to do. I'll give you the short version. Which is Legally in the US your employer owns your contributions if you're making contributions to any project open stack the kernel I don't know dive dive stuff that Linus does gnom any of them your employer owns that contribution under the US law For most of you who work for companies that are used to dealing with open stack This isn't a product used to dealing with any form of open source including open stack This isn't the problem none of you has probably run across this as an issue Principally because it's all been sorted out at your companies long before you started committing to the kernel or open stack or any other project But if you're trying to attract drive-by contributors They're people who work mostly in companies that are not used to putting code upstream and open source Anything you put in their way, but it's a barrier that looks like a legal document Actually has to go through a legal counsel at that company. So if Potential contributor to the kernel comes from a weird company. Oh, let's say Microsoft The only thing to make a contribution to an open source project that commit has to do for the kernel In Microsoft is get their managers permission. That's fairly easy if They have to sign the CLA they have to sign that legal document on behalf of their company the CLA actually says this at open stack and One of the things that no manager can do is give an engineer permission to sign a legal document All legal documents have to go through the corporate counsel in the company So this is already a barrier difference between the Linux kernel and open stack. It's actually a barrier I'm not just between the Linux kernel. There are lots of other projects that have this informal commit thing and The problem is that the actual corporate counsel is the guy who actually has to sign the contribution agreement Most of you who work for companies that are used to doing this with open stack I've already got a process for this your corporate counsel is already doing it. You never ever see this problem But let me show you what the corporate counsel for most companies who are not used to open source actually looks like This is the corporate counsel. This is the corporate counsel's job The intellectual property of a software company is the treasure pile And the job of the corporate counsel is to sit like a dragon guarding the treasure hoard If you want to go to the corporate counsel who is not used to dealing with open source what you're basically saying is Excuse me, mr. Dragon. Can I have a lump of this treasure? I'd like to take away and give to the open source community For a lot of people it won't just get you burned. It can sometimes get you fired The corporate counsel is not a person who wants to give away any of that treasure It is literally like trying to steal treasure from a dragon's hoard So for all of you in the room, this process has already been gone through by your predecessors But think about how it would look if you had to go and actually bait that dragon Try and force them to give up the treasure and to actually sign this agreement Because to the intellectual property dragon to the corporate counsel signing the CLA Looks like an agreement to give away the gold pile that they're sitting on hoarding This is why it is so difficult for people who don't actually work within your ecosystem to sign that CLA This is why the CLA is such a barrier to contributions in open stack So the one thing I'd actually ask you from this session is please go and support mark in his bid to remove the Contributor license or agreement and replace it with something else that gives you equal legal force But means that the poor old developer and say a dev ops or who doesn't contribute or some other organization Only has to get their managers permission to contribute. They don't have to go and be at the dragon of the corporate counsel okay, so Another thing we can look at is how would you actually go about increasing the prolificness in projects? And this is a problem not just for you an open stack It's also an interesting question for us in the kernel as well. We might look on paper like we're doing well, but we could always do better so The easy thing you can do is just increase the chances of acceptance getting rid of the CLA would be one way of doing that but there are obviously others and Obviously you have to do this without compromising quality There are spectrums if we had a project which was a free-for-all and we just accepted any patch that was presented to us We'd have commit statistics that overhauled the kernel by God knows what just because there was no quality control in the project Conversely if you have too much quality control, you're actually pinching the amount of contributions So you're artificially reducing the amount of contributions So either end of the spectrum is obviously wrong What we actually have to try and do is find the medium point in the middle and that again is why the prolificness Statistics I showed you are not things that you should take with too much Seriousness because it could be that the kernel is actually Accepting too many patches and therefore giving itself a quality problem looking at the regression statistics for our releases I don't believe this is true So I think we still sit on the quality control side of that bell curve, but I don't know how far we sit on there I think we could perhaps stand to loosen our process a bit and perhaps accept a few more patches But I'm not sure we'd actually do that I think you also since you sit to slightly the left of the bell curve of us could also do with loosening up your Quality control process, but again I don't know how much and there's always a danger to loosening the quality control process because you're it's not just Quality control on the back end. It's stabilization on the front end The more you loosen quality control on the back end the worst stabilization on the front end becomes your Stabilization period is a lot shorter than as well as 50% shorter than ours That means we have time to catch quality control problems that you might not catch with the shorter stabilization period So the perfect point on the bell curve for us is probably not the perfect point on the bell curve for you And so I'm not giving you any advice about doing this I'm just pointing out the comparison the comparison says the kernel is doing slightly better in terms of quality control and commits than you are because we're accepting more patches But they're still not giving us a huge stabilization problem, but whether you could do the same I don't really know because it would involve tampering with your release cycle to do that But one of the things that you and us all both need to do is to accept patches easily from non-core contributors in one way or another Harvesting your prolific contributors Nurturing them giving them free goodies in the dev lounge is a very good thing to do because they're the people who make most of the patches But remember Everybody who actually grows up to become a core contributor to either project had to start somewhere trying to encourage people who are starting out is also a vital part of this process and Like I said, I don't really have much advice to give you the biggest piece of advice I give you that I see happening on your reviews and our reviews is Basically, if you are a leader in your community like I'm a maintainer That makes me a leader in my field in the kernel community. Lots of you are leaders in the open-stack community Preventing bike shedding and reviews things which give people a negative impression of the community because they're basically wrong It's actually a very good service to the community If you see somebody give a bad review and a bad review means you've told you've given this person a list of ten Things to do seven of them are wrong. It's your duty to say so because if you don't say so What's going to happen is that person will do all ten things on that list. They'll come back The reviewer will be happy, but it'll get up to the commit of the maintainer like me And I'll look at it and go what the hell did you do those seven things for I'm not accepting this patch And that is very very dispiriting to the person who did that Trying to police the review community and make sure reviews happen is actually a very very core way of actually Encouraging people to come into your community and making them feel welcomed giving people bad reviews as a way of making people feel terrible Okay, so I think I'm going to leave five minutes for questions. Let's come to the conclusions So both projects actually have a lot of the same problems. There's a lot We could actually tackle cooperatively in the open-source world Bike shedding and lack of review is something we both have in spades If we did something about it, perhaps we could do something about it together Both projects actually have things they can learn from each other I think it's no secret now the colonel has learned testing from OpenStack Perhaps there's things about stabilization release you could learn from us But the testing thing was very simple. We didn't do it now. We're planning to do it the Altering the release process to try and get more prolificness that I presented in the graphs is not a simple thing to do It's not a yes. No, we do it. We don't do it thing. It's not the cable. Honestly. It's tight end It's something to do with the other end it's Sorry lost the train of thought. Oh, yeah It's it's something that you would actually have to debate and consider where they actually suited you or not because You sit at a different place on the whole set of curves that I was measuring from us And the reason you sit at a different place is because your release process is different from ours What works for us might not work for you So I can say the colonel learned from OpenStack. I can give you something You might be interested in learning from us, but I cannot say it's it's not a yes No thing like testing was it's something you have to go away and consider for yourselves So the good news is that both communities are already talking at many levels The current level that we're talking on which is the crowbar. I got beaten over the head It with is that the companies who fund the colonel are also a lot of the same companies who fund OpenStack So it's our funders who are actually discussing this and so things are actually being pushed down from the top towards us So ultimately you lot will hear about this as well as I heard about it in testing But that's actually a good thing and sort of me coming here and perhaps you coming to our events Would also be good things because there are a lot of common things in terms of building a community that neither of us get Right that we could actually look at doing common mechanisms for trying to sort out So with that if you enjoyed this presentation, it's actually done by using impress.js by Bartek Sopa I've got to say that this presentation is not the vanilla impressed.js That's in Bartek's account on GitHub I actually forked it and did lots of things that do some of the special effects that he can't do Which obviously goes to prove that I am a web developer in spite of everything My fork of impress.js is on git.colonel.org under my account, which is JJB if you want to look at it I'll probably put this presentation up online and I might even send a social media tweet about it So I'll tell you the URL so you can just download the source and look at it from yourself But with that thank you for listening to me and did you have any questions? Okay, so I think I saw one question there. You can either come up to the mic or I'll repeat it When I look at statistics for Contributions it seems that activity.org doesn't provide more granular stats Well, you're asking the wrong person since I'm a kernel developer But I can tell you that none of our kernel stuff provides the stats that I showed today I did a git pull on all of your repositories and actually ran a set of git scripts over them I looked at OLO stack-alytic and a few of the other activity things But none of them gave me the statistics I the only statistics I could get out was the initial graph I showed with the headline commits everything else I used my own analytic tools running over the git repositories for so and Obviously being a kernel developer. I can't really answer why open stack.org does something or doesn't do something But I know that neither project gave me the statistics I wanted to actually do the analysis I presented here today Why they don't do it? I suspect is partly in the reason that I gave you which is that trying to match these up by Statistics is fraught with danger. I can spin those statistics either way if I did the scaling one I can show with those statistics that open stack does miles better than the kernel if I show the raw ones I can show the kernel doing much better than open stack at the end of the day without the Explanation of the comparison base the statistics don't really mean anything, which is probably why they're not tracked Okay, so this is a question about the politics of open stack the question was what actual core projects did I count for that analysis? The answer is I knew I was going to get this question because what is core and what is not core is a really significant political problem With an open stack at the moment So I punted on the problem and I went to Olo anything Olo counted as core is what I eat So that commit graph was basically straight from Olo Olo seems to take non incubator projects that are actually sitting in open stack Which is where it gets most of its statistics from but I knew I was going to get the question So I basically sidestepped it. I have no real good answer for you There are people who are working in the open stack board to try and define what your core projects are But right at the moment I haven't really seen an answer come out of them And I wouldn't presume to give you one as a kernel developer from this podium Okay, any other questions? If not I'll say thank you very much for listening and please do turn up and support mark in the design summit. Thank you