 Hello, thanks for coming to this talk. This is about measuring the impossible, contribution analysis for open source. My name is Lucas Gons. I'm the head of product at Marico. I've done a pretty good amount of open source work, including contributions to the COVID tracking project to Creative Commons, to Mozilla, to Music Brains, and just wherever I could make myself useful. I was also the leader on XSPF, the format for play listing in an open context. I founded the seminal play listing community WebJ, and I've been product lead on a few things that have come to an exit. In this talk, I'm building on previous work by Hejong Yan and Jinglei Ren, my colleagues at Marico. Hejong is the CTO at Marico. His research while working towards a PhD at UC Berkeley on quantifying the value of code contributions forms the foundation of Marico's technology. Jinglei Ren is the CEO at Marico. He was a researcher at Microsoft Research, and he earned his PhD from Tsinghua University. His earlier talks, and especially a talk entitled Beyond the Dichotomy of Open Source and Corporate Worlds were foundational in this work, and what I'm talking about here is an extension of that. The hypothesis of this talk is rather wordy, so I'm going to break it out into parts and then I will explain each part in different segments of the talk. Number one, compensation calculated by measuring developer value creation. Two, is an emerging technology which is surprisingly doable. Three, it has limitations. Four, but it will have a big influence on open source. Compensation calculated by measuring developer value creation means first of all that we look at value creation after the fact. So once the work has been done, then we apply the metrics that are relevant to compensation. Of course, compensation in an open source context can be very indirect. It does mean cash sometimes, but it also often means things like reputation, being the author on an important paper, business networking, paid consulting gigs, or getting an interesting job that's relevant. This way of measuring value created for the purposes of compensation is in sync with the ordinary open source project management style of coding first, figuring out the practical details, especially money later on. Volunteerism generally relies on a persistent commitment, but has to work around the schedule of the volunteer. A lot of this value and a lot of these technologies also apply in commercial contexts, but we're interested in bringing these technologies to an open source context, which is different in ways that are subtle and important. Now, when we talk about value creation, we might talk about business value. However, what a programmer does is related to business value, but not quite the same thing. A programmer writes code, so we're using code creation as a proxy. That said, metrics can be devised for any sort of activity which generates signals. That includes active coding days, bugs written, comments on tickets, story points closed, and pull requests reviewed. Part two, an emerging technology which is surprisingly doable. When I say an emerging technology, the first question is what is it emerging from? And as you would guess, the roots of this technology are in lines of code, LOC, which is venerable, indeed dates back to the very first programs. I have read that in the days of punch cards, programmers estimated the amount of work they had done by the size of the stack of punch cards. And UNIX files, of course, are line oriented, so it's hard to imagine that the very first programs to be stored in files weren't counted, weren't sized by counting lines, I imagine using the line count LC utility. That said, lines of code are highly deceptive, and in some places I tell you the opposite of how much value a programmer has created. They can be misled by pretty printing, by comments, and by coder style. Lines are white space, white space don't cost anything, but they do throw lines of code measurements based off. Also LOC rewards programmers for repeated code, so copying and pasting code repeatedly across many functions is rewarded, and a programmer who writes highly modular code and puts that copied and pasted code into a shared function, which is clearly the right thing for the quality of software, would be punished if LOC was the metric. Badness shouldn't be surprising, this is a really old technology, but like everything, it has been superseded by generations of work, and in current generations we have substantial progress. Is the current technology good enough? Well, it's important to come to this with skepticism, and I welcome your skepticism. I hope you'll stick around after and in fact discuss and try out the hard questions, don't be afraid, it's totally fine. So is it good enough? The shortest answer is this production, this technology is in production use already in many commercial shops, however it is in the hands of managers rather than developers. The open source world needs to start taking advantage as well. And is it good enough in a way is a misleading question. The answer is it is a set of technologies which help you understand how you're doing, and if you're skilled in being able to read the metrics, you can apply them. You have to learn how to use performance analytics. So to give an example of learning, our analytics are often used in conjunction with intuition. A customer may run analytics against source code and verify their accuracy by seeing how well they conform with intuition, but you have to be aware that automated analytics are at their best when they are counterintuitive, when they tell you something you don't know. So you want to be on the lookout for missing important facts based on confirmation bias. Data insights like these are already in wide use on GitHub. GitHub's insights tabs shows this on the React repo, and this is a graph which shows the date range of commits. It's useful for telling when developers were most active. And I think it's evident that this does tell you something you can believe in. You can believe in this information. So for the programmer on the lower left, you see that she contributed earlier on, and for the programmer in the upper right, you can see he contributed later on. Another use case is an open source project called claps.dev. Claps.dev is a cryptocurrency contribution tool for free and open source projects. We partner with them. They use a number of different metrics for allocating compensation among the different contributors to projects, including our metrics and also lines of code and commit count and just splitting the money evenly. Another example of metrics that are already in use is get stats, which is a very early approach to this. The source forage repo was last updated in 2013, and get stats illustrates sort of the earlier approaches that we are now emerging from. It looks at lines of code, in fact. The total commits, the number of authors and the average commits per author. The choice of what to measure is key, and it comes down to understanding the difference between activity and achievement and measuring the things that matter. You want to reward people who get things done rather than people who generate a lot of work. Or if you're looking at your own work, you want to reward yourself for getting things done rather than for getting excited and worried and panicked. You want to reward focus. So Marico focuses on code and commits for analytics. There are a number of projects that focus on JIRA. We also look at JIRA, but we find that code and commits provide the most signal or sort of at the sweet spot. Algorithmically, our most important metrics are for productivity and impact. We combine them using weights. Then we sum the contributions for all contributors to a repo or a group of repos, and we find the contributors share. Lastly, we track get history over time, and the history is often the greatest source of insight. Our metric for productivity is called Dev Equivalent. It is a generational successor to lines of code with a lot of important improvements. First of all, we don't work on lines of code in a text file. We convert source code to the abstract syntax tree. The abstract syntax tree strips out white space, strips out separators. It gets rid of code that is irrelevant after the parsing stage, which is one of the biggest sources of noise. Then we compare two different generations of code, the previous version, the version of the last commit, and the version of the current commit, and we do a tree diff to find mappings. We find nodes that are the same in those two generations of abstract syntax tree. Lastly, we count the unmapped nodes. Anything that is new to the current generation, any node that is new to the current generation counts as productivity. This captures additions, subtractions, and modifications. A developer who refactors and gets rid of code and is ultimately adding value is rewarded. We're capable of counting the improvements in the code rather than only the increases in the size of the code, which are often not improvements at all because they make it harder to maintain. This is very roughly equivalent to diffing two files and counting the outputs, but it's much less susceptible to noise. Our impact approach is to measure the importance of functions. We look at their location in the function call graph and the amount of work done. The name of our algorithm is dev rank, and its output is dev value. You can look us up on the internet to learn more in detail. The paper you want to look for is quantifying the development value of code contributions by a John Yen. If you ask me after the talk, I'll give you a link and help you find it. Dev value incorporates three major things. First of all, it is a variant on a page rank, dev rank is a variant in the original page rank paper on which a Google search was founded, but it applies to the topography of a program based on the function call graph, the call graph, rather than the topology of a network using links and pages. The first factor is the centrality of a function within the call graph. That is inbound rank, in rank suggests that a function is important and that edits on that function have higher impact. Secondly, the number of edits performed on that function suggests again that it's valuable and it matters. When bugs hang around for a long time, it's usually in features and functions that are relatively peripheral and unimportant. And central and highly important and impactful functions are usually cleaned up pretty soon because any bugs they contain have fatal results rapidly, fail fast. Lastly, the complexity of the function is used in the impact algorithm. All this said, we believe in our technology and in this growing generation of technology, but it does have important limitations. First of all, there are technical limitations. Measuring business value is sometimes different than measuring code value. It's hard to put a price on code without seeing what buyers will pay. And in a lot of economic systems, the only time when something has a valuation is when a buyer appears. For example, equity in a startup is like that. Our methods are computation heavy and as a result, they can be expensive. The utility of the metrics has to be high enough to merit the cost. So there's a threshold of how much value these metrics must bring to your work before it makes sense to pay the cloud computing bill or the bill for whatever the iron is that you're using. Thirdly, classifying the non-structural purpose of the code has an important impact on our ability to understand the value of any piece of work. It has an impact on our understanding of the impact of the code. For example, new features typically have more value than regressions. And in Agile, there's a clear difference between a chore and a feature. So it's not impossible to identify this stuff, but doing that is ongoing work on a technology level. There are non-technical limitations as well. The most important one is that there isn't enough money to go around. The illustration you're seeing here is from an essay by Andre Staltz titled Software Below the Poverty Line, where he looked into how much contributors to open source projects were earning. To do that, he used metrics from the Linux Foundation that were super valuable and we appreciate those being available. Staltz wrote, the total amount of money being put into open source is not enough for all the maintainers. The core problem is not that open source projects are not sharing the money received. The problem is that in total numbers, open source is not getting enough money. And in fact, we looked at the same data and we found that it is the same story that the amount of money coming into open source through donations is relatively small. It is dwarfed by other sources of revenue such as corporate partnerships like the one between Facebook and React. So the question of creating more money or finding more money is quite pressing. For example, MongoDB recently switched from the Afero general public license to SSPL, which is their fork of GPL 3.0. And Redis switched from AGPL to Apache 2.0 with a commons clause. In both cases, this was about finding revenues from cloud services like AWS that are not obligated to pay based on hosting cloud-based versions. In a related development, Bruce Perens has been thinking about what's going to come after open source and kind of thinking big about how to address the deeper problems including finding more revenue. He suggests that revenue can be grown by improving enforceability, simplifying compliance, making it easy to pay, and automatically allocating payments to the contributors. Notice that making it easy to pay and automatically allocating payments are tightly coupled. And these are places where algorithmic compensation made possible by contribution analysis is a baseline part of that. To make it easy to pay for open source, a payment processor like Miracle can figure out where money should go. And although we focus mainly on package level, repo level, contribution analysis, the same algorithms can be applied in a broad number of contexts. Once said, it should be trivially simple to calculate what should be paid. There needs to be a technical way to figure out how to apportion income among developers. A related issue is that when value is created on the front end and then captured via the relationship with end users, it puts open source developers who usually are contributing to the infrastructure in the position of being value generators who lack the leverage to bring in revenues. A relatively small part of the application may be on the front end, a very large part of it may be on the back end, and the back end is generally where open source resides. So one limitation is that contribution analysis doesn't necessarily address the issue of value capture happening at a part of the stack that is the most likely to be proprietary. That said, contribution analysis can help with this problem by allowing contributions made through the front end, even through a required contribution, a payment, to be automatically distributed to the various creators who made that possible. Part four, but it will have a big influence on open source. The limitations of contribution analysis on bringing in more revenues are not as big a factor as you may think because there are many answers to funding for open source software. And contribution analysis applies to those as well. For example, corporate partnerships like the one between React and Facebook, or the freedom focused projects where the point isn't to bring in revenues, like for example the Free Software Foundation, or the creativity focused projects, things that are really pieces of art like the white space programming language, it's not supposed to be generating revenues. And the form of compensation is not directly monetary for that. Grants for research are not about contributions, so for example LAPAC and C-LAPAC were funded by the NIH. And of course open core projects which are a combination can be helped by our sort of analysis, but of course donations for projects like Vue.js can also use contribution analysis. A key insight is that contribution analysis solves a particular problem in the open source world that's relatively unique to open source software. And that is that developers in an open source context are highly independent. They often have their own motivations. They want to be in sync with the project's goals. They want to not be contributing a style that is out of sync with conventions for that project, but they pretty much manage themselves. And contribution analysis to this point has only been available for management driven projects, typically in a corporate context. And managers do add value. I know that is debatable in some ways, but managers do bring context. They understand how one developer compares to others. They provide feedback. Contribution analysis can be that management layer. It can automate much of what programmers rely on managers for. So developer driven engineering is how open source does its thing, and it is behind because contribution analysis to this time has only been available for centralized management driven cases. Things that contribution analysis can bring to the table include knowing when you're ahead and when you're behind, and knowing when you need to pick up the pace. Contribution analysis can help with seeing where you need to improve. It can help you see that you are not doing as good a job as you could. It can help you know which work mattered the most. Things like an impact analysis can show that you may have been putting your time into a module that is not that important. In fact, that's an important occupational hazard among programmers. It's easy to get off the track and focus on something that won't have that big an impact. And it's difficult to stay focused on stuff that really matters. As an engineering manager, one of the places that I bring the most value for developers is in helping steer them towards things that are really going to matter. Also, developers want to progress in their abilities and their skills, and contribution analysis can help measure how well they're doing, what they're learning, and what new things they're achieving. We can see here an example of contribution analysis for an engineer to work on their own performance. This is fake data in a mock-up in Figma, but these metrics are for documentation coverage and test coverage, which we can automatically find. Maintainers can influence contributors to do a code that is in sync with project goals by helping steer them towards contribution analysis tools that surface quality factors that are important to the project. Developers should be able to see how they compare to other team members. You need benchmarks. Sitting by yourself coding in a cafe or in your office at home after your day job can be isolating. So we have the ability for developers to see their own performance ranked against other contributors on a number of facets. Note that our ranks are generally anonymized. It's about you competing with yourself rather than sort of in a head-to-head way. This goes to the power of decentralized management to motivate and guide programmers to do the right thing on their own independently. We have metrics for reusability and modularity showing that contribution analysis can go pretty deeply into assessing and providing feedback on code. Our modularity metric provides an estimate of how independent and interchangeable components within the code base are. Reusability measures redundant code is similar but not exactly the same as modularity. It's not an exact match but a high enough degree of similarity. Let me summarize all of this by returning to the hypothesis and putting the pieces together. Compensation calculated by measuring developer value creation after the fact. Code first, figure out money later. It's an emerging technology. We're coming from the dark ages and moving into an enlightened period. It's surprisingly doable, especially based on cynicism around LRC. This is not easy. It clearly does have limitations. There is much ongoing work and everybody who's interested in this is making progress and still has important progress to make. But contribution analysis can have a big impact on open source. It can help with raising more money for projects that would benefit from an easier payment throughput and it can improve the development process in decentralized engineering contexts, especially where developers would benefit from improved self-management. My name is Lucas Kahnz. Thank you for paying attention, giving this your time and thought. I hope that these ideas will stick with you and will continue to yield benefits for you. Feel free to reach out. I'm more than happy to answer questions or chat about anything. You can connect with our team using the link tinyurl.com contribution analysis. Feel free to ask the hard questions. Be skeptical. Your skepticism is good for the technology and we're grateful for your interest. Thank you so much and have a good day.