 Hello there, my name is Denise Cooper and I'm here today to talk to you about inner source. This isn't the first time I've talked about inner source at OSSF. In fact, one of my very first inner source speeches was done at the first OSSF. But this is becoming a very hot trend among FinTech people and so we have some things to tell you about that and hopefully will pique your interest if you've never heard of this before or if you've been on the fence maybe we'll convince you. So let's get started. First of all, in the simplest form, inner source is just the use of open source methods inside firewalls to fix engineering problems that may have crept in over years of practice. It doesn't mean that your code will eventually be open source, although a lot of people that undertake this work that have things that need to be fixed in their engineering practice are also trying to figure out how to get to open source and that is the reason that I initially instituted this at PayPal because they were in that situation. They very much wanted to do open source but they didn't really understand how and they likewise had very little collaboration across the company and that was a goal internally. So we used inner source to build that internal collaboration and it yielded more understanding of how to do open source. This picture is Tim O'Reilly and he actually coined this term in 2000. So this is not a new idea but it took a while for it to actually land as an idea that could be actionably implemented and that's because it took a while for open source to win in the market and I don't know if you guys remember but 20 years ago, 15 years ago, there was not a lot of certainty that open source was actually going to prevail. There were a lot of very deep pockets pushing against it and a lot of people didn't think that it could possibly survive and then we won. So five years ago, it was clear to everyone that open source had won in its goal of becoming absolutely essential to the tech industry. There had been lots of successes along the way but about five years ago we started hearing some very conservative sources saying that open source was the way to go including Microsoft which was of course the big shift and that paved the way for inner source to really do well inside of companies that had not yet achieved understanding of how to do proper open source and the reason that it worked so well was what the change was was before open source won it was really easy for middle managers to block inner source by saying yeah well that open source thing is a crock it's never going to happen but now they can't. It's well understood that it's a more efficient way in some important vectors to get your engineering cleared up so that's why I started talking about it at PayPal and now I can't go forward anywhere there we go. So my motivation for getting interested in inner source again was that I've had a long and successful career in open source but I still think that five or ten percent of the total engineering workforce is allowed to work in the open source way even though we know it's more efficient even though there's lots of evidence that says that more people should work that way and so I think of those poor eighty five percent of the engineers as still being stuck in the salt lines and I feel like we should work as an industry to upgrade our skill sets for across all of our engineers if only so that we have a wider work pool that we can workforce that we can hire into it's just so much nicer to work this way but it's also faster and better and better for the company and it buys you out of a lot of technical debt alleys that people fall into so for instance silos raise your hands if you have a silo in your company I'm sure you all do so the problem is that at some point ownership excessive ownership culture feels like a good idea and the reason it feels like a good idea is because it creates some pride in that work it rewards people without giving them a lot more money it it does a lot of good things right up until they become a cult and they're not sharing information with the outside world anymore because they think they've got job security or they're so overworked that they don't have any time to do that which is also very common and there's technical debt that has entered into the picture and they are loathe to clear it out because they understand how everything works and they don't have time or the heart to destabilize it enough to say cut it up in in Institute microservices in some of the companies that I've worked with all of those initiatives have been tried as individual initiatives usually as part of a digital transformation around something like agile or microservices and the engineers just they they they can't decide how to do it documents prints are also something that I've seen fail in these organizations where there's a lot of siloing because they they haven't had to explain it to anybody that wasn't one of them like new employees they probably have a process for doing that but it probably over a period of many weeks as circumstances arise where that person has to be told another piece of puzzle it's not a comprehensive this is how it works and it's straightforward and self-evident from the architecture which is what you're supposed to get out of microservices so in this picture that amount of craft is represented by the rust on the silos right and so intersource one of the main reason that intersource gets adopted these days is by companies that have excessive ownership culture and are having trouble with it either they can't get the top people in the company to share what they know effectively or there is no cross-company collaboration or they've got that you know craft code that the silo people can't clear out because they can't distinguish between the craft and the normal stuff or there's a lot of executive escalation going into how how decisions are made about what to do the first project that we did at PayPal which is written up in a book called getting started with intersource that was a very important pivotal piece of code it was the checkout stack and 90 percent of PayPal's business was touched by that checkout stack and the manager that took on intersource as a change idea was so tired of the fact that they would plan a sprint of work two weeks of work every single time they were supposed to and they'd never get through it because 65 percent of their time was being spent on executive escalation you know where the one executive calls the other one and says why isn't my feature happening yet and they they barter or they or they intimidate each other whatever it is that happens but eventually a directive comes down to the silo saying drop everything and do this thing they were giving 65 percent of that and they would have done almost anything to get out of it so there's lots of reasons why silos are something that you want to bust if you can another reason and this is sort of a consequence of silos although that's not the only way you can have this problem is bottlenecks complexity and bottlenecks that make it hard for people to get stuff done because they have to wait through a traffic jam or they have to wait on a feature request until a silo gets around to prioritizing it or until somebody's boss calls somebody's boss which we've already identified as an anti pattern but it still happens a lot so clearing up bottlenecks so that everything can flow more freely you get better utilization of your resources because everybody can stay busy because nobody's sitting around waiting is another main reason why intersource is attractive to all companies not just fintech companies and then there are some companies that have shown have proven to themselves and to us that intersource can be effective to create opportunities for innovation so the reason this picture is is of the chinese army doing a formal drill is because there are companies where most of the engineers follow the rules even especially really large companies where there are lots of engineers it's really common for the bulk of the engineers to be pretty good citizens because they've been rewarded for playing the game as scripted but they get stuck and they can't they can't step out of that line in order to innovate it's too scary and there are some companies that have figured out that intersource can break up that pattern and i would name drop a couple of them but i don't want to embarrass them there is writing about this the last book that i did is called adopting intersource and it's got a fair amount of information about how that works and then one that people don't think about very often but it's a real thing if you think about it the people that you're going to be able to hire in another five years are all going to be people who are digital natives they've grown up with open source as a normal course of business they understand that they want transparency they understand that as individuals they want agency probably more agency than most traditional engineering organizations are prepared to give them and this intersource as a method to work inside companies is going to become a recruiting tool we're already seeing it in some places when everybody wanted to work for google partly it was because they worked this way when they started out so this is going to be a thing and if you think it's expensive to recruit and then lose your new recruits because they think nothing of changing jobs on a dime if they see something better you know that's going to be an expense and a problem going forward for companies that haven't ingested this and then of course there are companies that go out and get github enterprise which is a fine tool we're very happy that it exists but they think that they they don't even maybe realize that they want the intersource effect that's why they went and did it and it turns out that that github alone is insufficient to get the intersource effect to happen there are few reasons why that's true some of it is there's a little bit of tooling missing and we'll talk more about that in subsequent slides they're working on some of that and we spent a lot of time talking to them about these missing pieces but the other bit is most companies that go get github are not actually comfortable with all the code being available to all the people and the most common scenario is whatever you had before be it clear case or something else the people that implement github will put in all those same walls between departments or divisions or areas thinking that if those existed before they must be there for a reason so we better maintain them intersource requires that everybody be able to read all the code or almost all the code i mean there are examples of things that you don't host wikipedia hosts literally every line of code except the admin keys there's always something but if you're finding that most of your code is still fenced then you're not going to get the intersource effect you might get it between division or within a division but you won't get it across the company and if cross company knowledge full stack knowledge is something that you need and you do whether you understand that you need it or not you definitely need it this will be difficult to achieve even with a great tool like github without some understanding some force applied to making sure that it happens and definitely more transparency than you're probably automatically comfortable with so welcome to intersource this is the intersource logo notice that it both connotes an i for intersource and like maybe there's a party going on in that room that's because it's really fun to do intersource just like it's really fun to do open source the idea that an engineer could send a pull request instead of a feature request read somebody else's code figure out what they need to get done and express that in code and then learn through an interaction with a trusted committer on the silo side what they haven't done properly how they could change it so that it will be mergeable they they gain knowledge they get their thing done faster and they have more influence about how it turns out but also they get to take their career into their own hands a little bit because if they're clever and they're able to learn this stuff becoming a person who is well known for having broader knowledge of how everything works automatically makes you more valuable and it's more valuable to the organization to figure out who those people are so intersource is a is a virtuous circle that can really help everybody and as i've said before at the end of the day it can also help you understand how or come to understand about how to actually behave in an open source context if you have concerns that your brand is at risk because you're going to get it wrong before you get it right why not get it wrong first inside the company figure out how to collaborate properly and then come outside and try to do it there are many companies not so much in fintech but it's starting to be something that happens in fintech that require open source projects where you're going to be opening new code for the first time that you've developed inside they require them to develop an intersource following for that code before they allow them to open source it to see that they know how to build community with people who aren't themselves right all right so intersource is an all boats will rise kind of experience inside your company but it's also part of a larger experience of the open source effect which is where there are unintended consequences coming out of the connections that are made so some of those consequences that happened in our original project at PayPal that when I talked about what that was the checkout stack their original goal was to cut down on the amount of escalation work that they were getting that was keeping them from doing their planned work but they got some stuff for free one of the things they got the first day the guy that as I was explaining to the manager how it was all going to work and I did a version of this talk but with a whiteboard where I drew some pictures to he ingested it very quickly surprisingly quickly and he turned to me and said okay so you want me to put a trusted committer in place as a gatekeeper for pull requests that are coming from people that don't work on my team and I understand why you want me to do that but you want me to give that person only that job for the duration of a sprint and that person is going to be pretty bored unless some pull requests come in from the outside it's going to take a while to create that engine so in the meantime can I just send all of my own teams pull requests through that person to kind of rehearse them on the code review thing and I said yeah of course you can do that and then he sighed this big sigh like huge relief and I said okay what's going on over there and he said we hire engineers and we give them commit access because we've hired them to be engineers but it's not always a good idea even the new we know they're new some of them some of them are really senior we hire them away from other groups in all cases they get commit access and then we say we're going to do code review but we don't really do it because velocity and the imperative for velocity always gets in the way people rubber stamp each other you know friends rubber stamp each other's code the more senior you are the more likely it is that you're never going to get a real code reading but if I can make them do it but through this means of training this person up I'm going to definitely have better quality coming out of that now at PayPal they know exactly how much quality costs because there's this they pay for a lost opportunity cost when the system is down and merchants are unable to complete business because PayPal isn't working so they know exactly how much it costs anytime that happens and he knew what his number was and the amazing thing was that he got a 25 bump in quality just in that first week and and that persisted because real code review was suddenly happening so that alone was a huge help but then the other thing that happened that was interesting was as the trusted committers and we suggest cycling through your engineering resources he had 35 employees and he put the the most senior of his employees into a pool and every sprint the trusted committer changed so they work for two weeks as a trusted committer and then they go back to coding until it was their turn again and they you do that because people who have been their whole career evaluated on how they code and they've gotten to top coder they don't want to stop coding they're afraid to because yeah this is the whole dilemma of do I want to be a manager or do I want to keep coding they've gotten pretty good at it they like it they don't want to give that up so so he was doing a rotation right but after a couple of rotations they started realizing that maybe some of those people that they thought were lesser engineers because they worked on the edges or they were they were working on localization or they were working on front end stuff maybe they were better engineers than they thought they were they found several people that they really thought were very good some of them became so adept at writing code against this silo that was not really theirs that they became trusted committers in their own right and so they had an opportunity to spread that work across this community they were building around their code they also found people that they wanted to hire away from other groups so that's one of the things that happened so it's an unintended consequence but it isn't necessarily a bad thing because it helps with that problem of needing to build full stack developers okay so transparency is a non-negotiable part of both open source and inner source and at first it's going to feel really uncomfortable especially for you guys and I have a slide later to talk about regulated industries and how that all works but I want to say initially that's the fintech companies that have been talking to us and we have quite a few of them that have come and spoken at our events for the broader community have said that it was surprisingly natural once they made a commitment to inner source to become more transparent at least within likely pairings of of people you know and when I say pairings we talk about a host being the people that own the silo and guests being the people that are going to be contributing into that silo and finding pairs where both sides are motivated to get it right at least initially as you're doing your first experiments that's a really good idea because you don't have any problem you know making the code transparent between those two sides I will tell you that I've talked to a fair number of companies as a consultant where some part of the stack is pretty sure that they're the secret sauce of the company and therefore they need to stay secret because the company doesn't trust its employees not to inappropriately share that information but when we actually have gone in and looked and done a code review the truth is that code is pretty crafty and not very well maintained and they don't want it to be public for that reason not even public inside the company so that's not a good reason to keep it private right it is a reason to clean it up before you open it so you don't cause problems for those those engineers because there's lots of extenuating reasons why that happens like under staffing and too much workload or you know change of employee base and and not enough knowledge transfers so people will rewrite and leave the craft sitting there because they don't know what what's going to happen if it goes away there's lots of reasons why this happens but the reasons why people push back on transparency should be carefully examined because they are not always apparently as presented it not everything that gets written should be considered secret sauce and if if obscurity is the only way that you can keep your employees from knowing the full stack I mean that's not what you're trying to do you're keeping you're keeping secrets safe I'm here to tell you that security through obscurity is actually not a real thing anymore most malicious code is written by or insertions of malicious code happen through disgruntled employees and whatever checks and balances system you have in place now is not as good as a guy who is senior reviewing foreign merges or all the merges ideally and really reviewing them because it's on him if that if something gets through that shouldn't be there so all right I have been thinking a lot about the future and I already showed you a slide with the next workforce this is the one after that these guys are are going to be interested in a level of collaboration that you've seen if you have kids but I want to help each other my kids got in trouble for helping that they reversed cheated the the smart kids were helping the slower kids get better grades because they couldn't stand to see their little hearts break right and it's it'll make you uncomfortable if you are a lawyer or a doctor or from one of those professions where being at the top of your class is really important but the truth is most work in the world isn't like that most work in the world this is the most valuable thing that you can teach and it's not easily learned so intersource will be a way to teach it to the adults that you have around you now but I invite you to think about other ways that intersource can be useful one of the famous stories from PayPal is we you guys have probably seen stories about the Silicon Valley buses and how Silicon Valley companies that are physically located down in the valley found that their cool employees all wanted to live in San Francisco which is 45 minutes away on a good day so they started running buses big buses like the tour buses in Ireland you know with a big window big mirrors facing back that look like arms so 75 people at a time bust between San Francisco and their Silicon Valley office and then physically bust back at night with Wi-Fi on the bus and music and you could start working as soon as you sat down because you could log in right but that process of getting people on the bus was administered by admins who were down in Silicon Valley and had not spent much time up in San Francisco and so the bus schedules were wrong just flat out wrong they didn't know how to say exactly where the bus was going to stop you know they didn't they there were lots of things they didn't understand so intersourcing the bus schedule so that people could make their own pull requests against the bus schedule turned out to be really really a good idea because it got the idea across to non coders to the marketing people about why it was important and you might think about ways to touch your culture with intersource that are not the standard ways and think as well about you know like bring your kid to workday think about inoculating them with some sharing ideas and the extent to which that is a valued skill in your company because we got to do something about the education system and getting getting this to be a more common thing to know because at the end of the day open source is people this is people at a Drupal con this is a normal open source conference and this is just a tiny piece of that large audience I used to show pictures of the various years of Drupal because it started out with like 10 people Apaches like this too started out with 13 people and then all of a sudden there were a few thousand 10,000 25,000 right this is the open source effect so the reason that you want to think about those sort of rules of kindergarten and within the implications of transparency and also celebrating wins celebrating people for their wins the same way that happens in kindergarten is because that is how you build the stickiness that makes people want to keep doing the thing if you just blitz them with it as a campaign the way that you do in well so many failed transformation campaigns in corporations you know if your employees roll their eyes when you throw out that first announcement from the CTO that this is a new thing we're going to try that tells you that you've already kind of burnt those bridges you're going to have to find another way to do it one of the things that we heard in a recent fintech session that we did at intersource summit was just admit yourself this is a multi-year commitment I'm not going to tell you this is going to immediately change everything in your company it takes a while to get this right but you need to do it for the same reason that you need to clean the cruft out of your the back of your fridge or they frankly your code base or the way that you do things right so remember that that the people aspect cannot be underestimated in this you have to come up with both intrinsic and extrinsic rewards that make this work we talk about this endlessly at intersource summit and intersource commons we've got a whole bunch of patterns where different kinds of extrinsic rewards have been introduced we tell people not to get attached to measuring immediately do not apply OKRs to this at first do experiments first do a cultural inventory try to figure out what's going on in your culture why isn't this happening without you intervening and then secondly do some little experiments that will validate your hypothesis about why it's not happening and what dials you have available to turn and remember that helping people feel like they've done a good thing overcoming their natural reticence or abhorrence of change is also on your plate in doing this kind of a transformation OK here's that regulation slide so we talked we had a panel with seven members of the fintech community talking about their recent intersource experiences and how it went for them and we got some really surprising quotes which i'm just going to tell you about so the advice that you focus on celebrating the culture focus on your your extrinsic rewards on celebration right and remember that again this is going to take some time but there are some things you can do i would i was going to say to gamify it but it's not that trivial it's it's about motivation so there are several companies that we heard from who used the idea of a marketplace that showed where a given person could could contribute now it's easy to understand why the guests want to contribute to the host stack it's harder to see why the hosts would ever look outside of their silo to do some contributing and yet there's some of the most senior engineers in the company usually so you want them to do that how can you make them do that well at PayPal we had a disused fellow track and we got the idea from some of these senior engineers that if we could figure out how to get them on the fellow track or or signal that they should be considered for this fellow track this gave them an opportunity to influence the outcome of their own careers again in a way that they didn't have before so if every single project every single code base in your whole space if they're if they're participating in intersource has a list of things that they would like to see get done that they're not getting to that gives those senior engineers a target for where they could show up and do a little good and in so doing they can rack up successful merges in other people's code bases even advocacy that they're doing in those other code bases and if they rack up enough of them they can signal that they're interested in going to the next place becoming an architect becoming a fellow doing more for the organization than they can do from within their silo so there's an example of a really good hack okay other stuff we heard we heard that when picking your projects you need to be sure to focus on what's unique to your org but also don't compete with something that should be open source remember I said before that intersource can be a way to make sure that your open source is going to be successful but you don't want to keep stuff in the intersource pen if it makes sense for it to be open source how you make those choices we can talk about another time since this talk is about intersource but it is possible to identify key strategic moves to open source there's a paper that my former colleague Dwayne O'Brien wrote and also a talk that he gave about something called the four questions which is a good place to start if you're trying to think about how to get to open source but just remember that there are things that will always stay intersource and there are things that will start intersource and then manifest as open source projects here it's really helpful to do intersource if you're trying to work on frameworks or common reference platforms or common tools one of the most fruitful places to start is if you're implementing continuous integration and and deployment you have a couple of options there but what you don't want is for every single group in the company to do their own version of that and yet most of them will rebel against a version that has been built to be you know one size fits no one so what works better we've noticed is if you let them figure out their deployment on their own and you come up with an integration framework that's extensible give them an API and let them make it fit for their case or vice versa let them figure out the integration but give them a consistent deployment with some hooks that they can localize to but this is one of the main ways that people get intersource established in their company is by pointing out that that code reuse is the strategy for implementing continuous integration and deployment and then reward that right um there are definitely places where the risk of the regulation problem is mitigated by what you get out of the program so at PayPal they got velocity now they didn't throw the regulation stuff under the bus but they definitely relaxed it in some places where it had grown teeth if you know what i'm saying and you know not to put too fine a point on it but we all know people who work as you know policemen or or security guards that have authority things going on and they want them they want to grab more and more and more a lot of times your internal company regulators fall prey to that same problem because it makes them feel important so looking with a critical eye and how close you can shave that is probably worth doing if it means a big savings or a big change in the way you can deliver for your customers um and then be making working on discoverability both of documentation and of code is a huge benefit that doesn't hurt anybody's regulation regulatory framework just making people able to find things is enormous finding a block of code so that you don't have to write it again or as a starting point can be a huge help to engineers and what you don't want them to do is start copying and pasting but you do want them to be able to reuse things that are you know hardened and useful and they can find out everything they need to know about it if you also make documentation discoverable so these are some of the reasons that many fintech companies are overcoming their extreme fear of regulation and looking again and how how close they can shave it to get to the inner source place and we're seeing it blossom literally all over the world all over Europe all over the US all over Asia we do inner source summits we've just did number 11 everyone is bigger in attendance bigger in the kinds of speakers that we're getting and bigger in the stories that we're getting i'm getting ready to do another version of adopting inner source of volume two with more case studies because we found so many more interesting stories to tell it's just exploding and i think if you're not looking at it you're missing a trick because your competitors definitely are looking at it so here's a bunch of them these are obviously not all fintech but you're going to be surprised how many of them are and there are some interesting companies in this list so i'm just going to call out a couple of them microsoft up there in the upper left hand corner of course famously was not a friend to open source for most of the history of open source but they've recently turned a corner and they did it because their new business azure was needing that kind of transparency in order to compete with a ws and some of the other clouds out there so they recruited some people from apache initially into the open source office and then transfer them into the azure office and they got a great outcome out of that they felt like that was a really fruitful outcome so now they are actively pushing inner source throughout the company to all the rest of their employees because they want those same benefits they're basically admitting that the way they've engineered for all these years needs to change which i find really interesting because they want to understand more about this collaboration thing capital one i'm sorry comcast which is sitting sort of next to um microsoft there comcast is doing something interesting they're the ones i was telling you about earlier that that started this idea of you have to inner source before you can open source which i think is really interesting they've been doing open source for a long time too but they find that they get better outcomes from their open source efforts if they're certain that everybody can collaborate first and then i i will say something about damler who's over in right here damler of course a car company we heard recently is starting from a talk that we had at inner source summit is starting to use inner source across its supply chain so that third parties that write software for them are working with them directly in an inner source way with transparency across the code base and they're finding that it really saves them time and makes them more competitive against a company like say tesla which is delivering so much through software in cars but the rest of the industry is having to catch up i've got stories behind all of these names but i'm going to leave you with those three if you'd like to see your your logo up here come talk to us we're at inner source commas dot org here is our home page of our humble little 501c3 that we set up it's been in existence for five or six years now but it was under the auspices of paypal and we recently pulled it out to become its own thing it has a board that is made up of contributors it is run like a patchy in that you can become a member as an individual you can be nominated for membership and it is an individual membership it's not your company that's a member because you might change jobs but we like the work that you're doing if you demonstrate that you're not only fixing things in your own backyard but you're working to make the foundation better and make it more available to more people um we work under chatham house rules which makes it relatively safe to talk to us and ask us questions however we're finding that some fintech companies are still not comfortable with that they they'd rather have a negotiated nda laden um you know cone of silence or cone of trust and for that reason um we have an announcement which is that we are looking at putting together a special interest group at phenos for fintech companies that are inner-source curious we will make uh inner-source commons the upstream to what's happening at phenos because it will be a special interest group in the sense that the the issues that are facing fintech are different than many of the other kinds of companies that you saw in that big slide so we'll do our very best to make sure that um we stick to the fintech stuff but if anything fruitful comes out of it especially if there are patterns identified that mitigate risk or or enhance yield or if there are tools that come out of this special interest group they will be delivered upstream to inner-source commons so that the entire inner-source community can benefit so this is really exciting news um we have some partners in this and uh I'm looking forward to getting this started with many of the companies that I was talking about earlier including Morgan Stanley um Royal Bank of Canada, Deutsche Bank, inner-source commons and phenos who are the initial members of this but of course everyone that's a member of phenos will be able to come so we hope that that's exciting to people and welcome to the party I hope that this is enough to make you interested make you want to come in and be part of it it's it's a great community I can honestly say that they're a very nice people and smart people and you are going to find a lot of value if you get involved in this and thank you so much for listening to me if you're interested in my company Nirform we do consult on inner-source and if you follow that particular URL or you can just do nirform.com forward slash inner-source I think also works you'll find a little video by me you'll find some testimonials from customers and the way to get a hold of us there and there's also my email address and thank you so much for your time thank you to phenos for um letting me speak again at OSSF and I hope that you guys have a great conference thanks so much bye