 Okay, so thank you for being here and sorry for the delay. It's I got lost It's a huge. It's a huge conference center and I don't know you but I couldn't find the Sally on on the map Oh, Parsi Pasi. All right, so I'm welcome and thank you for being here. I think you're here to learn about how collaborating How to best how to improve collaboration inside your companies and I've been talking to quite a bit of managers in the past months to understand exactly how collaboration works inside companies inside corporations and because My role at the Open Sack Foundation is to exactly help companies Understand the Open Sack way of doing things and integrate developing teams development teams inside the the the Open Sack workflows and processes and It's interesting to me today to follow the keynotes by Jim Zemlin and he was saying exactly what the Linux Foundation Recently reported that business managers and executives of companies members of the Linux Foundation They've been saying that it's very important or somewhat important for over 90 percent of them to collaborate inside for their business and Not only it's important. It's something also that is going to be increasing over the next years. So with for Open Sack, this means that we You know, we all start to get better we need to get better and We need to get better at collaboration and I may have some suggestions for you in this and I'll start by giving out the Results of my Early investigations I would say the early results is that you will in order to improve the collaboration most likely Companies will need to inside your company inside your team You will have to tweak how you develop products and and how you serve your customers You have to start thinking about how Your company and your team is not just your team. It's also part of a Another someone else's team and you have to expect results over the long term It's probably most likely you're not gonna have short cuts or short-term results very easily So I give you a little bit of an overview of what how Open Sack is made so that you you know, the things that Those those explanations those early answers will make a little bit more sense. So, you know What surprises a lot of people approaching Open Sack for the first time is that it's got no traditional management structure you cannot easily buy into A seat you cannot easily become the Owner of anything in Open Sack you you earn Influence and you because because every most of the positions, you know, the technical committee or the board directors or even the user committee We all get elected. I mean most most of these except the platinum members from the board of directors every other position is an elected position and so you can think of Open Sack more of representative democracy then then traditional traditional management structure and So Open Sack is also made with a very predict predictive predictable rhythm We release every six months that we've been releasing for Quite regularly for many years now This cadence keeps the whole team working on Open Sack upstream a very focused and Helps with also the newcomers to to have that sort of Idea as a when things will happen and what will happen. So for example This is on the on the right, you know, the picture shows the calendar for the past release Juno We met in Atlanta in May mid-May for a week Just like we're meeting now in Paris for for a week to define what the roadmap will look like in the design Sessions, which will start tomorrow for the next release and then after After Atlanta after meeting in person. We all go home with ideas and things to do and And over the next month pretty much every six weeks or four to six weeks We get new milestones until we reach a very important time of the year of the release cycle where We don't accept any more feature Proposed linear proposals or everything that is also work in progress needs to be stopped by The third milestone and at that point we start cutting release candidates and Call for testing and then we have the release and then we start over So once we go home after the design summit we have We start working on features or bug fixes and they all share more or less the same the same road the same Cycle we have something that is called a blueprint that gets filed with the blueprint when we want to have also specifications proposed and the specifications are a larger description of what the feature will look like more like an architecture type of Document and these specifications to get discussed they get discussed openly they get discussed across all of the participants in the in the collaboration Space and once the specification is approved then This the blueprint with that depends on Specification gets also scheduled for a release into one of the milestones and then the real work starts or The next phase of the real work starts where code gets developed and proposed up for review again until it's merged and the blueprint is closed and each Code patch and each change proposal that goes through this review workflow Which I'm not gonna give you details about but this is just to show you do the complexity How things get made because you know in the end what we realize by looking at the amount of people participating in today Open stack is huge not only millions of lights of codes and 2600 contributors to Over 180 code repositories make make open stack extremely comp big It's also something that moves very fast So we release every month use you heard this this morning during the key notes a lot some of the users and super users We're mentioning how every release comes with new feature sets and and That creates stress create this velocity The complexity also a woman stack has been mentioned during the keynote So I'm not gonna spend too much time because that doesn't matter how you look at open stack Whether by layering the services on top of one of another or you show at the individual architecture Open stack happens to be big complex and fast So that that's what I think when I think of big complex and fast and the users And the newcomers to the community trying to do that they They can get hurt And looking at speaking of downstream so the ecosystem of open stack of the months that join open stack Why do we do it? Why do they do it? and I kind of think of the large picture of the ecosystem as made of Two large chunks the large one chunk is Companies that are producing something Based on open stack and something and companies that are consuming something based on open stack and and I say companies but it could be departments or Teams within the same company that have these different roles The producers inside the produce producers of open stack. There are two main pieces One is made of company groups and companies the firms who are Deeply committed in open stack and these are producing probably 90% of the code It's this these are the large chunk at the top 10 at the top 20 contributors to open stack that you see quoted in statistics and the companies involved are Developing contributing less, but they're the largest majority of the companies And if you notice all the the pieces of the puzzle have the same size because I don't I don't think that Anyone is less important than the other Operators are consuming open stack in terms. They're consuming whatever is produced upstream and they eventually have to plug into the upstream pieces and We were starting to have a look also at end users. These are developers of SDKs or consumers of SDKs to develop Applications on top of open stack and we just started to look at it stat this group now most likely and Or based on my experience companies tend to have this sort of approach to The products they develop that they develop and they have in mind a very classic product value chain where Everything is consumed and produced and procured within the boundaries of the firm and this is an approach that starts really to fade and Put into crisis as a model when you look at Collaborations and especially with the within the open stack ecosystem because technology development and procurement for example When you when you get to to open stock and you get to open source in general You start touching things that are much larger. So for in the open stock collaboration You don't have only pieces that touch that touch an interface with the firms only and things need to be integrated across the board company boundaries and So that vision that approach with monolithic way of procuring developing new New products so R&D or procurement of software really gets into into the way and friction accumulates When when you have that idea that everything depends from a direct Line chain of command inside a company friction starts to accumulate around All these places especially so when the specification is proposed when a specification is discussed When the blueprints is assigned or for for a deadline and during the code development life cycle and Friction accumulates because managers expect to give an order and have it executed by their the people that they depend They that they control and they have a long history of being being a Long history of obedience from their from their Employees but when when When someone is assigned a task and that task the execution of that task depends on someone else approval outside of a company boundaries those That that time and at that place that's where friction accumulates and I noticed that the companies that end up succeeding more in Developing OpenStack and contributing to OpenStack are those where the development cycles and and the model for to approach for new product development is more open and this is a The open innovation chart and that I have blatantly Used here to describe what some of these models where the OpenStack collaboration effort is part of a larger way a cultural approach and and business processes and that are inside the firm inside the companies that take large advantage of Collaboration in in any sense so the in in in a research and development case Collaboration is is allowed to happen inside the research phases until you know with Sourcing of other places until you get to the new market or you serve the current market with new products when when companies with companies They have this sort of vision for Product development. These are the ones that are doing But I have a are having best results inside OpenStack and they're not spinning their wheels so simple simple suggestions that I have for for You if you're interested in in is that Successful companies in OpenStack keep the tempo. So they have they have aligned their product release cycles. They have aligned their Services also on the development of the OpenStack release since it's OpenStack releases very clearly very very predictably every six months We have milestones. We have free show freezes that we publish At the beginning of the release cycle the companies that keep that in mind. They have more success stories to share The other important thing that I noticed as a pattern the companies that are successful like contributing to OpenStack and have less friction when They have people who work inside the OpenStack community and they are known Because in the end it's not who you know It's who knows you and it's something that we tend to forget Participations in summits participation in all the meetings that we have aligned all the relevant conversations doing code reviews for specifications and Actual code those are the things those are the activities that tend to have a large impact on your Developers get known by other people inside the community and we have governance spaces where also you can participate another another thing another approach that I noticed works very well with developers is When when teams are organized so that they adopt the open source principles something that really doesn't work is having agile teams scrum teams working and Working inside the same room developing features discussing architecture shipping code all Inside one room without sharing any any Result of those in-person meeting with With the outside world and this is something that Is really disruptive and doesn't really help the collaboration because you have to think that OpenStack never sleeps it's literally a community that covers all time zones and When you're meeting in one room and you're making a decision Inside one room and you're not sharing the results of that position Outside on a mailing list or anything that doesn't have a URL You're not very you're not really doing a very good job at communicating with the others Might be surprised to see your code being proposed without a discussion And if your team even if it's inside the same company without this inside the same time Zone if your team doesn't get used to the idea of having to share something over a URL You you're probably gonna have friction accumulating. You're gonna be spinning your wheels Asynchronous communication is also something to help so We all love stand-up meetings because they're short because they're on point but having having Collab communication over email is something that needs to be also Educate I mean it's an education process that would help Locking points that need to be avoided is also something that I've seen happening quite often where every large contribution from one firm comes After it's finished inside the firm itself and when it's pushed up for review to the open-stack community, it's usually Refused or it doesn't have it's not accepted immediately. So pushing the reviews Pushing the code as a working progress is a very good idea. We had people asking me how how to do that how Recently on on the mailing list. It's something that needs to be Adopted and it's a very good thing to do Because there are things that create disappointing results and it's quite visible From my point of view when there are teams organized around the product cycle internal at the firm I've had to rescue a couple of companies recently because They came up with an idea of adding support from to one of their hardware piece and The whole decision to join to our add that driver the little piece of code to support open-stack inside their company For that particular product line was done inside the company itself We were never contacted they never reached out to us and what what happened was that they They had the release cycle there the release time for that product at the same time Open-stack more or less being released. So it was around April but what they didn't know was that before You can land even 200 lines of code in a driver So it's a very separated confined environment they have to I mean that the code needs to be reviewed by others and And there is for these reviews to happen. They need to go through feature freeze like anybody else string freeze like everybody else and When they proposed the code that code got rejected and they missed the deadline and they couldn't ship the product with open-stack support As they promised to their customers They also made another classic mistake is to have only one engineer allowed to contribute upstream and This engineer was actually being the performances and the performance of you for this engineer were only measured for results that were Or things that he would do inside the company So this person was had no incentive to Do code reviews or for others inside the open-stack community He had no incentive to even read the mailing list and in fact He totally missed a deadline not only because because the mailing list is the main place where a deadline was moved so I mean it's it's it's something that happens and We I also notice at another company where developers were actually physically Blocked from accessing the communication lines with the open-stack community that there were firewall rules and Now they removed them that prevented IRC communication to happen Get review was also blocked because it works on a on a strange port. I mean a non-standard port We have places in remote Asia where well in in China some some Most of the developers cannot access directly some places of open-stack to org Because it's being blocked by the great wall the great wall the great firewall and I've seen another another company where every single commit Every single proposal for a change had to go through one single person So there were the individual engineer There were teams of engineers large teams of 15 20 people working on features, but they could not Propose the patches themselves. They had to go through one person So this created a very strong incentive for these teams to work on large changes before they pass them to the one person who was then allowed to contribute to patch upstream and during all these processes the there was a I don't know was all so doing all these steps the The original the original authors of the code that they were not allowed They were they they were not allowed to participate into the conversations the discussions and the debates that happened for them for their own code so that was really complicated and these are Definitely ways to have huge collisions of interest large masses large body going at different speeds so You know going back to the short answers that you need to tweak your organization You need to change the way you think of your Incentives your business processes Your development processes The short there are there is there are some shortcuts that you can take before you can definitely change the way you do business and One simple way of doing things is to join the upstream training that we do before each summits We we did one session in Atlanta we have three sessions before Paris over this weekend We'll definitely gonna do it again in the Vancouver and these training sessions are offered by the OpenStack Foundation To developers in order to get exposed to the way of OpenStack the way we do things So and it's what today's training the first day We teach how to use this the development tools the very simple fairly Fairly simple easy to grasp me for a developer But we also teach the more difficult things that the the social interactions All the all the processes that are complicated to to get used to We also play a lot with Legos and you're gonna see some of the results of the Lego exercise in the in the expo hole These days it's it's on the on the table Another very simple tweak that you can do in your companies in your organization is to give them to Free your developers to do the work that they want to do on their own This is here some people refer to this as the 80-20 rule that Google kind of is Known to have so four days of four days your developers can work I have to work on whatever you tell them to work on and for the fifth day They can do whatever they want and when they do whatever they want the if you tell them to work on OpenStack You know that they have free time On OpenStack and this will buy them visibility This will enable them to be visible to others and when they're visible to others they can they Earn karma they get their own Possibility to influence things and with that karma they can spend it when it's time for their code to be Accelerated or you know fast forwarded into granted an exception exception or Things similar and what you end up gaining is Lots of Avoid, you know being rejected. Nobody likes that to be told that they're late their contribution is late Developers there are free to use their time are also going to be the ones that will be reading the mailing list and Get and know the deadlines you Get also if you follow if you your developers are aware of what the surroundings are that you you get more Less chances to to Replicates work that has been done by someone else and one thing that happens is sometimes is that developers pick a bug to work on They go on and work and when they go back and they try to submit the patch They realize that someone else already fixed it because speed of open stack is such that these things sometimes happen Right so your team will be fixing stuff and Yeah, the feature freeze is something that really surprises every a lot of people every time although it's predictable again, but The During feature freeze is where everything really at the end of the release cycle Things get complicated and lots of people ask for an exception but those exceptions are only granted to the ones that are known to the technical leaders for Being able to deliver good code on time and I Underline good code on time because good code is something that is rare to come to come across So going back to the short answers to remind you, you know, you will In order to minimize friction you will definitely need to tweak how you develop product and how you serve your customers It's something that you cannot do without you cannot keep doing business the way you used to Without and and expect to be collaborating into open stack successfully without friction Definitely, we need to start thinking about Measuring things outside of corporate boundaries So there is there are talks during the rest of the day in this in this room about how to measure community contributions So you can keep track of the work that your developers do outside of the of your organization and you can incorporate that data Back into your into your reporting and your KPIs And also think about long-term. It's not something that you change in the tweak and you immediately get results You you need to have that long-term strategic view when you when you decide to invest in open stack So I'm early and I'm available for questions and comments I kind of thought about that you would want to have questions. There is a microphone in the room If anyone Better One of the things you commented about the lack of project product management and architecture and part of the governance Whereas a lot of the challenges that companies are going to be facing is with aligning their product Development process, which is very frequently Pushed through by product managers and product owners Do you have recommendations on how how to align those how to engage? Because the product manager may not be technical enough to engage directly with an engineer on open stack They don't always totally language. Yeah, absolutely so we do have recognized we have recognized that we haven't been good at tracking and and involving product managers inside the open stack communities and We are starting this week actually so we have a meeting starting at 2 for the first time at 2 p.m With product managers of open stack related products services we want to get to know them and enable product managers to Influence roadmap in open stack or even you know just simply Tell us what we need to do in order to help them out, but so That's The the one thing that we're doing the other thing that we want I mean the other suggestion is that we we We need to improve also on the community side We need to make it better a mediki easier to communicate deadlines For example the one thing that happened this cycle that one of the projects neutron moved the deadline Anticipated the deadline for feature freeze without on just simply saying it on the mailing list and without a larger communication was a mistake I think that we will try not to repeat anymore So that product managers can be can be informed and You know, I'm thinking of also come up with a Keep investigating the keep this research that I started and try to come up with more more detailed suggestions and more detailed use cases than what I could cram into a 25 half an hour presentation on now because there are some companies that are being better than others They all face the same challenges. I mean don't get me. There is no Silver server bullet. There is no simple solution. It's it's a complicated. It's complicated and requires changes in the culture changes in Business processes requires a very strong alignment in strategy From the top who the whoever makes makes the decision to join OpenStack to the bottom to the individual developers And right now, I mean until recently did the main the main driver or We counted a lot on the goodwill of the developers themselves to be able to do the right thing Inside inside OpenStack and inside the corporations But we're getting to the point where this doesn't scale or we saturated the amount of Developers there are used to open source in general and to open stack in specific There they're not anymore. So the new companies that are joining OpenStack every day they don't have access to those talents or to those educated people and they need to learn and The learning process right now. It's too complicated Hi, I'm a my name is Tassos. I'm a student and I was a little bit wondering about What's the plans about? Including other communities that work in a more shorter terms like Google summer of code I know that OpenStack this year had his first successful Incorporation with those people so how about those people they have shorter terms on working and how you can include students Because that everyone is a professional right? Yeah, that's that's a fair question. So this this year we have Participated for the first time to Google summer code and we have been participating into another Another program like that to incorporate new developers called the outreach program for women and We keep doing that One of the things that I'm thinking of establishing is a more Specific program for mentors. So an easier way for developers to find mentors So that they can be guided into the OpenStack way of doing things but honestly, I think that we have done a lot of work on Inside the firms to educate them how to use open source or how to join open source projects on the legal aspects And we have done it a lot on the engineering aspect What we have not done very well or it's not very well known. It's how to do that whole Business processes and development process at the management level. So at the higher level than just the developers We we don't have I don't know there. I haven't met Companies where they have a very specific way of tracking KPIs for developers who work mainly upstream They are I mean there are companies where They trust their developers to do the good thing or they trust their operators to be able to contribute upstream or they have very vague ways of Assigning priorities and tracking those priorities. So there are companies where I've seen Where the developers have to be able to land a patch within a certain release cycle So within kilo we need to have this support for this specific piece of hardware or hardware line and It's sure. I mean it's something that you can measure. It's a very simple indicator But there are companies where it's not really that easy to to to say that I mean to identify a deadline and give it six months a window to achieve that that Milestone and you know get a promotion and get a bonus because you do that It's also very difficult to to be accountable in into this way into this Frameworks of large collaboration So lots of managers gets really get really confused by that and they don't know how to act and we're we need to do a better job at find out best cases and Highlight what doesn't work for sure and I I started this investigation this research because that because I think that we're not doing a good job At this and we can improve. Hi, I'm Adrian Otto I think it's important to recognize that the development process that we're used to when we do agile is a completely different Development process that we do when we need to collaborate across companies To learn how to do agile. There's lots of books. There's lots of resources lots of training What resources do we have to learn the distributed? community development process I Haven't seen much Yep, cool. So Right now here is gonna write something about this and I think that there are other smaller articles. I think Now I don't remember the name of the authors But there there have been some descriptions of the open source way of doing it's called the open source way by Do you remember? No, the Fogel, yes Carl Fogel the open source way There are other there are other articles that you can find online if you look for the Apache way so, but I think they still talk about very Self-confined environments and when open stack has changed in my opinion a lot of As challenged a lot of these models because of its size the size and the amount of people and companies involved and the speed of it at which it's moving and The way it is also managed. So the combination of things where we don't have a dictator we don't have We have these elections every six months who change might change the the technical leaders every six months we may change the Priorities for the next cycle for one cycle We may be going fast towards new features for another cycle for the next six months We're gonna be refactoring all the code and that confuses throws people off a lot of the time. So It's hard There are some places where you can learn about how to apply the open source way to your your scrum You're agile places and I gave some some suggestions like even if you do scrum Try to be summarizing everything in to summarize the results of your scrum meetings and your sprint planning into a public place where other people can plug in but It's not just that there is something more than it's to be done and we are at the beginning of a of a journey, I think How are we time? other questions Hey, I'm Jim rollen Higgin Going back to the release cycle have we investigated other release cycle models other than the every six months? That may be better for either users or developers Like I think a lot of times waiting six months to ship a product isn't really viable True so I think that there have been discussions about how the Tweaks and changes that we could do to the release cycle There has never been a real Change so Because because it's it's hard So initially we were releasing every three three months at the very beginning and then we went to the six months So one year is probably too long at this speed We might think there's gonna be discussions next month next week. Oh, sorry starting tomorrow. It's already here about Alternating cycles where there is one cycle with focus on features and one cycle is more about fixing bugs and catching up with paying off the technical debt We'll see how that goes Because some projects are suffering suffering from that and right. I mean products product releases over six months So, you know from that aspect also there are there are some Operators where they try to stay close to trunk. So they never have the concept of a release. They basically release every day And some other operators are waiting for the new for smaller clouds probably they're waiting for the packages to come from right out to boom to suze Other distributions out there. Thank you. No, I don't remember when I was supposed to finish 20 a foot. Oh, sorry. Okay. Well, thanks everybody