 You guys, do you have a lab up here or do you need anything? Sorry. Okay. Well, here you got one of the beer. Yeah. Okay, I'll flush. All right. So, are you good to go? Great. Do you have any audio or video with your presentation? Nope. All right. Lab is off for now. Just... I'll turn it back on. I'm going to start talking. Novel seemed okay and I'm loud anyway. With you ready, we're supposed to start at 10.40, but we may probably give people another five minutes, just let everyone filter in from the keynote. We still hear someone talking. I don't know if they're still going or not. Those just arrived. We are starting at 10.45, approaching speakers. Yep. So... Okay, people are still filtering in. The talk is not super long anyway. So, people might still filter in because it sounds like there's still chatter out there, but let's get started. So, that's me in all of the places. I'm a structure engineer at a company called Ride Cell and I'm here today to talk about open source sustainability. Before we talk about sustainability, let's lay out a little bit of scope. A lot of people have very specific opinions about what constitutes open source. I'm going to take a much more general stance away from the specifics of what precisely makes something open. I'm also not going to wade into open source versus free software at all. So that other way, let's look at a few types of projects of what I consider open source. A majority of projects buy the numbers. One person writes some code to do a thing and they put it up on GitHub because why not? This is how I got my start in open source writing plugins for a bug tracker called track. But the more visible projects are the big ones. These are projects so vast and omnipresent, it's easy to just think of them as part of the fabric of the universe. Some of those very big projects eventually actually have to build legal entities around them, usually projects that are partnerships between multiple large corporations. At a newer venue for open source projects has been VC backed startups. So the company acts as a steward for some form of open source project and then they sell stuff on the side. We'll get to the selling later. What I call mirage projects feel like an offshoot of that so big it's part of the fabric of the universe idea, except these often have frighteningly few core maintainers. We all saw this during Heartbleed where the entire world sort of simultaneously realized that there were a single digit number of volunteers maintaining this enormously important core infrastructure. Some older projects, especially big Linux packages, tend to feel a little bit stuffy compared to the newer wave of GitHub generation projects. But these projects have very important history. They've been designed to stick around for a long term as individual contributors come and go. Some open source projects can only be described as crazy ridiculous moonshots, usually led by a single person and sort of organized around, we just have to get this done. And in my very wide view of open source, there's a lot of things that aren't software too. Some of the legal specifics differ on how you license and maintain these, but the overall sort of contributor dynamics are basically the same. It's clearly not a complete list of every type of open source project out there, but it gives you a taste of the kind of breadth that I'm going to be talking about. So that's two words from the title, what about the third? Literally speaking, sustainability means taking a project that exists today and ensuring it will exist tomorrow and the tomorrow after that and so on. Money and time are important and we'll get to talking about those in a few minutes, but the fundamental limiting factor on most open source projects is the rate of burnout of contributors and maintainers. Most projects fail not for technical reasons, but because people don't want to work on them anymore. The most common symptoms of burnout in open source are these three things, a cycle of feeling guilty about letting down your users, leading to dread about working on the project, and finally bitterness towards the project itself for sometimes the users of the project. Open source projects have truly taken over the world. In almost every facet of technology, communal resources have pushed us further and faster and we want this pace of progress to continue. We need these communal and shared resources, but with this cross-pollination comes interdependence. While open licensing helps to ensure the possibility of someone picking up a fallow project, it doesn't help us if there's no maintainers there to pick up the fallen banner. And if I can digress for a few moments about why I think open source itself is important, I think we can build things together much better than any one team or organization. A lot of problems in software, there's so little value to the market that it's hard to define how the market would even come up with a solution for this. But collectively, all of us together can and we're all better off for it. Open source rules the world and I think that's great, but clearly I also think there's problems with the system or I wouldn't be giving this talk. How do we end up with a worldwide ecosystem of software projects built on the backs of a relatively tiny number of volunteers? If we want a globally simplistic answer, infrastructure investment has been lagging across the board, putting time and money in things like bridges and dams doesn't show returns immediately if a return can even be quantified. If your focus is only on the earnings report for the next quarter, things are gonna slip through the cracks. But a deeper structural answer, at least in my view, is in how we make things. I can only put my brain towards building one thing at a time, but while I'm doing that, I'm using probably hundreds of other projects and tools. Even in the best scenario, there's always going to be more users than contributors, and that imbalance has spiraled far out of control for most open source projects. Even a simple project like a Ruby on Rails web app, I just sort of made this up off the top of my head. The Rails app at the top of the list sits on a mountain of projects and libraries. As we build better and better abstractions, which is great and we should keep doing that, it becomes easier to forget the shoulders of the giants on which we stand. But while this interests sustainability lately, way back when, open source was just some crazy college students, several of them actually from right around here. Basically everyone that consumed open source also worked on it. So maybe I worked on GCC and you worked on Bash, but we all used the same sort of suite of tools and we all benefited from each other's work. Over time this balance eroded. More people pull value out of the system that are putting it back in. As the third-ish wave of open source developers are reaching middle age and deciding that maybe other pursuits are more important, often there is no longer fresh faces willing to take up the banner. And so here we are. It's hard to pick a patient zero, but it was probably either the LAMP stack or Ruby on Rails that got the ball rolling on corporate use of open source. VC funded startups, usually in the early 90s, realized they could switch to free as in beer tools and save money. And the VCs didn't know about this, so the checks stayed the same size. And when being cynical, these startups used that money to just pad their own profits. If I'm being nice, they used that additional money to double down and figure out how this whole web thing was going to work because that was still pretty new at the time. But all good bubbles must come to an end. And by now startups and big companies alike know that zero cost software stacks exist and many are of extremely high quality. Budgets have all adapted to this, so companies often have no choice, especially small new startups. They can't afford anything else. At the end of the day, open source and capitalism are at loggerheads. Capitalism requires or at least strongly encourages privatization and open source is the exact opposite of that. As long as the bulk of companies continue to draw more value from the commas they put back in, we're going to have a sustainability problem. Much of the interaction between capitalism and open source is related to intellectual property, but I did a whole talk on that last year, so that URL will be up again later if you want to check that talk out. So that's the general answer to why things went wrong. What about some specific projects that were ultimately unsustainable in one way or another? Two that are very close to my heart, React and RethinkDB. These were both originally VC backed startups, but they were open source projects from the beginning. The founder of RethinkDB put up a fantastic postmortem on what he thinks went wrong with the company side of the equation, and I think it should be required reading for basically everyone at open source. But roughly it was an extremely saturated market, both in finance and in mind share, and they just couldn't get a foothold. Whatever you think about MySQL and Postgres and whichever of those you prefer, they've set an extremely high bar for what is available for free, and people are just not really willing to pay for databases anymore. Big player in the Android open source world, Cyanogen and their former OS distribution Cyanogen mod. In 2016, there was a big management shakeup of the company, and they looked at their books and they decided the benefits of being open source in terms of contributions they were bringing in were vastly outweighed by how much they were spending on user support and the amount of money that they could pull in from people that would pay for it if it weren't free. So they decided it just wasn't worth it anymore. Two notable individual cases, Why the Lucky Stiff from the Ruby community and Mark Pilgrim from the Python community. I say this not as a call out or critique of either of them, but it's cautionary tale if you put too much of a burden of a single community on one person, it rarely works out well for either side. And a more general example, drive-by contributors, people that just sort of show up, they do one commit and they leave. Drive-by contributors bring really important fixes. They tell you when your user experience is bad. They fix weird rare bugs. And sometimes you can convert them into regular contributors or even maintainers eventually. On the other hand, it takes a lot of time and energy to shepherd that first contribution and doing that over and over and over again can be extremely draining. There's not a great answer here, but just be aware of the problem and step back from mentoring first-time contributors if you just don't have the energy to deal with it anymore. That's okay. So that's things to track from sustainability. How about things that improve it? A big part of any project is how it's run and that can have a similarly large effect on sustainability. A benevolent dictator for life is where most projects start from, projects created by one person and more contributors slowly join, but that one person remains in charge of figuring out the overall direction where the project is going to go. This can work very well for a young project, but eventually having to run all those decisions through one person can slowly create more and more friction and frustration over time. So some projects that started out with one BDFL will eventually fragment into a number of these smaller BDFL-like maintainers that all just sort of exchanged patches. Linux is the canonical example of this. There isn't actually one Linux kernel. There's like 40 of them and they just all work with each other strange ways. This can reduce friction for the maintainers that are exchanging patches, but it can be intimidating for a new contributor trying to figure out how to get a patch in and where it even goes. Some projects have a more rigid hierarchy for decisions. This makes it really easy for engineers to work out disputes. There's very clear dispute resolution processes, but it brings a lot of social overhead with it. Others use a federated council approach. Usually these are called SIGs, but that's sort of the newer term. Each group has final say over their area of expertise. And sometimes when the contributor base is stable enough over time, you can forego basically all the rules and just say, work on rough consensus when enough people agree just do the thing and if we need to change it, we'll change it. It's great for longtime contributors, but it can also really easily result in hidden power structures like the one person that is in charge but is not marked as in charge. And that can be utterly incomprehensible for new people. And there's a lot of other models. As with the project types, I can't possibly hope to exhaustively list these. Every project isn't going to be only one of these. They're going to pull together a set of governance rules that matches hopefully their specific circumstances and people. Conway's law states that the output of any team is going to eventually look a lot like the org chart of that team. It's intended to be a bit of a tongue-in-cheek joke, but it's also remarkably accurate. Choice of leadership and decision-making structure doesn't just affect the humans. It affects the software that those humans produce. Unfortunately, it can be a bit hard sometimes to tell which direction to move in to improve sustainability for your project. For some projects, more structure can help to reduce the burden on overloaded maintainers. On other projects, giving them more paperwork to fill out will push them even further into burnout. So when you're playing with governance structures, always make sure to listen to the people that you are governing and make sure to change things if it's not working out for them. As I mentioned before, open licensing is an important component of open source. It's not directly related to governance model, but I needed to put this somewhere, and one often influences the other. A good license ensures that even if all of the contributors from a project move on, at least in theory, the source code will remain available. It will still be up somewhere and people can use it, they can continue the project. What about cases where the license can be more than just a backstop? Full intricacies of GPL or copy left general licensing are better left to that intellectual property talk I mentioned before, but the idea of copy left is in its own way sustainability. GPL doesn't actually care about the money side of sustainability, but we'll talk a little about dual licensing later, which sort of gets into that. Exactly how it works depends on the GPL flavor that you use, but the overall idea is that if someone is going to modify your code, it has to be available to the users, and thus probably to you as the maintainer. One failure mode of those really big projects I mentioned before is that if they use a permissive license, someone can fork it, change it, keep the patches themselves, and in theory, that can siphon off resources from your project onto their fork. Again, in theory, if you put your project under copy left license, it heads off this whole cycle because you can always get their patches, and hypothetically, if you want them, you can introduce them into your own project. But really, how often has this happened? It's definitely not zero. Big projects, notably the Linux kernel itself, definitely have issues with proprietary extensions causing friction for both users and maintainers, but my personal view is that this is probably not going to be a thing for your project. I think that copy left probably for most projects causes more problems than it solves, but maybe your project is different. But okay, in the end, maintaining open source is about time, and time is money. The simplest funding model is to just not have one, be a volunteer project, and those are great. Some people just like coding for its own sake. Some people want to improve their professional portfolio, but remember, volunteering your time is an act of altruism. I speak from personal experience. It's real easy to end up in that cycle of guilt and bitterness that I mentioned before until you just never want to see a computer again. The next step up is working on open source on your normal work hours. This might be a project that your company maintains, that you're improving, or contributing a bug fix back to a project that your company uses, and it's a great option when you can do it. But a lot of companies don't really understand someone to give something away for free, so not always an option. The next step up from one person using company time is everyone using company time. It's not a long-term solution, but it can be a way to get a new or nascent project off the ground. You'll need to look into some of these other options to eventually repay your investors. That's the idea, at least. So now we have to do the round-up projects that are trying to be self-sustaining, to bring in enough income to support at least some of the time that you are spending on the project. By far, the most popular method is to sell support contracts. But this can be really hit or miss. Small companies often can't afford anything that takes away from their launch window, and big projects just think we'll support ourselves internally. Examples of support contracts, there's a lot of them, but MySQL and Java are two particularly popular ones. Sometimes the project itself, or the community that stewards the project, isn't really set up to be a commercial entity, so the maintainers will sort of split into a bunch of smaller companies, and they'll sell support contracts and other services through those, but again, with the same idea of funding their work on the original project. This is the dominant model in the Postgres community, for example. There's no Postgres corporation, but there are a sort of fleet of smaller consulting groups around Postgres, and a lot of the contributors work for those. After selling support, the most common models are based around having an open core and then selling related close source components or differently licensed components that add additional functionality to cater to a specific use case, usually big enterprise software. Sometimes there's clear places to add these paid add-ons, like big enterprises love SAML login, they love cross-team reporting, et cetera, but other times it can be difficult to find a place to sell these features without harming your open source users in one way or another. This has been very popular in the DevOps space, so Chef, Pop and Salt and Ansible all sell commercial add-ons. Dual licensing is a special case where you sell the same software they could get for free, but under different license terms. So the common case is you have a copy-left-based project, somebody wants to fork and modify it, but they don't want to make the patches available, so they purchase a license from you that doesn't have the copy-left restrictions. It's not very common these days. QT is probably the best example of somebody still using this. Similar selling support contracts is professional services. This is usually some mix of consulting to help them best use or deploy your product. Customizations, making custom plugins, custom modifications to the project for that customer, or hosting if it contains a server component. Again, examples too numerous to list, but Pivotal software is a big one. Pivotal services around cloud foundry. Grants have been a mainstay of the academic world since time immemorial, and we're slowly seeing more of them available for software projects. Like with other kinds of grants, you write a list of things you want to do, why you think they're important, how long you think that'll take, and use that to come with a dollar amount, and you go and ask someone for that money. I'm sure the PSF wasn't the first software open-source grants but the first that I ran into. A lot of other groups have copied them in the Python ecosystem and offer project-specific or ecosystem-specific grants. Going a bit out from that, the Moss grants program, or Mozilla grants more in general, are another example. Rather than being ecosystem-specific, they're looking for anything that helps with Mozilla's foundational mission of improving the open web. And then one step yet broader, places like Google's open-source programs office, they have a mandate to support basically every project that Google uses. That's basically all of them at the scale of Google. But a big problem with grants, like with VC funding or any other non-renewable income stream, it's highly dependent on how good your time estimates were. If it turns out that you miss some complex requirements and it takes you an extra six weeks to complete the project, that might end up coming out of your own project or the project might fail. And we all know that engineers are super-great at estimation, right? So, we reached the option most people go with. If you like this, how about sending me a few bucks on PayPal? This is so common I want to dive into some specifics. So the symbol of solution, straight-up tip jar on PayPal or Venmo or Bitcoin, if you like that. Really easy to set up and also very rarely useful. It can be heartening to get a few dollars every now and then, but if you're not okay with working for free anyway, it's probably not going to help on the actually funding your time approach. As a maintainer, I'd much rather, and I see a word of thank you email from one of my users than like $5 from one person once. A more stable solution is some kind of recurring donation. This lets you plan out how much income you'll have for the month. You can figure out how much time you're going to allocate. Patreon basically owns this market right now, but they haven't really reached out to the software world, so it can be a little bit difficult to get people aware that your Patreon exists and they don't really have support for corporate giving very well, so it can be difficult to get donations from companies, which leads into a problem that I think both recurring and one-time donations share. Basically, it almost always ends up being friends paying each other rather than the companies that are driving economic benefit from my software donating to me. In general, I don't want individual users to ever feel like they owe me for my software. What I want is the companies that are making money off it to get some of those resources back to me so that we can both benefit. But okay, some other donation options. I did actually run the Kickstarter to fund some open source work. It was successful, technically, but it was extremely stressful, and I really wouldn't recommend it. Like I mentioned with grants, I misestimated my timeline by about three months. And Kickstarter's like a grant. The amount of money you have is the amount of money you have. You get it once, and you better hope that that was correct. Some mid-sized projects that are looking to temp donations from companies rather than people will try to get nonprofit status, so their donations can be tax exempt, at least in the U.S. But creating a nonprofit from scratch is really hard, and I don't really recommend it. So it's usually a better bet to join an umbrella group, someone that already has nonprofit status and can accept donations on your behalf. I especially want to highlight the great work of Software Freedom Conservancy. They've helped dozens of projects set up this umbrella donations system. If you're interested in doing that for any of your projects, I highly recommend talking to them. You've got a contact form on their website for it. Outside of the simple logistical issues, a common problem with donations is working out how to divide them if there's more than one maintainer. Sometimes you can side-set this by spending the donations on project-level resources like server hosting or buying t-shirts or whatever, but it can very easily lead to hurt feelings or perverse incentives in the project if you end up paying by the line or by the commit. While money makes the world go round, many donations take the form of discounted or free credits to services. So for example, Rackspace has very graciously hosted part of the CI infrastructure for my chef cookbooks for many years. It's often easier to ask a company to donate whatever their service or their product is than to go and ask them for money. But okay, enough about donations. On to some less common but more interesting funding strategies. Various projects have tried to fund themselves by selling books. Most notably, the Lua language did this, but it can be a really difficult line to walk because you still need some free documentation if anyone's going to use your project, so you have to figure out how to balance those. Oftentimes these days, it ends up being that the book is available online for free, but hard copies are available for purchase, but that'll severely impact your revenue. Selling ad space can theoretically be a very lucrative option. Ad sales basically run the entire world these days, right? Setting up Google ads on a website is super easy to get started with, but doesn't actually bring in very much revenue. Negotiating your own direct ad deals can bring in a lot more money and give you a lot more control over what ads you show, but it's very time-consuming and difficult to get those kinds of things set up, and it all might be for naught if your highly technical audience are all using ad blockers anyway. Bug bounties are most common in the infosec world as a tool to try to incentivize people away from selling bug reports and attacks to the bad guys, but they're starting to appear for normal software, other non-security-related things like normal features or bug reports, but they're probably still too rare to really qualify as a sustainable funding source. Maybe we'll see this get better in the future. Worker co-ops are a really deep rabbit hole and they all run differently, but we're starting to see some based on open source. Snowdrift and Tidelift are both getting more interest, but we'll see over time if they are successful. I think right now it's still too early to say. And so we look further out into the future. With Science Medicine in the Arts is a lot of national and even international organizations that provide funding for stuff like basic research. Basic research in science is things that do not really have direct rewards immediately, but they form the basis for future work. That sounds a lot like a lot of infrastructure-level open source projects. So maybe one day we can see a national institute of software. There's some government grants available here and there, but it's not really a coordinated system. And if I can dream really, really big universal basic income would allow people to spend time building things they think are important, even if the market doesn't directly value them. So hopefully by now I've convinced you that sustainable software is important and you've got some ideas brewing for how we can all move in that direction, as I stressed before, the enemy of open source is most often burnout, not technical failings. Practice empathy both while using and maintaining open source software and it can help a lot. If you make money using open source, which is almost every company these days, be on the lookout for ways you can help or donate to projects that you depend heavily on. And most of all, always be ready for change. Maybe the only maintainer of a critical library for you will have a kid and decide they don't really care about it anymore. That's okay. Maybe you maintain a library or a tool or something and you get a new job and you decide, man, that thing, it's just not fun anymore. That's okay. And maybe the entire ecosystem will change. Maybe the DAOs on the blockchain will actually take off and that'll be how we write all software forever. That's okay too. I don't believe that last one's going to happen, but that's a different talk. We've all only been writing software like this for a couple of decades. We have a lot to learn today. Thank you very much. There's too many people that I can thank if I had time for their advice and wisdom over the years for how to run sustainable projects. But I'd like to extend an extra special thanks to Nadia, Russell and Eric as true luminaries of this space. I highly recommend all of their writings if you're interested in diving deeper. And we have a few minutes for questions depending on when we're supposed to end. When are we supposed to be done just so I can keep an eye on the clock? As someone contributing to like a 10-plus-year-old project with multiple contributors, I'm kind of feeling mild burnout about like, oh, I don't want to change this. I might break, I could probably break users. I mean, the only thing I can think of is just try to have both, you know, a long-term support version and a branch and a more actually developed branch. Do you have any other advice? Unless someone is like you, you do not owe them long-term support. I mean, if your project is robust enough and has the maintainer backing to support an LTS release, that's great. But otherwise, just don't. I mean, yes, users will be annoyed, but like, you have to factor in what is going to be best for you as a maintainer, as a group of maintainers. And like, you don't owe your users that much. It is important, but it's not safe. Just a footnote on that. It isn't necessarily a binary thing. You know, I mean, Rails, for instance, is more aggressive than a lot of people would be about, you know, ditching features that no longer seem like a good idea. But what they do is, you know, what they do is to deprecate and delete on a schedule. Thank you. But, you know, that is a good approach, but users will still be mad. One of the projects that I maintain, we went two years, two and a half years, doing no breaking releases because we were very worried that our users were going to get mad about breakages. And it was terrible. We're never doing it again, ever. Like, the users loved it, but basically all the maintainers said, if anyone ever suggests this again, we're all quitting because there was so much technical debt to clean up at the end of those two and a half years of being in it. And in the end, I think it wasn't actually better for users. Like, burning out and making all of your maintainers hate their job is not actually in the best interest of users, even if they think in the short term that they would rather not have breaking changes. So, it's, I think, a lot about communication and communicating expectations. Thank you. Any other questions? Awesome. Well, thank you very much and enjoy the rest of DevCon. Thanks. Bye. All right. Thank you. We'll start. Hey, good morning. Welcome to this flip screen. Welcome to this talk. How open source enables cloud computing in China? Our presenter, this morning, is Chou Huynh. Yeah. And Chou Huynh has more than 14 years hands-on experience in IT industries, solving with companies such as Stevens, who are a life oracle. He's also been contributing to a number of different projects, so I've got a BT and a TP livered. With that, we will turn it over to Chou Huynh. If there are any questions at any point, just raise your hand and I'll try to get a microphone to you. They want to make sure they record everything. So, please, go ahead, Chou Huynh. Okay. Thank you. So, let's start. So, my name is Chou Huynh from that heart community based in Beijing. So, today, I want to present a job called open source enable cloud computing in China. So, this is the agenda today. So, first, I will review the status of cloud computing in China. Secondly, we are integrated to insight into how open source technology interacts with the cloud computing in China. So, I want to share some experience of how Chinese companies benefit from the open cloud technology by engagement with the code to the open source community. The final part we are talking about looking for the future and the challenge ahead of us. So, this is the IT market 2017 worldwide line script. You can see here the channel totally on the market will hit to 2014 five billion US dollars. So, specific to the industry in China, this year it will reach to 140 billion US dollars. And on each year there is 100,000 graduate programmers. And now there are 370 billion internet users accounted for more than 46 near 46 total population. And in terms of the cloud aggregate market in China, you can see that the market will grow at least 30% year-on-year in the coming three of five years. And in the 2020 the total value will be hit 140 yuan. So, this perceived public cloud market. So, the public cloud market will grow 40% year-on-year in the coming five years. So, much like other countries, the public cloud in China also one company play dominant role. So, this is Alibaba cloud. It is almost accounted for half of the total market size. Just begin following the Tencent cloud and the cities that not only produce the cloud technology based on open source, but also they consume the open source cloud technology. So, this city is three representative cities. The first one is Shanghai. Shanghai is China Finance and the market center. So, in Shanghai where is the finance or stock or even insurance industry use the cloud technology based on the open source. The second city is the Beijing. Beijing is the high court of university or research center and also is the international center. So, here a lot of cloud technology based on the source are developed here and also whether it's individual or the enterprise use the consume the open source technology. Shanghai is the high court of Alibaba cloud. So, it is the newly emergency cloud technology center. So, if you have been travel to the China, you should could be by pressure by the three things. Actually, if you carefully examine the reason behind that, you can see all these cloud technology. First is the Douglas Bicycle is used our technology now is open source. The second is the AliPay or WeChat Pay. It kind of in Monday millions or thousands of transactions, those transactions should be powered by the cloud technology back end. And China high speed railway. Actually, China railway used the open stack set up the private cloud only the build up their online system and online shopping. Whether it's JD.com or Alibaba they they have not only the public cloud provided, they also use public cloud technology online online shopping. So from here we can see that today China is society nearly powered by open source technology. So the second part is that before we talk about how open source technology interact with the cloud in China, we just have a quick review of the open source journey in China. Lovely is until now the people just want to learn the open source even just propagates the open source technology by translating the book into the Chinese. So at that time in the people mind open source is just equal to Linux. So the second phase starts from 2004 to until 2006. So it only appeared with the brilliant idea that they created something like single tank. The single tank tried to bring out ideas and organized open source community events to promote open source technology in China. And the third phase is starting from 2007 to the 2011. Some power companies however still contribute with more to the open source community. So the fourth phase started from 2012 until today or even in the future. So open source technology since this phase is booming whether from individual or from government or from enterprise they have set up common sense and the open source they are really needed. In this period they are biological not only to consume the open source technology but also contribution back. Some power companies such as far away Alibaba by Dutensant they contribute much of back to the open source community. So a lot of events or promotion activities hold up in each year even for the individuals we can see on the GitHub there are many of them from China. We will examine the open source technology succeeding across the world so open source extent play a very important role. So success model also apply in China. So let's talk about open source success in China first is developer roles now on the GitHub there are 220,000 active GitHub contribute from China and also just I mentioned a lot of local organization with in a large community such as this foundation C&C and Apache foundation to try to promote open source technology in China for the user use panel view in China the largest population in China individuals or under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under under The cloud industry is count 93, and the prime use of MSOS is count 10%. So together there are more than 90% there have to want to use MSOS tunnel in their cloud industry. So how does it implement? So 90% of, at least 90% is they just use the in-house development machine. That means they set up their anti-center to develop the solution of product based on MSOS cloud technology. 16% also is used open source provider. That's mean like the buy from the solution from the third party owner, such as Lightheart and other companies. And the majority is used both two ways, in housing or buy from the provider. So cloud development, we have a sense of way that how many would like the server development. So a lot, most of the majority focus on just 100, 100, 1000 virtualizing server also. And for the container department, so most of the companies that they use the content technology just for the office automation and the elastic scaling. And for the server development department, most use the technology for the file storage and the block storage. So there are various industry in China, different industry have a different maturity model and they have different department about open source cloud technology. So the X axis indicates the development stage. They have found the integration to transformation, to innovation. The Y axis indicates compound annual growth rate. So you can see that the manufacturer industry education and government, those stay in the integration stage. So the attitude to the cloud is to try to use the open stack or some kind of open source test of some private cloud and then they care about added insecurity. For the second stage, for energy, transportation, health care, there's actually is already set of four stack cloud technology, whether it's the path or it's the eyes or some kind of technology. They try to emerge some kind of technology based on the access to the data. And for the innovation stage, there is the telecom, details and other kind of industry. So this industry has a lot of emerging technology based on the underlying cloud. I'm sorry, what is FSI? What is FSI? I think it's a foreign service or in my case. Oh, okay, very good. Yeah. So let's examine the four cloud stack in China, really powered by the open source technology. So from the bottom line, they use the underlying operating system in some centers or U-band tool for the virtualized laser, the cloudized layer, they use the KMMCEN is the hypervisor, they use the open source as the virtualized managed layer. For the content technology, they use the criminalize missiles as the orchestration tools, the use underlying environment is Docker. For the mid-layer or past layer, they use the open source circle, my circle, multi-dimensional kind of things. On the operating layer, they use the IoT open source, they use the Jenkins pipeline, they use the TensorFlow for Spark or at Hattubu, they use the TensorFlow for AI. So from the bottom to the top, you can send for the full cloud stack, they just generally related to the cloud technology. So kind of the third part is the, so if we know or not, we organize open source technology is very successful in the cloud industry. So if we finally find out the reason behind that, we can find out what we are doing today is just circle around the key element. One is project, the other one is solution, the other one is value. So this key element is something like innovation skills. They work together to just spin the innovation and move the open source technology forward. So in TensorFlow project today, we are incredible number of the project on GitHub, even millions of them. So the most important one, I think that we try to need to, how project can incorporate, include the solution. Those solution is really adjust the customer need and really resolve the real life issues. So in China, you can see many companies just work with the international community to bring partner together to select some projects with meeting this criteria. So furthermore, if the project cannot bring the value to the company which contribution highly, it's not a good project. So the project, good project with solution could be bring some benefit. Let the company to earn the money, then can bring money back and put more community source on the project and makes the whole project move forward. So in terms of the ingredient in open source, so first thing in Codemy is the code contribution. Yeah, that's important part. But beyond that, we also needed to put code into some valuable project, not just a common project, because if the project cannot nearly resolve the customer issues, no one buy that, the code is waste. So in China, a lot of companies just trying to, in order to achieve this, a lot of companies in China try to be, want to be a member of the project, try to be the technical leader or could be strategic interest group leader or even be the board member of the foundation, some kind of a thing. So just talking about code contribution, I just give one example. So for current project, we can see that this run by country, Chinese from China side is the long-term force. And for membership, you can see that the top level of the leader foundation, they are two companies from China, it's around five ways and it's 10 cents. The gold member is the Alibaba Cloud, Baidu. For the CNF, the top level is the Plenium membership. We are Alibaba Cloud, we are Fawei, we are JD.com. And the gold member from CNF, they are Baidu, Tencent, and ZTE. So also in the OpenStack from Direct Board, we are three members from Chinese company. So project contribution, this company contributed a lot of project on the OpenStack community. So I don't want to go through that each day so you can just click the bronze. So now kind of the final part is the look for the future. So OpenStack technology, especially in cloud, it's like a train, it's already started in China, so it cannot be stopped. So if you think I had three or five years later, we can see some trains here. Number one is cloud Olivia. So you can see that today in China, whether it's individual or enterprise users, you cannot leave without OpenStack technology. When you are in daily life, you buy things online, the backup technology is a cloud, it's from the OpenStack technology. So you can see in the current future, some kind of represent the technology, such as OpenStack technology, Kubernetes, KM, and IoT. Those OpenStack technology will be widely, more widely be adopted in China. The second one is cost in my local focus. So I can give one example. Let's find way, help China Mobile, the largest mobile operator in China to set up some cloud based on OpenStack. They set up a server cluster, but they want to have the same kind of plan to manage the server cluster. At the last time, on the OpenStack, OpenStack don't have the solution. So for the company, just to help the China Mobile resolve the issue and the company build the code back to the upstream. So in the future, you can see that the industry adopt the cloud. We are spread from mobile internet, into the traditional industry, such as China real way, some kind of things, from non-PCNP critical to the machine critical, such as the finance or bank systems, some kind of things. And number four is the local company try to set up open source ecosystem. They try to set up strong partnership. I can give one example. The buy the company, the open source Apollo auto driving platform is the intelligence driving car system. And it's almost cost more than 30 billion US dollars. So the open source act, and they try to bring the partnership from international or from local car play producer. So challenges that has security. So if we asked what's the number one concern when adopt cloud technology, so number one should be security. They still have some concern about this for when they adopt open source cloud technology. The second one is the open source governance. I can give you some here, the Tencent, which has the largest WeChat, right, the owner WeChat. They try to open source a lot of technology, but the code includes some kind of prototype only software and the same open source software. So they try to make learn to handle that successfully to open source that. So it kind of take time to force open source governance. Mentally education in China, in the college, we actually have some open source courses some Linux, send out some open science, some kind of things. So it's take time to educate from the high school students or the college students or the kids. So it's time for that. So Maglet to open source technology smoothly. We know that in China, a lot of people still use the legacy at infrastructure. So it's easy, not easy for the Magletian legacy at infrastructure to open source support infrastructure. Cooperation among the community and with the extended organization. We know that we are a lot of local community that just work with the internal community to try to have more events or activity hold up in China. And they also want to work with some internet standard organizations such on the 5G. They work with the International Telecom Union, some kind of things, work out NFV spec, some kind of this. So in the future, I think that they need a more some kind of cooperation. Okay, that's it. Yeah, question. So the longer I work in open source and have an opportunity to work with other people across the world, I realized that the basic cultural norms within a country or a group or anything really dictate sometimes interest in working in open source. I think when you mentioned some of the, on the last slide, you mentioned some of the challenges. Do you think that, and I asked this with all good intentions, but do you think that the general cultural feeling within China is one that can operate in this meritocracy as Chris talked about in his keynote, is it, are people comfortable with the interactions they receive or the feedback they receive, positive or negative? Do you know what I mean? Go back a little here. This company is how the largest, in China largest VChat, VChat, no. One year ago, they are not actually past based on need confidence, but this year, we have some need confidence holding in. So just the goal, one step is, it becomes a top member of that. Before that year, they don't need to consider to open source any technology. Now, there opens a lot of projects, only one year, one year. One year, that's great. Yeah, so whether from the enterprise or individual, they actually know that open source, they are needed, they are nearly the same. So I couldn't mind knowing that even in four or five way, there are more than five hundreds and you need to work on open stack. Yeah, so I just saw that in second session, let's assume this one. This kind of things, if we have a challenge simply in China, you are question about these four things. Those things could be all powered by open source technology. So I think that from the human many sides, they literally, little by little accept open source, they are needed, yeah. Great, thank you. Thank you. To follow on to his question and something that I was thinking about during the presentation, because this is all very exciting, of course. My question here then is, are there clear or open signals coming from governmental levels, proposing to communities and others that, hey, open source may be a good solution for handling some of your problems. Do they openly adopt the ideas that, hey, open source is good for people and they promote that? Or is it something that here comes from the grassroots as it were from contributors? Yeah, I can assure you. Those community is half government. Yeah, half government. It's founded by government. So a lot of the events or communities events actually get money spawned by local government to promote open source technology. So we are, go back to your question, we know that knowledge of China, US trade is better. One of the key concern is the IP issue, right? Intellectual property. So China government realize that how can we resolve that, open source, right? Developers don't have IP issue in most cases. So why not we use government source? So they encourage local enterprise to just participate in activity more in that, in the upcoming future, I think. So as you say then, the Chinese government and developers see this as a way of sort of removing the selves of the burden of, say, restricted IP and opening up to a more fruitful development and implementation for government, or for problems that need to be solved, giving them new tools to do certain things and getting them away from the restrictions of proprietary technologies. So would you consider it then an overall positive outlook on the part of the government toward open source and where they wanna go with it? So they look at it as an overall positive thing. Absolutely, they look at it in a very positive attitude, yeah. That's like, you can see my single is the why there's so many companies in China as the membership, very high level. Sorry, yeah. So this company is not, China is 72 years. Not five years, only 72 years, yeah. In the Lin's Foundation, in the 2007, we have another exact number, it's more than 15 companies from China join the Lin's Foundation in one year, yes. So in one of the slides you mentioned that ISVs play a role between the users and the vendors. Yeah, I see, you can see. So my question is that how important do you think a role of ISVs such as Red Hat and Microsoft is to encouraging more Chinese companies to be involved in open source? Like you said, in the last two years, you've seen companies become platinum members and open source. So can companies like Red Hat push other companies a certain example that I think you should open source or do you think that? Yeah, so you can see that for ISVs, for example, Red Hat, they just use the way to set up strong partnership with local provider, Fawei or Alibaba. We have set up cooperation partnership on the urban sound technology. How do these customers trust our solution or products? So that's the one way. Another is that we have some collaboration project with Collegy or University. We try to, we have some intern each year. We have in the front, different University. We try to learn to, at the beginning knowledge. We have tried to Chinese marketing that have some advice. They try to cooperate with us, University, not in Beijing. They have send us some people regularly to talk about urban sound technology in the Collegy students, yeah. So it's tech time and I mean that it's not gonna be achieve in one night, so yeah. Is it, are there possibly also security concerns with regard to the adoption or expansion of the use of open source technologies? That is, when you're locked into proprietary solutions, you're stuck at fixing things at the pace of whoever is making that product. With open source products, if you have a good sustaining base of engineers and developers, you can respond to security challenges, perhaps a bit more adeptly and more reflexively. Is that a possible consideration in this or is that not well known? I mean, there are concerns about security from many perspective. One of them is about data privacy because now the system is quite complex. They're different layers and they incorporate a lot of code. So when they use that, some companies don't have some export, technical port to handle the security policy. So that's why we need SV, yeah. For that, we guarantee that we have longer support, whether it's the bug fix or the security point of view, yeah. So I think that's for many large enterprise and they still trust that we are from the SV, yeah. Yeah, there you go, yes. All right, well, thank you very much. Thank you. Not really. Ah, actually, that is a little bit of a problem. Okay. Let me know how this sounds. Don't look convinced. Okay, testing again. One, two, three, four, five. No, nothing, right? Should have walked, showed with a collar. Is this okay? Can you hear me? Okay. See if we can do any better. One of us has improved anything. No, all right. Think this was the best. How's the volume now? Yeah, I'm looking up, but it just sounds like it's very good. Testing once more. One, two, three, test, one, two, three. I'm just gonna keep on talking until I get any feedback. Yeah, okay, awesome. Thank you. Oops. Grab some water, are you good? Yeah, are we waiting for the moderator or to just start? Okay, I'm waiting for him. My name is Simon. Our next is presenting Automatic for the People and I'm sure he can introduce his talk better than I can. If you have any questions at any time, let me know and I'll deliver a mic to you. Let's just start, Lund. So, hey, good morning. Welcome to DevConf. I'm Alon Murenek. I'm a senior manager at Synopsys, where I manage Node.js and .NET agents for Seeker, which is probably the best I asked product out there, but that's not what I'm gonna talk about today. First of all, I want to apologize for those of you who saw the title and thought this is gonna be about the greatest rock music of the 1990s. It is not. If you are expecting a talk about that, feel free to leave, no hard feelings. If you're still here, what I do want to discuss is realization I got in the last couple of years. So, at least for me and possibly for all of you, as we, or as I advanced in my career, I found out that less and less of what I do is actually coding. And this doesn't matter if you're an engineer and you're being promoted to senior or principal or staff engineer or whatever they call it in your company, or unofficially, if you move up the ranks, those so-called ranks in your, excuse me, open source project and become a committer or a maintainer, or even God forbid, if you turn to the dark side like I did and become a manager. Less and less of what we do is coding. We're suddenly expected to do all sorts of things like define policies and do code review and mentor people, all sorts of soft skills. I have absolutely no idea why they call these soft skills, though really, really hard to do. And most of us, oh, sorry, I will not speak for all of us. I personally am just not good at these things. I've come to meet a couple of engineers that share the same problems. Excuse me. And even if you are naturally good at these things, you just don't scale. We're just humans, there's just so many hours a day. And some of us also take some of these hours and do other things like sleep or interact with your family, things like that, which takes away even more of your time. So we're kind of in a problem here, or at least I am. The good news is that as engineers, we are really, really good at coding. At least I hope we are. Otherwise we're in the wrong profession. And what I'd like to propose today is perhaps a mind shift of how we take the skills that we are naturally good at and apply them to this new problem of soft skills. So how do engineers solve problems, generally? We automate things. This is just what we do. And what I'm gonna talk about today is how we apply this mindset to, as I said, a different set of problems. A short disclaimer, I'm not gonna discuss any specific technology or any specific framework. If I do, this talk will quickly deteriorate into Travis versus CircleCI versus Jenkins Talk. And well, I have a plan to catch in about 10 hours so we won't go down that route. There's a bunch of project logos here which are all links to the respective projects or share the slides afterwards. You just feel free to check them out. And the second disclaimer is I'm probably not gonna show you anything new. I'm probably gonna show you things you already know, already do, and just I want to get you to think about them in a somewhat different perspective. For example, why do we have unit tests or why do we need unit tests? That's like an open question to the crowd. The wrong answer is to find bugs. Another wrong answer is to prevent regressions in the code. Excuse me. Well, not exactly a wrong answer, but an incomplete answer. And this does not address the issue of soft skills. From a soft skill perspective, unit tests are a way for me to document my software, to document how it's used, how it should be used, and what I care about when I develop it in a language that's really easy to share. Because a lot of projects, sorry, try to document the code in this awful language called English, if any of you have heard about it. English is a horrible language. It's full of ambiguities. It's full of double meanings. If you ask two people from two different sides of the Atlantic Ocean, they can't even agree on how to spell stuff. Have you ever seen color with a U? Some people do. And worst of all, for a lot of us, English is just our second or third language. Now, if I want to contribute to a project, let's say in Python, I suddenly need to learn two new languages I don't speak, Python and English, which doesn't make a lot of sense to me. So if I can just learn one language, Python, and get a documentation in that language too, it's much easier to transfer knowledge like this. So from a junior engineer's perspective, when I am introduced to a new project and when I read through the code, I want to only read through the actual code of the project. I'll read through the unit tests or system tests, integration tests, whatever. And this will kind of tell me what considerations I should care about, how different parts of the system are already used, how they should be used. This kind of hints to what I should be doing. As a senior engineer, someone who reviews new code, I now not only review the code, I review the tests which are submitted in it, with it, sorry. And this is kind of a shortcut for me. Takes away a lot of these conversations of have you thought how your code behaves in a multi-threaded environment? Have you thought how this code handles the front encodings? These are really easy questions. If you have a test for it, you've thought about it. If you don't, you did not. And since we like to automate things, code coverage tools are your best friend here. Now I'm not saying every patch can have 100% coverage. I'm not necessarily even saying that every patch should have 100% coverage. But if I review the coverage report, this is again another shortcut for me to find the glaring blocks of code which are untested. And I can give this quick feedback. Hey look, this part of the code isn't tested. You may want to look back at it. So unit tests are kind of an obvious example of this. Why do we need linters? The wrong answer is to make sure the code looks nice. Has anyone ever had this discussion about tabs versus spaces? Does anyone know what the right answer to tabs versus spaces is? I'll give you a hint. It's not spaces. I'll give you another hint. It's also not tabs. The only right answer for this discussion is not to have it. Because tabs versus spaces isn't a technical discussion. It's a religious discussion. And like any other religious discussion, it's fine to have it, but you won't convince anyone. If you aren't pro-tabs walking into the discussion, you won't be pro-tabs walking out of it. Linters are a way of not having these discussions. First of all, linters are a way of documenting a consensus we already have in the project. And a consensus can be reached really early on when there's a single contributor, and then that consensus is really simple. I decided to use spaces, this is what we're gonna do. You can document this consensus in a configuration for a linter. Check this in as part of the source code. And from now on, you are not having this discussion anymore. Every patch that's submitted gets run through the linter, automatically fails if it doesn't adhere to your standards. And that's it, problem solved. Not only does this save time for me as a senior engineer that needs to review stuff, it saves time for the junior engineer that needs to submit a patch. Has anyone ever submitted a patch to a project he didn't know as a junior engineer? So you submit it, you wait a day till someone takes a look, get 25,000 different remarks, please remove trailing white spaces. Take your patch, remove 25,000 trailing white spaces, submit the patch again, wait for another two days till someone takes a look, then someone does. So yeah, well, we like our curly braces, up here and not down there, so you go through that again, wait another two days, and a week later you finally get an answer. Yeah, you know what, this idea can't work. You just wasted the week of your life. If I could get this feedback immediately from my local build, I won't even submit a patch with these problems. I'll get the feedback, effective feedback much faster. Also, from a perspective of someone who needs to review the patch, I then go through four cycles of mindless reviewing of I can't look at this patch, this tabbing is all wrong. I can actually focus on stuff I care. Now, most of these tools are highly configurable and you can tweak them in whatever way you want. Most of these are also pretty easy to extend or quite straightforward. The tool passes the code, you get back an AST and it's usually pretty simple checkers that go over the AST and either fail or don't. But so you can add your own rules. But even if you aren't using a tool like that, these are immensely useful. Has someone ever used GoFMT for Go? So I don't quite remember the phrasing, but it says something like, GoFMT forces a style which nobody likes, but everyone can agree upon. They just took away the entire argument. Which again, just saves time and makes this easier for all of us. My personal favorite, why do we need static analysis tools? So there's kind of like a hint here. But this is the wrong answer again. Static analysis tools from a soft-skill perspective are a really quick hack to amp up your knowledge. Even us so-called senior engineers don't know everything, we can't know everything. The world is just too complicated and we can't think of everything. Static analysis tools, if you think about it for a moment, is just this massive knowledge base where a lot of people who are usually experts in a certain really small field, like synchronization issues or encoding issues or I don't know what, took all this knowledge, coded some simple rules, simple best practices, rules of thumb into an automatic tool. And now I don't need to know everything about everything. I kind of have a mind trust encoded in my software helping me. And this isn't just for senior engineers. Even as a senior engineer who's been doing this for over 20 years, I still found myself learning new things from find bugs, from spot bugs, definitely from JS hint, just by doing something a bit different in my job and exploring something I wasn't very proficient in, like, hey, wait a minute, these guys are right. Never thought about that. Did my deed for the day? I learned something. Check. And of course, this has a much higher effect if you're a relatively junior engineer that doesn't have so much experience. Also, as a senior engineer, as someone who needs to teach and impart his knowledge, the key point here isn't a tool. It's usually the ability to configure the tool. A lot of these, most of these tools allow you to say what type of issues you care about or don't. There should be errors or warnings per file, per module, per directory, per whatever. And these are more than just personal preferences. If I configure my tool to ignore synchronization issues, it's not me alone saying, yeah, I don't care about synchronization issues. This is a codable way of me to transfer the notion that this is a simple utility library. We don't care about synchronization issues here because we don't have enough context to do so. We don't know where we are being called. A caller needs to do this. And by configuring these tools, I again share this knowledge with my junior colleagues and practically with people who are going to use this library too. Once you kind of get over this hump and stop thinking about these tools, it's just how I make the code better once you start thinking about them in the sense of how can I share knowledge? How can I offload mundane tasks off of me and have someone better do them, someone better being a computer? There are a world of tools out there. Far past these three categories I discussed for anything and everything for spell checking, which is kind of obvious. We all grew up using spell checkers. Not enough of us do this in code. Have you ever had a blocker bug on the last day of the release because someone had a typo somewhere in an error message? Yeah, so million lines of code, 40 minutes before we are supposed to release, one blocker bug about data corruption, someone else will handle that, and a typo and an error message. Absolutely no reason to have these. Licensing, if you want to make sure that all the third parties you use and all the third parties they transitively use have a compatible license to what you want to do. There are good tools that do that. Accessibility, I think there's a good talk about this fortunately right now in a different room, but do you make sure all your images have alt text for accessibility issues? I never remember to do that, but I have a tool that helps me. Security issues, which is kind of what I'm doing nowadays. Security is a very interesting field and frankly not a lot of people get it properly, not a lot of people get it at all. Those who do, a lot of them do not get it properly. Those who don't get it properly will usually brag how the software is secure because they did XYZ and of course they have no idea what they're talking about and it costs excuse the French a crap load of money to have a security expert shadow every developer. Let's have a tool do this. And so on and so on and so on. Basically if you can think of anything mundane, repetitive, which a computer probably can do better, a computer can do better and there probably already is a tool for this. And if there isn't, please go right one. Please ping me up afterwards. I'm always happy to contribute to such projects. My pet peeves in my free time. Well, so I can't finish this without an obvious lesson. Never believe anyone presenting in the conference. We're all liars. I said I won't talk about specific tools, I will. But one tool I just have to mention is Editor Config just because it's so awesome. So Editor Config is an open standard which allows you to really easily configure how you want your source code to look with regards to tabs, spacing, new lines, position of brackets, et cetera, et cetera. It supports any programming language I can think about. Any reasonable IDE supports us for those of you asking, yes, VI and VIM are IDE's. They both support Editor Config. So either natively or with a plugin, I can't imagine an editor nowadays who doesn't have a plugin for this. So it takes about 10, 15 minutes to write a config file. Just check it in with your source code. From that point on, anyone checking out a project, no matter what he or she is using, will format the code the same way. And you get rid of all these stupid arguments. But it looks great in the clips. Are you opening it in IntelliJ? Come on, it's not 1995. We should not have this argument anymore. Problem solved. This is the one tool I will specifically and explicitly talk about. One last lesson or one last point I want to make is there's a proverb about an early bird getting the worm. And in our context, the earlier we give this feedback, as anyone who's done a team lead training or something awful like that will know, the earlier you give the feedback, the better. Earlier in our context is the earliest possible in the development cycle. A lot of these tools will integrate into your IDE. And if I can write my code, then as I'm writing, get warning saying, hey look, you should have done this. Hey look, this is a problem. This is great because I'm already writing my code. I'll just fix it as I go. I don't need to think about it. If you can't do that, at least have your tools run in your local build so I can write the code, run the local build, make sure it's okay, then submit my patch. If you cannot, even if you can do that, the next level of feedback is unsubmitted patches. And again, regardless if you're running Trevor's or CircleCI or Jenkins or a little gnome who executes things, it doesn't matter. And so this way I submit my patch and get back some feedback. Hey look, you failed two unit tests, your indentation is wrong, blah, blah, blah. Even if you do have all of these in your local build, please, please, please leave them in the CI also. Developers are horrible people. I know I am one. If I write code and some tool prevents me from compiling it, chances are, depending on how much coffee I had today, I'll just remove the tool from my local build instead of fixing it. So please, please, please have this in your CI regardless of whether it's available or not in the local build. And of course, if you don't do that, you can always fall back to running stuff periodically or nightly, which is not a bad idea, but doesn't really, doesn't really have the same effect. I won't name names here, but I used to contribute to an open source project which had a release every quarter, give or take. And once a quarter, we would all get an email saying, okay, we are about to release in two weeks. We run, we run find bugs on our code. Here are the 260 errors. Here are all the issues with licenses, et cetera, et cetera. If any of these look familiar, please fix them. Two months after I submitted a patch, I have absolutely no recollection of what I did, no way to correlate errors to code. What usually happens is some poor sub, usually me, ends up once a quarter going through tons of errors created by other people's code and trying to fix them without having any idea what he's doing. This is awful, please do not do this. So I've covered quite a lot of grants here. Chances are you won't remember any of this. People usually remember one sentence from a session if the presenter is lucky. So there's really just one thing I want you to take away from this. As software engineers, as senior software engineers, as we progress, our job becomes not only to make the code better, but to make the other developers around us better. If you think your job starts and finishes with a code, you are wrong. Luckily, we can use the skills we have as developers to help us with this new job, even if it isn't just about a code. And with that, I'll open the floor to questions. Yeah, it's not a question, but just a reminder. I've been lead of a project for 10 years, and we have about 30 different developers going in and going out, and we just, two years ago, started discussions about coding styles, etc. And we found that just having discussions helps quite a lot because there were some frictions we didn't know about, people that didn't want to touch other people's code because that person uses British English, I use American English, etc, etc, etc. And just having those discussions help quite a lot just to have everything on one plane if it's possible. But sometimes we just have to say, okay, we can't agree on this, then different developers have to do different styles. But I think quite a lot of people are just scared of having discussions because it's like everyone has to do their style. But sometimes it's actually possible to agree. Okay, so I'll maybe rephrase something I said earlier. I'm all for discussion, I'm all for communication. Having discussions is great. The problem with these is they don't scale. It's great to have this discussion and decide we use British English or tabs or whatever. It doesn't make sense to have this discussion every single time a new developer joins. So we kind of got to a point and now a new developer joins and says, hello, I like tabs and I'm going to write in Australian English. Why? Because we just can't keep on having these. We need to reach a consensus, document it in a way that will actually fail your make process. And then it's kind of obvious, you don't get into these discussions on why are we using British English? No, we use British English. You will see for yourself the code just does not build if you do anything else. Let's stick with that. Can you say the takeaway again? So the takeaway for me is one sentence. Our job as developers isn't just to make the code better, it's to make the developers around us better. That's one sentence. So with that, I will thank you for your time and enjoy your lunch. Thank you. Thank you. Let's try it again. Yeah, probably that number. That number in five? Yeah. Five is probably fine. Okay. But at least now, we're going to skip ahead. That will be at 1.30 in the morning. Okay, cool. Go ahead and start when it's 1. Or you can tell me when it's 10. I'm going to have to go wait a couple of minutes long. Okay. So we've got like, there's someone that's, I guess this is running in the background. Yeah. I mean the lunch after the talk after the lunch, either everyone's sleepy or still eating. I don't know if after or right before lunch, which one is the worst, but there will be beer in the talk. I wish I could give it out too, but I think the university frowns about unregulated distribution of, we can start and then people can filter in. Sure. Good afternoon. This is Aaron Albridge. Aaron Albridge. Aaron is a community builder at Elastic in a family who organizes the DevOps CT meetup and DevOps days art work in 2017. So he's interconnected. He is passionate for connected humans and music technology to enhance our natural information and connect with each other. And there are references to this that you can find in the crazy part. I'll revisit those too. His talk today is course to craft, building co-ed community. And with that, one more question. Hi. So yeah, that was one title. And I've also called, that was another, called this talk, craft as in beer to kind of play on some of the phraseology you're probably familiar with from free and open source software. The pointer wasn't on. And that's where you can actually spell how to find me for email or Twitter. I'd love to connect about this talk on Twitter because this is an idea I kind of was passing around. And I wanted to get some feedback on it. So I'd love for people to connect or like mention me in the middle of the talk if you're like, oh, I haven't thought about that. Like that's totally cool too. So for some level setting, I want to talk about free software. It's kind of where I borrowed that craft terminology from. And I didn't realize how much red hat presence there would be. So I probably don't have to be as detailed in this part of the conversation that I might have to otherwise. But I want to go through it a little bit just so at least we're all on the same point and same understanding of the past here. So in the beginning, software wasn't covered by copyright at all because largely it's hard to copyright like the order of holes on a card. And like computers looked a lot different than two. So it wasn't really a concern until the mid-70s. Contu, the government agency whose name I forget but remember the abbreviation, argued that basically as far as software represents the original thoughts of the author, it should be covered under copyright laws. And by 1980s, it is actually defined and added to the existing copyright in the US. And for those who know the history, recognize that about 1983, that finally bothers Richard Stallman in the left to leave the MIT artificial intelligence lab and form the Free Software Foundation. Basically at this point copyright exists in the way the licenses are written. So restrictive that even if you wanted to share something like in MIT with another department, you might not be allowed to share the code between each other. You can't really make improvements to it now because it's restricted. And we're still at very early ages of computers so there's a lot to change and improve upon on a daily basis. And basically that's the gist of it. We are always taught growing up that you should share. Like if you've got knowledge or a tool that can help someone out, you should share that with them. Growing up you're not told, hey I'm sorry you can't let someone else play with your toy because they haven't actually bought the license for that toy so they're not allowed to use it. Like that's not how society works. So it didn't make sense that this is applying to software too. We have to have a giant picture of Richard Stallman if we talk about Free Software, I think that's part of the GPL license. So to go over briefly the four freedoms you might know but to refresh, it's freedom to run the programs how you want so you're not restricted to an operating system or a way to actually use the program. You should be free to study and change how the program functions, which is pretty much means access to source code is your requisite for that. So source code is going to have to be available so you can see how the program functions and alter it if you want to. Freedom to redistribute copies, if I say hey I know you're having the same problem as me, this is the software that worked, how about you try it? Like that's part of Free Software. And the free to redistribute modified versions. Yeah I had that bug too, here's my fix, try it out it'll probably work for you. That's all the basic freedoms of Free Software. So of course that's a good new. Richard Stallman forms GNU, starts trying to write the first open operating system because there isn't one at this point, all the operating systems have to be licensed. And partially successful, built a lot of the major tools that make up an operating system but the more interesting portion that came out of that was copy left and the actual legal frameworks for how to have free software. The problem with just releasing software public domain without copyright is that someone else can then change it and copyright that and then you lose all those fixes and you lose that connection to the community there. So the actual licensing that came along with the GNU-GPL is what's really important to come out of the beginning of that project. That's what enables all of the future growth of open source and free software. We do eventually get a kernel that's open source and Linus Torvalds releases in 1990. The Linux kernel and you get the GNU slash Linux operating system using tools from both departments. What was actually the interesting fact I found here was in listening to them talk about it, GNU is trying to build this micro kernel structure. Very Unix thought each program does a different thing and Linux beat them to the punch mostly because they built a monolith. So interesting conversation as you think about how we talk about micro services and monoliths today. Just thought it was interesting. Adoption grows pretty rapidly of Linux. These were numbers from Forbes but you can see pretty much as internet ubiquity goes up so does adoption of open source and Linux because that's where largely the community lives that wants to use computers. It's a lamp stack. Linux basically takes over the web. It becomes kind of the de facto way to run websites especially if you're something like GeoCities and you've got a bunch of customers that want to run a bunch of websites and Microsoft and IAS only runs one website per box at this point. It becomes more cost effective to run a bunch of Apache servers than a bunch of Windows servers and IAS. Today that's pretty much what Linux adoption looks like on the web. 97% of the top 1 million domains are run on some form of Linux. 92% of all EC2 instances are some form of Linux. So pretty successful model as far as just adoption of the software. How many people have read or know of the Cathedral in the Bazaar essay? Most people, cool. You kind of know the gist. Lost his name, Eric Raymond. So Eric Raymond writes the Cathedral in the Bazaar to compare the two existing software development models. The Cathedral is very much the current proprietary. There's not a lot to say from the community but they work on it and then eventually it's released to the public and people can partake on it. Really long feedback cycles and not really taking outside input. Meanwhile the Bazaar, while it can be a little bit chaotic and messy is constantly changing and getting feedback both from the users and the people that are working and maintaining and using that software. Everything iterates very quickly. It's interesting to note that it's not that one is necessarily better than the other but they produce different things and they look different. Everyone remember this logo? So another successful move there is in the 2000s, of course Netscape Communicator has a problem. Anti-competitive practices and philosophy aside, they can't make money when 92% of the web is browsed by Internet Explorer given away for free. So taking inspiration from the open models in general and the Cathedral and Bazaar essays, they decide to open their source code and invite the community to take part in the actual build process. Which forms the Mozilla Foundation. So the Mozilla Foundation forms and eventually of course release Firefox as the big first open source wildly successful browser. So that's really cool and shows how you can have success with this open model but what's really interesting about it is where all that success comes from. It's not just because it's free as in you give it away but it invites the community to be part of that software development process. You're actually going out and saying hey, is this useful for you? And you start getting things like hackathons where people use all this free and open source software and start sharing tools to solve real problems in their communities whether civically or niche interest. You have a lot of mentorship that builds up inside this open source community. You have meetups, they had install parties because we didn't have an official support source. So the community turned to itself and said hey, I want to get started, how do I get started? And you'd have the experienced members come up with all free and open source software. How can I help you run it? So that largely drives the success of this whole format. There had to be people to support the open source movement. And we of course can say it's successful because free isn't beer but that's kind of problematic in that it's talking about giving it away but that's not really the exciting part about it. Besides if I say free beer you're probably not expecting to get like a bunch of Trillium or Treehouse to you. You're probably expecting like a bunch of Budweiser. So there's kind of this connotation that if it's free given away it's probably not the best quality. And it's not even that apt of an analogy anyway. It's usually more free as in puppies. Like here you can have it for free but there is some care and feeding and maintenance you're going to have to do. So it's probably going to cost you something if not up front. So again, not really compelling except for the part that we're talking about the freedom of the software. Free isn't speech. It has certain rights associated with it that lets you use that software how you want in your own manner without being dictated by a company. So if I were really clever I could come up with some like free isn't speech, free isn't puppies, free isn't beer, free is in some other thing and we could have some cool freedom discussion but I got distracted by the beer portion so we'll have some cool freedom discussion. So craft beer is really interesting. I want to do a little bit of a history run through here as well. On 1887 there's over 2,000 breweries in the U.S. Those in U.S. history know there's a problem in about 30 years. Now there are officially zero breweries in the United States because it's now illegal to brew and distribute alcohol. It ends in the 30s and World War II happens. So if we're really building up this industry again it's a vastly different world that they're building it up in. A lot of the tradition is gone. Suddenly you can start shipping things nationwide. You can start advertising nationwide very easily. So the popularity comes from beers that are consistent. I can get the same beer in Boston as in LA and it's going to taste the same. I'm advertised to, they're largely just generally appealing not like the niche flavors so it's pretty much the least offensive things start spreading. By the late 70s which is now as far from prohibitions were before there's only about 44 breweries in the U.S. so a far cry from 2000 even if that's underestimated. And that's mostly because homebrewing is still illegal. So 1978 homebrewing becomes legal again. In the 1980s there are roughly eight craft breweries. So we wanted to find craft beer a little bit so we'll go through what actually is craft beer. They're small so they have less than 3% of the total like barrel production in the U.S. Independent meaning they're not also owned by a company that has more than 3% of the barrel production and traditional. So they care a lot about where beer came from. The flavors still come from the brewing and fermentation process. It's not some completely off-the-wall artificial alcoholic beverage so like Mike's hard lemonade doesn't count as beer. So that's kind of how they're trying to lock that definition with traditional. So you might recognize some of the early brewers from the 80s. These are all the craft beer companies that existed around that time. Anchor has a great history like they're part of craft beer legend because they almost disappeared in the mid-60s. They got purchased, had to kind of modernize their whole infrastructure released actually the first American IPA considered to come out of anchor and so by the 80s they're pretty much up and running as one of these local breweries that could still stand and purchased largely because Fritz Maytag didn't want to see his favorite beer go away so he bought the company. New Albion was one of the first ones where a home brewer turned business owner only around for six years but largely inspirational for every home brewer who said I bet I could sell this and of course Sierra Nevada is still around today and largely responsible for the quality of hops that are available to home brewers and micro breweries. Did you say Fritz Maytag? Did you say Fritz Maytag? Yeah, bought anchor in the 60s. Is the Maytag Maytag? I know it's the beer Maytag. I don't know it's Maytag. So, yeah, Jack McAuliffe was New Albion who was only around for six years. They've come back a couple times and Ken Grossman is Sierra Nevada. Largely the 90s come along and we have sort of getting a little bit more adventurous. Sam Adams is proving that even a smaller brewer can show up in your sports bar. They don't have to be like some niche thing. We're getting a lot of different flavors. We have the home bellies. We should use all of the hops always. Allagash is bringing over all these Belgian styles. So, Blue Moon pretty much wouldn't exist if Allagash didn't brew Allagash white in the 90s. And obviously we've got some of the major staples for today from Treehouse and Trillium kind of pushing the edge of what you can do with beer. Oscar Blues who made cans cool again and other things going on as well. Pretty much the statistic is within about 10 kilometers of a brewery. This is not quite the smallest state in the U.S. but where I live and that's all of the craft brewers in Connecticut at this point. Yeah, so it's pretty popular. In fact you can see there's quite a few compared to 8 from 1980. There's now over 6300 operating in 2018. And what's really interesting is the way it sort of follows this dot-com boom and bust if you watch it. It grows through the 90s and kind of plateaus at the end and then jumps up again when 2010 comes around. So again the economic success is interesting but isn't what's really compelling about the story just like free software. So I want to tell a couple stories to sort of get at what's really interesting about craft beer. Does anyone know Smuddy Nose? They're relatively close by. So the great story about this is the owner of Smuddy Nose bought it by accident. He was currently and still is the owner of Portsmouth Brewing Company, the Brew Pub in Portsmouth, New Hampshire. And I forget the name but some other brewer owned the location and the materials from Smuddy Nose and went out of business and it was up for auction. So he came by not just to buy it because he needed equipment but he wanted to be there and welcome whoever bought this into the craft beer community. Like hey welcome, what can I do to help you out? Do you need information? How can we help each other? And of course if you're at an auction you throw in a lowball bid anyway because why not? And then he won. So he suddenly was faced with oh no I need a company and beers to brew because I already have one so I need to go reinvent all these recipes. They're able to do all that as you know they're around but the kind of the cool thing is they relied on the community to help them build this company that they weren't expecting to have to build. All the labels that come are pictures of people in the local Portsmouth community. They use a local photographer to take them. The original old brown dog is one of the brewery employees dog at the time that posed for the photo and they took a picture of. So it's really cool. They really gave back and made sure the community was part of their actual growth of their company. So Booth Bay craft brewery you might not know about. They're a bit further away and I don't think they distribute much outside of their area. But I visited them a number of years ago and they had a really great story about what they were doing. They hadn't finished yet so now I get to tell this whole thing now that they're done. When they started out they wanted to do their first beer about Booth Bay kind of established themselves there. So they gave out hops to the locals in Booth Bay. All the community were part of that. They had them grow them throughout the growing season and the harvest season invited everyone back for a big picnic and party and beer and food. Harvested all the hops that the community had grown. So they actually used these hops to brew their first pale ale. It happens to be 6.33% ABV which is like the area code for Booth Bay. So they obviously name it 633 and release this community collaboration beer like the whole community actually got together to brew this beer. But they didn't just want a brewery they wanted to build a brew pub. And a true public house that was a part of the community like where you go to talk about what are our civic matters? What are matters of our community that we need to fix? So they started building this brew pub and asking the Graph Beer community for support and companies like Dogfish Head and Allagash actually sent them lumber to help build this big... You can tell it's all the hardwood I mean construction that they build this out of. So they have lumber that's carved with like the Dogfish Head logo and the Allagash logo and the community started donating that too and they carved their names and location from where all this lumber came from to build this brew pub. Now it's completed and you can tell it's not just for people who like craft beer but their families can come in and bring their kids and they can be part of the whole experience that's there. They become that gathering place for the community. And then one more is about a beer store. So they're not a brew pub but they sell beer. So City Beer is in San Francisco and when the owners opened it everyone said you can't sell just beer there's no way you're going to end up adding wine you're going to add spirits like you're probably not just going to sell beer out of the store but they tried anyway. And it's set up kind of like the idea of those wine bars where you can taste wine and then buy and bring it home kind of idea. So not only were they successful craft beer becomes successful and everyone comes and hangs out there but when people would come they'd bring friends who might not drink beer and they could get in that situation and they'd say hey do you have any wine and they'd say no and that could be the end of it but they would start saying hey but what do you like to drink like what's your favorite wine and they'd describe you know I like a merlot whatever it is. I'd say awesome what do you like about the merlot like what's interesting to you about that and they'd drink it again and they'd describe maybe the flavors that they like about it or whatever cool line words mouth feel that's probably one of them and they'd say awesome you know I when you describe that I think of this particular beer do you want to try it and just see I know you don't normally drink beer but do you want to try it and more often than not they'd have someone like it and say you know I'm so glad you didn't have wine here because if you did I would have just drunk that and not known about this new thing but because you didn't have it I tried something new and discovered this whole community that I can now become a part of so just by including people that normally would be outside that community and say hey do you want to try and see if you like this they got a lot more people to join and grew their local following so how does this all tie in is I want to maybe coin a phrase called craft software it's a little bit hard to define as either free and open or measure it as small because it's really hard to measure the output of software like do you have less than 3% of the total barrels of code that are output is kind of a weird measurement to figure that out and I fully understand there's some balance between being able to give away your code and needing to like put food on the table so it seems unfair to exclude people that charge for their work I understand that and for some companies like for many beer brewers their code or their recipe is what makes them apart that's what makes them different than their competition so I had some ideas I want to wrap around this and I thought it was more interesting to get at the heart of what really drives both of these two industries of free software and craft beer rather than what defines the edges of who qualifies so I think if you're doing craft software you're proud of your work and you care about its impact that it has on other people and the community it's released into I think there's an innovative tradition there which is sort of picking up ideas of the past paying respects to where they come from and seeing what else we can do with those maybe turning them over in a new light like this talk for instance is not whole cloth grown but built on the shoulders of all the past of free software all the past of craft beer putting those together turning it over a little bit differently and coming up with something interesting even our software development movements like devops isn't wholly new it's kind of looking at where open source came from and where agile is going and saying how can we apply this to operations and does the supply back now to development openness and sharing I think is characteristic and while I want to say you probably should open source everything you can it doesn't mean you have to open source the whole company companies like Netflix and Twitter open source tools they use that aren't what set them apart necessarily from the competition but our ways they found hey we started doing this practice and we found it really helped us maybe this can help you too and they talk about it on stages and they go to meetups and they open source chaos monkey or other things that aren't their secret sauce but help other people get better at what they do and I think the community is big I think that's the biggest part is that the community has to be involved in it in some way they're invited into the production process of your software whether it's again through you get to be an open source company and you can just fully say come and help us build this or it's that tight connection of how is this working for you can we do something to improve it or it's even looking at the community outside the context of your product and saying hey we're in Denver or we're in Boston what's happening in Boston that we need to care about that we can give back to is there other things happening that we can help our community because we're bringing in venture capital like how can we give back so that's sort of the four hallmarks I think that define what I was thinking of of Kraft software I'd love to hear feedback on that so I don't know I have a little bit of time I think right if anyone has questions or thoughts I'd love to hear them now and if not I'll throw up my contact information again so yeah we got a microphone going to you thank you first let me say a wonderful wonderful presentation and thank you for the analogy that you use between Kraft Spear and software let me self-identify I am not a developer I head up a nonprofit I came here because I want to be more informed and learn about the community so that I can ask common sense questions of my webmaster but thank you for the wonderful presentation a question that I have I applaud the community for what you've created I think that it would be great if other industries followed what you're doing it would be especially the financial community, financial and the educational community so my question goes to your last point about community involvement because it's not new to the community that it's lacking a little bit of diversity in terms of women and people of color so my question and I will also preface it with a quote by Albert Einstein so my question is how do you think the community is addressing it at the root the root of having more diversity specifically women and men of color people of color and I would preface it also with a really powerful quote by Einstein who said you cannot solve a problem with the same consciousness that created it you must stand on a higher ground I have some ideas of solutions but I would like to hear from this wonderful group of people who are attending here in this particular workshop and I'd like to hear your thoughts thank you so I'm going to end up giving a famous answer for the company I work for too and say it depends which means I think it's a really good but really complicated question and there's no easy answer for me to say oh it's addressed this way I think part of it depends on what you consider the root of the issues and part of it depends on which communities you're looking at addressing it I can only speak for where I've been involved part of it for our DevOps days and actually DevOps days everywhere for the most part are explicitly required to have codes of conduct and they're usually explicitly inclusive or aggressively inclusive meaning they make it a point to reach out and invite people in not just hey we're here if you want to join but we really want you to be here be a part of our community and we're going to try we're going to make sure that's a safe place for you to be that it's not going to be invited in as an afterthought and then ignored once you're here it's about the inclusion not just the diversity of attendance I've seen a lot of tech conferences do a lot better as far as someone's just being conscious about it so in the past you might set up a conference panel or a speaker list and it's just whoever submitted you selected what you thought was good I've seen a lot more conscious effort in making sure people see themselves represented at conferences as well that you can be part of that community I'd love if anyone else has other things they've experienced maybe or other ideas then I would love to hear that too just to follow on his comment this conference has a code of conduct and it is rigorously enforced and there is a staff that is dedicated to responding to code of conduct complaints so there is a conscious effort on the part of companies and developer conferences to at least respond in a responsible way to the various pressures and problems that have occurred over time I just wanted to mention one specific thing are you aware of Outreachy? I just thought about it. Yeah, because that's a specific thing that was created in response to these kinds of concerns that is being done, that Red Hat is involved with so I just wanted to mention that one I appreciate crowd sourcing an answer and a talk about community too it's perfect. So I think it's important to not get caught up in solving the problem immediately especially for a problem like this it's not going to be like an overnight solution if you apply the agile method of software development it's all about iterative improvement so you make tiny changes that get you closer to your your goal so every time you get feedback for example from your community saying well you didn't include enough diversity in this last conference or what not or last activity you take that and analyze it and make an effort the next time you do it to reach out and include those people that may have been underrepresented so part of it is helping to some of the concepts for development and innovation of not thinking of failure as something you need to defend but as a lesson to be learned I think is a big way to apply that to your community development as well when you get that feedback of it wasn't whatever it was not an adverse enough panel instead of jumping on the defensive you can turn towards that thank for that feedback how can we improve if not let's see what we can do and iterate on this for next time do you have a question as well with regards to inclusion I've been part of a group organizing computer events in Norway we had traditionally about 5% female attendees our goal was 25% and we said ok we'll reach this within 7 years so quite a long span it was we had some different things and we said ok let's embrace failure we can try things and say some things will fail that's ok we did some things we said we have to focus on our volunteers we have 400 volunteers and 7000 participants so we're just on Facebook we show faces of different volunteers and their story why they got involved and we always focused a bit more on the female part than the male part and also the female the women that participated once and then disappeared we rang them up and asked why did you disappear what could we do to involve you more and we now reached 25% women and our next goal is and our next goal is like people of color because that's a small part of Norwegian population we want the event to actually mirror the entire population and yeah awesome are we ok so I'll throw up my this is some info where you can find the slides or I wrote a quick blog to also solicit comments there if you want to comment publicly or you can just contact me one way or another I would love to hear more in depth feedback or whatever as you I'm sure walk away and talk about it more that'd be awesome I'd love to continue a conversation about this because I think it's a cool idea thanks how can I find my slide so when you update it doesn't always look like I don't know if that's what you're having with yeah we can alright good afternoon and welcome to the job operating systems and TTR versus NTPF our presenter again is Erwin Olnich and if you're here in the last session you know we're going to develop over and if you don't we will tell you hi I'm Erin so I'm a community engineer at Elastic and I met a couple DevOps groups in Connecticut as well so I'm one of the organizers for DevOps Days Hartford and the DevOps Meetup that happens around that area as well so I wanted to talk a bit today operating human systems so if you were here for my last talk I sometimes take inspiration from weird spots so this is sort of a backwards application of studying how we operate complex computer systems and how it sort of applies to our interpersonal connections that operate those systems as well so some definitions and level setting will happen at first and then I'll kind of dive into some more practical things you can actually do so does people know MTBF and MTTR terminology cool I'll go through it so this is what they literally mean for computers is meantime between failure or meantime to repair so basically metrics for how your operations are doing either the time between actual failure cases or how long those failure cases take to resolve depending on which version of the abstract I have altered them to say meantime between fallout or meantime to relationship as it applies to people so if you were fine with your mother for the past year and you had an argument and speak for three days okay again that year was the meantime between fallout and those three days we were meantime to relationship so we also want to wrap some definition around complex systems because words like complex and complicated and simple are everyday vernacular but there are some specific terminology as we start to think about complex systems theory and how people and machines interact has anyone ever seen this graphic before awesome I'm really excited that I get to share new things with people so this is the Knephen framework on how it relates to complexity so these lines between it aren't really quadrants they're not hard barriers between things but they're a little bit soft they're more about categorization of either a situation or a system or whatever you're working with simple are really simple they should be understandable by pretty much anyone so a Lego set that's disassembled might be a simple problem you look at it and say these are Legos that are disassembled you categorize it this is the Batman mobile Lego set and you respond you follow the instructions to put it together so even a child with appropriate motor skills can put that together you don't need any specific knowledge in order to do it complicated things require some domain knowledge to be able to figure out so maybe your parents house internet is out and you need to go help them because they can't figure out the problem you have to apply some domain specific knowledge and figure out looking at this I can see the SSID isn't being broadcast so now I know to go check the router and I can see it's not plugged in let me go re-plug that in and everything's resolved so it might not be hard to fix but it required that specific domain knowledge of where to look and where to actually resolve the problem another example for complicated is like car issues the front right corner is making a noise you might not know what to do to resolve it so you bring to an expert to look at it and fix the problem so complex issues as they're different from complicated live in a world that's pretty much unknowable as a whole there's no way you can look at a complex scenario and actually figure out where all the moving parts are some of them might be concealed to you or more often acting inside a complex system causes the system to change so increasingly this has to deal with the actual computer systems we operate we're building micro service clusters on hardware that someone else owns in places we're not quite sure where it is that have unreliable internet connections between them or virtual networks we can't really conceive of the whole thing as they grow more and more or if you saw Noah's talk earlier talks about how any simple Ruby on Rails app might involve a ton of different libraries and dependencies that you just can't conceive of everything at once so usually in a complex scenario you have to do something first to figure out where you're at you can only really understand the effects of actions in hindsight so this is a place where you're usually doing new things you might build a prototype see how it works respond to that behavior and iterate on that again build a new prototype see how that responds go so on and so forth chaotic are usually incidents like something went wrong house is on fire so the first thing you do is make fire not there right like there's a problem act first then go back and figure out what caused it and try to resolve the situation there's sort of this like fold between simple and chaotic because one of the easy ways to end up in a chaotic situation is miscategorizing something as simple I know what this is I will do this now everything's on fire right so there's that quick terminology that's why problems don't sit specifically in each quadrant but can move around as developers and programmers and technologists we tend to flock towards the complex domain usually when we solve a problem even if it requires domain specific knowledge like how do I configure this server we then get really bored because we already solved that problem so now we automate that and now the problem is how do I maintain this automation process or how do I control a fleet of servers or how do I just keeps getting bigger and bigger we constantly move towards the new and novel or new emergent a novel is I did not expect that to happen and you end up quickly in chaos so I would argue also humans exist in this complex problem space largely even to ourselves we don't always know what's going on we often go to a doctor who might be able to deal with some problems and not others there's a lot of maybe this will work maybe that will work there's differences in how we act day to day the same like input doesn't always give the same output in complex scenarios yeah I think there's a lot there there's stuff that goes on behind the scenes in our heads that we still don't really fully understand and I think our relationships are even more complex because now we've got a bunch of complex systems we're interconnecting we might not even quite talk on the same level so now we've got all of these different barriers that we either see or don't see that we have to traverse kind of an example of this is I could say to someone one day hey that's a great shirt and they're like oh thanks in the next day I might say hey that's a great shirt and it caused them to be really sad because and they didn't expect it because that shirt was actually given to them by a really good friend who passed away a year ago and they haven't thought about them for a long time right so there's lots of moving parts with people in our relationships okay so does anyone know RCA as it pertains to incidents and computers root cause analysis awesome so don't do it anymore so it's not really valuable in complex systems there is no root cause in complex systems there are too many things going on and too many things that change and even now that maybe you've got a problem and you analyze it and you've seen a complex system space but even that analysis on the problem occurring has pretty much changed the system you can't say oh this will definitely cause the problem every time more importantly there's so many things happening you can't prepare for every root cause so it's not really valuable root cause is often a search for closure not a search for the actual problem you want to say oh this was the problem that we can move on for instance the root cause to every plane crash is that gravity pulls stuff down but drawing that conclusion from analyzing the crash isn't really useful it just lets me say now I know why the plane fell consequently there's no root cause to problems in relationships we don't always know what's happening so you can't pinpoint one specific thing that might cause a relationship to fall apart so the challenge here is when we think of root cause we think about preventing failure and trying to extend that time between failures we often build really rigid systems that don't allow for a multitude of problems we say oh we're going to avoid this failure case we're going to avoid this failure case there's a lot of you can't do it this way or you must do it another way and when we have emergent behavior and we have novel things occur rigid structures aren't able to adapt to that because they haven't prepared for those scenarios it's better to have resilient systems that can kind of adapt to their environment one of the other things about resiliency is it tends to get better the more often it experiences these failure cases so you know now what makes it break and you can start to address ways to get around that one of the problems with complex systems is it usually takes a lot of different things to go wrong it's catastrophic failure so when you wait to take care of problems it's not quite broken all the way you're not ready to experience that problem case you build a lot of tech debt and debt like anything else eventually has to get paid whether it's tech debt you could have systems failure if it's relational debt you could break off from someone and never talk to them again if you never pay down that debt from an interaction it can cause problems because anytime we interact with someone some sort of transaction takes place and it's not zero sum you might both walk away enriched from a conversation you might both walk away upset at the other person or maybe one person has no idea what's happened and just passed by but someone else is hurt by that conversation so something happens and if we just always transact and take away it's eventually going to be a problem that the debt comes due experience with failure is necessary that's what helps us understand where the edges of our systems are and what helps us improve those so with our computer systems we kind of want to know how far we can push it before it breaks so we know when to stop and we know how to expand those edges and for people it applies to if I understand when pushing you is a problem because it's going to damage whatever it is maybe it's going to cause some undue stress on your part I need to know when to step back but I also need to understand when some discomfort might help us grow together whether it's in a relationship or you as an individual it usually takes some amount of discomfort to grow so understanding where that edge is and how far you can be uncomfortable before it causes problems is important which means we're probably going to cross that threshold accidentally now and again and we need to address it okay so this is sort of the level setting portion I want to get into what we can actually do to start addressing relationships when we have these failure cases and we have problems sort of required operating conditions is a blameless culture if you're in a group of people that really doesn't care about fixing it but is really seeking that root clause or that blame to say oh this is what the problem was now we know what it is we can get rid of that thing you're going to have problems ultimately you need to be seeking the restoration and the learning from that failure rather than seeking what to blame from that failure there's whole other presentations about just culture or blameless culture but I don't have time to dive deeply into it but if you don't have that most of the rest of this won't matter so a few more lessons from computers so the first thing is sort of about distributed systems and how they interact and one of the things about the company that I work for at Elastic is we're distributed not just in the software we build but our company is fully distributed as well and strangely enough one imitates the other abstractions are really hard to deal with that is to say when I say oh I don't know if this solution gives us the best bang for our buck most of like half of my colleagues on the community team don't actually know what I'm talking about because they don't have that turn of phrase in Austria or Japan or India so I have to explain that right there's some challenges just in communicating from the fact that our systems operate differently there's also problems in the abstraction of asynchronous text communication when you're trying to talk to humans who use more than just the actual words to communicate most of the time inflection of voice and you know facial expressions play a lot into how we communicate with each other so one of the things we do is we always make a point to assume good intent usually when someone does something that hurts us or offends us in some way it's most of the time not intended most people in the community act together for the betterment of the community with the exception of specific bad actors but that's not what we're talking about in this case so starting from that frame of mind can really help you shift into that blameless mindset of this person probably meant to do something good but something bad happened how can we learn from that and improve it and sometimes abstractions aren't good enough to communicate so we have the slash zoom command in our Slack where we can just instantly bring up our zoom meeting room and we can just go face to face with someone so if you're having a discussion you recognize like hey we're not seeing eye to eye I've learned to recognize it for me is when I start typing a really long thing trying to explain a lot I'm like you know what we probably need to see face to face and talk our words directly to another person's ears to shout where our disconnect is so that's an important aspect to remove abstractions as much as possible so some lessons in relationship networks from computer networks two-way communication is usually better than shouting into the void I think almost always when dealing with people which I recognize sort of defeats Twitter's entire business model but nevertheless a really good turn of phrase is what you're saying is X and what I'm hearing is Y this is sort of like the human TCP handshake you're acknowledging that someone's delivering a particular message verbatim should be what they said but what I'm hearing might be all of the other things you're interpreting out of that so if you're having a really rough time with someone because your pull request got rejected again you might say you know what I what you're saying is I need to my code better run it through whatever but what I'm hearing is that I'm not really good enough to be part of this team I don't have the skill set I'm not really wanted right so you're acknowledging the other person that you've heard them but then explaining where you're coming from this tends to prevent that like everyone talking louder saying the same thing back and forth to each other because even though you technically hear each other you're not communicating so this isn't always successful either and there's a whole framework for problematic conversations and difficult conversations called nonviolent communication this is generally where one party has been hurt by someone's actions and we're trying to bring that back and rectify that relationship I sort of think about it like incident response for humans we've got a problem and we need to fix it and much like incident response you usually wrap a framework of behavior around that response of do these steps or check these boxes as you respond to an incident just like you know what they go through for how they deal with a fire so the nonviolent communication framework breaks the conversation into four specific pieces and this is again like troubleshooting we want to break down the problem into specific parts so we can address where we're not getting the behavior we expect so I will go into each of these a little bit more so observations first are observable data they're not evaluations of someone's behavior and they're not judgments on that behavior just what actually occurred that you'd be able to observe like what a camera or microphone would pick up not what you're adding on your end so for an example I'll actually go back and give an example at the end because I realize it works better at once so then there's feelings that come from that because try as we might we are still squishy things controlled mainly by emotions and hormones so when something happens you probably have some sort of visceral reaction to it that caused the relational problem these aren't thoughts they aren't things that happen up here as you analyze or think about it over and over again but there's sort of that visceral body emotional reaction that you have right away it's easiest to talk about feelings if you sort of simplify the spectrum language is very colorful but when you're trying to communicate it's hard to have all these different floaty meanings that describe lots of different things so if we can kind of paint with fewer colors we can make a more valuable image and generally we want to avoid qualifiers you're not necessarily very or a little bit in emotion you just kind of want to talk about what it did I found this sachet acronym really useful for sort of painting a short small spectrum of feelings it's just these four so sad, angry, scared happy, excited and tender sad is usually like a loss of something angry is usually either you wanted something and didn't get it or someone violated a boundary that you have where you say you know you shouldn't do this and they did it whether or not you're explicit scared is usually uncertainty of the future usually is what makes you scared happy is usually contentedness so it's sort of the opposite of loss you sort of have right you're pretty content excited is sort of the opposite of scared you're looking forward to the future it's probably more certain what's going to happen and tenderness is a connection with other people it's kind of like I'm feeling really empathetic someone might share a story that hits close to home and you're feeling tender for them so great once feelings are defined needs are the next thing these are usually basic human needs connection, well-being, honesty, play peace, autonomy, meaning or purpose this isn't an exhaustive list of your needs it's a good starting point to think of oh it's not what did I need in this specific scenario but what is it I'm after like at a human need level that I'm not getting from this interaction and then finally a request is made so what do you want this isn't a I want you to not something but it's a positive request because it's harder to not do things than it is to do things because when you're not doing things you're always successful until the one moment you're not able to achieve not doing something but if you ask a positive request that's an achievable goal I can do that thing you kind of want to think of smart goals in the request form as well so that someone can actually achieve the request to make but it is not a demand and it's not guaranteed to happen you can't demand someone do something and they're not obligated to follow through it is a request or else it kind of shifts the power balance the other way especially in this case we're not getting as peers so NBC framework in a nutshell an example for instance is an observation I asked you for help I have a deadline on Friday I asked you Tuesday afternoon for help looking over my code you said you'd get to it in five minutes it's now Thursday morning and I haven't heard anything there's no judgment of that behavior there's no evaluating what happened just simply what occurred feelings so I'm feeling angry because I requested something from you and you haven't done anything to make it happen I'm feeling scared because I've got a deadline coming up and I still need help and I'm not getting it I'm scared that I'm not going to do good enough work maybe you're sad because I thought we were closer than this and I haven't heard from you so I feel like I'm not as close to you as I thought it doesn't have to be all those but as an example you can see this sort of a spectrum situation and the needs here might be I do need that certainty that I'm producing good work I need that certainty I also need to feel like my requests are being heard I haven't heard from you I need to know that you're hearing it and I'm prioritized in your course of the day so my request is I ask that you do help me out in the next two hours if you can or we make another time and I request that I totally understand there might be something that came up that you just communicate with me communicate with me whenever you pass a time frame if I request help and you say I can give it in an hour if you can't in that hour just communicate with me so I know what's going on then it's up to the other person at that point to either accept that request or not and so on and so forth but it breaks down all this complex interaction of my quality and not feeling heard and not getting what I need into sort of consumable parts that lets another person know that their action whether they intended to or not upset me had all of these consequences that you know happened from that so the big thing I want to sort of leave off with is the goal for goal setting and all of that is to seek success not avoid failure if you're avoiding failure you can only do it so long and you're going to always have problems there's no like the success condition on meantime between failure is the number infinitely goes up which is not a real possibility you can't have no more failures in the future so rather than doing that we should make goals to actually fix things when they go wrong seek positive conditions out of it fix repairing the relationships not avoiding problems and then I have a bunch of sources and if anyone has any questions I would love to answer them yeah so one of the problems that I have at my work doing this sort of thing is a lot of people in a profession lack emotional intelligence so when you have people lacking emotional intelligence communicating it's get yaki so how do you deal with that so I found that those really specific frameworks of only use the spectrum for emotions and don't say very or little and you know this very specific like nonviolent communication process for instance which if you plan on applying I definitely encourage you to look into much more than just the crash course I went through because they provide very specific like actions and responses to those actions it helps when you don't have the skills to express yourself yourself to have this path to follow and say this is what I should do next how am I feeling I have to pick from these six so let me see where it fits best I think it helps provide a toolset to make the gap you might have in emotional intelligence and eventually you get the hang of it and you can branch out a little bit more from there other things as well but that's one of the things I've found anybody else yeah so I guess just trying to understand thanks for the feedback I'll repeat the question afterward if I so my question was is this something you've applied to like you said you manage teams is this something you've applied successfully to your team like how I guess how many success stories from this have you seen so far any specific examples I guess yeah so different different techniques use different places and successful in different environments for sure for instance our team does deal with all the initial stuff that's sort of baseline of assuming good intent and removing abstractions from conversation whenever we have problems works for us really well at elastic and in my team on community specifically I even recently had an issue with someone on the team we clearly weren't communicating at the same point you could tell both of us were annoyed with each other and it was just like you know we need to have an open discussion on this and talk face to face so we can figure out where we're missing like have the opportunity to respond and not read in your own emotions to all the words that are coming at you so that's always been really successful I've used a version of nonviolent communication and really they're high trust like blameless groups it's pretty much like a group therapy kind of thing but it's really really successful in that case where you might be in close quarters for a long period of time and being able to talk honestly to people that are willing to hear it and not pass judgment right away is huge for resolving those conflicts quickly especially in a group that might be like working together on a really important project in a short period of time it lets you get through that very quickly we've used that on we do weekend retreats basically that are kind of about emotional intelligence at base I can talk more about it maybe out off the mic and things like that but that is a technique that we use there to deal with inter-conflict while we have a very intense emotional like 48 hours together do you find having elastic being remote if you're a partner that sometimes having two people in the same room is more intense? yes it does make it easier harder it depends on the problems it is nice in the way that I can step back and sort of process something and recognize if I respond right now it's not going to be good I need to go take 10 minutes and come back to it without a weirdness in the conversation you don't have to explicitly state that you can just walk away from the keyboard but it does make it harder to for instance say you know you're three hours behind me I really need to talk about this but you're not available so we can't talk until Monday and now I've got to deal with the burden of carrying this conflict until Monday right when we can actually sit down and have the conversation so some of one some of the other yeah so at elastic you guys at least have told everybody else and I believe to be true that you are like a truly blameless or like the company believes in this or a company that might believe less than that how might they transition and either from like the top down or from like more grassroots type thing yeah the blameless and just culture which are totally worth looking into the resource here I think is about patient safety like there was a talk last year at Dev Ops Days Boston that talked about the just culture at Brigham and Women's so that's a really good place to understand this so applying it is tough because it's cultural change like anything else so you run into a lot of the same problems when you say how do we do a Dev Ops initiative at our group that hasn't never done that before because there's a lot of changing hearts and minds that comes into place I would recommend starting at something practical so thinking of blameless postmortems for instance and you can apply that specifically to that condition or whatever you want to apply it to but being able to start at that practical non-personal level might help move into the more personal side of things and say hey we do this for our incidents can we do this for this argument we're having right now and try and retrospect what's going on I hope that answered and maybe we can yeah right yeah all the companies that may not have the same blameless culture you have we do blameless performance but in those blameless performance or retrospectives remember what I call them we're often looking for root cause in a complex system that is like what would you say to somebody that says hey but I think that there is a root cause to your complex system so what would I say to someone who says there is a root cause in a complex system first study a lot of John Allspaugh who is one of the biggest and most prominent proponents of there is no root cause and furthermore is it human error is a myth is the other side of it too as far as how we study reactions of systems so there's again a lot more where I would probably start having that conversation I think is to address what you're actually after when you look for the root cause I think like I said earlier when you're looking for root cause you're more often looking for closure on the incident rather than how can I fix the problem you just want something to say oh this is what caused it now we no longer have to explore or learn from it so if you can get to the point where that statement is explicit in the group and you sort of understand that collectively I think gives you the best way to move past it from there I think that's it great thank you I just wanted to announce that after three so we have one more talk today and then at three we all we're all going to go to the keynote room Metcalf Large and we're going to do like a raffle so three o'clock raffle in keynote room Metcalf Large hey hi Paul hi Mary do you give this talk often? so I've given this talk in do you know what a night format for it's like a five minute talk and it's one of those four minutes I've done that way and jumped into open discussion afterwards this is the first time I've given it so I have here nice to meet you this thing is exactly the way it was oh that's great alright good afternoon we're going to begin our next talk which is you can see this title is singing a missing role in agile team and of course our presenter is Deepak Kohan who is an Associate Manager in Quality Engineering as is stated on this slide so we'll go further ado alright am I audible at the back okay so my name is Deepak I'm an QE and I started in 2007 this was the year when the global economic recession also started incidentally and this Selenium and the WebDriver projects also merged which would then go on to become the de facto web testing standard so I did five years in PTC which is parametric technology and before coming to Boston I was not aware that PTCs had quartered 19 miles from here and after that I did six years in Red Hat and I've been doing the same thing for last 11 years like accepting user stories, writing automated test cases and finding bugs so you'll say it is a monotonous monotonous job right but if you have an eye for it it throws a bit of nuances and challenges every day alright anyone in this room whose responsibility is the software quality alright so there are couple of people who fell in this trap so this was already discussed in one of the earlier sessions that quality is everyone's responsibility and the agile recipes my talk is an agile recipe so it is only fair that I give you the disclaimer that the most important ingredient of any agile recipe is the people which practice that success recipe so you cannot copy a successful agile recipe and paste it on a different team and expect similar results so it can either do really well better than the other team or fail really bad alright so let's start with the heartbreaking story so has anyone seen this progress bar on a web page GitLab has it so it was back 4 years ago I was working on a user story and it was a ticket management tool which had let's say hundreds of fields and out of those hundreds of fields two were the fields which used to be pulled regularly because we had to show users the correct data a very updated data so what we used to what we used to do till that time was we used to discreetly update those two fields by pulling making two network calls in the back end of the user so in that sprint there came an enhancement request that you need to give users a visual clue that you are fetching the fresh data for those two particular fields so my developer friend used this JavaScript library I really liked it because it was on GitLab, it was on my favorite e-commerce site so I thought it was cool so as a QE I tested on different browsers because JavaScript and then I did regression test as well and one Friday morning Eastern time and unfortunately the Friday evening India time we released it and I had this habit like teenagers when they put their cute selfies on Facebook or Instagram they refresh the feed to see how many likes so after every release I had this habit of monitoring emails IRC as well as the to see whether these are not the actual flippings I just made them so there was a huge outrage so everyone said that this is a usability disaster this change this particular loader was reacting us from working and if we don't work on cases, Red Hat loses business so we quickly reverted the change and to bring back the sadness in the system and then we had a retro like every team we had a retro and out of the retro everyone said that this happens we could not we could not do user behavior we thought it was a good way of implementing this change but we could not so we actually brushed it under the carpet but then I did a retro with myself so I'll explain in another slide that the retro which you do with your inner self is the best retro ever so this was the struggle so this is the user support engineers in our case and this is the QE and this is the developer if you look at these three categories of people which two are more likely to go for a drink together or maybe share a laugh or go on a team outing like QA and developer because they have that team bond proximity and because of that proximity which is a psychological thing you cannot change it because of that proximity what QE does normally in agile teams is they test the change they don't test why did the why did the developer implement that code change so very existential question but QEs mostly fail to do that and more so in today's time when we don't have traditional QEs every QE is also a developer of some different kind so they are writing code day in and day out so once you see a piece of code on your developer friend's machine your mind becomes biased now you have three ways of breaking that code which you have already seen but it's not a black box to you now so there were hundred other ways of breaking that code which you will miss now then we thought about why don't we do one thing why don't we bring fresher perspective into the QE thing so same work stream we bought another QE but without making any capacity changes so there was user story one there was user story two and there was first QE and primary QE and secondary QE so primary QE will work on user story one for 50% of his time and the same thing secondary QE will do so because no two people think alike I haven't made it up it is neuroscience our mind is connected like our fingerprints it's different so different people perceive things differently so if you look at crowd sourcing testing platforms which have you know boomed up recently they exploit this thing so they take a website and throw it to 500 testers all over the globe so we thought that this will at least make some improvement in our process it did but there was still a gap because to work in this system both the QEs needed to attend all scrum ceremonies like daily stand up planning retro everything and the proximity challenge which the primary QE earlier had the same thing happened with the secondary QE as well right then we came up with a new thing called cynic the cynic is someone who is not part of this work stream a completely different person of maybe a different project or someone else on the bay right so what does cynic do so how do you the first thing is how do you find a cynic so these are the qualities of a cynic cynic should not belong to your work stream or project which means that he is shielded from the project related discussions right the cynic should be naturally a vocal and SRT person so we don't need to make any any effort on finding these people in red hat right and the third and most important thing he should be willing to help again in red hat you will find because people contribute to different projects so you will find cynics and then if you don't find then you have to incentivize right now what does a cynic do cynic is a very fresh and mostly end user perspective to your work because he has not seen the code he has not been part of planning or grooming or scrum so he is looking at your changes as a user and cynic is your accountability and if we go back to this line what we have to do is we have to nudge this person from here to slightly here and cynic does that he nudges the primary secondary QE towards the building the right product side now the take away is from this talk so if you look at what happened with after that incident that usability incident first we tried to introduce a secondary test then we identified the gaps then we tried to implement a cynic so we were constantly improving finding gaps and improving the system which is the essence of an agile team and then again the thing which I said earlier best retrospectives are the ones which you do with yourself right because the actual retrospectives are very formal the third thing which you can do is you can sign up a cynic you can find a person in your bay who is not part of your project and ask him if he can give 30 minutes per week to you and see if it works for you as I said this is a recipe it can work for you can work better for you than it worked for us because in our team we have this we have a UI work stream a mid tier work stream services work stream so the QE working in UI work stream automatically become natural cynics of because they are the consumers of the service layer right they become natural cynics of the service layer work stream so what you can do is you can find a cynic a very assertive and vocal person and try to sign sign him up for your project and some physical takeaways so if you look at this Tiffin carrying jute bag I bought this three of them from the mountains of Kashmir in the north of India so I actually want the feedback for this so for that I am incentivizing that's a nice bag so sorry maybe but right now I just want feedback for this talk so whoever uses this hashtag you can take the photo as well so whoever uses this hashtag in next 30 minutes and provides a tweet as a feedback and because I have bought them with my hard-earned money so I will choose the best three tweets and I will give away these bags any questions about the cynic as part of the team and that aspect to it but I noticed that there was a bit of a similarity between the secondary QA and the cynic because you are already pulling in an engineer from some other there is no similarity between cynic and secondary QA secondary QA is part of your work stream so he has to attend your scrum ceremonies but doesn't the cynic also have to so for cynic you have to the only interface between your project and cynic is the primary QA so just like code review you get you write code you get your code reviewed from your body developer right so let's say all the stuff which was developed came on the QA environment now I have to test it what if before testing I would do a cynic to get a fresher perspective 30 minutes for per split it's not much to ask for so my secondary question so if QA chooses a single cynic and like frequently demos with them after some point they are going to become buddies and so they are going to have to find a new cynic and so maybe Redhead won't have this problem but at smaller companies they might have trouble finding enough cynics to see if primary QA and cynic become buddies that's okay that won't make any difference because it does not put any bias in cynic's mind towards the work which developers are doing if they are buddies I think there is going to be a little bit of a but he is a QA right the job of QA was initially to start with to break the code right but then we went into simply rubber stamping the code we need if you go to Jim right the most common example of accountability partner is the weight loss thing right you start with weight loss you make a new resolution you start with weight loss and over a period of time you stop going to Jim right but if there are two people who are each other's accountability partners then there are chances that this weight loss resolution will go till March or maybe mid of April thank you any other questions feedback me thank you everyone thank you for your time I don't want to take them back because I have done shopping from Boston