 Cool, I think we're gonna get started here. So thank you everyone for coming to the Kubernetes deep dive So this is a session about working with a global team and also with a global community So we have some representatives here from IBM and AT&T to talk about how it's worked with their distributed team So people in the audience. I'm guessing all of you work in a globally distributed team Like two time zones three time zones Anybody four time zones? Wow, that's it's tough for meetings, right? Yeah So I'm Emily Hoganbrook as you probably guessed for me being the only girl up here I'm an advisory software engineer at IBM. I work with Bob and our team stretches across the US and China And I have a great panel here. So let's get started with some introductions. Thanks. I'm only my name is Matt McEwen I'm an associate director at AT&T and I help lead our community upstream development for OpenStack on behalf of AT&T's integrated cloud platform and I'm Andy Yacasic. I'm a senior systems engineer at AT&T and I I was instrumental in actually in launching the team that Matt now leads and I work with one of our working groups that we initiated a little bit less than a year ago called the large contributing OpenStack operators or LCOO Which is has a global, you know globally distributed membership And I'm also part of the product working group and I have you know been working with well Distributed teams for probably 20 years My name is Bob Hansen I As Emily had said I work for IBM and I've worked on a variety of systems management related products over You know IBM offerings over the last 20 years and I have to say out of that 20 years I can't remember a team that I either led or worked with it wasn't distributed, right? They've been all over the planet. You know as Emily mentioned the current team We have members in OpenStack, but we've had you know members in Europe Brazil You know just rattle off your favorite countries wherever any kind of technology development is doing now And we've had people that have bumped into that most recently. I'm working on the OpenStack stuff I primarily do the continuous integration work for the the ZVM operating system Cool, so let's get started with some questions I think the the first question is how do you build that team camaraderie with a globally distributed team? You know what do you do for for team building exercises like trust falls are a little bit difficult across continents, so Andy you want to start? Well, you know, it's it's it's not always globally distributed, too when you when you ask that question, I think of our our well our Community team that Matt Matt spoke of when when that was first launched that had a lot of young team members And they were you know distributed all around the United States I think it's significant to you know whether your distributed team is one where you know you have You know one or two people here another one or two people there Or if it's sort of more in clumps, you know where you have a large number of them in one place Maybe another clump in another place and in our case. It was more like the ladder Yeah, but so we did you know we had to do a lot we did a lot of deliberate things And you know since was a young team, you know We had team building exercises and you know different things icebreaker kind of the things and in our meetings And so forth to try to you know build camaraderie as you say like You know like asking somebody out to you know, what kind of shoes they're wearing and what does that say about them? Things of that nature Bob or Matt you want to chime in a Really hard problem, right? I know most of the projects I've worked on at least if when the when the people are actually within the United States or travel was actually easy like from Europe, right? We really did try to organize somewhere to meet right not not too unlike what's going on here Right, these happen every six months and most recently there was a PTG So we would find you know some you know design thing or something to rally around Right so that we could at least have lunch together You know and have some conversation that we can do and we then try to shore that up with with the basic tools That we're all familiar with you know, we've done IRC which is Effective with some groups and not so effective with other instant messaging kinds of things We've even tried to do video chats, you know There's anything to just try to make eye contact, right? It is really big to get people to kind of you know essentially Call less a little bit and so that we can work on some of those harder problems together Yeah, and that that a face-to-face is huge when you can get it and that's why you know Andy and I work together every day and and it's great to actually see him face-to-face But the same thing holds true, you know on a day-to-day basis with engineers, right? And being able to pick up the phone is really valuable to teleconferencing of course with video Because you know IRC is the lingua franca for communication an open stack, right? But any kind of if you're relying solely on text-based communication It just opens up a real opportunity for miscommunication and If and you have to keep an eye out for that, right? You almost want to within a team have intentional periodic Verbal or video sync ups just to avoid falling into that trap Yeah Just following up on what you said Matt Yeah, it makes me think because you may you know you mentioned IRC for example and when I first started You know I first you know wanted to get involved with the product working group You know I started to go out attending their IRC meetings and so forth and you know and frankly, yeah, I Was I was trying but I was no participant Yeah, it was very difficult to follow You know what really was going on why you know where what the focus wasn't so on it wasn't until I went to the product to their Their meetup their mid-cycle meetup where we you know met in person all day for you know for two days That everything clicked together and then after that subsequent to that the IRC meetings are very effective You know, but it's because we all already know You know, it's like the conversations already happened and now we're just putting down notes, you know and providing status You know, but you know when no one there are things that we still you know They're really call for more discussion Then there are you know side meetings where we'll use phone bridges and you know will work that way Yeah, so Next topic meetings, you know other than IRC and and phone You know it what works best for meetings And is it good to have a mix of of ways for people to communicate in a meeting so Bob Do you want to start with that one? So I've had actually had really good results right with with some kind of a meeting at some frequency where people try to get On the phone and talk to each other now. Obviously with a team that's partially in Beijing. That's extremely difficult, right? So so I have to say what Emily had done was actually we kind of split the pain, right? So like one week, you know, the meeting will be at 9 o'clock in the morning, you know, Eastern time You know depending on what standard time is or not and the following we would we would pick a time That was a little easier for the team. That's actually in Beijing That's a good second best to have those annual meet those those meetings at a certain frequency because you get to talk to each other Now it seems that English seems to be the language of choice for most most people right obviously, you know In a country like China the English sales might not be you know as good as you might like But we seem to find ways to struggle through that and and we're able to you know Just have a conversation right because I'll have to say you know the IRC's and the chats and the instance messaging The written word is is definitely and I'll second your point is often up to interpretation We're just the simple inflection of your voice Actually communicates more than you probably realize so we've had pretty good luck with that You know we try to do it once a week sometimes it's every other week But at least get get you know have that communication happen on a regular basis One thing I'd add to not to disparage IRC or tech space communication too much because it is incredibly valuable one thing You know like Bob and Andy were saying right we come from companies that have been doing global telecommunity Teleworking for our entire careers, so globally distributed teams is nothing new It's always been a part of the way that we all do our jobs But then the interesting thing for me coming into OpenStack and coming into community development Is that you're entering a culture where that's the only way of doing? teaming and as such from the ground up everything is organized around Distributed teams and so when your team adopts OpenStack norms and OpenStack conventions, you know the that have been laid out as you know guidelines for project teams it actually can alleviate some of the challenges that you know within AT&T or otherwise that you have with globally distributed teams So IRC for example, right? Maybe it's not the only solution for meetings, but it does a great job of Handling the challenge of folks in all different time zones working at all different hours really well Yeah, so has anything not worked well on your teams? Any any horror stories or that that time you type that thing in and somebody took it completely the wrong way I don't have a horror story. I I mean just just something is probably would be pretty obvious But in in LCO. Oh, we have you know members from all around the world Yeah, and so and as a working group, you know for us It's really not effective to try to have our meetings with IRC We've tried that sort of thing, you know, it really calls for being able to have discussions. We're a working group we're you know and so Scheduling those though is is just difficult because no matter what time you pick It's going to be very inconvenient for somebody, you know, and so we the best We're really able to do there is is to just rotate the burden So that you know, it's in not always inconvenient for the same people But it it is a problem because Inevitably, you know, you have a meeting and then some of the people won't be able to attend because it's for in the morning Where they are, you know Bob or Matt you want to The only the only horror story I would say is just in general what we mentioned before miscommunication miscommunication miscommunication It is it's it's the in my experience the biggest danger with the distributed team is just getting people off sync from either a technical standpoint or a you know communicative You know interpersonal standpoint and a lot of problems just go away as soon as you get these folks actually Like you know syncing up, you know verbally or on the phone something like that So that that's something that I'm sort of hyper aware of now As a lesson learned and that we're actively working around I Guess I can give one story of a combination of tools that didn't work out real well And I say this tongue-in-cheek right because this is I'm really am joking I was working with a team that we used to have our weekly meetings And there was one in the individual that we did have difficulty understanding I mean, it's just it's just the communication was wasn't kind of there So so what I had done was I used the combination of the tools right? So we were on that we're on the phone and then what I asked them is Do me a favor, you know, can you also type it into our instant message client that we were using at the time and He did that he was very patient But when that phone call and ended I got a phone call from him And basically I got called out because I couldn't understand his English and all that and it was just a very negative different things So the reason I bring that up is you need to be very careful Right on how you communicate things like language barriers and that sort of stuff Because you can make mistakes like that where you actually you find yourself offending somebody and that's really not what we're after You know when we started AT&T's team of community contributors it was This was a situation where you know, they were clumped and the majority of the people were Well, we're located in st. Louis and then there were smaller numbers that were in some other, you know, outlying regions and and What happened was is the people who were in these other outlying regions, you know started feeling really marginalized and Were, you know being in void, you know and spoke to me told me about you know How frankly very unhappy they were how because they felt, you know, just like they were Out outside of the loop, you know, they weren't and so we you know We had meetings as a team and to address that and and you know We made a number of decisions to be very deliberate in how we change the way we communicate it so that it would be more inclusive of everyone For example, you know because of course when you're all together in one location your tendency is going to be You're gonna want to just meet face-to-face have your discussions face-to-face. It's quick. It's easy but then you've left out your other team members and so, you know We made a point of we started setting up chat rooms so we could make make use of that That was very effective. They were everyone was in that all the time that really helped a lot You know, but we also and we also established team meetings where we used You know video conferencing and so everybody, you know was present and We used we were able to create an environment where you know, everyone could see everyone and all these things really began to work together to change the culture and Change the way people communicated Which is is actually it actually just brought things to be more in line with the way it is in the open-stack community Where everybody's so distributed so you just begin to adopt those practices. Yeah Yeah, and I'll add a story about not using any colloquial isms or You know local language at one point We were having problems with a lot of new team members always stomping on our test systems and and bringing everyone down And so we were adopting a team motto of don't be a bull in a china shop So it's a US phrase for you know, not stomping around and and ruining everything for everyone else And and one member of the team came up to us and said, you know That's really offensive. I'm not Chinese. I'm Korean and we said, oh wow We we totally did not realize that that would read as as offensive It's just a saying here in the US and she had never heard it before And so yeah, we learned after that to be a lot more careful about using those colloquial isms But let's talk now about the open-stack community So what do you think the open-stack community can be doing differently to make it? You know easier for the the team around the world to To contribute and and get more involved I can add one thing I mean first the open-stack community has been Incredibly welcoming as a rule and so I I think that it's a really outstanding place for new contributors to get involved So I think it's already starting out really well. The only thing I'd add is That you know, there's probably a small minority of folks out there that are just innately suspicious of contributions coming from large companies like like ours And you know, don't don't err on the side of being suspicious everyone who works for large companies and comes in and is contributing is at least At least from our team is coming in with really good faith trying to make a great difference in the community With the best of intentions and a lot of good work. So, you know, just Give people the benefit of the doubt Andy or Bob any any thoughts on how you'd like to see the community work differently I'll let me yeah Let me say that one of the area that you know the area where I'm focused in now is something that that the community has really been trying to nurture In particularly over the past couple of years and which is is to be to really to try to inject To a much greater extent the needs of the users the needs of the operators the users of open-stack Into the open-stack roadmap into what really drives, you know, the direction of the development evolution of the platform And so, you know to that end there's been a number of steps that they've taken You know one of them that you'll you'll experience, you know here this weekend for the first time, which is the forum and And so This is one of the things I guess I would say that you know, we're we're really trying to do better is to try to try to open up The voice of the user and there are a lot of barriers to that practical barriers that People in the community don't really Tend to think about and once you're once you're well embedded in the community you become something of a community insider You've got your nose in the mail lists and all the different things that are going on and you're aware of all sorts of things But other people simply will not be and the tools that the community uses Can be pretty Obscure to many people out in the typical business world and and so trying to tell somebody that you need to contribute Your idea is about requirements and so forth in in a text file in Garrett as is a bit of a barrier It's if it's going to take them, you know more than 30 minutes or more than an hour to be able to To just even set up, you know an environment to be able to do it You know, they're going to disengage and so these are some of the things we need to address Yeah, I've definitely noticed that with Women of open stack and the way that their meetings tend to be run versus like the tempest group meetings You know tempest is is really good about having, you know, the agendas set ahead of time and You know, we all come into IRC and have a lively discussion And women of open stack tends to do more phone calls And and do things that way More so yeah, definitely I can see the the difference between the more developer technical focused Versus the the working groups and and people who are maybe not that Techie background necessarily Even that they're very technical, but they're just not, you know, they're working in the normal business world They're not working in the open stack community world, which is its own culture So I think now we're going to open it up for some audience questions and in the theme of trying to be more inclusive and Use different ways of communicating we've set up an etherpad that you guys can go out to So it's there or I'll bring it up on the screen here If anybody wants to Type in questions on the ether pad That's one of the things you'll experience in the forum sessions to this weekend Is that everyone has every session will have an ether pad and there those are meant to be discussion oriented You know the conversation really flows from what's happening in the ether pad Can I just talk at the microphone instead cool? So one of the questions I've got we work I work for a Dutch company based out of Netherlands But we're in I don't know 12 time zones or something So we have a very distributed team and we've tried to implement things like follow the sun and When you're off hours on one region you're supporting another one You know and trying to use the environment and your infrastructure the most efficiently with open stack Are you seeing a better model for large groups collaborating globally and running the same infrastructure and supporting each other? Or is it not gotten to that level of maturity yet? So I can speak to part of that. I don't think it answers your entirety of the question But one thing that I've seen work really well and work really poorly is that whole follow the sun model, right? And I think this is one of those areas where doing things in sort of the community focused way works much better There have been times that I've worked with distributed teams where the developers on one in one time zone are not really Involved hands-on person-to-person developer to developer with the team members in another time zone There are various kinds of proxies in between them right to sort of mitigate cultural differences or Organizational barriers or things like that, right? So it's a little bit more of a throw it over the wall Just from the get-go because this that's just the way it's set up but you know again open stack is as a community project is geared entirely around developers talking to other developers and That has been really key in my experience to achieving successful follow the sun kinds of approaches where you have a relationship established between a couple of guys who can Hand off work in a meaningful way and have it continue Bob do you want to weigh in on that? I don't think I have much to add I mean today that particular question because I think what I would give is the fall of the Sun pitch right and and And I'd have to agree that that that only works if the teams actually are very you know knowledgeable of each other's technologies and And most of the cases they're not right. I mean that's just the way we structure our teams right by by locations in in the product working group actually what they've done is They know we have we have two different meetings. There's one you know There's one that's you know timed for the people basically in the Western half of the world and another one that's for those in the Eastern half of the world and so you know, there's sub teams, you know two sub teams and Why that can work is because Four times a year we all come together face-to-face and those on it really honestly those you know those intense long term You know sessions together they they set the tone then for all the work, you know That's being done in the intervening period So that everyone is in sync because we all understand the objectives and what we're working on our goals and And then of course there are you know various You know side meetings and things that still connect people but that that actually is working, you know pretty well So I think we have a couple questions at the bottom of the ether pad here to scroll down So I think there's one here about mentioning that you have chat rules to help remote people not feel excluded from the Conversations, so what are some chat rules that that you like to set up? Think we mentioned maybe like idioms not not using those colloquial isms as it is one of them Well, it's certainly basic courtesy You know and to be positive it's always I think it's you know It's certainly important to be positive about things or at least to try to express things in a positive light You know we've thought this isn't quite what you asked but you know we've also had you know Situations where you have a meeting, you know like this only it's a discussion And at the same time you're in an ether pad kind of like this in fact that's how all of the ops meet-ups work and and that actually can be really effective because the people who are Less vocal, you know and so forth. They're still putting their stuff in the ether pad or You know with somebody states something in the ether pad other people are putting little plus ones plus ones plus ones to show That they support that idea and and this stuff's all kind of happening dynamically or you can ask a question You know and put a few different lines in the ether pad for answers and everybody's just kind of voting You know and they're all this is all happening as a subtext into the conversation And it it can really work very well because you know as you're moderating you you can see The kind of reactions that you're getting from all you know in the audience And it puts everybody on an equal, you know equal level Yeah, and I'll say it doesn't have to be ether pad I know that there are a lot of different Technologies out there that let you do the sharing so we've used like box notes on our team And some of the like Microsoft Office 365 even lets you do some of that stuff with word documents or PowerPoints Or or something like that so you can really work on them collaboratively as you're talking about them If I can add on to that, you know we mentioned before how you can have sort of a clustered kind of Thing where you have a big group in one area and other groups in other areas, right? And it really takes an intentional effort on the part of the folks that are Co-located to to go out of their way to make sure everyone else Doesn't feel like they're not part of the main group, right? It's it's not it doesn't happen automatically and you know if you're in that that center of gravity You probably don't feel like everyone else is an outsider But the other folks might so it really behooves you to go out of your way And one way, you know pulling us back to the question that we Approach that recently is you know we had some folks in that location doing whiteboarding and Explaining some things to the located team and some other folks on the team were like oh man I wish that we could we could participate in that and We did we just set up a webcam pointing at the whiteboard and the guy was you know going through the exact same Conversations and it worked out really well everyone in you know in the meeting room Locally there was you know no fidelity lost and for the remote folks. They got quite a bit out of it as well So that's a very simple low-tech approach to introducing remote whiteboarding Cool have you guys used mural at all? It's a whiteboarding app. We've tried it a little bit Mostly good for like sticky notes like can ban boards and stuff like that I see that there's a question about any advice for supporting cultural differences or is it best to just just avoid culture in a team Just just pretend we're all robots right where they're at a code 14 hours a day right there there are yeah, there are absolutely cultural differences, you know that that factor in We you know Americans in particular we tend to be pretty outspoken. We tend to be pretty brash in comparison to some other cultures And so you know what I'll find as well in a meeting say within LCO We'll ask a question or you know, we'll have you know, we'll be having a conversation And if there's you know one or two seconds of silence, I I feel like I need to say something, you know I need to keep it going But we have to just sit for about five seconds or so before many of like our Asian members will then speak up They're much more, you know, they're they might say polite or you know, they don't just barge in with their comments and And so you know, yeah, as you said, you have to be intentional about all those kinds of things One thing that is really important is If the team members will go out of their way to help their remote team members That can cover up, you know a whole lot of ills, you know, a whole lot of misunderstandings If you go out of your way to help your teammate And I mean really go out of your way and demonstrate that you're working to help them They'll cut you a whole lot of slack in the future and you've made a friend and a colleague So it's another one of those things not to take for granted. It's worth going out of the way for it It helps build your team With the question on the work-life balance, do you guys use anything to differentiate? I need an answer whenever you're free versus I need an answer now Like I work in operations our QA teams in Lviv, Ukraine And there are times I'm like when I'm finishing up my day and I want to hand off work or any questions But I don't care. I don't want them to get get out of bed to answer my question. What do you guys use any specific? Ways of differentiating what you how much you need something I Think a lot of that Like if like operations, you know things can be need to be fixed like now, right? Whereas whereas development questions, you know or some question about a technology could probably wait, right? So for the operation kind of things at least we have a management structure in place that helps us with that, right? So I'm not going to call somebody in Beijing, you know at four o'clock in the morning their time because I have a question about their API Right, but if I am supporting a system that breaks down that affects our customers Then we need to run the management chain to solve that so so big, you know in some ways It's back to the cultural thing right let the management team in that country handle it the way that you know They do there right and we don't tend to interject ourselves in that kind of thing You know you do it you have to have an understanding Which again and means you have to actually talk about it and decide, you know, but you know how things would escalate, right? you know because Certainly, you know and certainly the tools really are great as far as making it possible for us to You know to for people to be you know engaged in conversations in their own time You know offline by going in and contributing a comment or what have you but then you know So there's you know things that where you might be putting comments and you know on a wiki page or something But then there's also something that you might send somebody a message about in Slack or you know or IRC You know the way in my mind and have worked it out with some others You know if it if it's really serious then you send a text message, you know And what I actually what I when you first asked the question of what flashed my mind immediately was Just a little system that my my girlfriend and I have which is that yeah She'll call I'll be in a conference call or something. So I won't pick it up But the deal is if she calls right back again, then that means it's something really important And I'm I'm gonna put my call aside for a moment. I'm gonna take hers And that's the understanding Yeah, I'll say I know our China team as well uses we chat and obviously they don't put anything confidential on there But I'll notice that that if I have a question that that needs to be answered right away And and I ping any member of the team then they'll put a ping out on we chat, you know Oh so and so can you respond to Emily's email within the the next couple hours So having having like a code that you can use in some outside chat If you really have to to get in touch with someone is useful, too and like like Andy was saying there's a We have so many different kinds of Communication tools and we probably talked about a dozen of them that we all use up here on stage And that wasn't even getting into the ones that are internal to AT&T, which we probably have another dozen and so they've just Naturally kind of fallen into a prioritized order on the team So depending on how urgent a question is you just use a different communication mechanism and like Andy Said at the end of the day if you need to get a hold of somebody faster, you just text them or call them Yeah, so I guess we'll end with an easy question. Maybe what's what's your favorite method of communication with your remote team and Bob if you want to say lotus same time, I don't Think that's probably our least favorite one around the lab That's the one that seems to work the best though And it's the one that we like the least so But that one that we're mentioning it isn't persistent So if the person isn't actually, you know logged in, you know, they lose the message So so you usually it's email, right? That seems to be the best way, right at least for me I'm gonna go for the easy crowd Please are here and say that my favorite way of communicating with my remote teams is to meet them here at the summit Them drinks, right? I That's actually something it's It requires some thought Because on the one hand, you know We all you know the milieu in which I work at least a lot of a lot of developers a lot of technical people Everybody seems to love the instant messaging tools. They just for whatever reason It seems that people don't want to communicate verbally and yet of course Verbal communication in my experience is the most effective form that we have You know, you can get something, you know communicated much faster much more effectively verbally yet yet We really don't do that very much And I'm in so I often think how how could we improve on that? You know, how could we you know Make it easier for people to do that. But of course when you communicate verbally it requires Attention and it you know it requires immediate, you know engagement and response, you know Whereas you know popping something into a chat room You could ignore that for a few minutes, you know, or what have you and then you can respond You could take your time to think about it. Then you can respond Emails the one that I actually hate the most It's it's just the bane of my life There's just a constant flood of email and just even you know sorting through it to see which ones are even You know worth answering just takes away too much of your time Yeah Well, cool. I think we're just about at time. So thank you everyone for coming and hope you have a great summit And we'll we'll keep checking the etherpad and and responding to questions on there periodically too So if you think of something later