 Hello, it's four o'clock, so I guess we'll get started. This is pair programming from 2,000 kilometers away. Strategies for distributed agile teams. If you were here in the last talk, welcome back. If not, my name is Jen Spinney. I'm a software engineer at Hewlett Packard Enterprise. I work on the Diego team. I'm a full-time contributor. I work from Seattle, Washington. Right now, there's no one else on the Diego team in Seattle, Washington, so that means this is my life. I do remote development every day, eight hours a day. I want to talk a little bit about what I'm going to talk about here. Diego is an agile, distributed team. In some ways, those words, agile and distributed, are kind of clashing, and a lot of people say that agile can only be done in a co-located way, but I think the Diego team has worked out a couple of strategies that I think are working pretty well for us in terms of doing distributed agile engineering, and so that's what I'm here to talk to you about. We're going to talk about the tools we use, the procedures that we use, and our team culture, what we do to make it work. So who is Diego? We're made up of nine engineers and one product manager. We're made up of three different companies. Right now, that's Pivotal, IBM, and HPE, but it's been other companies as well in the past. We're a constantly rotating team in general. In the Cloud Foundry core, people are switching between teams every couple of months. We have engineers in three time zones. The max time zone difference is three hours. That's the US East Coast and the US West Coast has a three hour time zone difference. We have five cities that we work from, San Francisco, Seattle, New York, Raleigh, Regina, and we've become increasingly more remote over time. And so I think over time we've also been building up this muscle about how to do agile development and I think we're in a pretty good spot right now. So what is agile? There are tons of books about this, tons of internet, flame wars about this. I don't want to give you the definition of agile because I'm sure I'm going to be wrong in some way or someone's going to think I'm wrong and I don't know if I'm going to be able to give you a definition that everyone's going to agree with. So I'd rather not go into what is agile, what is big A agile, what is little A agile. I just rather talk about how we do agile on the Diego team and in Cloud Foundry in general, like what that looks like for us and I can tell you exactly how we do development and what principles we use. So we do a type of agile that's called extreme programming but I think even that we're maybe not doing exactly at extreme programming but our own brand of it. So I'll tell you exactly what our principles are, what agile development means to us and then I'm going to go through and go through this list. I'm going to first explain what each of these principles are and how this works in a normal co-located environment and explain each of these principles and then I'm going to revisit and I'm going to go through and talk about what all these mean in a remote atmosphere and what we need to change. So the most important aspect of our team culture is pair programming. All code on Cloud Foundry is pair programmed. That means we have one computer, two mice, two keyboards, two mirrored monitors. So we have two engineers working eight hours in a day sitting next to each other, working together on the same machine on the same problem. Typically we have roles called a driver and a navigator. So the driver is the person actually doing the typing in that moment and clicking and stuff like that. The navigator is the person that tells them in kind of broad strokes what to do next, the larger vision and kind of guides the next steps. And these roles are changing frequently, like it can be just like within a minute, you can switch back and forth or it can be for an hour, there's one person that's a driver and navigator. It's just kind of how you feel with that person in that moment. So we work a 40 hour week, that means eight hours of pairing every day and in the ideal world in a single time zone team, that means that we have everyone come in at the exact same time. So at nine a.m. we start and at six p.m. we leave and we all go to lunch at the same time, we go to lunch at 12.30. This is not to be strict for the sake of being strict, this is so that we can maximize the amount of time that we're actually spending doing pairing as opposed to working on code in a solo way. So this is an example of pair programming station. Here we have two monitors, two mice, two keyboards. In this case we actually have a third monitor in the middle. This is just one example set up. They all kind of look basically like this. We have two monitors side by side. The developers are either gonna be standing or sitting right next to each other. And you can customize it a little bit on the team, like individual teams or individual people might have slight preferences, like in this example we have a third monitor to have slack in the middle, but these details don't really matter so much. On the Diego team, people love their individual keyboards and mice, so the first thing that happens usually at the beginning of the day is everyone takes their own mouse and runs to wherever station they're gonna be and plugs in their mouse and their keyboard because everyone needs to have their own special one. So the first thing we do every morning is stand up. The people go around the room, one person from each pair will say very briefly what they did the previous day. And we decide what the pairs are gonna be for that day. So generally we switch pairs every single day. Sometimes you might stick with a person for two days in a row, but anything longer than that, at least on the Diego team is just not done because we have a large enough team. For a smaller team, maybe you'd have to do that more often. So then the pairs are assigned for that day. This is usually up to the anchor. The anchor is the person that's been on the team usually the longest. And so they just kind of orchestrate this meeting. And we have a couple of roles that we assigned to certain pairs. So we have someone that's, we have a pair that's the interrupt, that's the pair that other people from outside the team can tap on the shoulder and interrupt with random questions throughout the day. We have someone that's responsible for monitoring the build and another pair that's responsible for monitoring GitHub activity and stuff like that. And so just as a reminder, I'm going through right now what it looks like in a co-located world. This is sort of the ideal situation of how we do development. And then after this I'm gonna come back and we're gonna talk about what it's like when you add the remote element to it and what parts need to change. So at the end of stand-up, this is our result. We have a whiteboard that has all the pairs listed out and the tracks are gonna be working on that day. It'll have the roles that each pair is gonna do. So this way we have this whiteboard on display in the Diego work area. Someone from another team can come by in person and look at this board and say, oh, I have something, a question related to the SSH track or something like that. I guess I'll have to go talk to these guys that are working on SSH. Or I have just a general interrupt, so I'm gonna go bother the interrupt pair and stuff like that. And so usually this is generally on display for everyone, both people in the team and outside the team in the Diego pairing area. Another huge aspect of our culture is test-driven development or a TDD for short. You may have heard the term red-green refactor, the idea being that first thing you do when you set out to do a work item, we also call that a story, is you write a failing test or a red test. So this makes sure that the test you're writing is actually testing what you mean to test. So you're testing what the story actually requires and you're not letting your own implementation guide what you end up implementing in the end. You don't like drift. So at the very beginning when you start, you say this is what's required and so you write a test and you have it clear in your mind this is what's actually required in order to deliver the story. It also makes sure that you don't write your test afterwards and you accidentally write a bad test that would have passed trivially. And in general, we want to only be writing code that makes tests pass. If we have code in there that didn't make some test move from failing to passing, then we don't really know that test, that code is actually tested and if someone were to go and refactor that, no test would break and we're in kind of a more maintainable situation. So on refactoring, in general, we only want to refactor when all the tests are green, that is when all the tests are passing. So we don't want to have a broken test and then start doing some random little method changes here and there because then we don't know that what we actually changed was broke anything or not because some of the tests were failing anyway. And so in strict TDD, we say only write code if it makes a test pass. And so the result of this is that you have, sorry, the result of this is that you have a lot of a concise code because the code should only be there in order to make a test pass and it also means that you have a ton of tests because it means that for every bit of code you wanna write, you need to have a test that corresponds to it. So you end up with a ton of tests and pretty concise and maintainable code. Another key principle of our team is shared code ownership. So there's no one person or one pair that's responsible for a section of code. The entire team owns the entire code base and this has helped a lot by our frequent pair rotation. The idea is we don't wanna have information silos where one person just knows one area really well and then that person gets sick and then we're like, what do we do? All that knowledge is in that person's head. So that's why we try to always be rotating pairs so everyone has eyes on it and everyone has contributed to it because the idea is like more people's input can catch more bugs and prevent sort of one particular person's specific style from showing through too much and everything is a little bit more standard. So we also share responsibility and blame that means. So we don't point a specific developer and say like, oh, his code broke, everything or something like that, because everything's our code. Like every line of code is probably written by four engineers over the course of like, if a particular story takes two days to write, so maybe that's three engineers or four engineers that have worked on it. Another key principle for us is continuous integration and release, continuous integration or CI for short. This is the idea that we're constantly integrating our code, our develop branch into our master branch. So we always check in directly to our develop branch through Git and then it goes through a huge series of tests to make sure the code is stable and doesn't break any existing functionality. And then when it's ready, when it's past all those things it gets merged into master. Our master branch is meant to always be solid validated code. The idea is you can always release from master at any point in time. And so this means that we can release frequently whenever we need to for any reason. And the master branch has always kept clean and we don't push directly to master. This is our pipeline. It's pretty overwhelming right now. But basically when you push to develop on the Diego team your code is gonna enter from the left-hand side and then go through all these boxes. Each box is either a test or a deployment step or something like that. And once it passes all these things, you can see there's some red right now which is not, you know, that's not great. But once it passes all these things it comes out on the right-hand side and it's been merged to develop and it's ready to be made into a final release that can be published. Another key aspect of our development culture is regular retrospectives. So every week we get together as a group. We talk about what went well, what didn't go well in the past week. This is absolutely key to our entire process because the team dynamic is constantly changing because we're always bringing in new people and we need to always be adapting to our current engineers and learning and making ourselves the best team we can possibly be. We gotta make sure that if we made a mistake this week that's fine. We just gotta make sure we don't make that mistake again next week. So in general the way the retrospective works is we have a whiteboard with three columns. We have the good, discuss, and bad column. The first 10 minutes of the retro, everyone has a marker, people go up to the board, they write things in all three columns, whatever comes to mind. And then the next 50 minutes are spent actually going through all those two topics and discussing them and creating action items that are gonna be done in the next week. The last thing that we do is collaborative story estimation. So we have a weekly meeting where we look at everything in the backlog that we haven't talked about yet. So we have this backlog in Tracker and our PM creates new stories and he writes acceptance criteria down and then we as a team, the engineers come together, we talk about the acceptance criteria there. It's our opportunity as engineers to give feedback to the PM and say, we don't really agree with this acceptance criteria or we can talk about the broad implementation details and how we're gonna do this. So we kind of get all on the same page as a team before one pair goes off and goes to implement it. Then we point the story, we estimate the story with the Fibonacci numbers. The point of story pointing is twofold. One is so that we have some velocity estimates so we can say, well, we know in general we can do about like 15 units of work in a week or whatever it is and then when we see how much work we have left until a certain feature is done, then we can say, well, I guess it'll take a month to do that feature because it has that many, sorry, complexity points. The other advantage of doing this collaborative story estimation is just a chance for us all to sit down as an entire engineering team and talk about something at a broad level and make sure whoever the pair is that picks it up is kind of doing what the entire group had in mind for implementation. So that was just the principles that we use in a normal, co-located environment. That's just the way that we work on the Diego team and in Cloud Foundry in general. So now I wanna go through and talk about what changes with remote development, what parts of these stay the same, what parts need to change. Some parts are totally unchanged, like the TDD, test-driven development, that doesn't matter if you're remote or local. Shared code ownership, same thing, it's just a philosophy and continuous integration doesn't matter. But there are some other things that need to be modified. The most obvious one is pair programming. So we're not actually physically sitting next to each other anymore. We have a couple of different tools that we use for this. For Windows and Mac, there's a tool called Screen Hero. This shares your entire screen. It gives you two cursors and it gives both people control over clicking on things and typing things. In general, it works pretty well for us. The downside is that it doesn't have a Linux client. So, and some of the people on our team have Linux machines. So for Linux, we use a different combination of tools, depending on what we're doing that day. If we really do need to share the full screen, then we might use Google Hangout Screen Share or Appearance Screen Share. And we have, in addition to that, we have a voice element as well because Screen Hero also provides voice on top of screen sharing. So usually we have to do like a two-pronged approach where we have one tool for screen sharing and one tool for voice. And we also use Chrome Remote Desktop. The other thing that we more commonly do if we don't really need to share the screen for that day is to use T-Mux. That'll just share the terminal and that works pretty easy. And that's also pretty good if we have one person that has a bad connection for the day or something like that. T-Mux is a lot faster than sharing the entire screen. Standups. So we don't have that physical board anymore, that white board where we have the actual Polaroids of the team members' faces and stuff like that. But instead we have a Google spreadsheet where we've uploaded the Polaroids of everyone's face. And in a way, the spreadsheet's actually a little bit nicer. I think this is one thing that's probably even better than it used to be because we can do analytics on this afterwards and we can see, oh man, that person's pairing way too much with that person or that person and that person never pair together. And we can kind of like, after the fact, look at trends and make sure that we're actually pairing people together with the correct distribution. And this is also fully, it's totally public. Anyone can, sorry, anyone can view it. And we can put our pairing roles here as well. In general, I think this has actually been an upgrade from the co-located solution that we had before. For the retros, we don't have a white board anymore. So no one has markers or is going up anymore. We just use a Google Doc spreadsheet. It's pretty simple. The only slight disadvantage, I guess, is the people who are actually in the conference room because we still have a couple that are in San Francisco in a conference room. They have to pass around a keyboard to each other so everyone can type in the spreadsheet, but that's not really so bad. So the retros have been pretty unaffected. We just gotta all get onto a voice call or a video call usually. That's pretty easy. Story pointing, that's similar. We're all in a big video call together. One change that we made was to start making sure that all items to be discussed that day are tagged with the IPM label. This means we can just click on that label and it brings up everything that we're gonna talk about during that IPM. When we were first doing this, we would say things and we're like, well, we're talking about this story. Kind of scroll halfway down and when you see that, then it's the one that starts with this. But so that was a small thing that we did that made it a lot easier. Make sure that, we make sure that camera's on because we point using our fingers. We hold up the number of finger to say how many points we wanna allocate to that story. Otherwise, it's not really that different than in person. One challenge in general, this applies to pairing, but it applies to really everything about how the team runs is dealing with the time zone differences. So like I mentioned, we have people on the east coast of the US and the west coast of the US. So it's a three hour time zone difference. When we first started having a large east coast presence, we just kind of kept doing things the way we were doing it before where we'd say, well, our standup is at 9 a.m. west coast time. But that meant that the east coasters had three hours where they were online and it was part of their work day but they didn't know who they were working on. They would maybe grab what they were working on the previous day but it wasn't really clear what they were supposed to do. So we started doing these time zone specific sync ups. So we have a 9 a.m. east coast sync up for the east coasters and then a 3 p.m west coast sync up for the west coasters. And this is a chance for us to talk to the other people in our time zone and say like, oh, you're working alone, I'm working alone, let's get together and pair and join tracks. And it's a time for us to reallocate the different pairing roles because we have that interrupt role and the build role and stuff. And so making sure that we have people that are covering all of those roles. And this is not as long as stand-up. Stand-up's pretty quick but this is even quicker. This can be just two minutes literally. The downside is, you know, we're not hitting 100% pair programming anymore. We try and I think the sync ups help a little bit but it's inevitable when we have the situation that there's gonna be like three hours on either side where in a lot of cases there's gonna be people soloing for three hours a day. It's not uncommon now. So it's a little bit of a downside of doing the cross time zone development. But it seems to be working all right for us. We're gonna continue to see how it goes. But so far it seems like not pair programming 100% of the time hasn't seemed to be a huge issue for us yet. There's also just a general willingness of the team to do remote development. The team has to actively work towards this. It has to be an entire team buy-in from the remote people and the people. If you have people in one like central office, they have to be super conscious of the remote people and it has to be something that people are actively working towards. So there's a lot that kind of rests on the in-person people. So in this case, it's the San Francisco people. You know, things like speaking into the mic, having only one conversation going at a time when you're in a meeting. And when you have someone interrupting on the phone or they're making some sort of gesture that they wanna interrupt, you kind of give them a little bit more of the ability to interrupt than you might give them if they were in person because you recognize that it's harder. There's some lag time, it's harder to read body language and stuff like that. And so because it's harder to interrupt and be part of that conversation when you're remote, as someone that's not remote, it's your responsibility to make sure that person has the stage and is given a voice. And if not in that moment when they're interrupting, then you say, okay, hold on one second, Jen, we're just gonna let this person finish and then you. But you acknowledge them and you actively give them the stage because you realize that as the person in San Francisco, for example, you have a bit more control over who gets to talk and stuff like that. The West coasters in our case, so we have the West coasters that work for three hours at the end of the day, where the East coasters have already signed off. So West coasters need to be aware to write down their story progress at the end of the day. So after they've worked that additional three hours, they need to update what they did on the story and be very clear. I got exactly this far. My work in progress is on this branch for this repo. And that way an East coaster can come in in their extra time in the morning. They look at what progress they're a pair made from the day before, for example, and they can just pick up from where they left off. And there are just like lots of small things you can do. This is part of just like building up empathy for the remote people and the willingness and the desire to have a remote team. Sorry. So for example, saying like, instead of saying the meeting starts at 11 a.m., you might say the meeting starts in two hours. And these are small things that you just notice over time. You might catch yourself doing, oh yeah, I just said 11 a.m., but that doesn't make any sense to that person. But by saying something like it starts in two hours, it just kind of like makes it clear that everyone's on the same playing field and it's not like you're trying to say one time zone is the right time zone or something like that. But like I said, it has to be a team priority and people have to wanna do this. And there's a lot that rests on the people that are in-person if you do have an in-person core. There are also challenges that we face. In-person interrupts are tough. And by in-person here, I mean San Francisco in our case because we have a large, many of the core Cloud Foundry teams are all working at San Francisco. So when I'm pairing with someone that's in San Francisco and someone else from another Cloud Foundry team comes by and interrupts my pair, at first a year ago, my pair might have muted the mic because he thought it was rude to be talking to me and talking to someone in person at the same time. But we learned over time, it's actually better not to mute the mic unless you're talking about something like private to your company. You wanna keep that mic on just so the remote person feels more included even though I can't interrupt the person in real time and say something, it still feels nice to be able to hear what that discussion is and I have some context. Oh, the Cappy team just came over and asked that question. It just helps to keep the mic on. Obviously ad hoc discussions are just a natural part of software development, especially when you have everyone in an open, shared space working together side by side. You overhear some other pair mention something and you kinda interrupt and be like, oh wait, I have an idea about that. And then you start talking and then you go to a whiteboard and obviously that part's a lot harder when you have a remote person. But there has to come a time when you have that and you have remote people where you say like, hold on, let's just pause. This is turning into a little bit of like a design discussion or a meeting. Let's actually get a conference room and call and tell people if they wanna come join the call, they can just call in at this number. This might look like a joke, but no ping pong. So in San Francisco, they have ping pong tables and it's a great break in the middle of the day. You've been pairing, you're focused and you say like, let's just take a 15 minute ping pong break. And it's fun, but it's also pretty useful to actually get up and be standing and doing something physical and be doing something with your pair that's not programming. I think it's actually really healthy. It's a healthy part of our day. And so it's unfortunate when you're pairing that you don't have that and when you're remote pairing, you don't have that. And it feels a little bit less casual or less familiar sometimes. When you're in person with someone, sometimes you feel like you can joke around more. You can be more casual, but when you're on the phone suddenly, especially if you've never met the person in person before, it can feel a little bit more formal. And so that's just something you have to be conscious of and it's something I try to be conscious of. So I try every time I meet someone new over the phone, I try to be really friendly and casual and let's not make this too stiff or boring or formal because I'm working with the person for eight hours. And I wanna feel like we're humans, not just developers. And obviously there's problems if you have a slow internet connection or something like that, but usually we can get around those using Tmux instead of a full screen share. So but other than that, I would say that remote pair programming and remote agile has gone pretty well for us. And as someone that's doing remote agile, I think it's actually working really well. I go down to San Francisco every couple months as well, but I don't feel like there's a huge, huge difference between when I'm working with people in person and when I'm remote. And I think part of that is just due to the amount of effort the Diego team has put in to making the remote people a priority and making all of these changes. So with that, are there any questions? I have a question, but we were working remotely for almost half a year. I was working from Dublin to London. There's no time, no difference, but something that helped us is that we feel retrospective during the day, not the retro time, but something that we just feel it. It's much easier to remember, helpful. One thing that's, when you don't prepare, as I said, no people breaks, but you have to make time or something and might break it because you just don't see how it's like, you worked till lunch, without breaks and you just were tired and thought that it wasn't very tiring, but you didn't feel because you died something that doesn't die itself and you just were tired. Yeah. And you don't, you're quite satisfied just fine. And one more thing that I found, I was not from Cisco for months and it's common that you, if you need to interact, you go to the person. But when you work remotely, if you want to interact, it's much, much easier. Better to ask someone to open to appearing and you can, it's much easier. You can discuss with your pair, see his face and that another pair, it's like 12 people discussion. It's much, much easier, much better. Yeah, yeah. I agree with pretty much everything you said. So, quick summary is the Google spreadsheets for retro means you can write in the spreadsheet the entire week, not just a retro time and that's a nice advantage. And you can use Appearan for quick interrupts. Appearan is a voice chat, video chat and it's nice to actually see video, not just calls so you can see people's face. And I forget your, oh, ping pong breaks. You have to remember when you're doing remote pairing to always be taking breaks because it might feel a little bit, it doesn't come as naturally as when you're in person and like the ping pong break is available. Sorry. Any more questions? So the team, so the question is how often does the team meet in person? Well, it changes because the team changes all the time and so for individual people, there are some people that really can't travel that much. You know, they have young kids or it's just not possible for them to be traveling that often so they might not travel very much at all to meet with the rest of the team. We have maybe every six months to a year these inceptions where we kind of get together and talk about the team's direction and what the next big, large tracks of work are gonna be. I did one inception remotely and it was very tough. The inception is a full day long meeting and I found it was just in terms of the actions and stuff like that, it was just very, very hard to be a remote person for that so the next inception that we did, I made a priority to be there in person and we got the one, I think at that point there was only one other remote person, we got him to come in as well so we synchronized that but in general we haven't, there aren't many times when we say everyone is gonna come and get together at one time and it really depends on the individual person and like I said, the team is constantly changing so it really depends on the people. I try to go every two months or so, other people come and go to San Francisco either never or a different cadence. Yeah. So the question was about bigger times on differences. On the Diego team itself, no. So we haven't had people on the Diego team, well we've had people that work on a specific sub component of Diego that are kind of a self-contained team working in Europe for example but we're not doing direct pairing. We're not assigning pairs across like Europe and the US. We work very closely with Garden and Garden is based in London so we're conscious of these things, we always have to make sure like if you have garden questions or if we have a cross-team pair that we wanna do with Garden, usually we'll get a US east coaster to do that cross-team pair and we need to make sure we prioritize that in the morning and stuff like that but it's tough when you have large times on differences to actually do true pairing, it becomes harder and harder. So the question is about team size and so the Diego team right now has nine devs and that's the largest the team has ever been. I think it's been five devs in the past or something like that. So that's the team that we've worked with but the larger Cloud Foundry organization as a whole is obviously much larger than that. I think there's like 130 core contributors and so like I said for like Garden and stuff they all work in London and so we work with them but they're not in the same exact team as us. So our team is quite, you could say it's quite small with nine developers. So the question is how come we can do pair programming when we have an odd number of developers? Yeah, nine is not the ideal number and we try not to have odd numbers but at the point at which you get to nine developers that's quite a lot, that's very big for a Cloud Foundry team and so at that point there's almost, there's always some reason that someone's gonna be out that day and so even if we had eight developers you know someone's probably gonna be soloing that day just because someone has a doctor's appointment, someone's sick, whatever. But yeah, obviously nine is not ideal for a pair programming team. Yeah. Question is when we're doing solo programming do we do any code review after the fact? Actually we don't. If I'm solo programming usually, I mean usually I don't work on an entire work item solo. Usually it's like I'll start with someone and then someone goes, they leave for the day and I'm still working and then I'll just finish it up on my own. So there is some amount of pairing that goes into pretty much every story, like it's rare that an entire story will be done by a single person. But yeah, so we don't have any code review. So the question is about having a scrum, we're not doing scrum, but do we have like a scrum master? You mean like. We have, so we have two special roles on the team. We have the PM, they're the project owner essentially, they're the one that does the acceptance, they write the stories and actually test that things are accepted. And then we have the anchor, which maybe, I don't know that much about scrum, but the anchor might be doing more of the scrum master type stuff, like they do a little bit more of the day to day, like they run stand up, they run the retros and stuff like that, but there isn't really that much that needs to be done there. So we're about five minutes over, I think there are more questions. I have a little bit of time if you wanna come up after the fact, but I think I'll let the rest of you go now.