 So, good morning everyone, hello, I'm Harrison, I work at Pantheon, and I'm on the mission control team, and we're responsible for the system that makes the different pieces of the platform able to communicate. And I'm on a distributed team, there's four of us, there's two of us in San Francisco, there's one of us in Cape Town in South Africa, and there's one of us in Prague in the Czech Republic. My team isn't the only one that's distributed here, and sorry this screen is so squished, we have like VGA cables, so it's, but here every lightning fist represents at least one pantheon as we call ourselves worldwide, and you can see that we're everywhere. And our headquarters is in the far left on that map in San Francisco. And this talk is about how we've successfully built a distributed team that's both efficient and sane. And here's a photo of us at, oh no, our, oh well, whenever. Here's a photo of us in South Africa at our latest meetup, and if it hadn't worked then we wouldn't have climbed Table Mountain together. And so in this talk I'm going to explain why I know my colleagues on the other side of the world better than I know the one who I sip aside every day, why I'm more productive from a cabin in Northern Canada than at my desk, and why working with people on the other side of the world has made me better at long-distance relationships. But before we start, quick poll, how many people in the room work with someone who is fully remote? All right, that's like most of us, how about someone who works in a separate office? All right, how about someone who works from home occasionally, all right? Or someone who would like to? So everybody's raising their hand, that's pretty sweet. So this talk is on building a distributed culture, but before we talk about how to do it, let's define a little bit, right? So distributed systems are autonomous systems, which by the nature of their implementation are more scalable and resilient to failure, and they also communicate and coordinate their actions by passing messages. And a distributed team is a team of people that doesn't even have to be like full-time distributed or even that distributed. A team can be spread across town or it can be spread across time zones and continents. And so a distributed culture is taking the mental framework of working with distributed systems and applying that to working with people, and that means giving autonomy to colleagues and teams that they can function independently in space and time, and more so in granting that into company culture from how to distribute work to how to conduct meetings and to how to actually execute homework. So why would you want to do that? This is Conway's law, which I'm not going to read, but Melvin Conway was an early computer scientist and hacker known for a seminal paper on co-routines, which is one of the backbones of modern-day computing. And to paraphrase it, communication structures of organizations directly impact communication structures of systems. And so what this means for people is you want to put a stronger emphasis on planning such that people are afforded the freedom to organize their work and personal lives responsibly. And as anyone in the room who's in prompting meetings can attest, impromptu meetings are extremely expensive and context pitching is extremely expensive. And so when you start to distribute a team, you have to start to think about how you approach the process of work. So this means that there will be more persistent communication and better documentation, which leads to the last flu interrupt for not only distributed employees, but also local employees. And so as communication becomes asynchronous, it becomes harder to ask what something is or what something does or where something is located. And so while there's a lot of wins here for distributed teams, there's also a lot of wins for local teams as well. And so let's say you work at a company that either works with hardware, software or customers in another time zone. And these things tend to get upset at often it can be need times like at 4 a.m. And so if something gets upset, like a service or a customer who you're working with on the other side of the world, you've got to respond to a page or some kind of notification. You've got to get a laptop. You've got to wake up. You've got to get up to speed on the state of the system. You've got to view alerts out of context to make sense of what's going on and then kind of build a narrative around it to see what you can do to try and debug it and rectify any dangers and mitigate it. And that's misery and it takes a long time. And especially with people, these problems can get worse the more you wait. And so if these things happen when you're awake, the time to resolve is much quicker. But at the same time, even if you do have to wake someone up, you can do some preliminary investigation and mitigation to hopefully lessen customer impact. And so how that manifests for us at Pantheon is that we've got two pager duty shifts, one for the Pacific daytime hours and one for the central European daytime hours in case something breaks. This means that someone is rarely woken up in the middle of the night, which makes us much happier overall. So the end result is that a part of why we'd want to distribute a team is that it leads to highly available colleagues for a product or a service that's supposed to be highly available as well. But this isn't only for emergencies. You can also use this for support tickets. And if you're able to distribute people in sort of the three time zones of like the Pacific, central European, and then Asia PAC, you can bake in follow the sun support. And this means that if you're working with customers in any time zone, you can expect them to get one day response times on their tickets. Another reason why we'd want to distribute a team, who here commutes? Not many of us, all right. Well, in the States, the average commute time is about 30 minutes. And let's say that's an additional 15 minutes, like getting out of the car or getting into the car each way. And so that's about one and a half hours a day or seven and a half hours a week. Or if you take three weeks of vacation, which we're in Europe, so that's laughable. But in the States, that's what we get. That's three weeks per year commuting. And that means you spend more time in your car or on your bike or whatever than being on vacation. And so cutting down on the commute leads to more time with friends, more time with family, and more time doing what you love and doing your hobbies and so forth. This also frees you to work on your schedule. Let's say if you're an evening person or a morning person, it lets you work on the hours that you're more productive. And overall, this lets itself to more productive and happier employees and colleagues. For example, I have a good friend in San Francisco whose girlfriend lives in Hawaii. And he would go to fly and see her twice a month, which that's a fairly expensive plane ticket. And it's a fair ways away. It's like 20 hours every month or something. And so he was actively looking for a new job that would let him work remote until his manager caught wind of this and offered him the opportunity to work remote his current company. And he's much happier for it. And his current company kept like a fantastic and loyal software engineer. Another reason why you would want to consider distributing a team is that recruiting and onboarding are expensive, not just in terms of time and money, but also in terms of opportunity costs. What's more is that when someone leaves, losing their ingrained knowledge is very expensive, even more expensive. And so as any business, you want to do what you can to try and mitigate that and lessen and improve retention. And so for example, needing to move houses due to life circumstance shouldn't require you to necessarily move jobs. But in this case, just a caveat, skimping on salaries is a terrible idea. Instead of thinking about how much or little can be saved in terms of money, think about how your company is investing in a loyal employee. What's more is that that employee will know that they're getting a great deal, in fact, better than they would get if they got a local gig, which will make them harder to poach. Employees as a business are one of your most valuable resources, and so you should do what you can to keep them. For example, I have another friend who had a dream of living out of a bus and traveling the states. And so his company was actively supporting that because they knew that if they didn't support it, he would leave. And again, they would lose another great engineer. And so they gave him a MiFi, which is like a little personal hotspot. And so he was able to live out his dream while his company kept a great software engineer. And on top of recruiting, this map shows, I'm sorry it's so squished, but it's this VGA cable, but this map shows the location of the tech companies in the world. And so by building a company that is only local, you're limiting your candidate pool to the tiny dot in which your company is located. And this means that you're competing aggressively for talent in potentially a very tiny pool, but you're also chaining employees to where your office is located. And so should you have to move offices, you also require that all of your employees move with you. And again, this can definitely cause people to leave, which is expensive. And so I'm going to be talking about when and how and to start the process of distributing a team. And I'm going to kind of dive into the Pantheon Toolbox and the different tools that we use and so forth. And so distribution can be across town or it can be across time zones. And as with all things, the cost-benefit trade-off of each differs from where you choose to work remotely from. It's not in either like you work locally or like you work locally in your company's headquarters or you work from home on your couch. For example, working from home affords mild risk and mild benefits. But there's also co-working spaces which afford a higher social health, provide motivation, and provide a good balance between the all-home and all-office situations. And then there's working from home full-time or your boat or your plane or your spaceship or whatever, which affords the most freedom, but it's the most challenging to stay motivated. So it can really happen from anywhere. And when to start, right? Well, company culture grows over time. And so if you want to build a distributed culture for all the wins and all the whys that I just kind of went through, then it's best to start as soon as possible. There will always be an excuse, but the beauty is that you don't have to dive right in. In fact, committing entirely to it from the outset is a sure way to fail and encounter resistance. It's a big, scary change for a lot of traditional organizations and even small organizations which the thought of hiring your fifth person like on the other side of the world, it's kind of scary, right? So you have to give yourself some time to adapt to it and see what kind of works for you. Start small as a first step. Maybe start work from home Wednesday and then make it mandatory to work from home one day a week. And this builds empathy for remote workers and starts to make it a cultural norm. And then people start to think about how can you improve the daily process, such as like meetings or scrum or updates or whatever, to also include remote employees more efficiently. And so this image is from Salyat-6, the Soviet space station program in the late 70s. And it was initially laid out to show the full 96 flight path, including the 1500 sunrises and sunsets the cosmonauts experienced. And the Russian space agency thought that it would be a great idea to give the cosmonauts like the full layout of how their voyage was gonna look. And it quickly became overwhelming to see how far away from Earth they were gonna be and for how long. And so they folded it up into two-week chunks, first counting into the voyage and then counting down to their re-entry back into Earth. And so as with spaceflight and living in orbit, it's important to start small and work in incremental steps and then build from there. Committing to a substantial remote policy without prior experience is a sure way to encounter resistance and failure. And if it works for cosmonauts in orbit, it can probably work from your living room. And now to the meaty part of this presentation where we're gonna talk about the how and then get into the toolbox. And so there's some kinds of work that are better suited to local colleagues and there's some kinds of work that are better suited to remote colleagues. And the key is in leveraging the strengths between the two. At Pantheon, we've seen that work generally either is iterative with a high frequency of changes such as feature development or working with customers closely. Or it's more hands off and slow and monolithic such as rewriting the platform or in the case of like an agency maybe updating like your core product or any proprietary plugins that you have. And so the environments that are best for these kinds of work defer greatly. For example, in a highly iterative environment, being close to product and being close to communication is immensely important. And what's more is that being pulled out of your flow, air quotes, is a little less expensive just because the communication is so important. Excuse me, whereas if you're remote then back and forth as in the case with working with someone on the other side of the world can take up to like one day per email. And so we tend to assign high touch work with customers or product or feature work for example to local teams just because the main stakeholders are right there. Whereas we assign writing the testing frameworks and then rewriting our core products to folks who are less prone to be interrupted. But then when scheduling these meetings and kind of figuring out the back and forth and the plan and sort of the story layout it's important to take someone's time zone into effect when building cross time zone meetings. When you rarely see someone communication via text becomes immensely important. As in work and in love, the relationship is directly, the quality of your relationship is directly correlated with the quality of your communication. And so you should do what you can to try and keep it positive. And so at Pantheon we've seen that communication generally takes place in two forms, synchronous and asynchronous. For synchronous communication, our toolbox includes Slack which is an instant messaging and extensible chat application. Kind of like hip chat if you've used that at all. We conduct our formal communication here but we've also set up a water cooler room for link sharing or off top discussions or meme generation or whatever. We've also set up a stand up room should anybody need to drop their updates in. It's hard to get someone on the phone who works on the other side of the world like one day or one hour every day. Because for example, working in the Pacific time zone, our 10 a.m. is maybe European like 7 p.m. and you don't wanna have to interrupt your evenings every day, right? And so having a really lightweight way to keep tabs on someone and feel the pulse of someone is incredibly important. Even more so for seeing if they're blocked on something. We also make heavy use of GitHub's Hubot which is a chat room robot. By augmenting him with custom scripts we can perform actions on our servers and see who's on call, browse endpoints. And not only that but it also lets us listen to listen to certain texts and then drop cute memes in the chat which keeps it kind of fun. And not only does this leave a persistent record of some of the operations that someone has performed but it also helps us implicitly learn from one another and so if someone's trying to debug an issue or see what's going on on a site, we can kind of hop into the room and see, oh okay well this is the command that someone typed and then because they are working with someone who's elsewhere they have to type everything out you can kind of see okay well here's the command, here's the graph, here's kind of what they were thinking and kind of follow their train of thought. And this helps then more junior engineers and people coming on be able to diagnose problems. It's like having a mentor basically without any of the added overhead. For meetings and more vocal communication we use hangouts with the Polycom which is, and Polycoms are like multidirectional speaker phones with like little satellite things so if you're at a long board table you can hear or remote people on the other end the call can hear everybody at the table at the same time. Excuse me. Company wide meetings also have speaker setups that remote colleagues have as much of a voice as the local ones as well as widescreen webcams you can kind of see what's happening. And then also for even larger meetings we have a, it's called a shout box which is a microphone in a plushy queue which you can toss around the room and it's important to repeat questions that you want other people to hear into it because your voice doesn't travel to other colleagues or elsewhere. And we converged on this setup through months of painful audio visual experiments. We had a retro program every Monday morning at our engineering meetings where we were talking about what are we happy about, what are we sad about and what are we kind of indifferent about, right? And for six months AV was up there until we started this mandatory work from home one day a week thing. And that started to build empathy for remote colleagues at which point it became top of mind and we started to do everything that we could to improve the situation. Also with meetings, since my team is in both the Pacific time zone and then the Central European time zone before we were always asking our European colleagues to stay up later or it was like at inconvenient times. But again, once we started to work from home and we started to travel around a little bit, we started to try to line it so that it wasn't it wasn't too inconvenient for anybody regardless of where we were. So now they're like engineering early like at nine a.m. or in the evening in Europe would like is, I guess it's six or seven. And so it's important to start to build empathy with people that you're going to be working with because you're not going to be seeing them and you're only going to be communicating with them via text. A synchronous communication is a little bit easier to handle because that's by default in text or email anyways. So in addition to emails, we have a pretty strict pull request system which not only enforces implicit learning but it also helps ensure product quality. We also use waffle.io, which is a cool thing on top of GitHub issues which is almost like a Trello board. It's like the GUI that GitHub is missing. It's GitHub issues are a little bit difficult to convey narrative between issues or identify things as part of a story and so this is a good way to draw them together. And instead of using whiteboards, which I'm sure we all like to do because we're in tech, we use stickies.io which is a free sort of asynchronous just like whiteboard sticky program. And so now having pushed code and experienced meetings in not only the headquarters in San Francisco or a colleague's living room in Cape Town or even here in Barcelona, I can say that this works really, really well. And finally to manage the on-call schedule, we use pager duty. And not only for engineering failures as I myself as an engineer but we also have a pretty hefty customer success pager duty shift which accounts for because we have customers all over the world and we want to make sure that they have a very quick ticket response time. And so we have a very strict overlapping schedule. It's a great tool to formalize incident management. And again, for the engineering shifts, it's offset by 12 hours so that again, nobody gets woken up in the middle of the night as to one of the first slides. And I know that someone's got my back which lets me sleep a lot more comfortably. The final thing because the secret to a great relationship in both work and love is not only communication but empathy and so communication only takes us so far. And so when someone comes from out of town, you want to do what you can to build a strong personal connection with them. Working alone or working remotely, being social creatures, it's easy to get sucked into lapses of low morale or cabin fever. When you don't tightly interact with a team of colleagues who you work closely with regularly, it's important to find social fulfillment outside of work. And at this point, finding groups of people who share a similar interest is immensely important. And there's a lot of great resources for this such as meetup.com or clubs or finding your local jazz choir or a sports team or whatever. Even so, when someone comes from out of town, which we stress like an entire one unit team gets together for about three weeks, twice a year. And it's a great time, as I said, to build personal connection. But they don't necessarily, like remote colleagues don't necessarily have the same social network that local ones do. And so by default, they don't have their evenings and weekends planned out. And so you can take that time and you can take them around and you can show them out. And that's a great way to build that intense personal relationship. And so for example, at our latest meetup in August when my team came, here is our South African and myself sitting in some of the famed California redwoods. And then when I went down there, he took his set of table mountain. And now whenever there's an issue that comes up, like I know that he's got my back and I know I'm like quite well and he knows that I have his back. Even if we need to cover each other like in convenient times or on the weekends or whatever. However, distributing a team is, it's scary, it's big, it's a little bit different. It's not, let's say like a usual, it's not what we generally think of when we think of a company, right? And so that we've talked, now that we've talked about how and when and why, let's talk about some of the scary and difficult parts. So you've got to learn to trust your colleagues. It's a little bit easier when you work locally with someone you kind of see what they're up to every day. But the truth is, if they weren't gonna do the work locally, then they wouldn't do the work. Or if they didn't, if they wouldn't do the work remotely, they would still find a way to not do the work locally. And if you can't trust someone to not do the work, then why are you trusting them with company secrets and credentials? Moreover, working with the distributed team, team members ties to the company are a little bit looser. And so should someone leave, maybe it's not necessarily such a bad thing. Security is also important for distributed teams, even more so. And for this, I'd recommend looking into two factor authentication for important sites, such as GitHub and Google apps. And maybe a VPN is the right solution for you as well. At Pantheon, we use certs, but that's because we're more engineering and developer focused. And if you have to use your own servers, we also use these things called Ubiquiz, which carry your private SSH keys. And you can only access your servers if you have the actual physical key plugged into your computer. And so that means even if we have people distributed everywhere, we know exactly who is logging into our servers and we know exactly where they are. What's more is that we're creatures of habit. And so not only is finding social fulfillment important, but it's also important to build routine. And not surprisingly, or surprisingly, it depends how much you like to work. Building routine helps to prevent overwork. And this is very true in the example where let's say that you're working on distributed team and you wake up in a time zone that's like three hours ahead of the rest of your folks. And so you start working at whatever time you feel is appropriate, like nine or 10 or whatever. And then by the time you're in the middle of your flow, everybody else sort of comes online and then you feel kind of swept up and carried away. And before you know it, it's like end of day where they are. And that quickly burns you out. And so it's important to not only, it's important to work on hours that you find are the most productive for you, but like build that into, build a routine of that into your life, even if it's a loose routine. And again, back to communication and ambiguous feedback and passive aggressive comments, the negativity of that is amplified even more so when that's all your interaction with someone. And so if there's something that could be construed as snippy or passive aggressive, emojis are fantastic. And we use emojis everywhere. It helps to convey humor and it helps to kind of lighten the mood sometimes. Here's a bunch of resources that help explain a little bit more or provide more detail or provide alternatives, but they're all immensely helpful. There's a good GitHub link in there. GitHub is like 60% remote and how they've done it is really fascinating. They were the folks who actually invented Hubot so they could start to learn implicitly from each other. So in summary, there's a lot of wins for both local and remote colleagues. Distributing a team affords people more freedom, specifically in how they choose to live their lives and blend work and life. It's not for everyone and it can be difficult and it can be scary, but it's also extremely rewarding. It's difficult to stay self-motivated and it may not be for everyone, but if you can find a way to make it work, it's fantastic. And overall, it improves employee and colleague happiness and leads to direct monetary savings. Thank you. Any questions? Yeah, so the questions were, it's sort of like the same question. It's how do you onboard someone whether they're new or junior? And so oftentimes we either fly them in for a period of two weeks or if we're bringing on a bunch of new people, then we fly a small team out and train them up all at once. And we sort of give them the core knowledge and then point them at the core resources that we use and then with the chat rooms, like everything is very collaborative and so you can kind of learn from your peers as you go. And so everybody kind of becomes a mentor. Is that kind of, we haven't, I mean, I guess, yeah, we do have, our teams are quite small and so it's sort of, it's not like we have a one to one buddy system. It's more that we have, everybody kind of looks out for everybody else on the team but maybe the team is like two or three people, right? So the question was, what was the name of the GitHub tool? Hubot, H-U-B-O-T, yeah. And it's very extensible. You can write your own, like if, let's say you wanna fill your chat with memes, like you can do that. If you wanna, it has this, you can say, Hubot, animate me thing and it'll go to Google and it'll get you a GIF of the thing and drop that in the chat, which is, or like, YouTube me this and it'll get you a YouTube video of that, it's fun. But you can also do like, Hubot, spin me up a new server if you choose to write that script. It's pretty powerful. So the question was, how do we involve clients in the distribution process? It's important to be very upfront with the client in the beginning that you are distributed just so they know. But then if they're a big client, then we fly to see them and then onboard them onto the platform. I mean, we're a platform, right? So it might be a little bit different. Or if they're small or they're local, then like we bring them in. But again, as with employees, it's a very intensive period and then we have very open communication. So for large clients, we do bring them onto Slack and we have private rooms, but we do have, we have our customer success folks spread everywhere and so they can be in very constant communication via emails and desks, for example. And then if there's a more serious question or something that the local CSE can't, doesn't know, then it can be redirected back to San Francisco and we can give them like the long-winded answer. That sounds kind of creepy. Yeah, so the question was, do we have experience, I guess like almost being on a video call with someone sitting beside maybe a colleague on the other side of the world, right? Yeah, personally, so I haven't done that since the guy I work really closely with is like right beside me and we just like paying back and forth, but I know that one of our infrastructure, well actually our whole infrastructure team, they're on a Google Hangout like all day, every day, just like constantly like on their microphones and it's like they might as well like be sitting at the same desk. I haven't used squiggle per se, but I think it works quite well for them, yeah. In the AV discussion, did we decide on our best headphones or microphone? We used Polycoms, so P-O-L-Y-C-O-M, I think, and that's like a really good speakerphone system and then we also use for smaller, and those are quite large and those are good for boardrooms and then smaller things are called JOBRAS, J-A-B-R-A-S, maybe the S, I don't know, but that's again, just like a little bit of a better microphone and speaker than like what your MacBook has. I have few questions, but one is, how do you handle planning sessions and especially like long planning sessions and two, how do you get someone online if they're suddenly not there and you can't really get in touch with them? So thank you. Yeah, the questions were, how do we handle planning sessions and then should someone like drop offline, how do we get in touch with them? Yeah. Yeah, so planning sessions, our product team is very local, so we do a lot of the planning there and then we try and do as much planning as we can when we bring people together just because when half your team is remote, they may have a different opinion or like a vastly different like idea of like what to do than the local folks. And so we try and like when we bring people together, it's mostly just sitting around the table for three weeks talking about what we wanna do for like the next like quarter or the next like six months or how we like sort of what the underlying things that we wanna achieve are and then we kind of go from there and we're like, okay, well like these guys wanna do this and so maybe we can like get this done while also getting that done. Should someone drop offline? So I guess there's kind of like two cases where that can happen. One where that's intentional and one where that's not intentional. In South Africa, for example, they don't have enough power to run the whole country at the same time and so they just shut down parts of the country for three hours at a time. And if you don't check the schedule, it's just like as I was there at like lunchtime on like a Wednesday, it's just like the entire central Cape Town went out and it's just like, you know, it's like, oh great, okay. And so now that like we know that it's like, okay, well like he'll be back up in like three hours or something. Also, if you have like your phone has like Wi-Fi like routing, then you can like plug your laptop like sort of through your phone and then call back in because that would have like enough battery and whatever and the cell phone towers like aren't connected to power. If that is intentional, we haven't really had a case where someone needs to be there and then they're not there when you need them. And I think that's because if, like if you're gonna be working on a distributed team, you have to be very clear that you wanna do the kind of work that you're doing. And so you're there because like you wanna be there. Like nobody's pulling you into the office, like you're pulling yourself into the office. And if like you don't wanna do it, then you're gonna leave, right? And like maybe that's why if someone leaves, it's not necessarily such a bad thing because like it could improve the overall team efficiency. For example, if they don't really wanna do what they're doing. Thank you, cool. So the question was if you do need to, let's say provide feedback to someone who's remote, how do you do that? And so part of the benefit of bringing everybody together and then forging really strong personal relationships is that like you feel fairly close to your colleagues. And so should, you know, like maybe, like maybe you can get away with it, being kind of subtle and like friendly about it. Just be like, hey, like I know like you weren't really around this week. Can you, you know, I don't know, like be like really gentle about it. If you have to be more aggressive about it then it is more difficult when you're not with someone. Obviously you wanna be as close to someone as you can get and so you wanna like bring it up via Hangouts or whatever. But there's really no substitute for like, if you need to like let's say get rid of someone, which thankfully we haven't had to do, I can't imagine like that would be any fun at all for anybody if that was done via Hangout or like a text message. It's kind of like, I guess it would be like ending a relationship via text. Like you don't wanna do that. Yeah, so the question was if there is some kind of issue, how do customers track it, right? Yeah, so they do have open communication with support folks. Let's say if there's like a big issue, we have a status page which like would notify if something big is happening and so you can kind of check that on your own. If it's like something that's only specific to you, then you would have to, I mean, you could see like the status of your ticket, like if it's been resolved or like if there's like some back and forth, but you would still have to ask if like, if you haven't seen a change in it in like two days, you'd still have to see it. Like oh, it's like this being worked on or something. My good friend David over here is familiar with that. That's sort of his calling and so you could like talk to him afterwards. I just volunteered you, I'm sorry. Anybody else? Cool, all right. Thanks everyone.