 I think that's my cue. I'm also just trying to blind me for fun. All right, so thanks for joining me in this session. My name is Richard Sarota. I'm the VP of Product for CenturyLink. We're going to talk about DevOps, how we do DevOps, what have we learned, what have we done good, bad, ugly, just give you a sense of what does it mean to have a 50,000 person telecommunications company do continuous delivery to organize differently to deliver at scale. Real quick, so I do work at CenturyLink. We're a telecommunications company in the US and internationally, co-location provider, public cloud recognized by Gartner and others, private cloud, we do children's parties, we kind of do everything, so lots of good stuff. I work for Pluralsight doing training, write for InfoQ and some other things there. So we can talk about, so Gartner, of course everything they say is true, so 25% of companies in the Fortune 2000 or Global 2000 are moving to DevOps. So we know that I'm not telling you all anything you don't know, that this is clearly something everyone's looking at. The tricky part is, can you find the right fit, especially if you're at a large company? It's one thing when you're 12 people trying to figure out DevOps, it's another when you're larger and you're trying to change a culture or change thinking. So in this case, we'll talk about how did we make some of those changes and what do we do on a day-by-day basis to actually execute in a DevOps fashion? It's the whole point for us specifically is how do we move faster? How do we move more effectively so that we're shipping code faster, so we're delivering services to customers faster, be more responsive when someone has an issue? So we're gonna do three things today. I'm gonna go through what are our core values real quick, just to level-set why we do what we do because doing agile or doing DevOps for the sake of it is meaningless. Doing it because you're actually trying to adhere to some core purpose is what makes sense there. So what things do we care about? What drives why we do what we do? Some of the logistics, exciting things like organization charts and team structures which clearly is what you've all come for as we'll talk about things like that and even how we've set up our space. We recently redid office space and tore down walls and created team rooms and things like that. So we'll talk about how that fosters the culture and then finally go through on Monday through Friday. What the heck does it mean for us to execute in this fashion and how can you maybe take some of those things back with you? So what are some of the values? What kind of things do we care about? Well, we care about empowerment, care about individual people having more control over their particular area. So in our feature teams and others, it's that we've also learned that people who have more latitude to do things end up learning faster, doing more good work. So it's a core principle that we try to empower staff but we also make sure they're accountable. That you can take responsibility but then you have to turn around and own it if something goes wrong. Teamwork matters. We don't really believe in the lone wolf. We don't believe in the lone genius who works in the corner furiously hacking and typing code and shipping it. That doesn't work for us. It's gotta be in a team culture. It's gotta work in a way that when there's a crisis we can come together quickly. When there's a success, the team succeeds. There's not the same individual rewards as if the team has to succeed. Communication matters and not just dumb everybody in a meeting for eight hours a day communication but how do we smartly make sure that people are kept aware of what's going on? There's a lot of people who care about what we're working on. So how do we not get inundated in status meetings and one-off nodes and this and that? So how do we figure that out? And then being obsessed with continuous improvement that we can't be satisfied with the status quo. We have to be trying to change things and that drives, again, a lot of the decisions we make is not ever being satisfied with our point estimation process when we do our agile sprint planning. We change it every few months. It's on purpose as we do retrospectives and learn how to get better. Then finally, probably shouldn't be last maybe first but delighting the customer is important. How do we do things that are in the best interest of our customer? It doesn't always mean we do everything our customer asks us but it does mean that we're focused on doing things that sometimes are above and beyond or at least done thoughtfully with a customer in mind. So those are some of the values. Being empowered, working in teams, that's what shapes the next things we're gonna talk about which is how are we physically executing on some of this? How do we arrange ourselves or structure? So a year ago, if I were here talking to you I would have told you we were shaped like this and we're still doing DevOps. We still had our VP of ops and engineering working closely together. They were peers. They could deliver each other's status reports or standups, very much a collaborative group but it was very matrixed. You had to work that span teams and it wasn't bad but developers don't really love working in a matrix culture. It doesn't always help them. So communication was trickier in this model and frankly this doesn't scale very well. Now most companies have scaled this but what you end up with is bigger and bigger departments and you lose focus on the customer and the experience. Adrian tweeted this last year and I think we agree with this that DevOps is typically an org change. It is not a technology or things like that. It requires you to actually think differently and it often means restructuring yourself to prioritize differently. So how are we looking now? So right now we have two sets of groups. People like myself who are called overhead which is fine, I'll take it personally and then feature teams of about 30 some odd that are broken up into each individual self-sustaining pod almost like a franchise model. Each one of these can do their own thing. They're self-sufficient. Each of the developers or engineers is single focus. They don't span teams. They're focused on an individual feature team and as you can see there's about five roles. There's a product owner from my team. There's a product analyst from my team and the product owner owns your backlog. They own the strategic priorities. They own what is this thing supposed to be? What does the customer need this to be? They hand off sprint plans to an engineering manager and then they're involved. This is a physically co-located team when all possible. Product analyst, what's the customer usage? What's the profile? How are things going? What are the requirements? Engineering manager is the only person who has any sort of direct reports. We have a very flat structure. Engineering manager has the developers and ops people report to them but they should be doing 20% of their time on management. 80% is individual contributor. We don't really believe in people with manager titles. It doesn't work well for us. And then software engineers and infrastructure engineers. Everybody's an engineer. You're building things. You're building the robots that run the product. You're building the product itself. Whatever it is, that's the pod structure and it's kind of the Amazon two pizza team size which each of these is at most a dozen or so. But the point is that we try to separate this where the people at the top, the overhead are connecting. They're the ones who are trying to connect these different pods. As you can imagine with 30 some odd feature teams, it's really hard for each of them to know what's going on. Yes, it's harder for me but that's what they're paying me to do. So I gotta keep track of these and know that hey, you're trying to ship this. You know that you could use this object storage service that they're building. Oh, what about this service here? And a lot of time, you're a broker at the top. You're a support structure. You're not a dictator. You're, instead, you're connecting the pods. You're connecting the fabric and the company. So what's the tool chain? What do we use on a day-to-day basis to operate at a DevOps function? So we do live in Trello a lot. Love it or hate it. It serves a lot of functions for us, for backlog management, for retrospectives, for sprint planning. And again, it might not be perfect. It checks the boxes. It's internet accessible. I can do prioritization. Easy to add people. Easy to see what's going on. I can see work in progress because we typically do a Kanban, scrum style, agile fashion. So as we do all that, this provides us a great way to do retrospective sprint planning backlog management. Each feature team has a Trello board that has their backlog and then how's all their work in progress for a given sprint. We do use GitHub, as I guess most of the modern world does at the moment, but we use this for code we use this for our knowledge base. All of our actual content for our website is in our knowledge base. And if you go to our knowledge base at centrallycloud.com, that's all open source. You can contribute changes. You can add new articles. You can do whatever you'd like. We open source that on purpose. We do use Slack or big believers in Slack at this point. If you're not using it, I have pity for you. Slack is awesome. It's amazing. I don't get emails from engineers anymore. Some of them have actually put out of office notes permanently on their email saying just don't email me anymore because I only use Slack. I can't do that. But it's a great way for us to collaborate. We have team rooms. We have conversations in there. We're able to share content and you don't have things stuck in email where somebody might miss the whole thread. So obviously it's just kind of a chat on steroids but it does provide a nice collaborative culture for us that we use this for public facing things that our sales force uses on what's the health of the cloud that we can publish to. And it integrates with our tools. So GitHub, fit check-ins go in here. Continuous integration build things get published here. Deployments get published here. So it's kind of a live look at what's going on in the environment. We do use End Desk for ticketing, for feature triage. If you send something into our user voice or feature queue, we're able to review things. And again, the key with a lot of these tools is we're not saying they're all perfect but the fact that we've committed to these and if we don't like them, we will change them but we'll change them together. So we don't want everyone using different ticketing systems and you simply lose data. So if we find a better one, we'll switch but for now we try to keep people on the same sets of the tools I've just showed you. And then sometimes you're stuck actually having meetings. For us, a meeting's usually a failure because it means you couldn't have solved it some other way but you do have to get together sometimes. So we use things like Webex or GoToMeeting or Hangouts or Skype. We have not solved this problem because I used like nine individual collaborative tools but ideally this should be like one thing. So what's our workspace? How are we actually arranged? Our cloud headquarters is in Bellevue. If you're in town, feel free to visit and tour us around. But we use team rooms. We built this specifically. We opened this last October but every team is arranged in a team room. Each of those feature teams focuses in that and the point is you sit there together. The conversation is relevant. Your developers, operations staff, whatever. You're all functioning in the same team room. You can notice there's a few logistical things. We don't get a dog with every room. That's only special rooms but the desks are standing desks or sitting desks. They're meant to be long so you can always have at least two people working side by side at one of them. You see there are couches and there's televisions in each one. Not because we're watching, murder she wrote marathons or anything but because we're doing stand-ups there. And when there is a deployment, the team will typically throw a movie on because we're just gonna sit as a team and hang out until the deployment's all completed and checked and verified. The couches are there for collaboration. So it's meant to be a workspace where that team can pivot between developing and working on things and sometimes doing a little bit of social. One of our lead architect developers explained the difference between an open space and a team room. It's a fundamental difference that open spaces which is those low cube walls where everyone just chatters on their phone and you can't concentrate, that is not a team room. That's just madness. So ideally a team room is an enclosed space where the conversation is relevant and you're actually collaborating as part of a team. You're not just chattering over what everyone is working on individually. The common space is key though. We have a big area in the middle where we have lunch a few days a week but that you're not allowed to eat or take phone calls in your team room. So it forces everyone out to the common space to make sure that you sit at a table for lunch. I did two days ago and I'm sitting next to our support people, a couple of developers, one of the guys on the analytics team. That's great, that's the whole point is it's not about having little kingdoms of people who never communicate. It's about smashing people together sometimes against their will. Executive engagement, we did have tricycles for a while but then somehow they went away I think because we were crashing them a lot but the point was is that that's our senior VP who now just rides a scooter around instead but the point is that we're trying to make sure that nobody should be in their office. We only have a handful of offices in the whole building and the point is that you're not supposed to be sitting in there. You're supposed to be regardless of your job engaging with the team, talking to customers, getting some feedback, sitting alongside and understanding what's being built. It's about not having a bunch of just helicopter executives who drop in from time to time but people who are intimately engaged in what's going on and not meddling or micromanaging but actually being collaborators. Our architecture still comes from our leadership team in many cases, it's not just the teams but it's contributions from those at the top. And then using that space to also focus on learning continuous improvement. We just did a Raspberry Pi kid hack day a couple of weeks ago where we had 50 kids in the office doing, hacking around on Raspberry Pis and learning stuff and some of our engineers were there helping out, that's awesome. We do a cloud foundry meetup every month in the office. Just trying to use the space to have people stay after work and learn something if they'd like to or contribute to something. All right, so how do we execute? What do we do Monday through Friday? Now not all of these things do happen on a weekly basis that would probably be a little insane but for purposes of presentation, I'm kind of cramming in what does our DevOps look like on a Monday through Friday? So what do we do on Monday? Cloud stand up every day, 10 o'clock. We all huddle together and we go through the health of the cloud, why do we do this? Well, we do this to start that feedback loop. This is the start of the day, it's not the start of my day, I'm at work at seven but for developers it's the start of their day, 10 o'clock and then we're figuring out what happened last night, what are the focuses for the day, what are some major things that have happened that are going on and this is a daily synchronization but it's just people who need to be there. It's representation from each major team to maybe a dozen people and it's over in about three minutes. It's the point, if there's an issue, quickly outline it, if not we all get back to work but it does start to set the tone for the day so that's a continuous thing you'll see every morning. Feature triage, every Monday at 10 30 I huddle with a dozen or so people and we go through everything that you all as customers or partners or sales people have submitted to features at ctl.io or our user voice channel and review feature requests and the reason it's a cross-functional team is that something I might not think is relevant the security person in the room says that's super relevant we should actually do that, don't discard that or hey maybe this is really important but the support person says they've never heard of a ticket about that. So that sort of cross-functional feedback is really important it's not just a product team sitting there going yes, yes, no, no, no, it's a bunch of people from customer sat to support to security to product teams, all of those coming together even our UX team participates in that. Batman, so Batman is our on-call person because on-call is not interesting. Batman or Batgirl is interesting so that person does not have to wear a cape although that would be pretty awesome. What they do though is get assigned the pager and it's not a real pager as I was asked before it's a page, we use pager duties so they get text messages but they're on-call for the week and that's the developer or one person on each feature team is on-call so hey guess what if you build a lousy product you're the one getting paged for it at 3 a.m. in the morning so there's a good shared sense of ownership because each of those feature teams if you build it you run it there's no handing it off to a shared operations team you have to take ownership of that so every Monday morning there's a handoff between the Batman and Batgirl and corresponding person and they get to build some empathy with the customer if you're that developer or that operations person you're hearing frontline questions it also helps those teams understand the platform more you have to be in Batman rotation within your first month or so in the job so you have to quickly learn what this product is and not necessarily be concerned about digging into the bowels of the system What do we do next? Every Monday night we send a giant email to our CEO, CTO, CMO, all the C's all that go out there and so what this does is this summarizes the health of the company our health of our business how many, what was the money last week how many new workloads were created in our public cloud what was the failure rate of some of the provisioning process what's our roadmap looking like from last week to this week what's the business cases we're proposing for new work any staffing and hiring changes what are our top 20 customers asking us about right now we send that out every week to that executive team and you might think well that's a lot of work and that's just a big status but the key is each person covers their area I spend four minutes a week on this updating my roadmap section everybody jumps into a shared document updates their section and then it's sent out so it's a quick process it's pretty lightweight but it's really informative and more transparent than any other unit at the company that we get to send this out on a regular basis that transparency helps it saves us from a bunch of random questions coming in about what's going on this is really helpful and it shows that we're not trying to hide anything and gives people a shared sense of ownership of what we're building so Monday's fun and we get to Tuesday stand up gotta do that every day that's important gotta keep that rhythm going with that and then my product team does a stand up but so does every feature team we're doing agile but you don't do stand-ups for the sake of stand-ups if you've read a book on agile and say we need to do agile, let's have stand-ups stop doing that, that's not the point the point is to make sure you're actually delivering differently but our product team does do stand-ups just operational things each feature team does as well as we do releases we wanna make sure our frontline support people are trained so people like myself or others will meet with our support team and talk about the upcoming release what changed did we fix some bugs that you've constantly gotten tickets about we should let you know about that because that's something that now if somebody calls and asks you can say hey we just fixed that so not just assuming they're gonna go pull our release notes and see the differences but actually help them understand what's different so again it's important to build that shared ownership make sure the communication is solid and that we're giving the right information to customers now we do deployments all the time we do typically 21 business days so all 30 plus some odd teams ship about every 21 business days and so let's say one of the teams our core platform team does their deployment they put a movie on usually takes a couple hours to just go through and QA the whole thing but we use continuous integration continuous deployment tools to ship this out but it's important to keep that rhythm the whole feature team owns it so they've gotta make sure they're there throughout the duration make sure it's successful and that they go home when it's finished what's fun though is other teams can follow their own process so while I said some of our teams of course we say hey use Trello use Slack use Zendesk but within a feature team you may use some different technologies our website team uses Travis CI for some of their continuous integration that's not what we use in the core platform team that's fine you know it's okay if there's some expertise within a feature team that might use a different tech stack some of them are using Ansible to do some of their builds while the most of us are using Chef and that's okay we're not gonna be too directive on all that because sometimes you're forced to make sacrifices when you build these shared common things that don't satisfy anybody so in our website team we push that four or five times a day if you go to centurylinkcloud.com that's always an updated Node.js app it's all fed from GitHub it's a real fun process and that team usually deploys right after we push to the main update so they can update the website alright so Wednesday's on cloud stand up you know I gotta keep the rhythm every day that few minute period then sprint planning so how the heck do we do sprint planning so a product owner maintains a product backlog that's my job or folks on our team's job you maintain that backlog and then a week before we start that sprint we talk to the engineering manager going this is what I'm thinking we're gonna tackle the next sprint what do you think they're thumbs up thumbs down Roman emperor style but then we actually dig into a little bit and find out did I make some assumptions wrong or is this on track and so that gets queued up for a particular sprint and then that team owns the estimation so as a product owner or as some sort of mustache at the company I can't tell somebody you're going to do this work this month all I'm gonna say is here's your list of priorities when their velocity is full when they have 100 points of work they're finished and they don't take any more work in and it's up to me to now reprioritize if I didn't get everything I wanted to get in there so that team owns when they say they're done what's valuable there is now they own it and if they don't ship that that's on them then they might have to do the extra work but they can't say well you made me do this extra work that team signed up for it they said we can build this much work with our team we're gonna take this on and it makes sure that the right people own the right steps here so that's a big part of this process sometimes it's long sometimes it's quick but it's a good flow that we do every 21 days a lot of our teams like to do brown bags and they use that shared space but shared learning is important when you have these sort of distributed pod culture because you don't always know what everybody's working on so our analytics team a few weeks ago did a great brown bag on their progress what are they building with Spark and how are they using Cassandra how are they doing these technologies to suck in all the data about cloud and do predictive analytics that's great stuff the team loves to see that we bought a database the service company three weeks back orchestrate.io and they were there last week and they were doing brown bags for the team going this is how you use our tech so don't go stand up a MongoDB database anymore you should be using our NoSQL service that's now available all over the world don't build your own stuff and so educating that means we're also using our own technologies learning from each other I walked by yesterday and they're all sitting on a set of couches with three different teams talking about how they're managing Logstash and our Elk stack and I love seeing that that means you're cross-pollinating and learning from each other ask the experts this is when executives at the company do a once a month for anybody at CenturyLink con call to ask questions and it's often asked the expert because somehow I'm the only one who always participates in this but the idea is that you're able to ask anything you want about architecture about product roadmap product strategy and the whole point is how do we increase the transparency if there's rumors or concerns like hey is this thing getting deprecated or are you going to use this technology we can talk about that as a team and things don't spiral out of control so having those forums to talk about patterns and talk about strategy has shown to be pretty important as people do sometimes bundle up their questions or they use tools like Slack to just ask them as one-offs our knowledge base mentioned in GitHub anybody can contribute and so we'll see on a regular basis pull requests submitted by people in the field who want to go write a knowledge base article about a pattern they're using or they want to fix something because I can't assume that all of mine are right all the time so people want to go ahead and fix those and submit that so I like seeing that because it's shared ownership again right it's our product as a company it's not just a small subset of content people who produce everything that's going on Thursday isn't that today yeah Cloud stand-up hey I missed that this morning they'll forgive me product team stand-up so what's important is while my product team on Thursdays we spend a half hour together instead of 15 minutes and I bring in guests whether that's our consulting arm whether that's another feature team to try to do a little bit of education sessions so that we all feel like we're sharpening the saw a bit we're learning tech more bringing a vendor for a few minutes to talk about their stack or bring in another technology team to talk about how they do agile so we do this a lot so every week in our Thursday meeting we just talk a little more just how do we get better at being product owners product analysts technologists again that stuff's fun a learning culture is a lot more interesting than just kind of punching the clock every day ops review this is a bloodbath but it's also somewhat enjoyable once a month all the feature teams go through and talk about what's the health of their particular feature team and not you know hey we're happy it's what's your metrics do you are your customer satisfied what is your failure rate what's your success rate how much revenue what is your team generating because of your product so it's a quick review each team has five minutes it's not meant to be you know hours and hours and hours but each team talks through this and our customer sat team talks through tickets and whether the top things our customers ask for and this is that one forum for all of these teams to read out what is their health what are they working on what's relevant again the stuff matters for transparency and information sharing is when we do this it's really easy for the executive team to quickly say I think this is going off track here how come your failure rate's going up or hey your plan for next month sounds interesting but based on this feedback maybe we should work on this it provides a good shared forum we don't love to have team meetings and shared meetings but this is one great setting to make sure that everyone's tracking the right stuff we focus on execution so it's even for teams that were formed three weeks ago in real life I'm expecting them to be doing an ops review read out the first week of June what did you ship even though you've been around for three weeks that's got to be the expectation not I drew an awesome UML diagram that doesn't matter to us it's did you get feedback from customers did you ship something did you learn something how is that feeding back into the product service interruptions hey it happens disruptions do happen for various reasons gremlins goblins whatever else and so these things can happen unexpectedly if you do have a plan disruption that's fine if you knew you were doing maintenance but let's say hypothetically not every Thursday is there a service interruption or we have the worst SLA in cloud so I'm not saying this is a weekly occurrence just saying this is an example so Friday do stand up we'll probably talk about that outage right what happened was there disruption or any sort of thing that might have happened a poor communication what do we find out in stand-ups that we can do something differently we'll at least bring it to everyone's attention so that we know if we hear from customers how to talk about this and then escalations let's say as part of that ticket was escalated this could go to the feature team this could go to Batman or Batgirl whoever's on on call and so the key is that we have a frontline support team but then they hand off things that are tier tier one tier two is shared but tier three, tier four that goes to the feature team that product you built might have a problem go fix it right it doesn't go to some shared pool that has to get figured out it's that team should be addressing their own product and building the most stable instrumented product they can outage retrospectives these are always fun for us but every time there's an outage disruption, communication failure anything that involves we need to learn something from this we do do a retrospective and what happens so the day before whoever's running the retrospective sends out a link to a new Trello board and that board has what went right what went wrong and an action plan and everyone's meant to fill out the first two columns what did we do well what didn't we do well and these are not blame sessions we don't call these post mortems because there's no dead bodies we call them retrospectives we're trying to learn something from this so people list them all out typically they do actually list them ahead of time we go into the meeting we spend the first 10 minutes voting on which things everyone else cares about things start to bubble to the top what did work well what do we all agree hey maybe we responded really well the communication was great or we resolved something quickly or conversely boy this was a dumpster fire we all completely lost track of what was going on there was no coordination whatever it is what worked well what didn't and then we explicitly take the top 10 items or so and do an action plan and that has to be owned by somebody so if something didn't work right do we need better process was the process broken you're not allowed to skip the process because the process is actually supposed to fix the process so go back to the process or documentation might have been bad and the one person who knew something wasn't available what are all those action things and sometimes this is a two hour meeting sometimes this is 30 minutes but what comes out is it's a tangible set of things of how we improve and someone has to own those and then that has to be implemented you're we like to say you're allowed to make unique mistakes at the company not allowed to keep making the same mistake right when you move fast you're going to break things if you keep breaking the same thing there's a more fundamental problem there so how do we use retrospectives to constantly learn some more and gain some experience then of course it's not all negative so once a month we do it all hands meeting and this is fun this is just in our shared space it's one of the days we usually have lunch brought in and it's just a what are new customers doing what's happening in the platform what are some shared metrics and we do this on purpose to also build some of that some of that shared urgency shared sense of ownership that it's not just people building products for fun there you are building products to support our customers you're building it to support a company that's trying to use those products and services so understand that hey there might be some big customers coming that you should know about or maybe there's some other big changes technology-wise or organizationally so we do this it's usually 30 minutes 45 minutes and just kind of get together and afterwards people hang out and if it's after five we tap the kegerator and that's fine just the team should get to know each other better spend some time together that's an important part of it finally I'll at least do a Saturday Saturday Batman sometimes gets paged and then by Monday boy Batman submit some great feature requests I get the best feature requests on Monday mornings from Batman from over the weekend hey guess what we need to fix instrumentation I bet we do what time did you get called so those are things that really play a great part I like seeing that they don't like seeing that and they want me to be Batman sometimes and I always refuse it but it's important they we learn how to make our platform better when the people who build it and have to own it are also on the hook for it and that's make a big difference so when Batman does get paged we stand up a slack channel and have a conversation if something is actually an outage is happening or if there's a disruption if it's just sort of a ticket thing that they need to work on they can deprioritize that and work on it later but it's a big part of it is that that person needs to have some ownership there so I'm trying to talk you through a journey you know your core values matter don't do any of these things because they're on a slide that you do them because your company culture says we want to you know we want to increase transparency we want to improve empowerment we want to have these sort of things that's why you do these things you have to go to your core values and this is just a representation of them so figure out what they are and then look at your logistics if you're saying we're going to do your CEO says we're going to do DevOps and nothing changes you're clearly not going to be successful and sometimes you might reorganize sometimes you might change your seating area sometimes you might change your tool set all of those have made a big difference for us actually being able to deliver the way we do at the pace we do and then finally you're looking at some of the execution of that what makes sense don't have meetings for the sake of meetings which ones are adding value are the right people there no lucky lose in the stand-up meeting we don't allow that it's people who are actually stakeholders in this and that's it so figuring out what your schedule is what's your process what's your engine that's made it repeatable for us it means when new people come to the company they can absorb this very quickly right you get into this culture very quickly there's no sign posted on the door with all these rules or all these meetings you start to just accept this as part of the culture so people live it it's much simpler thank you questions there's a microphone up there use it to ask a question sing a tune yeah so I was curious with your feature teams how you make sure that dependencies between them are resolved yeah that's arguably my hardest job is making sure that that person who wants to ship this in June and says it's the end all be all the team they depend on isn't going to work on it till August how do we prioritize that so a lot of my job in the product owner's job is as they we actually use slack to even here's my sprint plan for next month and here are my dependencies and so I talk to them individually but it's also socializing that with the rest of the team so when we say you need this team to ship this to ship your thing first of all that may not work because that may just not be that team's priority so it's about at least bringing it up to the surface and sometimes we do do some horse trading to make sure that that priority if it makes a big difference for a roadmap objective that they had promised or committed to gets achieved and I have to move something else around but typically falls to me to make sure that if things do have to shuffle because they're misaligned then I shuffle them or we communicate outbound that your thing won't get done when you think it does because that dependent team can't work on it yet. Okay and kind of related to that when you embed like operations people inside the future teams when you fast forward a year how do you ensure that they're still acting like operations people and still paying attention to that versus developers? Yeah just more focusing more on the next feature that needs to get out there. Yeah I mean really the key role of arguably any operations person moving forward is building the automation to run their products in any company I mean that's the future of system administration is caring to the automation not actually doing those tasks yourself arguably. So the role of these operations people as well as the developers that's where it starts to blend for us is that everybody's gotta be able to build a supporting infrastructure to have HA and know that this service can recover and have discovery and so those people are gonna be focused because the knowledge is still within the teams and there's gonna be turnover we're gonna have teams that rotate and things like that so the institutional knowledge is in the team and somewhat documented but the fact that you're still operating it and running it means you can't just focus on the next feature you're not running projects you're running products and if we treat that mindset there's never a project it's just about adding capabilities or building sustaining things on what you're already running so if you try to hit that mindset hard you don't just leave behind the old and busted and work on the new stuff it's all the same stuff it's all your product that you have to keep up. So clearly what you're doing today is different than what you did when did you start this? Well we switched to the full feature team model I showed you there I'd even argue in January but we've been doing a DevOps-y sort of structure about two and a half years. Okay so then it was an evolution kind of thing so a question then is really like how did you move from the old way of doing things to the new way of doing things what was the aha moment when you're saying we just can't keep going this way and then moving a large organization getting the sponsorship just to rip down walls and put in dog space and stuff like that this probably was a big change how did you make that turn? Right that's a good question so I came with a company called Tier 3 which CenturyLink acquired about a year and a half ago and as we were Tier 3 you can imagine as a cloud startup there seems to be some other big competitors we may know some other names to compete at that rate you've got to also work differently functioning in a siloed fashion we would not have been a startup any longer we would have been a footnote so we had to change from that imperative but it also came from a couple of executives who joined the company one of them who was a founder of Agile and came aboard and really started to put some discipline so if it doesn't come in with some sort of mandate saying we've got to change and whether that's market driven like we are going out of business unless we get better or it's an executive with the credibility to come in and say let's pivot this and change that that was the start of our change and it changed further once we got acquired CenturyLink saw that we'd like to reproduce what this little team is doing which is out producing every one of our large teams at this company we need to bottle that up and so success is the best way to keep this stuff going for us we got the mandate to build a great office because we were shipping and because we were proving that this was working if we had first off said go ahead and do this for this rag tag and a band of cloud people just trust us I don't think that would have worked I think they saw that there is a success here so success beget success and I think that was the key part of this you often have to incubate it we're not changing a 50,000 person company overnight but now we're a 500 person organization next year we'll see we continue to try to change bigger and bigger parts of this by demonstrating that this is a better way of working than having a bunch of silos with hyper managed people who don't get a lot of ownership are people enjoyed doing this work and so far it's proved successful yeah another question so I have a question about back on the feature teams is we have very much a similar structure to you and one of the things that's challenging is the feature teams tend to be built around a feature a particular feature that feature graduates over time it changes the team's morph and I'm curious about the ownership of the feature and the support as those teams morph over time yeah that's a great question I mean I would guarantee you our feature team structure of today will look extremely different in six months but what we also have is one feature team called SWAT and that SWAT team maintains mature in-market products that we don't expect to do a lot of re-engineering on and that's our biggest team by far it's a lot of people who take care of in essence whether a couple 100 products in market many of them don't require anything but some of them do so there will be a graduation process potentially for some of those things that don't have the same place in the platform anymore but as we build this platform and this automation I expect that you'll see teams shrink in size will spawn off some teams we have some teams now who are doing more work than one team should do so we're gonna slice that off and make another one and so I expect that this will be relatively fluid and that there will also be a retirement process for some of these as well as an absorption into a steady state feature team that manages a bigger portfolio so my question is about communication and collaboration that you talked about in particular Agile style I had a team that's dispersed over four continents seven countries, all developers the operations people are in different countries as well how do you, and I'm facing a real life dilemma how do you adapt to something like that? Yeah it's a good question it's a religious debate on one hand because it seems like some believe that you can truly do completely virtual teams where I mean look at automatic the WordPress company who have a great book talk about that the year without pants was a great book sounds like a weird book but it was a good book but it just talks about the idea of having a truly virtual team all over the place and some teams can make that work what's tricky especially with operations or outsourcing or things like that that if you're not aligned if you're just a shared operations team it's difficult to bring that into the fold that you almost do need to carve it up into particular feature teams and even that way right now we've got you know our headquarters is Bellevue I live in Los Angeles we have people in St. Louis our company headquarters is Louisiana the company we bought's in Portland so we're not all sitting in the same place we do use these collaboration tools pretty heavily to make sure we can all jump in a slack channel and ask questions we can quickly turn on webcams and a hangout and that's okay but you've gotta we still get together face to face at least quarterly the whole product team and I travel up to Bellevue a couple times a month I'm actually moving now cause I can't take it anymore but so face to face matters for us but the same time we've learned if we use the tools right and if we're transparent enough and people feel like they're plugged in you don't have to be sitting right next to the person so some of these communications we do the weekly updates and the all hands calls if you have enough of those we have plenty of people who are even by themselves in San Francisco or this or that that still feel like they're part of the team but it just takes a really concerted effort and that feature team structure really helps cause if you were just a faceless operations person that's really tougher than saying I own this part of the product I own our backup services and I work with three other developers and two other engineers I have a product owner I have a sense of ownership of this thing versus I just handle tickets generically across a portfolio. Any other questions? Good I appreciate you all coming to this stuff this is fun if you'd like to chat any further feel free to ping me or come on up and again thank you for the time.