 I will stand here. Hey, so I'm very excited to be here. This is only my second DevOps Days event. For those of you who don't know, we actually have our data center facility here in Denver, as well as our credit facility. So Denver's kind of a second home for Nordstrom. And we're also hosting our first DevOps Days Seattle. So if you haven't been to Seattle and you want to attend that event, it's in mid-May. So sign up for that. I'm going to talk about the journey that we've been on at Nordstrom. And it's been extremely challenging, as you can imagine, large enterprise. We have a lot of different systems in our environment. And really, I'm going to focus on what it's like to live in this hybrid world. I mentioned we've got a data center. We actually have multiple data centers. We're also moving to the cloud. So I'm just going to talk about some of the complications with that and what we've learned along the way. And just for a brief introduction, my current role is I have accountability for the program management and engineering functions for our customer-facing teams. So website, customer mobile, in-store technologies, personalization, anything we're doing in that space. Brief context setting for Nordstrom. How many people are Nordstrom customers? OK, thank you. So the way we think about our business is in two spaces. We've got our full-price offering, which is really our Nordstrom full-line stores. Nordstrom.com, the website, our full-line apps in both iOS and Android. And then we recently acquired Trunk Club, which is a capability you can go and say a little bit about yourself, and then you'll get connected with a stylist. And the stylist will actually ship a trunk to your home based on your style. It's kind of a cool offering. And then we have our off-price components as well. So Nordstrom Rack, the store. And then we have Nordstrom Rack.com and Hotelook, which is a flash sale digital offering. We also recently opened stores in Canada for the first time. So that's pretty exciting. And we'll continue to expand into Canada as well. So as you can imagine, technology is a key enabler for us. It's basically everywhere in our environment. And we're really focused on strategic flexibility, really bringing the experience element into our digital properties. Because if you shop our stores, it's very high-touch. It's very experience-based. But the online offering doesn't really match that. And then kind of the flip happens in the stores. It's not as convenient to shop our stores. So if you want to get in and out really quickly, we don't really have a model for that. So we're trying to figure out how to bridge those gaps. And then we're all about speed. We need to move faster, as you can imagine. We're sitting in the same location as our friends at Amazon, and they move very quickly. And we need to be able to keep up with that competition. And then reliability. So that hasn't always been the primary focus. And I'll speak to kind of some of the similar things that Dave said about how we're creating a culture of accountability and really making sure that what we deliver and the people who deliver that, they also own and run and improve that code as well. All right. So I'm going to talk a bit about modernization and how we define it. And then I'll talk about two case studies. One that is really about our modernization, and then one that is still about improving even our existing capabilities. So as you can imagine, we have varying degrees of maturity within our environment. And we kind of think about it in these three stages. We've got people who are crawling, walking and running. And then I always joke that we have people who haven't been born yet because they're not even really thinking about these concepts. And people are really evolving in different states. And even within the same teams, we have teams that are in the web team that are in both a crawl and a run state. We've got teams that are bridging between crawl and walk as well. So this is just kind of illustrative to show our web team. We've hired a lot of external talent. We've got engineering thought leadership in that space. We're starting to invest in understanding the lean practices and mindsets and really starting to migrate to the cloud as well. I'm going to go through this quickly. All right. So this case study is about the money maker on our website. So as you can imagine, our product page is kind of the key component of our website. So prior to this modernization effort, our website release cycle is basically five weeks. So every five weeks, if we want to deliver value to this product page, that is basically how the business was run. So queue up a bunch of changes, push in big batch every five weeks, and you can imagine how wonderfully exciting and successful that is. So this team basically went through and said, how can we take the monolithic code base that we're currently stuck with in our environment and really take the product page and isolate it, put it in the modern stack, and really get to a spot where technology is not the constraint for delivering value to the product page. So our initial condition, and I say initial instead of current because this team is now at the target condition, there was no trust. Our customers in this context, I mean, our product management team and our extended stakeholders, they just did not trust our ability to deliver. We're very slow, as I mentioned, five weeks is an eternity, especially if you're tacking on, I got to conduct an AB test and I got to learn from that and then eventually I'll put it into production. We had and still have a very big code base and we were having site failures during peak traffic time so we would encounter critical incidents as well. So our goal was really to improve our trust with our partners, increase our speed really to the point where on-demand releases were just the error that we breathe. Moving to microservices, really leveraging CI and CD for the deployment pipelines and creating environments that were reliable. So within that five week really cycle, the number one blocker for teams was environments, environment availability, the data stale. They're not treated in the same way from a 24-7 support so we would see a lot of leg in our cycle time because of that. So when we started, and I love that Dave kind of hit on this too, tons of silos. So there were multiple dev teams potentially contributing to this code base, centralized support teams so everything got kind of thrown over the fence and another team tested it and another team released it and another team supported it and so we had all these handoffs along the way and that's inclusive of the infrastructure capability. So we really moved into a model where we brought all of the engineering capability into this single team and we created this on-call support rotation. At first we tried to just put everybody on-call and we just rotated within that existing team and we found that the team didn't have the capacity to actually fix the things that were broken in production because they would just get right back on to the next feature. So this creates the capacity for the team to actually improve and focus on things that are keeping the team from going fast. So this turned into eventually, I mean, you can imagine, it turned into really strong sense of ownership and accountability and really focused on that operational mindset up until then our engineering teams and our dev teams were really accountable for feature delivery only. If there was an incident in production they were not ever really engaged unless we absolutely had to pull them in. So it created that mindset and then we accelerated our feedback loops as well. We moved away from managing all of our source control in a central way and really having our data center as well. So I mentioned that we have two data center facilities. One of the data centers supports the website 100% so we started moving away from leveraging that on-prem infrastructure and really moving into the cloud. And then also getting out of this five week release cycle that was constraining delivering value to our customers. So now we can release multiple times a day, technology is no longer the constraint. We have the rollback capability so before if you were part of the five week release cycle you put something into production, if it caused an issue it was roll forward. There was no version of rollback. So there could be multiple days of customer impact until we actually were able to resolve that issue. Now we can roll back within minutes. And we can handle peak which is probably not shocking. That's one of the drivers for moving to cloud. We don't have these issues where if we understand that we actually need to add peak capacity we've got to procure hardware and get it into our data center and now we don't have to worry about that. So from a logging and metrics. So our old model was really literally parsing through IIS logs and trying to find what the root cause was. Antiquated ways of measuring performance and the other key to this was the team didn't define these metrics. We had a shared service organization that would define the metrics and then push them to the teams which creates this environment where the teams don't really feel like they're accountable to those metrics and also they weren't helping anybody. They weren't actionable. So the team really defined what do we need to know within our product page in order to be able to support it appropriately. Which was huge because now they're able to influence those metrics. They leverage them. They're all actionable and so it's kind of flipped that mode away from having this kind of central team pushing it. Oh and the other thing really using business telemetry. So our old model was is the system up. So if you can hit the homepage, we're good. If you can hit the product page, we're good. And in reality, we wanna be looking at things like we're not seeing people add things to bag. Why is that? So it's really more focused on scenarios and modeling than it is about this page is available. And then we got out of the centralized support model. So as I mentioned before, it was really separate team would get called if we had an incident, they would work the incident and then maybe the root cause of that incident would get found and then highly unlikely, it would actually end up on the right backlog and they would actually really resolve root cause. So we never really got to the bottom of incidents and then we never really got resolution to those. And now they have proactive alerts. They're resolving incidents sometimes in minutes and really being able to own those incidents and have that sense of accountability. So meantime to detect has gone down. Meantime to recover has gone down and we really have less incidents which is probably not surprising as well. All right, so here's what we've learned and I know it's a lot of words, but as I said, this is only my second one of these so I'm still learning as I go. But absolutely agree with Jez Humble and it's funny cause when I prepped for this, we did a bunch of interviews with the team that actually works on the product page including our product manager. They actually said a variation of Jez's quote without saying Jez's quote. They said, we just, we created pain and when the pain came we were like, we're gonna make this better and we're gonna own it. We're not gonna shy away from it cause old behaviors would be, oh you created a production incident, oh we need more change advisory boards. There's somebody, we gotta pull everybody into our room and we gotta analyze the change because obviously we need a risk management program because that's why it went down and so there was all these old patterns and this team was like, nope, we're gonna feel pain and we're gonna solve it the right way and then if we have more pain that's actually a good thing. So complete agreement with that approach and there's a true behavior around continuous improvement. So the team's passionate about it, they wanna work the backlog items that actually reduce the pain. The product manager really understands it which is a huge win cause we have a lot of product managers who they're just like, eh, I don't wanna care about the underlying capabilities to support my features, I just wanna deliver features. So having product managers who understand that performance is a feature, the availability of the page is a feature. So it really shifted that mindset and having that strong partnership. We're really, it says smaller feature increments because I'm not allowed to say single piece flow yet but that is the goal. We really wanna get to the point where our features and the work going through this team is in manageable increments because before that it's these big batch approaches. The ability, this might seem small but having the ability to experiment. So beforehand we would have all the stage gates and you basically would put something into like a UAT environment and then everybody would hit UAT and provide feedback and then kinda go through the system again. This gives us the ability to say, okay well we think we wanna add this capability to the product page. We wanna change how we're displaying images. Well let's just spin up an environment, we'll give access to the business, we'll give access to the site merchandisers, everybody can get in there, test it out and then we can actually iterate from there. So the team can do that in a half a day, have something up and running even sooner. We no longer have sign offs which might shock some people but actually maybe people don't know so don't tweet this one out. We used to have these big, everybody's gotta get in the room, we gotta get security and privacy and operations and infrastructure and all the people in a room to basically say that we could do a go, no go. This team does not have to do that because they've created the ability to still, they still track their stuff, they're not just pushing things to production with no change records but it's all just part of what they do. So if we need to show an auditor or a security engineer or a privacy person what we've deployed, they can get to it just through the actual CI CD pipeline that they've built. There used to be another pattern where 100% of the team's capacity would go to features. So next release, let's pick the backlog, all features. Now this team actually allocates an amount of capacity to non-features. So what are the infrastructure related capabilities that we need to deliver and they get that support from their product manager because everything's transparent, it's all visible. They can go through that all together and really decide what needs to go in that next release. And the team is focused on business outcomes. What it used to be is teams would not really ever have line of sight to why they were working on a certain feature. It's like, oh, it shows up in my backlog. I do my work, I push it into the system and then go on to the next thing. Now they know, cause the business outcome is very clear. It's like we are going to drive conversion rate up by this percentage and they know that the features that they're delivering are in support of that and then they measure it afterwards and they're in it with the business so they know my work's meaningful, which is important for, I'd say it's probably the number one thing that is feedback from our engineering community is, I just wanna know that what I'm doing is actually providing value and can you give me line of sight to the work I'm doing? So this is a pretty easy way for the teams to do that. So we don't have it all figured out. Surprising, there is a whole lot of stuff that we still are working through. So this team is pretty autonomous, but not fully autonomous. There are still a lot of things that if they wanna do a certain capability, they still rely on other teams. So figuring out how do you orchestrate that and you're only gonna be as fast as the slowest link in that chain. And there are a lot of opinions about being able to move engineers around. It's like, oh, well, we wanna switch our priorities. We'll just pick this team up and we'll move them over here. We found that that's really not that easy to do. Especially when you get teams immersed in thinking this way, it's really hard to just forklift in both directions. Teams that haven't been operating this way have a hard time just moving into a team that is and then vice versa. It's hard to go back to doing work the old way. There's still cross-team alerts. So even though this team can proactively see the health of what they support, they still are reliant on other teams to give them information and not everyone has moved to this model yet. And still creating that operational awareness and mindset, it's not completely adopted across all the teams in the environment. And still have this decade plus old monolithic code base where we've taken, well now we've taken two sections of the website, but this was one of the first big sections. We've got a long way to go still. Which leads me to, you can't do everything at once and you can't take your eye off the existing capabilities because there's still value being delivered through those capabilities and you can't ignore it and we still have a lot of teams that are operating in this context. So I'm gonna talk about what we did to address that so that we didn't ignore that component of the company. So I talked about our five week release cycle. So any capability in our website follows this five week release process. Which is really three weeks of engineering and two weeks of what we call hardening. Which essentially is the catch-all for continuing to code, because not everything's ready. All the testing and preparation for the production deployment would happen in that two week window and it was a disaster. There was a whole lot of waste in that two weeks. So we basically said we are gonna take that two week hardening cycle and we're gonna reduce it to one. And it might sound like, well that doesn't seem very aggressive, but at the time it was a big deal for us to actually go after that. So we had a lot of conditioning in our environment where people were just so used to, we've got our, well we have time because we have hardening. We're always gonna have hardening so we'll just wait and we'll just leverage those two weeks and then we'll get the code into production. And it was very manual. Releases would take several days. The team would be in a war room for 24 seven. We'd have shifts of hundreds of people to get this code into production. And we had this exception process which I like to tell this story because it's like every exception that came into the hardening window, I approved. So it's like I have no idea what's going on. Like at the ground level and it's like all these emails are coming to me to say it's okay for this stuff to go through the exception. And guess what I'd do? I would rubber stamp it because the team was waiting for me. They're like, well we're blocked until you approve this thing. And I'm like, well I guess it must be okay. So let's move forward. And that's really not helpful. And so one of the very first things we did when we documented this value stream is we said we're gonna stop it. There's no value in this approval process. And there were people who were like, like how are we gonna know that we're putting the right stuff into production? And I'm like, we don't know that now. If you're depending on me to be that person, that is not a good thing. So yeah. Thanks. So super exciting. Target to get to single eight hour window. Like let's let people go home. Let's do the release and let people go home. And then I added this at the bottom. Our goal for 2016 is to continue to take 20% out of this because it's probably not ever gonna be done. Or I shouldn't say that. It won't be done in 2016. So we need to continue to improve. So essentially we brought 40 people into a room, 10 different teams, documented the value stream and essentially just went through the exercise to say, what happens during these two weeks? Like what exactly transpires as we're going through this hardening phase? And I can tell you it was really painful because we had people who were like, why don't you know where all the problems are? Why don't you just let me engineer our way out of this? Cause I know exactly what we need to do. We need to stop doing that. We need to stop doing that. And then all this magic's gonna happen. And in reality, nobody knew where the problems were. And until we did this exercise, we had no visibility to the reality of what was happening and we had no way to quantify it either. So this gave us true cycle time for lack of a better term for this window of the release. So this is a graph that shows just kind of what transpired as we continue to take waste out of that hardening phase and really just continue to work on what testing do we need to do? What needs to be automated? The really controversial one is that we put in, we were no longer gonna have our performance testing gate be a gate for releasing. So we would take our performance lab, it was always at the tail end, usually like seven or eight days into this two week hardening phase, we'd finally go into the performance lab. And we'd get results, they'd be mixed. We get things that would be like, you cannot put this into production and then we'd dig in and it's like, oh, actually the environment was configured incorrectly. So we're actually okay, everything's good, now we're go. So we're like, forget it. We're not gonna put this as a gate. We're actually gonna move performance engineering into the engineering teams. We're gonna do it as part of like our non-functional requirements and we're gonna test as we go. We're not gonna have this big batch at the end, which also freaked people out, but we did it anyways. And we have not had, and I don't have any wood to knock on, but it's like we have not had any performance incidents since we stopped doing that. So I always end with this. This is like my slide that I use every time. It's a huge change in behavior. These are the things that we're now seeing and when I talk about leaders, so my role in this is really to create that environment where we can set up the teams to be as productive as possible. We need to get out of their way, really understand the work that they're doing and meaningfully remove those blockers. So we've got a huge initiative underway around unlocking engineering productivity. Really, I love Dave's slide about the access denied to production. It's like that would be amazing if we could get to that point because we're still in a mode where there's all these I'll call them overhead, but overhead processes before we can release value and I think we need to continue to break those down over time. But it takes leadership to start that and to set the tone for the team because if we're not leading by example, then the teams don't see it as value. We did improve trust. This team is one of our models for how to deliver work. Actually, both teams improve trust. Even the legacy release team, they're seeing a lot of benefits from this as well. And teams are learning how to problem solve in a very different way and really practice continuous improvement and that lean mindset. All right, so I have no idea how long I've gone. So I don't know if I have questions. Time for questions. Okay, 10 minutes for questions. So if you have questions. Got it. Okay, so the question is, are we using new tools or the same tools in a different way? It's a combination. So in some cases we adopted new tools in order to really create that ability for that team to go fast. Some of our existing tools could be leveraged in that context. We just needed to think about them differently. One of our, you know, I talk about patterns and behaviors. One of our old patterns was we gotta buy everything. Like there's no version of open source that can actually help us. That has flipped completely. Like we are way more open to open source and leveraging tools that are optimized for the outcome we're trying to achieve. So that's another key change is what problem are we trying to solve versus prescribing? You must use this tool to solve that problem. So does that answer it? Yeah. For the, okay, so the question was what sort of tools are we using for kind of the continuous quality piece? Yes. So we actually had to build some things because really thinking in more of a behavior-driven and scenario-driven test cases, we actually had to, we had to build capabilities in order to do that. So I can talk to you more about it maybe on a break or something, but yeah. Yeah, so the question is I'm creating, like talking about creating visibility to the process and kind of how that was facilitated. Do you mean in the new way or the old way or both? In the new way, yes. So essentially what we did is we started with what we thought should be our target. So it was really how do we deliver on demand? Like we do not want to be in the way. We want technology to not be the constraint. So we set that tone with the team and then we brought together the roles required to actually pull that off. So we needed product management in the room. We needed our UX partners. We needed, at the time, we needed security to be at the table with us because we were gonna do some new things in the cloud and we wanted them to be on the journey with us, not a, oh hey, show up at their door and go, can you pen test this? So it was really like, how do we embed them in and bring them with us on the journey? Same with infrastructure engineering and live site ops. Like let's bring them into the mix and let's understand how we're gonna go about this and then let the team self-organize and really come up with how they can go about this and achieve that outcome. So that was the way we went about it. Oh, and we gave them the capacity in the air cover. It's like this team is gonna be doing things differently. So know there will be risk. Know that they're gonna fail because that's gonna happen. We need to allow it to continue to move forward and not give up. So, yeah. Okay, so I think I got it. So how are we working with teams that maybe haven't kind of opened their eyes? And do we have kind of a corporate mandate or are we doing it more grassroots? Yes. So it started grassroots and really started with a very small pocket of the organization where we said we are gonna lose if we don't change how we're delivering value into these customer-facing properties. So we said, we're just gonna, we did work with our business partners to make sure that they were on board with it, but we said, let's start here. Another pattern that we have at Nordstrom is we always try to boil the ocean. That's like, oh, we're all gonna start operating this way tomorrow and nobody, it never sustains. So it was like, let's limit the blast radius. Let's prove that this can work and then we can take it broader within the organization. So I wouldn't say that we have a mandate, but we have recognition that this is the way that we should be delivering value and more teams are starting to adopt these practices and it has visibility within our executive team as this is a way to get value to the customer faster. So yeah, so the question is how we've been able to give line of sight to the developers around business outcomes and kind of how we're going about that. Yep, okay. So really, it takes a couple things. One that we had to do right away is get everybody in the same building. So we were not, we were in separate locations and physical separation, even though we were all in the Seattle core made a difference. So all product management sits with the engineering team. They're like, you wouldn't know who reports to who. You like walk around, they're all one team. They actually use visibility boards to cascade their business outcomes and talk about them every day. So they all stand around it. They're like, all right, how are we doing? Are we achieving the outcome? What's on our backlog? Do we need to pivot? Should we try something different? So they're all together when they do that and now we're starting to actually implement that even further up into more of our kind of enterprise initiatives so we can cascade it all the way to the team and back so they can truly see how they're delivering against those outcomes. Yes, yep. So in this case, you know, it's conversion is the biggest one because it's our product page but we have things like we watch CSAT which is a huge indicator of how we're doing from a customer feedback perspective. We watch sales in demand. We're moving more into like customer engagement so if we put something new out in our digital properties, how are customers actually responding to it versus only looking at the conversion metrics? So those are some of the metrics we're using. Yeah, yep. So the question is about how we kind of went from having, you know, and we still have this where we've got a lot of environments, a lot of teams have a lot of environments and like how do you kind of break apart a component and spin up an environment and still have it be, I would say maybe operable. Like does it, you know, how do you, yeah, how do you kind of separate out and still get the value of an environment, right? Oh, infrastructure change. Yeah, yeah. So literally we stopped deploying environments into our on-prem scenarios, so within our data center and we moved to cloud. So, yep, yep. So all of that is spun up in the cloud. All of the configuration is automated. All the data provisioning, everything happens now in the cloud. Yeah. Questions? All right, thank you.