 So my name is Sinton Ryan. I'm an industry analyst, Red Monk, we're a developer focus industry analyst firm. And I'm delighted to be here today to be able to moderate this panel. And I would like to get us from Comcast to introduce themselves please. So Uma, if you'd like to go first. Sure. I'm Uma Tumala, working as a developer, working on the transformation of our legacy SOE platform to cloud native microservices at Comcast. And I'm Chris Trotina. I lead the technical architecture team for the department that I'm a part of at Comcast. Good afternoon. I'm Greg Otto and I lead the cloud product engineering team for Comcast. Cool. So obviously we've heard Comcast mentioned earlier on today. Abby was citing a couple of figures earlier. For people that are here for the first time and may not be familiar with Comcast's journey with Cloud Foundry, could you tell us a little bit about the background and kind of the scale that you're at at this point? Yeah. And in the spirit of rapid change, I'm going to have to apologize to Abby because her slides from just a few moments ago are already out of date. So we have over 1,500 developers on our platform. We have over 12,000 application instances. And then we also have about 210 million transactions that we process every single day on the platform. So we gained about 30 million since our keynote a few minutes ago. And then last year when I was up here, we were into tens of millions of transactions a day. So now we're into hundreds of millions. So things change pretty quickly. Yeah. We were joking backstage that Abby should put the slides up in GitHub and we can just do pull requests to update the stats as it's going through. The kind of cultural shifts and cultural changes that happen in organizations when a technology like Cloud Foundry is rolled out are quite profound. Could you tell us a little bit about kind of the organization, the way it was and the silos that exist and how you've kind of moved beyond that over the last couple of years? You mean when we were transforming our cultural shift? Yes. Sure. So when we got started back in 2014, we were facing a couple of key challenges that probably many companies are facing. The first was we did not have a way to express with our business partners the impact of the incidents that were occurring. Additionally, we had very stark divisions. We were very siloed in our methodology between developers, operations folks, and testers. On top of that, we knew that we wanted to train up those engineers, but we didn't have a solid plan for doing so. And lastly and perhaps most importantly, we just weren't able to move at the pace that the business wanted to in terms of taking some great new idea and deploying it up through to production. So when we began our transformation, we put in a couple of key changes. The first was we agreed with our business partners on some common language. We called it impact duration for how we wanted to communicate the customer felt impact whenever one of our applications had an incident. As Greg knows, this was key for us in terms of putting together a business case that we could put in front of our executives to get the initial funding that we needed to really kickstart our journey. The second piece is that we did some organizational realignments to put developers and operations folks together to form DevOps teams and then added testers like Uma herself here, that's her background, development together to form value teams. Teams that can deliver complete end to end business value all within the context of one two pizza box size team. And then we brought in Pivotal and Accenture to help really jumpstart us. We thought we were doing Agile and DevOps the best that we could, but once we brought those guys in, you haven't seen anything until you've seen them do Pure Agile. And that's what really then led us to be able to do the DevOps culture shift that started to enable the increase to cycle time. Just on that kind of training aspect and the kind of retraining and reskilling people, could you talk a little bit about kind of how that has changed over time? Like at the moment there's literally a tsunami of technologies coming up people and we can't just take people out for training courses for two weeks and put them back in as much as sometimes a training course is a nice thing to get away to. So kind of like the on-the-job type training, can you talk a little bit to us about that? Sure. Funny enough, in a past life we went through a similar transformation at a different company and it was very classroom based, which certainly has its place, but we weren't able to come up with the kind of iterative value that we wanted. So knowing that when we brought in Pivotal and Accenture to help us, we were very keen on, we wanted to be on the job training. We want our folks, our teams on the ground to pair with those guys on delivering real business value for actual applications that we were looking to build and do or to migrate. And it was hugely successful. We were able to get some of those folks, many folks now. I think we have, I don't know, 50 some odd teams probably on the platform using the existing knowledge that they had before. Cool. When you started out on this whole journey, kind of the standard method that a lot of people use is they play it safe and they choose something small. It's not something you guys, Dave, can you tell us a little bit about the kind of applications that you chose to start with? Yeah, yeah, this was one we really hit out of the park, I think. When we first started looking at what we wanted to tackle, our first temptation was to choose something very small, very easy to try and get a big quick win. But then we did a lot of analysis between Greg and Uma, and then certainly shot out to Todd and Rob over there and their teams for looking at what were the highest value applications that we could move and value in terms of traffic and number of incidents that we could avoid. So we chose three pretty good-sized services, three out of 100 some, but by migrating those we were able to move 40% of the traffic off of the platform by the end of year one and free up enough capacity where the following year we could do the remaining 60 some odd services. So all of this then ultimately ties back to developers and developer productivity. So Uma, could you tell us a little bit about kind of the developer journey and where you guys have got to at this point and how you feel it's all got its mood? Sure. So this was definitely a big challenge until 2015 because we had a solid line in between the developer, a tester and the operations. But we adopted a Comcast couple of the things to really shift to a DevOps mindset. One of the things we did is we introduced the new tools, technologies such as continuous integration, continuous development and also we did the test-driven development. We shifted the testing to far left and then start writing many, many tests early in the cycle because we were on this big monolithic platform, Heavyweight, which used it to take 250 million plus transactions on any given day. To scale that kind of application on a monolithic platform, it used to take months for us because when developer write the code for us to deploy that code to different environments, testers use it to open the ticket, put it in the queue, operations use it to take the ticket, takes couple of hours to deploy, testers find the defects, the process goes and goes. So there were a lot of manual steps in there, but with continuous integration and continuous development and also with test-driven development, we were able to achieve. I can give you one real-time example. At Comcast, delivering the new products and services to customer houses is our job. We try to deploy new devices to customer accounts on that old platform like Enterprise Services platform, which is a legacy. It used to take three to four months for us to deploy any new feature to production, but with the continuous integration, with the collaboration from the teams, we started deploying any new feature to the production in 15 days, even less than 15 days. So that is a major culture shift that Chris was talking about, and also we removed a lot of manual steps in our processes. So that was a major shift in the DevOps methodology. Cool. Would you be able to touch just a little bit more on the TDD part of it and how you feel that has changed the way people are approaching development? Yeah, sure. So we had to have a common-release schedule. We have like 20-plus development teams. We use it to share common-release schedule. So everybody put in the code on the same day by operations. But we were not talking to each other. We were not collaborating. Every team is in their own silos, their own issues. But once we started going into the DevOps, we were deploying all the applications into production at our own pace, but still we were collaborating. We started collaborating. We had a private Slack channel with like 400-plus developers talking every day, talking about the issues, solutions, different innovations. So that really changed the culture shift in Comcast. Cool. Just in terms of like this, the emergence of Slack channels, that this is something, I was talking with the guys backstage about this, that it's something that's coming up as an emerging team in lots of organizations we talked about where there are whole diverse communities of people who haven't spoken to each other before now, now talking. Could you talk a bit about kind of the cross-pollination of ideas and what you're seeing in terms of that? Yeah, I can take that. So when we started the Role-Out Cloud Foundry, we created a Slack channel. We used Slack for real-time communications and collaboration. And it was a really easy way for us to be able to share what was going on. And this kind of community phenomenon was happening naturally, which was really interesting. So we have over 750 people. Actually, I think it's closer to 800 people. So even I'm out of date. And what was really interesting is that development teams that normally would never even talk to each other, totally different products, are all collaborating on a real-time basis, sharing best practices, helping each other out. And it really started to see a shift in the way in which all the developers were collaborating across these diverse teams. And then I think Chris and Uma can talk a little bit more about how that inspired them to create product channels for their own development organizations. Yeah, like the XSP channel, right, Uma? Yes. So we rebranded our Enterprise Services Platform, ESP2, Xfinity Service Platform. That is our Comcast brand. So we introduced our own Slack private channel with the name XSP, where like 400 developers, even though they were all looking for solutions from the architecture team, if some other team already had that issue and they resolved it, they used it to pitch in, they started pitching in, they started providing the solutions and paid programming. If already a team experienced the issue and found the solution, they needed to pay them with the right folks, that's where we started collaborating. And one of the other things that we were doing is, any time we are introducing the new frameworks, new libraries or any upgrades, we started communicating using our development forum. So we used to take the constructive feedback, we keep on making the changes. So there was a lot of improvement from the initiation of the framework to where we are right now. So the feedback loops have got shorter in terms of all the conversations. Well, feedback is always welcoming, which really helped us to improve more and more on making the things very easy for all the development teams across the board. I think you referred to it earlier, Uma, as community-driven problem-solving innovation. Yes, true. So really nice kind of evolution from where things were. One of the other aspects that has struck me, I've heard Greg talk before during an analyst session here last year, the growth in terms of developer adoption within Comcast and kind of like from a relatively low base to the point you're at now, could you just tell people just a little bit about what that kind of growth rate has been? Sure, and we'll definitely ask Chris and Uma to jump in, but the growth in the developers, we started with very small teams and my team is not marketing this to the rest of the developer community, but once we had some of the wins that Chris described and his first services that we moved over and some of the developers were feeling that lift of anywhere from 50 to 75% improvement in productivity so spending more time on code and delivering new product experiences for our customers, it really started to expand because of all the developers talking to each other and encouraging them to try it out, so it really just kind of exploded from there. Yeah, I think our development community really craved the independence. Any initial hesitation I think was washed away once the first couple of teams migrated and started seeing the advantages that Uma described earlier around no ticketing to do deployments. The ability to deploy on hours without a CM, things like that, a change management ticket. Those things really encouraged the teams that moving to this platform was a reward and something that they wanted to do in order to provide more business value quicker. Which is a nice bridge to my next question, which is about the business value. So essentially technology is ultimately about customer experience and you're not in a large enterprise, you're not going to get investment in technology unless you're changing the customer experience. So could you talk to us a little bit about the business value aspects of what you've seen over the last number of years? Yeah, so I can take that and hand it over to Chris. So again, when we had some of the resiliency challenges with our applications that were impacting the business and our customers, that was really kind of the first challenge. So we think about the business value across three dimensions. One was the resiliency of the applications that moved on to the platform. The other one is around time in the market and improved productivity. And then the third being scale. So scale is hugely important for us. And just some of the numbers in the impact minutes that we've had, we actually decreased those 81%. And then at the same time, the amount of time that it took for the teams to resolve issues when they did come up was increased by twice. So they were able to solve problems twice as fast. And then in terms of the frequency, it was happening half as much. So all of those things conspired to create a really awesome story around the resiliency of our applications and then the improved customer experience for the business that resulted. And then from a time to market across the entire platform, we're seeing at least 50% improvement in time to market. And some of the teams, it's even higher than that. And then I don't know, Chris, if you want to talk about some of the specific examples from your group. Yeah, I mean, first of all, it's funny thinking back to the end of 2014 when we were sitting together, Greg, talking about that number. Because at the time we got the funding having promised that by the end of 2016, we'd reduced by 40% impact duration. Not knowing at the time how we were going to do that. And here we are today sitting before you having reduced by over 80% in just about the same time period, much of which has to do with our migration to Cloud Foundry and the additional resiliency that provided. One of the great examples that I love telling this story, near the end of 2015, Greg and I and a couple of other folks sitting over yonder were getting ready to present to our CEO the next day on, hey, here's what we built, isn't it great? The night before, we found a bug in one of the services that we had migrated. And we ended up fixing that bug and deploying all the way through to production with complete automated testing in a few hours that night. So it was pretty fortuitous going into the next morning to be able to have that story and say this is exactly the kind of value now that we're able to deliver. A fix that probably would have taken us at least a week with our old cycle now took us just a couple of hours and that was so impactful. Yeah, it really demonstrated the power of the platform and allowed us to kind of put it through its bases and definitely a fortunate moment that demos can go bad especially in front of your executives. There was also another pretty cool moment that happened and it was last year when Chris and I were heading out to the spring one platform conference and he had talked about it on stage when I didn't even know but literally the night before we were all flying out a couple of the folks from the team were a little groggy and that's because it was an issue the night before and again they were able to create a patch and then run it through all the cycles fully automated and in the production environment which is pretty sizable and it was like two to three hours which was pretty incredible. Yeah it was really nice last year to be able to share that on stage. I mean it literally had just happened as I was walking up to present and it was just a great better together story as you like to say Greg between the two teams the team running Cloud Foundry under Greg and then all the application teams like myself like Umos throughout the organization partnering together to solve a problem literally in less than 24 hours after we found it and live on ours. Yeah I was a little nervous when he started to tell the story because I was like uh oh you know Did I steal it? Well I wasn't sure if it turned out well because it was news to me at the time so it was pretty cool. So I guess one other business value or business metric that's of interest I think to most people in this audience is developer productivity and just getting things done faster. I don't have quite as good a phrase as what we were had from SAP because you talked to us just a little bit about the speed gains that you've seen in terms of bringing kind of products out and the iterations on them. One of the thing we had the challenge is scaling the applications because with 250 plus million transactions on any given day any issue that comes up in production and deploying the patch to the production for all different forms in the production it was a quite challenge for operations team monitoring perspective but now we were able to see in production across all the application instances and our strong monitoring was helping us to see how healthy is looking like so that was the biggest shift as well where we were able to see the issues before even our consumers our customers are actually identifying so that was another interesting thing that really helped us as well. Cool. I've never kind of set up on a stage like this we've got a timer that's timing down in front of us and if you've looked at the schedule we apparently have 40 minutes but I'd never like it to be said that a red mark analysts stop people getting to beer so we kind of have to have to do things on a little bit so you please know this is actually my last question but before I ask my last question if you're interested in hearing a little bit more about this, Greg is giving a talk tomorrow at 3.45 so pop along to that talk. So the last question two words, what's next? Yeah I mean we've come a long way in a short period of time we're pretty happy with where we're at but we're striving for more the next big two items are using cloud foundry as an enabler for multi-cloud we have a desire to run both in private cloud as well as multiple different public cloud providers and we think the abstraction that cloud foundry provides us gives us the ability to do that and then the second big item is around digital first that's where we really want to be able to give our customers the ability to interact with us on their smartphones on xfinity.com as the first line of defense Greg and I were talking just before and I believe the last time I saw we take 300 million phone calls per year and a lot of customer interactions from folks who I'm sure would prefer to interact with us through digital means versus waiting on the phone for a little while so you've probably seen some of the press releases we're making great progress already but we're looking to get there and really transform the customer experience Cool Thank you very much folks Thanks