 All right, so good afternoon. I'm Greg Otto and I lead the cloud platform engineering team at Concast and I have responsibilities for our pivotal cloud foundry environment It's really an honor to be here Any of the folks roaming around outside if you want to come on in? so I'm really excited to be able to share with you an update on our journey since last year and again, you know, we have the whole team here all four of them So we're really excited to talk to folks and you know, we're going to be here For the rest of the conference. So I'm going to leave plenty of time at the end for Q&A So please make sure that you if you're not going to ask questions during the session Feel free to stop us when we're walking around the conference So for those that don't know love it when the clicker doesn't work so Comcast is a Global and NBC Universal is a global media and technology company and A lesser known fact is that we actually have the world's largest IP network. It will be really awesome If my slides were working Sure Yes, so we you know, we were here last year So this is you know part two I am definitely not in marketing because the title of my session kind of stinks But yeah last year when we were here, we were talking about our journey So I spent you know quite a bit of time talking about how do we get involved with Cloud Foundry? And how do we start what were some of the challenges and then today what I want to share is I'm going to briefly cover that and Then I'm going to focus more on kind of where we've been since last year and then What were you know some of the things that we're working on now some of the changes in scope and scale for those that may have Seen us in the keynote with the panel. We gave some facts and figures But this allows us an opportunity to provide more detail more granular detail and again an opportunity to have an interactive Discussion which you know, we really can't have on the main stage. So again Comcast We have a lot of different products and services across, you know all the different entities that we have so We have a lot of great stories to share and with respect to our adoption of Cloud Foundry So for us it started again briefly in 2013 We had some real challenges around our legacy applications We had some brittle applications that were causing impacts to both the business and Then also for our end customers. So the challenge from the business was hey, you know, we need better stability We need more resiliency around the applications That are being delivered to our customers and supporting our care organization. So that was a huge focus But we also knew that this was a transformation and I say this all the time not just of the technology But people as well. So we were very committed to investing in our development organization creating a safe environment to change Mindset and work in a new way with these newer methodologies around agile development All the modern best practices for 12-factor apps test-driven development and so forth So I'm going to share some of the things that worked well And then we're also going to share some of the things that maybe didn't go so well some lessons learned and Hopefully that will help you in your journey. So as we go through From the part one up to up till now We'll highlight some of those some of those things so in 2014 we got the Foundation laid and that wasn't just with the technology in the infrastructure It was with getting agreement from our business partners at the executive level On what were the things? What were the metrics that they wanted to measure our success by right? So we heard about the resiliency Time to market was certainly a key element there and then of course with Comcast we have challenges around scale so We one of the things that went really really well was that we started this journey together So this is hard stuff. There's no magic pixie dust so you definitely want to bring lots of friends and You know going at this in a better together type of approach is you know really the best way to go that's my best advice and Then again what we learned was you know starting small getting some of the DevOps teams You know early adopters was really key build success with them first In 2015 we started to move some of the first services over and again what worked well was We tackled not just the easy things we didn't want to just you know get some easy wins We wanted to focus on some real high-value targets, and I think Chris shared during our keynote that we picked three services to move and Those were the services that were very very active it represented about 40 percent or about a hundred million transactions On one of our critical platforms that process around 250 million transactions a day so those three services are part of Over a hundred services in a monolithic application that we decided to move so it was pretty ambitious Definitely want to pick high value targets your ambition level might not be as high Be definitely want to pick things that are meaningful that you can demonstrate that business value One of the things that we learned really quickly was don't try to boil the ocean don't take on things that Are even more complicated so for us what that meant was deferring the data persistence issues until after the initial migrations So that allowed us to realize value more quickly and be able to start to see what success and the the expected outcomes would be as We started to move some of the first services over 2016 for us was a huge breakout year You're going to see that in our operational metrics that we really stepped on the gas We moved a ton of services a ton of traffic over our most business critical and customer critical services that I'm also going to show are All running on this platform Activation systems order entry systems any of that customer kind of transactions that flow through our back office our Services that we've moved on to this platform some of the things that we learned and went well really well was around embracing and Accelerating the DevOps culture so realizing the benefits of shifting things left with test-driven development paired programming And then of course the CI CD pipelines getting those established Realizing the benefits of having an effective CI CD process One of the things that we learned Was to include the BA you teams up front so again bringing things in you know bringing teams in earlier It's really a big help definitely want to make sure that again the more folks that you include in the earlier that you do that The better that your journey will be as well So here are some of our operational metrics. I will get to the business outcomes. That's kind of the big reveal We shared some of these metrics last year And they're significantly different so we had about 490% growth last year and then you can see from the top That we started our journey on our internal private cloud There's some other folks that have actually started their journey on public cloud and are moving things internal So our journey is kind of the reverse most of our Cloud Foundry environment and the transactions that are flowing through it are still running predominantly on our private cloud But over the last couple of years Again based on feedback from the development teams. We've enabled multi cloud delivery to include cloud and multiple cloud providers So that's a trend that we expect to continue and again. It's really all about choice So cloud Foundry for us enables that choice so the developers can still focus on their code and Their applications and building awesome products that deliver new experiences for our customers And they don't have to worry about what cloud provider that they're running on So that was really powerful for us and we're starting to see some of the application teams actually Move it to a multi cloud delivery model So again, you know the application growth total hockey stick, right? insane level of growth there The number of developers the number of developers has grown significantly I think when I gave this talk last year, I probably should have remembered what the what the number one I think it was somewhere around 400 developers now We have over 1500 developers that are innovating on our platform every day and One of the really proud Pieces for me is calling out our slackers. They're actually not slackers We use slack for real-time communication and collaboration and We created a slack channel for cloud foundry We have over we have almost 800 people I don't know it may have gone over 800 since the beginning of the conference, but having first of all having over half of your customer base actively Communicating collaborating in real-time on a daily basis is pretty incredible But what's really rewarding for me and this was not a planned outcome. It just was something that happened It was a residual benefit, but it's really a Another transformational moment for us because it's changing the way that we're communicating and collaborating So we have development teams that are otherwise never talking to each other because they're working on separate products And not only are they helping each other with respect to cloud foundry, but they're helping each other with what are some of the best Practices around application design What are some of the techniques that they're using? They're sharing across these different product teams. So that's really pretty awesome And then some of the application teams like Uma and Chris that were on stage with me on Tuesday They've taken it a step further and they've created their own product channels for their Individual products that literally hundreds of developers are working aligned to this product So instead of just a scrum of scrum type of format These development teams are communicating collaborating in real-time. They're seeing issues They're able to know what's going on and be able to jump in and help and you know really streamline the delivery And again, you're going to see that in some of our business outcome metrics in a moment But it's totally transformed how the development teams are working with each other So, you know that again the transformation is you know, not just around the technology but the people and also the way in which we work and the way in which we collaborate and It's even more open than what I would have envisioned it to be I'd be remiss if I didn't point out that I don't need more than one hand to count the number of folks from my team that actually support all this insane growth All the developers that we have any applications that were onboarding So those folks are up here in the front Hopefully took separate flights But it really is an honor and a privilege Because I do push them hard it really is an honor and a privilege for me to have such talented people Working on the team and you know, I know that the trust that the business partners have with us is just off the charts So and that's a testament to the folks that are actually doing the work I just get to stand up here and take the credit for it So again to make this real These Applications are right in the business and customer critical path. They're running Real applications both supporting our frontline employees the people that are taking phone calls or the people that are in your homes installing product and Then we also have some direct customer facing applications some of the URLs that you could go to right now today They are running on our platform so things like the splash page for the Xfinity stream app and you know a bunch of the other things that are up there They're you know, absolutely in the customer critical path And again, I think that I pointed out at Abby. I had told Abby that we were around 180 million transactions a day That was maybe a couple of months ago And now we've gained I don't know where but 30 million additional transactions a day. So You know the growth as you saw is just phenomenal and it keeps growing and growing so Last year I was talking about tens of millions of transactions a day And now I'm talking about hundreds of millions of transactions today I get a little bit queasy when I think about what next year might be but it's pretty exciting and We'll probably still have four people supporting the platform All right to the big reveal So some of the business outcomes that we've been able to capture and to share first thing was a resiliency that was the biggest that was the biggest issue that the business had and I know that there's a few folks like Chris and Ooma and my team that were a little nervous because we actually committed to 40% reduction in the amount of time that our applications were being impacted by outages and those outages were impacting our customers as well So I'm not going to read them all off, but just to put it in the context An 81% reduction in the amount of time that our applications were impacted is huge But then when you add the fact That when we did have issues the teams were able to solve them twice as fast and The occurrences in terms of frequency was happening half as much So we definitely got like a huge, you know check mark on resiliency totally, you know Even my wild expectations far surpassed So then time to market Business always wants to be able to deliver new experiences to our customers faster. This is an average across the platform So the average across our platform is about a 50% time to market improvement pretty significant There are some of the applications that are off the charts transformational So taking an idea and putting it into a product as a feature or a new service that we offer that process Sometimes I've seen it take three four months and now that takes less than two weeks There's instances that we have that much of a transformational difference between idea to feature so big big transformation and Another thing I want to call out is that developer happiness, right? so everybody talks about happy employees are generally more productive and we've been able to put real metrics behind it to show that The developers that have anywhere from 50 to 75 percent Improvement and productivity because they're not working on assembling all the infrastructure pieces They're not doing transactional tickets to get work done Those happy developers are able to get things out the market twice as fast on average and sometimes a lot faster than that So that's definitely pretty pretty transformational and then again just because Chris numerous sitting here one of the examples is Our services platform that's in the middle of Basically every customer interaction. That's the 250 million transactions a day. That's running on that platform That specific example is on the board there. So there used to be 18 releases a year Which is was an enormous amount of work across many teams in a fairly well-refined process But using older methodologies That same application product and all the development teams that we've enabled to relearn Adopt newer methodologies are now doing 120 in the same year So features are getting back to the business that much faster to our end customers that much faster The experience is improving for our customers that much faster. So huge transformational change there and then Scale is pretty important It's important to note that these applications that are scaling an incredible pace That we're improving our resiliency and Then accelerating time to market. They're not sitting still Right. So those same applications that are on the platform while we're moving them to the platform have seen an average Increase almost three-fold. So scale is really really important for us and to be able to do all of these things At the same time in a fairly short window. I think it's took us about a year. I think right so You know, it's pretty it's a pretty proud moment for us And again a big update from last year that we're able to you know We're happy to share these These metrics and these business outcomes that the business challenged with and the team totally knocked it out of the park So it's pretty exciting Now I'm going to shift focus on where are we at now and what's next? I'm going to share more granular insight into how we manage the platform and some of the things that we're doing to try and Deliver even more services and some of the things that we've recently done So what this picture really depicts is that even with that insane ridiculous growth that we had The four people that we have supporting the platform. We're actually spending less time managing and running the platform and More time with the developers and understanding what their needs are we're working with them in a real-time collaboration environment every single day and It's really just it's unbelievable to even say this, but we probably spend about 80% of the time Over the last year with the development teams Understanding what their needs are and delivering new services Even with that that growth which again, it's a testament to the talent of the folks that we have on the team So what are some of the ways that we're expanding the platform? I'm happy to announce and this just timing is everything and Nithya is here and certainly was a big part of helping us give back to the Cloud Foundry community So Sergei who's sitting right here? Definitely want to talk to him We've open sourced Telegraph for both the build packs as well as for Bosch deployment And what that means in English is that when application teams spin-off applications on our Cloud Foundry environment? One of the things that they requested from us is that hey those same metrics and telemetry dashboards that you share with us It would be awesome if we had that for our applications and Because of the integration with the build pack and Bosch deployments Now within moments of birth anyone that deploys an application onto the platform Automatically gets these metrics and it all happens automatically and it's no effort on the developers part Huge transformation and again, I'm happy to say that that's all open sourced and available for you all to use and Hopefully, you know add value to your development teams There's a few other pieces that I wanted to call out. I'm not going to name every single one of them one on the slide here But we started to use Cloud Foundry kind of even beyond just the platform itself So using components of Cloud Foundry to deliver services Even outside of the platform and actually a really good example. We've heard about Kubo in the conference Which has taken the pieces of Bosch and you know UAA from Cloud Foundry authentication and authorization and Delivering value for spinning up Kubernetes clusters with a single command So that same pattern we've used to deliver some other services that we've provided first within our Cloud Foundry environment And then shortly after to the rest of our environment things like global load balancing is a service So we talk about 12-factor apps multi-site application deployments Now we enable a programmatic way for people to be able to globally, you know get that global load balance services So that they can spread their applications wide across multiple availability zones And the global load balancing services is not a ticket-based transaction And that's being expanded to not just the Cloud Foundry environment. So that's really key Then there's a few others there one. I want to talk about real quick is Jenner lemur. So Neville who's sitting up here as well. He wrote that and Folks may have heard of chaos lemur that runs around breaks the environment test the resiliency to make sure things are functional We created Jenner lemur which runs around and does clean up because Sorry, but the developers can be sometimes a little messy and we're lazy We don't want to constantly sweep up after them So this takes the application instances that haven't really been doing anything and goes and cleans them up for us Because it's so easy to deploy new ones anyway, and it allows us to spend even more time Calling up to our developer friends and helping them deliver these awesome experiences to our customers in the business And that's something that is going to be in process To be contributed back to the open source community So with you know the help from our friends like Nithya. We can get that pushed through and make that available so we're really happy about being contributing back to the community and We hope to get the gender to lemur open sourced Quickly so that you all can benefit from a little bit of clean up So more insight more granularity in terms of how do we manage the platform? so this slide depicts our overall architecture and strategy for metrics and telemetry and The strategy is really best depicted with the elements on top right the status and notification eventing and Then the trend analysis that we use most of this is all open source One of the things that we learned really quickly is that there's no one tool to rule them all So we took very much a microservices architecture approach to monitoring and telemetry and We picked the pieces that made the most sense for us that we can manage and scale in a loosely coupled way and Not going to go through everything But you know our infrastructure and platform type metrics influx DB is is the heart of the of the system of the environment we use capacitor to provide some business logic as we Publish the information that we consume and again that all the development teams can also consume as well and You know one of the things that we really want to point out that builds that truss that partnership as you go through the journey And I'm going to show you what our dashboards actually look like all of the information that we use a hundred percent of the Metrics and telemetry that we use to manage the platform is completely available to all the development teams So there's no question around how are things running my apps not performing right? What's the platform doing and getting that visibility that you might not otherwise get we completely make that open and Transparent the feedback we've gotten from the development teams is really really positive. They really appreciate the transparency And when we do have issues on the very rare occasion We hold forum talk about what went wrong We can show the metrics and then we can also talk about what we're going to do to prevent future occurrences So they really appreciate that openness and transparency So we have three examples. We're going to show I'm gonna go through it fairly quickly so we can get to some of the Q&A So these is an example of our platform metrics So these are on TVs and on people screens and so forth and of course any of the development teams that like to keep an eye on us This is an example of our metrics and telemetry that we use to show the health and status of what's going on in our platform environment Monitoring telemetry for applications, right? So we talked about the open-source tools that are available for you to download today and This is what we provide automatically for application teams And it's just a sample and then of course application teams can make changes and Add elements that are important to them But again this happens within moments of birth of the application without any user intervention at all And then this provides the infrastructure kind of underlying. Do we have enough scale? To support the environments to support the demand We don't want the development teams to worry about whether or not we have enough capacity to support What it is that you're trying to innovate so we keep an eye on these type of metrics to make sure that Our platform has enough horsepower to perform and again, this is for internal and external as well But yeah, we always want to make sure that we have enough scale to support the environment A few of the big lessons learned we've stepped on a lot of landmines for sure But this is representative of kind of the best advice that we can give and we talked to a lot of companies And this is usually encapsulates what we talk about If you're an infrastructure team that's working to deliver this I can't overstate the shift and focus from Focusing on the infrastructure to delivering platforms and services and getting closer to the development teams hugely important You can manage the environment using things like puppet and ansible and so forth but the faster that you learn things like Bosch and Concourse and have CI CD pipelines for how you upgrade and manage your own Underlying cloud Foundry environment the sooner that you're going to late make life a lot easier for yourself. So definitely that's a big Piece of advice because you know some things like Bosch are not the easiest thing to pick up There's a little bit of a learning curve but Sooner you do it the easier your life will become The emphasis on partnership and community Again just huge You saw the development teams up on stage, you know, we're really one team We don't really look at it in terms of Organizational boundaries So we're all in it together. You definitely want to establish that sense of community and partnership with the development teams and then simply stated If you make it easy you do not have to worry about adoption So that insane level of growth zero outbound marketing from us It's really just all the development teams talking telling a friend and so forth and being able to deliver value with much less effort and That's what enabled our environment to scale. There was no mandate the Alps shelves must not sit It all just happened organically and again because we provided a safe environment for the development teams To learn and grow and adopt these new methodologies. So that's a big piece of advice as well So just to wrap things up because I do Think that it's invaluable to talk to the folks that are doing all the work Get some questions going Start together bring friends No one wants to go in alone. So you really want to start this together. There's comfort in numbers We focused a lot on people the technology change can be complex but Once you get everybody committed and on board and Comfortable that you're in it together and channel people's passions Then it's a lot easier to make all the technology changes that you need to make Start small. Don't try to solve world hunger. Just feed the family first Be strategic about the applications that you that you start to migrate first so that you can capture the winds and communicate the business value as you Establish an open forum and a feedback loop make sure that you're actually listening to customers So all of the things that we deliver is Based directly on the customer feedback. So they feel that right so you want to make sure that you definitely listen to your customers Having a solid foundation is fairly intuitive Automate everything so we're able to support this environment because of just completely ruthless levels of automation Automation is in our DNA There's not anything that we do that we don't look for ways to automate that so we don't have to do it again manually and That enables us to really spend all of the amount of time that we do Collaborating with the product teams and the development organizations and then lastly but most importantly Join the community. You're all here as part of the cloud Foundry Foundation community and Give back give back and sharing the lessons that you've learned We learn a ton from you all as well So if you've created capabilities in your environment, please contribute those back to the community We're seeing a lot of growth in Contributions for cloud Foundry, which is awesome to see love to see that you know happen even faster and Even just sharing the stories so we're we're here to share with you But we're also here to listen and learn from you all as well So, you know, it really do mean it when I say that, you know, we're anxious to get feedback. We're anxious to have conversations We're going to leave time for Q&A now and But you know, it's not a one and done So we're you know, we're here for the rest of the conference and really excited to talk to you all Thank you very much That there's a question Yeah, that's a that's a great question. Certainly appreciate the the challenge So the the question was was around how Awesome has our platform and the work that we've done to improve the customer experience and you know I think that I shared some of the metrics with you, right? So we know that when our products are working people love our products, okay? We don't want to be Netflix. We've we've communicated that you know We have a premium product and we you know, our customers expect a premium experience and Folks interact with our product more so than most any other product I can think of right? So just the always-on internet and Wi-Fi and all the other products and services that we have so we know that When we can create a an experience That is a self healing when problems can correct themselves because of our ability to scale up and scale down applications Responding to business demands in real time. That's a way that we've enabled our applications to support our customers So that 81% reduction the amount of time that we're impacting folks ability to call and get service or To activate new products that definitely helps improve the customer experience We've shown pretty significant gains and again. We have a lot of work to do for sure But in our net promoter scores in terms of customer satisfaction We've seen really big strides in improvement and I can't share necessarily a direct correlation between you know net promoter score the customer satisfaction and some of these metrics but we do know that when our products and services are more available then Customers don't have to call in and the products just work So creating that experience where things are more self-healing and more self-service I don't know Chris if you want to talk about some of the digital first Type initiatives you want to come on up? Yeah, so for that one in particular actually there's a metric that we've been tracking for a while that we referred to as customer dissatisfaction and a couple years ago. We started correlating that to a line graph also showing the Availability or what was at the time the lack there of of our systems and there was I forget what the correlation number was But it was so incredibly closely correlated that the business was really pressing on us around That's really the impetus to improve So not surprisingly after implementation of the platform and all the journey that Greg just described Our customer dissatisfaction line has gone gone down in correlation to the uptime of our our environments The next leg of the journey as Greg just alluded to is certainly around digital first That's where we want to provide Mechanism for our customers to interact with us first through their smartphone or xfinity.com or whatever the case may be Without having to pick up the phone and call a person or more likely pick up the phone and go through the IVR And wait on hold for a while and then get the person So not unlike some of the web-scale companies like Netflix and Amazon We really want that to be the first way that customers can engage with us That's a way to start the Q&A that's cool Hey, we are open right so No, it's a great question. Thank you Okay Based on traffic Yeah So so let me so let me just repeat the question and tell me if I got it right So the question was we talked about three we chose three services were our strategic selection process and was that based on traffic and The answer is is that you know, certainly there was a lot of traffic these were the services that were called most frequently and There was also another dimension which was in terms of the resiliency. They were also the critical pieces of the services platform that Were most likely to cause other, you know, kind of near neighbor type of issues as well So the the three services that we we chose was not just based on traffic But also the potential impact for improving our customer experience So and again, that was a testament to Chris and the development team in terms of Choosing and being really ambitious with respect to the services we chose That's a good question anything else great. That is an awesome question and I'm just going to be kind of shameless and I'm going to give an opportunity to some of the folks to come on up so Yes, alright, so the question was and I can flip back to it so the question was around our monitoring and telemetry strategy, there was a lot of pieces up there and We do get a lot of questions about this So the question was, you know, can we provide more granular detail into how we selected the pieces and what the benefits are and I Wanted to you know bring up the folks that actually were responsible for sure. Thanks so So my name is Tim and this is Sergei so sir Yeah, they should work like thing one two three four thing one two three four Yeah, so I don't know he's number one so the so the strategy kind of built organically and One of the things that we really wanted to depict with this picture is that what Greg alluded to was that We chose the best tool for the bet for the job and decomposed our monitoring strategy into things in flux DB is a really great Time series metrics database. It's very fast and reliable capacitor does a really good job at Massaging those numbers and being able to do some business logic on it Grafana, you know that is it is a great obviously is it is a great Trending tool to be able to visualize some of these metrics and You know, we we just we just over time came up with this strategy and I don't know if there's really any Overall thing that we can say to say this is the best it's just that you know Look at what works for you and just employ that for that specific job. Don't look for something that's gonna Do everything for you at once actually, you know, I forgot right because I'm getting old and my memory is failing But so Jim you want to talk about the Bosch config box? It's kind of like a little box down there and it's not open source yet But again in the spirit of giving back. Yeah, I mean that was one of the challenges We had was that you know the platform is always changing, right? So so when you're deploying new VMs when you're scaling you're always gonna have new things you need to monitor so we have a method by which we can take a Bosch config and Translate that into a Nagios config and we use Nagios because it's simple It's extensible and and everything else and it does a pretty good job So we were able to to build a script that will generate a Nagios config based off of Bosch Bosch output or the Bosch API? Yeah, so that means that we don't have to worry about forgetting oops You know this component chains or the IP change or whatever the case might be It allows us to again to just spend more time with our development friends than you know worrying about managing the platform But again, that's something that we're going to work on contributing back and again, you know, if he is glad you're here I Think that you know folks might benefit from from that Yeah It is it is both and so this this topic actually my Need a separate session. It's a pretty big but I'll address this questions so from Selection and strategy perspective what strategy do we have to select this these components? well, one of the strategy is We don't want to commit for 20 years from now We want to build system not based on a you know some one single enterprise solution that is not replaceable But instead pick up pieces that can be replaced when there will be a new product that will fit better so work on work on Standards industry standards or or you know the standards Don't try to do too much proprietary things Select components that are play well with other vendors other products and Make sure that you can you can easily pick up Remove one component and replace it with something new technologies that will be available next year So that is that is on a on a strategy question about monitoring platform or and Applications we do both and Greg mentioned Telegraph build pack. So this is one of the things that we started to use very actively to monitor the platform, but also a Lot once once we start to advertise and release it to show to our consumers in Comcast Many teams started to use it right away. So what is this build package is doing you just create a telegraph dot-on configuration and You do CF push this configuration. That's it. You don't need anything else and it will Automatically spin out a telegraph instance based on this configuration. You can bind in flux database To this this application and it will start to stream your your metrics to that application. So our consumers application teams devops teams are Very active and love this feature. So they're using this for application monitoring as well And you know, we're here all week as well. I want to make sure that we answer any other questions I think we might be at time kind of looking at Bridget to see but Yeah, it surges obviously, you know very knowledgeable and proud of the contributions we made certainly you can look at the source code Do we have any other we have any other questions that we can we can answer quickly or we? Okay, so I'm sorry we are at time But what we'll do is we'll come down and we'll come and talk to you and talk outside so that we can get The room give the room back All right, thanks a lot