 Okay, everyone thank you for attending. I think this will be a very lively discussion. We're gonna have it and we really do want your Participation that can be very very informal. I'm Lou Tucker Bp and CTO cloud computing at Cisco And if you saw the demo yesterday, and you saw a mark up there on stage You're probably wondering so where's open stack? So the purpose of this Session here is to show you exactly where open stack is and how it plays into this the mark Thanks, Lou. Hi everybody mark meal with Comcast You know that's exactly what we want actually We don't want our customers to know whether things are running on top of open stack or anything else because our customers shouldn't care too much Right, we wanted to just work so Fortunately, we've got a bunch of experts here today to tell you how it is working and a little bit under the covers At least what we're allowed to share We're probably not as tight-lipped as the NSA But you know, we do have some approval processes that we have to go through to get things released But we would like it to be interactive. We welcome your questions and you know, we'll answer any that we can So let me hand it over to Warren. We also have Andrew and Bill here Bill's one of our big users and he's actually one of the primary users for the service that you saw yesterday our X1 platform so Warren, why don't you take it away and we're gonna have a seat and feel free to interrupt us with questions We really do want it to be interactive and hit whatever you guys are interested in. Okay, right? So we'll tell you a little bit about our experience with open stack So we really had to end up selling the cloud internally and we started off a little bit over a year ago Looking at building an internal open-stack cloud At that point it was pretty amazing to think that we went from essentially nothing to Yesterday demoing a real-life product Being powered by open stack So we took the attitude of build it and they will come we had fairly limited offerings of essentially Nova and Swift And then we did end up going on a century road shows going around to all of our Different engineering and development teams and they give you a little bit more background on that As you could probably imagine Comcast is an extremely large company not all of our products and Applications come from the same groups. So you have Kind of siloed development teams So we had to really introduce Open stack to them and sell them on this idea and what's even funnier is that Before that I had to sell myself on the idea and because I can't sell I'm a horrible salesman So if I don't believe in it So if I if I wasn't gonna believe in it, there's no way that I could sell it to someone else So we wanted to we wanted people to start thinking what if right? So what if you could reduce your time to market? at Comcast we have Pretty long processes to try and Procure equipment get it out there and then get your connectivity set up Right, so we want people to concentrate on the applications not necessarily on the hardware which unfortunately Means that as the cloud administrators we have to take that on we don't necessarily get some of those advantages So We also wanted people to have fewer network connectivity issues One of the ways that we sold that was that we're gonna put this in part of our network that's a little more open than other parts of the network and ideally you would have connectivity to it without too many problems and Even better than that if you just put everything on open stack, then you really won't have any connectivity issues We also wanted to give people distributed compute and storage and make that easily available the key difference here between What we were building an offering is that it's it was largely a self-service model So we do have other offerings within the company, but it's not truly self-service and I enjoy When someone comes to me and asked me for VMs, I tell them no, I'm not gonna give you VMs But I'm gonna give you the ability to create your own and So I think the best way to sort of demonstrate this is really just you have to show It off. It wasn't until I Got access To open stack itself and started spinning up VMs of my own and then it really really It clicked in my mind and so here's the the scary thing right we're gonna run a live demo It worked last night So let me describe what's going on actually. Let me make this a little bit smaller. Whoa a lot smaller But I think this little device So we developed one of my teammates Prashant Hari developed a Python script Which essentially would spin up instances assign them floating IPs if you wanted and Add them to HA proxy if you wanted as well. This was prior to the load balancer as a service coming about in Grizzly and So we we decided to undertake this on our own However in the future, we will end up going with Grizzly and and using the load balancer as a service So what we're doing here is we're spinning up 20 instances on open stack They're gonna get automatically added to an HA proxy. We'll come back to this in a couple minutes. That was weird Okay, so We had to you know, like I said, we went on road shows and we were talking about Open stack to the different architecture and engineering teams and development teams So what we were looking for at first is really our best friends, right? So these are the these are the people that really understand how to design applications which scale horizontally and are fault tolerant And they understood the concepts of ephemeral storage and how to use object storage because initially we weren't offering block storage They understand how to use orchestration and automation to get their instances rolled out and They just in general were designed to work in a cloud environment There's another tier of applications that we wanted to really bring into the fold But we may or may not have been necessarily ready to do it right then and there So some of these are like databases and other applications which require persistent block storage There are some applications that require extremely heavy resources and it may not make sense necessarily to incur the overhead of virtualization so Mass is something that we've been very interested in and then finally You know, it's not it's not necessarily to us ideal to do image-based deployments We really wanted to push people to do the automate automation and orchestration using things like puppet and chef and Then there's the third tier of folks and they're a little bit scary for our environment Right, so these are things like monolithic databases applications which have single points of failure and OSes which don't necessarily work all that well under KVM or Play well with the vert IO and Then finally at Comcast we have quite a few applications which Their deployment model is a 100 you know like something like a hundred page doc and someone sitting behind a keyboard if they lost VMs we'd probably have to run for cover. So there we go. We have 20 Nodes that have been deployed and added to HAProxy. I apologize there are our internal site for this is a little bit slow But these are a real live graph graphs So of course the the demo part work, but then I can't prove to you that it worked because the graphs aren't working, right? Okay, here we go so All right, so what we're looking at here is the sessions that are connected to this sort of test environment here here And you can see that we're there's a very consistent load because There's a team in the X one side that's generating some load test and you can see the This is a looks a quite a bit different on my screen than up there You can see that over the course of a week we've gone up and down and this is actually quite amusing to me We were told that we basically maxed out their load test before Being able to max out our servers And then you hear the servers that I ended up spinning up just a few minutes ago We went from here to here. That's a difference of 20. There are no inactive servers So all of them worked And normally you would be able to see the the servers over the course of a week. Unfortunately that graphs a little bit slow here So there there is There are all the servers that we've been spinning up and down over the courses this week And you can see last night here. I really wanted to make sure that it would work So I scaled it up to a hundred and then back down and everything worked fine All right, so I'm gonna let my manager Andrew Mitri take over here and describe a little bit more about what we're doing Thanks for a great demo So I thought we'd share a little bit on what our vision is around open stack at Comcast We're really hoping to build a unified environment where we can operate most of our critical and innovative initiatives So something that as we start to build newer products They can access the environment and scale out quickly to be able to serve our large customer base We want to provide that technical agility efficiency and velocity to our business and reinforce and capitalize on scaled design principles So kind of the question that I've been getting for the past couple days is What kind what kind of deployments do you guys actually have out there today? And while we can't say specifically I'll give you what date details we can we started out in our national data centers We have a few thousand cores tens of terabyte of RAM and hundreds of terabytes of replicated storage Primarily object right now and we're rolling out some block We have development lab and production environments stood up today But what we're doing a little bit different than maybe some of the deployments out there is we're really looking to go towards distributed data centers We have a desire to scale to hundreds of cells as opens that calls it now as we move deeper in the network and closer to the customer We want to distribute our compute storage and network fabric to support that elastic scale So that the application can leverage saying hey I need to be in these hundred different cells to be close to the customer I need to be in this region because there's a load here and manage that elasticity And some of the ways we're addressing that with our applications is we're really targeting green-field applications clean slate free of legacy requirements And we're looking to open stack As the virtual agent and scheduling layer to stitch this all together Why do we choose open stack? We feel that our problems aren't the hardest problems on the planet And so we want to be part of that community solving all these types of problems together We can learn a great deal from engaging in a broader tech broader technical community And we have a little bit to share to from the size and scale of what we do We feel that the community releases new features quickly We've been running on Essex now for about seven eight months six months six months I guess and yeah, too long and it's running well and stable for us knock on wood and And it receptive to broad and deep participation even from the heavyweights And we want to avoid the vendor lock and that's really important for us And What we're kind of looking for is we're moving forward is that abstraction composability and orchestration as key We want to make the underlying hardware more fungible so like we can swap out any piece and it's not going to affect our end user We're looking for intelligent resource scheduling Understood by the app but managed by open stack so the app can say hey I need this type of resource in this type of place or in this type of environment and open stack and handle delivering that We want to have an effective separation of concerns You know the data replication in the data layer So like if we want to be able to land data in certain places and replicate here or not replicate what not and we want to Encourage among our developers improve software design principles, you know, maybe things like chaos monkey or whatnot, right? So And then we want to separate the application development from the underlying infrastructure services So today we've been focusing mainly on infrastructure as a service, but we want to go both above and below So we have a lot of interest in metal as a service as well as a platform as a service We're seeking harmony among our apps quality infrastructure and network So we want to balance that cost reliability scalability and security And we want to simplify and better utilize the network storage and compute the app knows best, right? So we're trusting the app to manage that, you know And we want to move security and connectivity policy deeper into the application groups and control So I think that's it and maybe we can go into a Andrew. Can I ask you a question? Yes No, it was probably no more than about six weeks ago Maybe eight at most and we're having conversation. Well, what could we actually show at the summit here today? And I heard things about well speed test, you know, we could do that. I'm going well, you know For this audience, I'm not quite sure and I heard Mark then pipe up Well, we can just show it, you know, XFINITY X1 wouldn't that be impressive I want to know how nervous you were when you heard Mark say that And congratulate you and actually deliver it. So actually I want to give major props to Bill and helping us out with this So at six weeks ago, we were not running X1 in production We had billed and started testing some stuff exploratory phase But they stepped up and we worked together as teams in cross-functional across our organization and And actually made it happen so that we could show this in production bill I don't know if you had any feedback when you got that call from me. Is this Yeah, so the team that I work on Which is primarily X1 We have a lot of people that have to deal with standing up our own Infrastructure and networking gear and one of the things that we would like to do is obviously move faster And so what I I first heard that you know, we were we're being asked to do this I certainly was nervous at first and then I'd read up a little bit about OpenStack, but never tried it So I started talking with the gang here and got access and I saw that they were like here We'll give you a quota and here are some base images. Good luck kind of a thing I mean and willing to help but but and It took me about an hour and then I was like, you know able to using the API's like stand up and tear down Nodes and it was awesome. There was like no no like big process in the way No pile of paperwork no Giro tickets. It was we you have a quota to work from like and You know here you go and and the experience from my side has been awesome So there are you know some things that I'm looking forward to in OpenStack such as like load balancers of service and and DNS Integration and things like that but as far as my first interactions with it as a as an OpenStack user Not as a cloud administrator have been fantastic So unfortunately, that's one of the things that we don't talk about We don't talk about vendors specifically. We do use some of the technology from Cisco That's one that we've gotten cleared No, we want this to be ubiquitous we want this to be Vendor agnostic So we did we did get some professional services from one of the professional services group That's running around here at the conference just to help us get familiar faster and sort of give us more leg on the ground But we will you know one of our design principles is we do not want to do something that locks us in with a particular Partner and we want to stay on the core stock OpenStack. Yeah, we want to be on the trunk more questions We got to kill the audio messed me up yesterday, too Yeah, my non-answer. Yeah, I don't think so. No, no No, I think we want to we want to develop expertise in this area So we will work with folks and we will pay folks when necessary and collaborate with folks and maybe not pay them When necessary so that we can learn and mutually benefit and move Together, you know, I think they're the point to that's to that sort of second category is There are things that we can learn from someone and they can learn from us Including other users of OpenStack. This isn't always a vendor that we're talking about right so we're interested in collaborating with folks You heard the Bloomberg folks in the Presentation yesterday talk about 200 locations around the country and as I was in the back Following our discussion, you know, I was asking them. How are you guys thinking about that? What are you gonna do and I think there's you know, that's where the community is now we're at the point where real-world users are starting to solve their actual problems and Coming into the conference. I would have been I was gonna be surprised to learn that someone Wanted to take OpenStack and run lots of smaller installations But there it is Bloomberg wants to do it and you know, I we exchange business cards and I want to develop that community around Users that have similar problems to solve I could also add I know that when I Informed my management chain that we were doing this with Comcast the response I got back was Lou you do know like Comcast is one of our largest customers or whatever. Don't blank it up And my response was no, we're actually doing this together It's much more like co-development because we do have the OpenStack community We we have people here now who are experienced and trained in OpenStack and providing input and making the changes there So that's so that I think is a part of this new model about how vendors want to work with their customers Because we can come together and provide simultaneously provide that kind of vendor neutral vendor independence that a customer is always wanting But now being able to get the kind of support through us and through others that are that are vendors in the OpenStack community as well Just one more reality check and then I'll take that question in the back You know, we have 55 million devices in people's homes that are tied to video delivery So that's no small, you know legacy system that we have to move over and You know, I can't even remember if I said it in the talk yesterday I'm pretty sure I did but I kept it anonymous, you know those those the software development platform the systems engineering You know ecosystem around those 55 million devices is the old-style stuff that is difficult for us to manage and we need to Bring as much of that to the future as possible Just for the health of our business while also building a new platform that we call x1 That we're rolling out like gangbusters and we're rolling out to more markets, you know throughout this year So we you know, we're in we're a mature business. We're not a brand new business that's trying to figure out You know, we're gonna put out our first product and we've got this totally greenfield wonderful world that I sometimes look at longingly We've got to figure out how to bring our business along And so partnering with a company like Cisco Which happens to be one of the core suppliers of that legacy infrastructure has advantages to us, you know We we like that close working relationship because if we're going to be able to pull that those 55 million devices forward It's going to be with partners like Cisco and others in the back Yep, so the question is were there third-party softwares that we use to I think a century make life easier So one of the some of the key ones that we absolutely depend on are like cobbler and puppet So we use that to actually deploy open-stack We didn't we decided very early on that we didn't want to be in the business of trying to Install open-stack manually I'm sure many of you here have tried that on your own. It can be quite cumbersome So we were For monitoring we actually ended up just staying our typical route and we stuck with Nagios and NRP But we are looking forward to gathering additional metrics through like Solometer And let me give you a slightly different way to answer that question We try to get into the Intel presentation that was that was happening right before lunch, but we were kind of Turned away at the door because it was so full so we went into the overflow space and When the slide came up where Intel was talking about there, you know, this function was sort of down the left axis you know Compute storage monitoring deployment blah blah blah all those different functions that you need We actually whipped out our iPhone to take a picture because we want to you know We don't just like we said in our slide We don't have the answer to everything and we are still in the phase of looking to see what's out there And you know, we wouldn't be afraid for example to switch from A to B If we needed to because we thought that was a better answer more questions, yeah, so The question was when we went on the road shows what were some of the items that we got pushed back on So I'm gonna take a stab and say that We took a stance very early on and said that our VMs we were going to support them support them in a Stateless model right so you get them and if they get terminated you lost you know, whatever was there And so many people didn't understand that you know And they didn't understand how to to build a product on a platform that behaved like that You know, and that's not to say that You know open sack is immature. It's just that we Choose to build resiliency into the application as opposed to pushing it down into the infrastructure layer So that was a major sticking point right and so we still have another sticking point of block storage We do need to offer that and we're investigating our various options there I think those are the two main ones that we get correct Well, like I said, you know, there was some pushback on the the stateless model I mean, it's not like we go and at least not right now We don't go and intentionally tear down your VMs for no good reason, right? But we couldn't guarantee that they were gonna stay up through Maintenance what one cost issue that pops up Well has popped up. I can't say it's really happened more than once One of our goals is to do the DevOps thing where we're developing the product every product sort of in one infrastructure and it's going through its maturity cycle in that infrastructure instead of sort of having a lab deployment versus a production deployment and The production data center is more valuable, you know per watt or square foot or compute cycle or whatever metric you have It's more valuable and so we've had a little bit of noise in the system about Burning that precious resource on Development process that said I think you know We're working on the business case to show that we can actually build things faster and get it out more easily and it's you know less Carbon intensive as somebody else put on a slide that I loved to see today It's less human intensive to do all that work So in the long run I think it's much more cost-effective to have a unified infrastructure instead of a lab and production But that's an example of another of another issue that we wrestle with So from from my perspective as a user what we looked at what was being offered on open stack Which pieces we had that would have fit in no problem and started with those As far as other things like you know, we're interested we run a Fairly decent Cassandra workload and that right now that's it's not a good fit for what we have available But we're we're collaborating on you know What would make sense for us to be able to deploy Cassandra effectively on open stack and we're working toward that both on like a hardware and software perspective So let me I'll give you a simple answer We are users of VMware and we have a good Very substantial chunk of VMware that's sort of managed by a different part of the organization And I think there's space for both an open stack and a VMware world But most of what we do in our product engineering as opposed to things that are IT CIO back office kind of functions We look at those as scale out and we look at open stack as a much better Fit for scale out systems There was a question over here. Yeah, sure So I run a group called product engineering and so we we do engineering for video voice data Wi-Fi home security business services residential stuff That's the that's the engineering that we do in my team as part of that team. We have like a high-end operations group Not the folks that that you know get the first tier one alarm and react to it But it's like the people that really you know, they're every bit an engineer as the people figuring out how to build the products And in that group, you know, we had all of my other teams kind of draining into that group They're different architectures and they're different systems and their different way of building things and you know specifying their own Hardware needs and so on and we said well, that's enough of that and we needed to create some efficiency in that part of the process and we started looking around and We thought that there was enough of an opportunity to improve the flow of work from the development organizations into the Operations engineering organization and achieve efficiency within the operations engineering organization that we could provide these things as a service So we reallocated headcount, you know, we didn't go and ask for new folks But we reallocated headcount because we thought it was a good strategic bet And we've got a team of five to ten people that are pretty close to dedicated to this and you know We're still I guess the big joke at San Diego is we're hiring so we're hiring You know, we're still looking for more resources to help fill out this team Yeah, so That was one of the strategies that we So I'm The question is essentially Designing our applications to be resilient probably takes a different mindset and Let's see if I get this right What challenges do we have in sort of driving that to the engineering folks that correct? We like directness So I'll take a crack at this and I'm sure Mark will have his feedback on this as well I think as a system administrator for some large-scale Projects in the past. I've seen how it's done right when you scale out And I've seen how it's done when you scale up and quite frankly, I believe in the scale out model I think it works better. I think you can Design your resiliency at a higher layer, right and and stop trying to count on Infrastructure and other teams to build in resiliency for you and turning to maybe vendors to build that resiliency for you Yeah, I mean we have a little bit of an advantage because I own both organizations and so In my staff meeting every once in a while I soft I softly threaten the development leaders in my organization that I'm taking their capital dollars away And I'm giving all of their deployment capital dollars to the cloud team And so if they want their application to actually launch they they need to figure out how to get it to run on the cloud Now that's a bit tongue-in-cheek, but at some level it's it's real You know I am Investing disproportionately in the cloud infrastructure as opposed to the you guys can specify it and build it and buy it yourself In terms of getting people over the hump where they're actually sort of naturally thinking this way It is you know, it's a different design methodology, and I think the best way for Developers to learn is by talking to each other often And this is an example where when one development community talks to another development community talks to another one Goes to a conference learns about this and that the the comfort level and the skills and capabilities rise So it is a lot about going to the most aggressive, you know willing to take risk piece of the organization and Saying hey guys, would you be willing to help us pioneer this way of doing work in our development organizations at Comcast and help Us prove out that this is the right model and you know oftentimes they're willing to do it They put you know discretionary effort in to do that doing that And then we have a successful whatever successful launch or a successful story to tell and we start to pass that story around And then more people come but to your point. It is a difficult and Sort of constant process of getting people to to move to a different I think because this question actually comes up a lot, and I think it's almost we have this mythology around the scale-up model as being Hardened about mission critical and and therefore can't fail. Well, they do fail I mean failure is a part of this and I think more than anything else when we're thinking about putting resiliency in the application we're recognizing that failure does happen and That really the application is probably a better place to actually handle that and move to like you said stateless systems The other thing that we're using to our advantage here is that we're saying some of the hard stuff about building Resilient scalable things such as storage systems and where you do have to do is we're we're putting that into services So that's where you see Swift you see other other kind of technologies being developed Cassandra and others Which means the expertise in building these scalable resilient systems can be encapsulated and then provided to the application Developer so he doesn't have to worry about that So there is there is a mindset change about building Scalable horizontally scalable application, but also then as much as possible encapsulating the hard parts into these services And so that's why we're looking for it. So where is the next, you know Be in terms of like key value stores and no sequel databases so that the hard part can go and we can have the expertise and the Accumulated experience that all the applications can then benefit from and I think one of the things we're doing too is when we talked to a lot of these Development organizations were having them start off with a hybrid model So they don't necessarily have to move an entire application stack over to the cloud But maybe their web tier or app tier is a good fit and until they catch up and get their database problems Re-architected and whatnot to work in cloud. Hey start here get exposed to it and get them thinking in that club mentality And we try to coach them through that And within my organization the vast majority of our apps now are Scale out and when ones have single points of failure That's like a big red flag and we're really encouraging saying, you know When are you ready for you know our equivalent of chaos monkey to be Applied and so people are really aware when they're developing if they if they haven't solve that like, you know Single special snowflake problem yet that that really needs to be high on their priority list One more comment just because I've heard it actually in a couple of other sessions as well There's another real benefit which you can directly attribute to us a cost savings Which is that when you have to update things when you have to replace things if you've already built resiliency into your application You can take those systems down and update them and replace them And you can do that multiple times a day ten times a day hundreds of times a day You can update the applications because they are designed to to continue to run even though components of those things are being taken Offline that's a tremendous cost saving when you when you know that both in the time and the management and process and everything else That you have to go through to update things you can you can directly You know get enormous cost savings if you just build this kind of continuous deployment into your model right from the start We did that on all new hardware. We started off small and then as demand has grown we've added hardware We still have a lot of Older applications or legacy applications that we point to existing infrastructure, so we've been building it on new hardware Yes, so we definitely are interested in the elasticity of applications I think we have a lot of various Events and things that we support and that's actually one of the reasons we're not just interested in How opensack works today, but we have a lot of interest in metal as a service because some of those applications We want to scale actually do require bare metal and we might want to switch like one day be doing a Hadoop type thing in another day Be doing a different application. So here's a here's a real-world example. We didn't do this yet But this is the kind of thing that we think about for that question every single keystroke that we type on on this Service goes through our it's called the XRE the cross-platform rendering engine Which is one of the things that's running on top of open stack and so if you imagine Super Bowl Sunday When halftime starts how many keystrokes happen on however many Remote controls are out there in the world and come cast footprint. It's a large number I don't know what it is but we all can perhaps say it's a large number and it is probably Disproportionately timed at that moment to be a flash crowd or in that 10 second window or 30 second window We're gonna get a lot of traffic so we We actually plan and did this year because this has been rolling out for a little while now We planned this year to take special action to make sure that we were ready to handle Unique events like Super Bowl Sunday because it's one of the most watched television events a couple of weeks ago We did an event called watchathon week where we made things like HBO And other premium channels available to our customers for free for a period of time And it was tremendously successful and now we have more data than we know what to do with about what customers watched And how they navigate and how they found the content and all this other good stuff And so during the period where we're delivering the content on for watchathon week Wouldn't it be nice as I said we did not do this yet, but wouldn't it be nice if we could Reallocate you know reshuffle the cards so to speak and get a little more delivery capacity for that service and then after watchathon week reshuffle the cards bring down that delivery capacity and bring up the Capacity to analyze all that interesting data And then maybe two years from now or two more watchathon weeks from now wouldn't it be neat if we had the ability to actually do that analysis in real time and To fine-tune watchathon week for each individual customer so that when we figure out what you're interested in and watchathon week if you Want to try out games of thrones because you're not an HBO subscriber and you want to try it out now That we make it we figure that out and we do special stuff for you because you're trying to sample game of thrones Yeah, that's actually another good good challenge and good good barrier So the question was do we have organizational support for the capital expenditure in advance? The to answer your question for the company overall I think the answer is we're gonna do what's right for our business When you get down to the detail and figure out if I could actually write a purchase order and everybody who has to You know sign that purchase order is happy about doing it The answer is a little less rosy So We have not historically had the philosophy that it's okay to pre-buy data center space. It's okay to pre-buy compute and storage stuff Because this is a relatively You know up-and-coming approach and the company is so focused on innovation and so focused on You know analysis of information to bring better value to customers That attitude I think is changing because what we've done in the technology organization is connect the Value to the business to the capital expenditure So we say that if you guys want us to innovate more quickly and if you want us to build this stuff faster And if you want us to operate our platform with fewer people All of those are good economic things because you know some of them drive revenue and some of them lower cost Then we need flexibility in other ways to be able to do that And one thing that we need is advanced data center space advanced networking capacity and advance Storage and compute capacity So the attitude is changing, but you know if you walked around the hallway, you know I don't think you'd get universally supportive responses from from our from our team So we really want to make sure that when we started this that we offered Swift as a service the Usage of that isn't quite as high as we'd like right now There's there's been a slow adoption to Swift at the moment, but we're still pushing people to to really use object storage as opposed to Stop writing things on the file system when they really don't need to be there And then the second question was about the CDN and just to clarify the question Did you want to know you asked if we used to the CDN to deliver what? Okay to deliver the television Today we Materially speaking do not use our CDN to deliver live broadcast television We do use several flavors of CDN to deliver on-demand experiences So we actually have a thing that we call a CDN that most of you know Our internet colleagues would not necessarily recognize as a akamai-like CDN To deliver on-demand to a set-top box we use something that looks a lot more like a CDN to deliver things To deliver IP experiences, so whether it's to the Xbox so those of you who aren't Comcast customers You may not know Maybe some of you who are may not know that you can get our on-demand experience on your Xbox And so we deliver our on-demand library to your Xbox It's the same service that you see here on the set-top box But we use a much more internet-like CDN it leverages IP technology To deliver that experience to the set-top box or to the Xbox For delivering this no This is a this is a title six service. It's called. It's a regulated service and there are Requirements that we have to meet for example. We have to provide emergency alert service over over this product And we also need to deliver the service from our owned and operated infrastructure So there is no option for this particular service for us to look at at an outside party more questions When are we when do we wrap? There's about another because another couple of minutes to probably one more question or must do yes So, I mean, I think we're still in the early stages there We try to anticipate our customers needs via a lot of conversations and Surveying as well as looking at our usage and metrics. We are pulling reports out of we're running assets right now So what we've built kind of our own reports coming out of that But we're very interested in rolling out Solometer and getting more data to predict where we're going well right now We're also still kind of hedging our bets based on where we think we're gonna need things today. I Think we're probably out of time and I want to make sure you've got opportunity to get to the next session So thank you all very much for attending. Thanks a lot