 So, thank you so much to use the New York term schlepping from all the way to the other side. I had problems finding this place myself, which probably means I have less people, that's good. Okay. So, my name is Jacob Caspi, I'm a principal technical architect at AT&T responsible for our domain 2.0 and specifically our cloud architecture, which we have spoken about at length and at a keynote address in day one. So, figured we'd take a little time to expand on what Sorab said on day one and if we can just look at, so, what makes us qualifiable to actually speak here about this stuff? So, you know, some statistics, we are moving as a company to a virtual centric or software centric network, we virtualize our global network with target of 30% to be virtualized by 2016, this year and a goal of 75% of our network virtualized by 2020 and he says, so to give you a deer of what that scale means, right now we are moving about 114 petabytes of data per day. So, virtually that by 2016, if you do the math, that's about 30 to 40 petabytes per day moving through a cloud, which is a fairly impressive standard, at least if we are, if we are to impress ourselves, the name for AT&T coined for our cloud is called AT&T integrated cloud, we had several iterations early on, which I can go into detail before that, after this. 74% deployments to date, meaning there are 74 data centers or central office or other facilities where we have our cloud deployed today. On the IT workloads, we are planning to move at least 60% of our IT, strategic application to a cloud deployment and we are very serious about it, we are moving one application per day and we have thousands. One of the things that motivate this of course is our operational servings, we found that by moving to a cloud when we can utilize how much higher utilization of resources, we have reached as much as 50% higher efficiency than our current dedicated boxes. Of course, white box strategy, we are a contributing member of the OCP, open compute projects, if anybody does not know what that is, we should definitely look that up, open compute.org, and we are fully committed to open source. We have doubled a number of open source applications running AT&T and for anybody from a telco, we should know that that's quite an achievement. So specifically to OpenStack, as I mentioned, we have deployed over 70 sites with OpenStack clouds, 57 of those are actually in production. We are connecting millions, we are not a China telecom by any means, but we have quite a few millions of wireless customers that we are virtualizing. We are training our staff with OpenStack. We are an early adopter. We actually started deploying OpenStack in the Diablo version in 2011 and I should know because I was part of the team that actually did that. And a bit of an antidote is that when we deployed this, we decided to and got the support of management to do it in a fairly harsh way, meaning we did not publicize the AT&T that we were doing this, we were given a separate budget to go and do that. And basically AT&T incubated this idea to see where it will grow. And I remember there are people who actually knew about the project and were talking to me and telling me something like, you know, this stuff with open source and OpenStack, that will never work. And of course, here we are, five years later, we have a strategy based on that. Don't ever discount small efforts, for sure. 15,000 VMs, thousands of nodes, we have found that deploying resources on our clouds is definitely a lot faster than deploying it on bare metal servers from a resource allocation perspective. And that, the most important thing is that the unified set of APIs that make it easy to talk to the cloud and applications on the cloud with regards to how to communicate and make efficiency of efficient use of the resources. So what makes Carrier different than a specific IT workload? And that's really where it gets complicated. Our needs beyond just virtualizing and cloudifying our IT workloads, which most companies should be doing, is that we are actually virtualizing a network, which is a totally different set of problems and concerns in architecture. To give a few examples of that is if you're doing just IT workloads, you can have three or four data centers, so you have diverse geographical locations, and you can put many servers in there, and that's it. We have thousands of locations. We have central offices. We have head offices. We have video centers. Each one of them have different workloads. Each one of them has different needs. Each one of them have different requirements for various sizes. We come from a, you can have thousands of servers in one location and maybe half a dozen in another location. So that diversity leads to architecture. How do you make it all work so it all looks the same and you don't have to have customized deployment location? Again, the workloads varies from one to the other. Video workloads, network functions, there's been a lot of talk about NFVs, those get complicated. A bandwidth, I talked about high bandwidth demand, and our bandwidth demand are not just east-west between applications in the data center. They are north-south. We set a huge amount of data between data centers or between our data center to the cloud or data center to other functions, such as mobility centers and video centers, et cetera. And on SDN level, layer two, layer three, it's pretty much been accomplished by this point by several providers. But we actually need to move to the next level. We need to do re-switching, re-router, and higher pocket processing throughout the whole fabric. Talked about the different size of infrastructure in data centers. There's also not just that the size is different, but actually that even things like the power consumption is different. In our data center, we have DC voltage, 48 volts DC, which I don't know if anybody saw in the last OCP summit, Google actually announced that they are moving into the OCP world with a 48 volt DC power distribution, which people at AT&T were saying, well, we've been doing this for 30 years, 40 years, I don't know. Regulatory, we need some of those locations required in NEBS 3, which is a higher level standard. We have some locations which are staffed not full time, so if something goes wrong, they are dark. We have to come back or dispatch a technician. And the big thing that we see is that in a large data center, if you have thousands of nodes and properly distribute application workload, if a node fails, it's not a big deal. Application recovers, builds somewhere else, and you're back to your own capacity. When you have a deployment with half a dozen to a dozen servers, where each one of them serves critical functions, that becomes a lot more tricky. You can't just exhaust the server and then come back a month later and replace the dead server, you really have to tend to it a lot faster. Some of the things that we have done to remedy this, we have looked at multi-vendor standardization that whatever vendor we buy from, it's actually the same specs, the same server. So if we change vendor and we do rely on a multi-vendor surgery, so the server could come from any of the leading providers or white box. Our automation can recognize what the server is and do and provision to that regardless of what it is. We have started looking at the OCP and OCP inspired hardware to see what we can do to push the community to adopt a carrier, more stringent carrier specs beyond what Facebook and Microsoft have been doing. And the network also, of course, does change a bit as well because we are doing workload not just moving data from one point to the other, but also scaling out to the web and scaling up to multiple applications. We're talking about multiples of petabytes or gigabits and even terabits of information going not just through fabric internally from one server to the other, but actually going out to the white area network. To run the network functions, we have to do a fairly low over subscription. So our close fabric needs to be much, much wider than normal. Moving white area functionality such as MPLS or others directly to the servers requires more stringent functions on the switch such as deeper buffers and more memory for tables as an example. And again, the way we've figured out around this is by treating all the sites the same. So if we have some sites that don't require all that functionality, we still deploy the same type of hardware on all of the sites. So in theory, all sites can handle all functionality and all sites look the same. Again, going back to a little bit of history. So close fabric, which is the standard deployment of and most cloud applications was actually invented at AT&T in the 1940s. So it's sort of interesting to see how this topology that is now, so ubiquitous across all the web scale is actually something that our folks thought about quite a while ago. And they didn't even know what Ethernet was at the time. Trying to self simplify the hardware layer of the infrastructure with separating from the overlay from the underlay. And we've had several announcements in the press about that. For example, a few months ago, we have announced that we are using OpenContrail to as an SDN controller for our network, which will allow us to do more complicated functions that currently are not available in Neutron. And the most important thing that we have learned through this is that most clouds are local and they get attached to the wide-area network. Our clouds are virtualizing the wide-area network, which is a totally different thinking about what the cloud does. It's not just to process data or process, do MapReduce or calculate the payroll. It's actually virtualizing how packets move throughout the wide-area network. We talked about diversity of the network. We talked about the bandwidth. And the other important thing that we have to consider at our scale or at that from carrier is that we're not just connecting to the internet or maybe couple data center to each other. We actually are making point-to-point connections from one functionality of our infrastructure to another. So for example, from video to data center from, meaning from IT, from mobility to our edge points, etc. The other big difference that we see is a customer, customers could be anything from either IT workloads or even some SL users that needs to upgrade the operating system on their machine. Overlay perspective, moving from basic layer two, layer three with the GRE or VXLand to MPLS over GRE. The transport layer, IP version six is critical to us. We can't do anything with IP version, just IP version four. And looking at Google statistics, I think the last time I saw is maybe about somewhere between 10 to 15% of Google traffic is IPv6. 100% of our traffic must be IPv6. So we can't make any compromises on that. Network services, I mentioned before, is beyond just the bandwidth. We have to provide to a common network function visualization which actually provide services across the wide area network. So an example, a network on demand product, you can actually service change a load balancer and a firewall right through a portal running on this cloud. So you not only determine how much bandwidth do you need is you can actually say, okay, in addition to that bandwidth, I wanted to add a firewall to this. And our cloud actually has been configured to make that happen. Again, routing and switching can't just do standard vSwitch. It doesn't have enough capacity that we actually have to do. Integrated the routing and the bridging right in the router. Locally distributed, one of the ways to make OpenStack thin is actually distribute as much of the control plane as you can to centralize locations and then get them to be thin in the areas where you have limited capacity for hardware. And then of course you also have to remember on the big data centers where you have thousands of VMs running. You still have to have enough scale at capacity so you can get to the number of servers that you need at that layer. None of this will be possible without automation. Again, the 50% efficiency would not have been able to be achieved if we didn't automate anything or everything. If you've seen Serb's presentation on yesterday, he introduced example of a tool that we have developed that allow us to say, here is X number of tracks, here's the capacity we need, go build it. And that tool actually creates rack elevation, assign IP addresses, creates the hooks to a automation school, create a tool such as Fuel and allow Fuel to then automatically provisions all the servers from bare metal to an OpenStack node. And that has been a tremendous savings from a labor perspective. If you wanna do 74 sites like as we did last year and plan to do it across hundreds, if not even thousands. As mentioned, we've been using OpenStack and Fuel and have been contributing to it in the last year to make this automation go. And of course, transforming the organization from a standard orderful environment to a true dev app model that allows the CICDs and allow us to actually upgrade on a regular basis. So we don't have to be stuck with an earlier versions and smooth move from one version to the other. So, we started a little late and I wanted to leave some time for questions. So just to summarize, again, diverse workloads, web, big data, analytics, network functions, voice video. They all are different and they all have to be treated different, but they all need to run on the same cloud. And that's the only way that we found that we can actually make it scale enough to be efficient. Multivander standards, all servers have to be the same regardless of where you get them for. We see OCP as an important function. It has been solely focused on IT workloads or Facebook or Microsoft, but we are pushing very hard to make it a community that will do a much more focused effort towards carrier needs, which I mentioned as before, are more stringent. On the network on delay, we made a tremendous effort to move to 100% merchant silicon switches, so we can basically change vendors in the same way that we are changing on the servers. As long as the switch functionality is the same, it doesn't really matter where we get it from. Now, overlay, basically, again, virtualizing the wide error network, so getting wide error network functionality directed to the server is critical. And orchestration scale from centralized to controlling small locations to scale out deployments in the large data centers. And finally, three of the most important things about automation is just do it, automate, automate, automate. Okay, with that, I'll take some questions. No, so the scale, I wasn't saying that they are virtualized. What I was saying is that that's the subscriber level that we have. So the services to those subscribers will be a part of our virtualized network, yes. I'm sorry. Okay, actually, can you guys use the microphone since we are recording this? And there's a queue. No, the question was, what network elements of the wireless network are virtualized? So at this point, it's the control element. I really can't go into detail what specific network function does or what part of it is virtualized or not. Okay, but the plan is to virtualize all those network elements, right, like the MME, probably the control. 75% is our target, yes. Right, okay, thanks. Yes, sir. Can you disclose which network OS you're using on the merchant silicon switches? Sorry, no. If I ask you privately, can you? No. Okay, it will leak out. Yes, it will. I mean, our comparatory answer to Verizon just released what they're doing. We are not at the stage where we are. We are liberty to do that. Similar question to that. Can you disclose if you're using a particular open stack distribution or using your own? So we are using the standard distribution right out of trunk. Okay, good to know. Thank you. Could you go into any more detail on the virtualizing the WAN in the cloud, the transition between the neutron powered, more local networks and how that transitions. Do you encapsulate before you send it over IPv6 or just talking about that a little more detail? So we have several sessions that will go into detail on that. And I sort of rather not take the time to get into details in this discussion. But if you do, we have several discussions throughout the day, throughout the summit that actually will go into detail with that. Sorry. On the previous slide, you had a contributing to fuel. Can you say where that fits into the strategy or what it's used for? So you're using a standard open stack distribution. So fuel is an open source tool. You can use it with Mirantis operating with the Mirantis open stack. You can use it with standard distribution. It's not specifically to Moz, although we are working with Mirantis. Yes, sir. For your larger data centers, what has been the impact on your operational staff, both in terms of numbers and the level of training required? I don't have numbers in terms of staff. But I do know that anybody in them that is part of the AT&T domain 2.0 is required to take training courses on open stack and cloud. We have courses that we offered from AT&T University. And we have courses that we have partnered with other leading universities to provide, as well as running our own courses. So this has been a New York Times article about that. AT&T has made an immense effort into retraining our staff with the next generation architecture and moving people from making sure that our workforce is ready for the next generation of technology. Yes? OK. I think you have many, many open stack clouds. Most probably those are separated. But there are services running on that cloud. The service needs to run across the clouds. So I guess you have some orchestration or management software on top of that. So could you explain a little bit about that? So we have developed what we call the service orchestration layer, which is I only, I know the acronyms, and some of the audience may help me with what that is. It's called ECOMP. It's Enhanced Control and Policy, right. So, and actually I believe there's also a session about that, that we have developed internally and actually considering open sourcing that, that orchestrate across the data centers and orchestrate the various functions running on top of open stack. So if you think as open stack for us is the infrastructure of the service orchestration, ECOMP is the service orchestration layer on top of that, that does management terms of location distribution and services, et cetera. Thank you. I have probably what's considered a non-sexy question, right. So it's more about IPAM, IP address management. So from your slides, I understand at least for the under cloud or the open stack components, you know, you could be using any variety of, you know, open source or, you know, vendor specific IPAM software. How are you handling the overlay or the NFV type function, IPAM management, where, you know, encapsulations, tunnels, destinations, subnet ciders might get assigned, you know, you know, not randomly, but on demand, be torn up, torn down. How are you handling that type of auditing and control? And are you integrating that into the same tool as your under cloud or your open stack components or using a separate one? So to answer the last question, yes, we are integrated it. From how we're doing this, we are actually using Open Contrail to manage IP distribution and service changing, chaining and MPLS management. OK, so all your, all your pretty much your NFV overlay type IPAM is outsourced basically to the network controller versus open stack or some... Exactly, exactly. Thank you. Hi. Hi. So you talked about network on demand, sorry, bandwidth on demand and the ability to service chain a load balancer and a firewall. In your experience, how many times has it been requested and also, you know, provisioning a simple service chain is OK, you know, in terms of actually scaling it up, like for example, you scale up the load balancer and then actually the scale the firewall in accordance with that. Do you give templates so that people can lay out their own topology or what is the way you actually deliver this particular service? So network on demand is an AT&C service. It hasn't been rolled out in a wide fashion. I don't know how many sites, how many customers we actually have on it, but it basically, the customers have a portal where they can assign, let's say they have a 10 gig pipe, but they only need, on a regular basis, they only need, I don't know, one gig. So they can dynamically change the amount of bandwidth that comes to their premises. Now, in the past, we used to have a customer premises equipment, so if they wanted a firewall or if they wanted a load balancer, they would basically buy more hardware. The way we're doing this now is through that portal is they can add that functionality to their services, but virtually. So they don't have to have anything, any premises, any hardware on the premises. So it's a simple service chain, not something like a data center where you need to scale it out? Yes, so that's a scale out data center is a totally different deployment type. But again, the movement is from dedicated hardware for load balancing and firewalling to scale out functionalities of virtual load balancer, virtual firewalls, so you don't have to have all these big pieces of equipment there sitting there. OK, man, we arrived on time. Well, thank you so much for coming and finding this ballroom and enjoy the rest of the show.