 Hello, my name is Gene Bagwell, and I'm an Associate Fellow of Verizon's Technology, Architecture, Planning and Strategy Group. I'm here today to talk about building and managing VNMs and CNMs to support 5G at scale. Before we get started, just a bit of background. The Verizon Cloud Platform DCP is a telco cloud designed to support network function virtualization across all of our business units. As such, we do not host standard IT workloads. We have a separate cloud system for those workloads. DCP is a global platform deployed in 94 locations around the world. The majority of sites are in the continental U.S. supporting our 4G and 5G services. DCP is divided into three service offerings, Core, Edge and Far Edge. Core handles our common network functions such as INS and Bolting. Edge handles 4G and 5G EPC elements and common RAM functions. Far Edge is the virtualization of the RAM infrastructure at the cell towers, and this is what we're going to be talking about today. We are predominantly a Red Hat shop, and as such, you try to have OpenStack and OpenShift to support our VNF and CNF environments in their Core and Edge locations. For Far Edge, we need to meet very stringent latency requirements for transport and data processing, which I'll touch on more in a moment, but this led to the selection of a second vendor for our Far Edge PAS and CAS. The core of the network is located in the Network Equipment Center's annual seats. This is where our VCP Core platforms are deployed. VCP Core hosts our 4G, INS, SDM and Bolting VNFs in the new 5G Core NFS. Moving out from the NECs are the Service Access Points or SAPs. This is where the VCP Edge platform is deployed. VCP Edge hosts VNF for 4G EPC functions such as P Gateway, S Gateway, MME, SBC and MRF. For 5G, the SAP sites will host the UPF and SMF NFS as well as the 5G Home Fixed Wireless Access platform. The next hop towards the edge of the network are the telco aggregation points or TAPS. These are the aggregation points for transport between the cell sites at the SAPs, which is also known as backhaul. If latency is a concern, we can deploy 4G and 5G EPC elements running on the VCP Edge infrastructure in these locations. Finally, the CRAN and DRAN sites. Equipment in these sites controls the antennas and the conversion of radio access protocol to TCPIP. DRAN sites serve urban or rural locations and are typically deployed at the base of the cell tower. CRAN sites are found in metropolitan areas and control multiple towers. CRAN and DRAN sites house three components. The radio unit or radio heads, sometimes called the RU or GNOB, the baseband unit or BVU. This is the control unit, which has two parts, the data unit or DU and the control unit or CU. And finally, the cell cell router or CSR, which provides Ethernet switching within the CRAN and DRAN sites and also layer three for backhauling to the TAPS sites. Our 5G infrastructure is fully virtualized. This includes virtualization of the RAN baseband unit or BVU. As previously stated, the BVU has two components, the data unit or DU, which handles the user point functions of the BVU and the control unit or CU, which handles the control point functions of the BVU. CUs are deployed as DNFs on DCP edge in SAT and TAPS sites, while DUs are deployed as CNFs on DCP far edge in CRAN and DRAN sites. In a traditional RAN architecture, there's a one-to-one relationship between the CU and the DU. This one-to-one relationship is inefficient with 5G, where the cell size due to the higher frequency series is small. Smaller the cell size, the more cells that are needed to cover a different area. The increased number of cell sites results in more handoffs between the BVUs, which is less efficient and can impact customer experience. Separating the DU and CU functions into virtual data units or BVU and virtual control units or VCUs allows us to create a more than many relationship between the VCU and many VDUs. This is known as a CU-DU split architecture. The split architecture allows for many cell-to-cell handoffs to be handled by intergenal beam ability, improving network efficiency and the overall user experience. This is a typical DU-RAN site. This is a class II telco environment. That means all equipment installed must be hardened to withstand significant temperature and humidity swings. It also means that all the hardware in the site is powered by 48 volt DC. The CAN-colored cabinets in the middle house the BVU and the CSR. Cabs contain dust filters and fans for air movement, but in most cases do not provide air conditioning or heating. This is a typical CU-RAN site. This is a class I telco environment. Class I sites can be AC or DC powered and they provide heating, air conditioning and humidity control. Let's take a look at the hardware needed to support 5G and CU-RAN. At the bottom of the page are the remote radio units which are sometimes referred to as radio heads or radio units. For 4G, the R-U's connect directly to the BVU's at the top of the page. In 5G, the connections to the R-U's are relocated to the front hall switch unit or FSU. We'll see why in a moment. FSU's are also called front hall gateways. Connections from the R-U's to either the BVU or the FSU are referred to as front hall. Front hall connections are shown in yellow. Next is the VCP far edge worker node which connects to the FSU via mid hall. Mid hall connections are shown in blue. And finally the CSR. The CSR provides L2, L3 connectivity in the cell site and L3 connectivity back to the tap and sap. Connections from the BVU or BVU back to the tap and sap are known as back hall. Back hall connections are shown in green. These are through the radio heads used in the network. R-U's connect via fiber to the BVU. In 4G, the common public radio interface or CIPRI is used for signaling between the R-U and the BVU. Even though CIPRI is a standard space protocol, each vendor's implementation is proprietary. In 5G, front hall signaling is handled by enhanced CIPRI or E-CIPRI. The E-CIPRI specification enables radio and data transmission over packet-based transport networks such as IP or Ethernet. This is the first step in adopting an open communication standard for front hall signaling as defined by the O-Rent project. In 4G, 700 MHz, 850 MHz and AWS plus PCS radio heads are cabled directly to the BVU. For 5G, we need to split out the 850 MHz spectrum and send it to the BVU. This is the role of the FSU. During the installation of the FSU, the connections from the radio heads are relocated from the BVU to the FSU. New connections from the FSU to the BVU are installed. The FSU then acts as a splitter. The 850 MHz spectrum from the radio heads is split out and sent to the BVU for processing. The 700 MHz and AWS plus PCS spectrum pass through the FSU in a process by the long as the BVU. The requirements for VRAM timing and sync are dependent on the radio technologies deployed in the spectrum used. In frequency-divided spectrums such as 7 and 850 MHz, a relatively loose synchronization of 3 to 5 microseconds is required. Time-division duplex spectrums such as CPRS and millimeter-way require tighter timing to phase synchronization. The higher precision is needed to prevent interference between a blink and downward signaling, loss of service during a cell-to-cell handoff, and for correct scheduling of software tasks in the BVU. With timing requirements of 1.5 microseconds or less, IEEE 1588 B2 PTP timing is required. The FSU, BVU, and CSR all receive timing signals from the Global Navigation Satellite System of GPS. The FSU, acting as a PTP grandmaster, provides time-sink to all components in the CRAN or DRAN site where they're locked to GPS or in holdover mode when satellite signal is lost. During holdover, the FSU's built-in oscillator can provide stable clock signals for up to 10 hours. The next component is the Farage Compute Node or Worker Node. We were not able to locate God's platform so it could function in a class 2 environment and still comply with our NEDS requirements. To meet these requirements, we had to work with our hardware suppliers to develop new server platforms. What we ultimately wound up with is a 2RU chassis that supports up to 2 blades or slats. Each blade is self-contained, including a single socket CPU, memory, storage, and networking. The chassis provides redundant power supplies and both slats can be run off a single power supply if one should fail. For the initial rollout, a chassis and a single slider are being deployed to support the 850 MHz spectrum. Additional blades of chassis will be deployed later to support CBRS and millimeter wave spectrum. In VRAN, package from the RU must be processed by the BDU based on a very tight latency model. Consisted packet processing within a well-fixed latency window requires the use of a real-time operating system or RTLS. By separating application functions into self-contained tasks and implementing on-demand scheduling of those tasks, the RTLS can guarantee task execution time. Initially, we were unable to locate a supplier supporting an RTLS in a container-based virtualization platform. This meant partnering with a supplier willing to develop, support, and maintain a pass based on an RTLS. For us, that partner was Wind River and the Wind River Cloud Platform. However, the general purpose Intel Xeon process which is used in the worker nodes were unable to keep pace with the packet rate. This meant that some form of hardware acceleration required. VRAN L1 to L3 packet processing requires many floating-point operations to implement features such as successive interface cancellations in my mode or forward error correction. The general purpose Intel Xeon SPCPUs in the far-edge worker nodes are by themselves not able to keep up with the number of floating-point operations required to support VRAN. An Intel FPGA in every worker node provides the additional compute power needed to support VRAN. However, it presents another problem. Each BDU vendor implements their own methodology for offloading signal data processing operations to the FPGA. The vendor code for the FPGA is called an acceleration function unit or AFU. Right now, the installation of the AFU requires flashing the FPGA a task that can take up to 40 minutes to complete. Even with automation, the severely-impact deployment velocity and BDU lifecycle management. Future Intel CPUs will include new instruction sets such as ADX 512, better caching, and micro-architectures that will deliver more instructions per cycle. The improvement may allow for the removal of the FPGA or the replacement with an easier-to-manage GPU by the time we deploy other spectrums. VCP Far-edge, which hosts the VRAN applications, is a distributed system with centralized management. The centralized management system supports inventory, pass provisioning on the local and remote worker nodes, installation of the CAS, and deployment of workloads. Controller nodes are deployed as clusters in the SAP sites. The current version of the centralized controller platform can support approximately 200 worker nodes per cluster. Depending on the number of cells, sites in the geographic area, SAP may host multiple pairs of control clusters. We estimate that between 200 and 300 control clusters will be needed to cover the network. An automated provisioning system is used to deploy and configure the controllers. IP addresses in the BMCs are preconfigured during physical installation of the hardware, and an automated process periodically pulls the IP addresses assigned to the controller BMCs. When the BNC responds to a pole, the controller is queued for installation. During the first phase of installation, Redfish is used to validate the BMC and BIOS firmware versions. If an upgrade is required, that is handled by Redfish. Once the firmware versions are up to date, Redfish is used to configure the BMC and the BIOS. As installation begins with a building on a custom ISO image, this image contains site-specific information for the controller, such as the hostname and interface IP addressing. Redfish is used to mount the controller-specific ISO via BMC virtual media and kick off the install. Once the OS is installed, Ansible is used to complete the configuration of the controller. The same process is used for the second controller in the cluster sites in the worker nodes. To cover 200 million points of presence across the US, VCP Farage will need to support tens of thousands of worker nodes. The scale of Farage necessitates a zero-touch provisioning process for the hardware. We developed a custom UEFA firmware image for the BMC, which is installed by the vendor during server manufacturing. This custom image includes an 802.1x certificate that allows the compute node to authenticate itself to the CSR during an initialization of the BMC network interface. The authentication is to prevent someone from attaching a rogue device to the CSR and gaining access to the network. The CSRs are already configured to generate IPv6 routing advertisements, so once the compute node authenticates with the CSR, the BMC is able to form an IPv6 address using stateless address auto-configuration or Slack. Within a valid IPv6 address, the BMC will connect to a centralized database server and register its Slack address. One segment in the IPv6 address contains an encoded location ID. Location ID is used as a key to locate and assign a permanent global routable address for the BMC and to find the site-specific PAS ISO file for the node. A web book then kicks off the Jenkins job that uses RedFish to validate or upgrade firmware, provision the BMC in the BIOS, and to install the BMC permanent IPv6 address. Jenkins then uses an API call to a central controller where the worker node will be registered. The controller uses RedFish to mount the node-specific ISO file on the BMC virtual media interface and begins the install of the PAS. Following a post-install reboot, the node will register with the controller. A successful registration with the controller will trigger file configuration via Ansible. The worker node is now ready to receive workloads. On boarding and application for VCP Farage starts with uploading a C-SAR file containing the Docker images, film charts, YAML files, and any required scripts. Marketplace then breaks out the contents of the C-SAR file. Docker images go to Artifactory, film charts are registered with NFBO, which is our orchestration platform, all files and scripts are sent to GitLab. Once all the components are registered with their respective platforms, a deployment can be triggered by an API call to NFBO or manually initiated via the NFBO UI. Once container images are successfully instantiated, an install complete message is placed on the CAPTA message bus. An install complete message triggers other tasks, such as adding the CNF to alarming logging and monitoring. Once the Farage site is installed and the VDU workload deployed, both need to be monitored and logged. With the introduction of the VRAM project, we are deploying new monitoring and alarming tools to replace our legacy OpenView and NetExpert platforms. Data collection is handled by Telegraph, Collectee, S&P, and Redfish Events. Information from each of these tools is placed on a CAPTA bus and sent to a data lake. Log, stash, Elastosurfs, Kibana, InfluxDB, and Grafana. Mine data from the data lake to build dashboards to display metrics, view log files, or display trends. Right now we have one instance of the monitoring stack for each of the VCP platforms. The stacks were designed with scaling in mind, as the number of VCP platform nodes increases additional instances of the monitoring stack to be added. The implementation and scaling of these new tools are still very much a work in progress, but early results are promising. Development of the hardware and software needed to build and deploy a virtual RAM infrastructure for 5G services has been a two-year journey. This culminated a month ago with a successful completion of the first end-to-end, fully virtualized 5G data session in a live network. All of the NS involved from the core of the network out to the far edge of the network were deployed as VNFs or CNFs running on VCP core, VCP edge, and VCP far edge. The link in the slide contains more detail about this milestone event. Thank you for your time and attention. At this time, I'd be happy to take any questions. Our speaker is now available live to answer any questions. Please submit your questions via the Q&A chat panel, and he will be happy to answer them. I'm hoping you can hear me. So if we have problems with redfish, the BMC will reboot and go back to its initial state so that we can start pushing the PAS image again. The next question was what percentage of the RAM in the core are virtualized today? The RAM we're just starting. We have approximately 700 sites deployed today. With our core infrastructure, we're approximately 60% deployed. Are you looking at FPGAs only on VDU or also GPA and GPU and EA60? FPGA was used because it was the quickest to get into production, and a lot of the VDU vendors had already started development of their specific code for the FPGA. We are looking to move away from the FPGA because of the maintenance requirements and the time it takes to provision. GPUs are probably the next place that we'll go. We've looked at a couple of EA6, but they haven't turned out to be as promising as GPUs are. The next question was about did we build everything in-house? Yes, everything that's shown in the presentation was built in-house. We had a lot of false starts with trying to do our own development Verizon, or at least the group that I work with in Verizon. We're not really a development house, so we had to learn how to develop and how to develop CITD pipelines at the same time as we were trying to push this platform into production. Fortunately, it's worked out well, and we're now looking to deploy nationwide. The question about what's your approach between CNF and VNF security. For the VDU working in the far-edge implementation, there's only a single worker node there and only a single container that's there today. We have an internal strategy for securing all of our VNFs or CNFs as part of the marketplace aspect of deployment. In that process, the applications go through a very extensive screening process, scanning process, and then we perform pin testing on each of the applications. Once they're in production, we have a security group that maintains and looks at vulnerabilities and can trigger an update to correct a vulnerability that's found from a central location out to all of the VRN sites. We do everything. One of the questions is what cloud provider do we use? Actually, everything is on a private cloud. It's a private telco cloud that was built internally by Verizon. It's only used for hosting our 5G, 4G, VNF and CNF infrastructure. We are federally regulated. We can't use AWS or Azure for our systems. We are not currently using machine learning or ran intelligent controllers. That will be part of the next initiative as we move into C-Band and CBRS. Yes, we do anticipate going 100% CNF and migrating away from VNFs. That's the last one in the queue. Hey, Jean, it looks like we have a couple questions in the chat as well. Were you able to address those? Just saw them. I'm assuming with a question about how do you compare your edge and differentiate from AT&T. I'm not really sure that I can actually answer that one right now because of some internal issues with what we can say and what we can't say about our competitors. VNF security already answered that one. We actually, part of the question about using ONAP, HP NFED is our primary orchestration platform. We built a bunch of components around it, Marketplace and Atlas, but HP NFED is an ONAP compliant provisioning system. Do you anticipate Etsy compliance for core and edge? We've been trying to follow Etsy, but in many cases Etsy gets in the way. Where we can be compliant with Etsy, we absolutely try to be, but in some cases we have to work around. And that's the last question. Oh, one more. Yes, we will eventually use it to use the VRAN infrastructure to roll out millimeter wave in 5G. We'll be using a similar infrastructure for both 5G, CBRS, and CBAN. So in three to five years, our entire RAM infrastructure will be totally virtualized, and we'll be going away from the legacy BBU infrastructure completely. Do we have any other questions from our viewers today? Any plans for 400G optical infrastructure? Yes, we are currently looking at that. How we get that out to CRAN, DRAN sites is still a work in progress, but we are actively looking at that not only for backhaul from the CRAN, DRAN sites to SAP and TAP, but also 400G infrastructure within the SAP and the NEC locations to support the aggregation of all the traffic coming from the edge of the network. All right. Well, I think that concludes our webcast for today. Thank you, everybody, for joining. We look forward to seeing you on the next session. Have a great day. Thank you.