 Good afternoon, everyone. My name is Susan Combs, and I'm with Verizon Wireless HQ Planning, so thank you all for coming. My role has been to support service assurance for our virtual mobile provider network based on virtualized network functions, or VNFs, based on OpenStack. So our leading the way with our VNFs has been supported by the highest level company, Lowell McAdam, is our CEO. So we definitely have support for our work on VNFs on OpenStack. And many of you are most likely already familiar with Verizon. Maybe we even already have our service. But you may not know that LTE covers 98% of the US population, and our global IP network covers 99% of Fortune 500 customers. So this is diving a little bit more into the support that we give to our customers. So for our digital commerce customers, including Telematics, which is the number one fleet management company in the world, we're leading evolution toward virtual network functions on OpenStack while continuing our consistently strong operational performance. So the graph below illustrates Verizon's leading evolution toward VNFs with active calls on a geo-redundant VNF site pair on OpenStack serving the mobile provider network. So in this case, the VNFs are routers. It's just one single site pair. And these are active calls. And they are on our production network. We're very proud of supporting our production network on our virtual mobile provider network. That just started this year. So this is from the past month showing our support for over 50K or around 50K active calls peak each day, then dipping every evening. Then you can see the slight dips on the weekends. And what's also interesting about this particular graph is that it shows the transition from one VNF to the other. So this supports our very reliable service that we offer to our customers. Now, all of this network activity generates a lot of logs. And what I wanted to also say is, for our service assurance, we have many different methods of gathering data. It may be SNMP alarms. It may be SNMP polling. We gather extended statistics from our VNFs. And then we also gather logs. And the title of the talk, Finding Needles in the Open Haystack, is a reference to that it can be for alarms. It's very easy to see, OK, I have an alarm. I know what to do. For extended statistics, it may be calls active. That's pretty clear. For logs, sometimes it can be a challenge to know where to look for the log that you need. So we'll go into logs. So today, we gather about seven gigabytes worth of logs per day from our virtual mobile provider network. So this is the entire set of logs that we gather. And then we can go into detail and look at individual logs as well and search for them. Or we can split it out. Here, I'm splitting out just open stack logs or specific open stack services, whether it's Cinder or Ironic or whatever open stack service may be of interest. So I'd like to look at logs for. And here, again, we can go in and look at the text representation of logs as well. OK, so this is interesting. Going down even into more detail, what logs are created by specific actions? So just as a small lab project, I did a couple of tests to see what we can see in the logs from specific actions. So for example, we may want to create a test project. So here, I'm just creating a summit test tenant. And a test user, summit test user, validating that they've been created. And the role assignment, we can see that the summit test user is a member of the summit test tenant. OK, so far, so good. Now what log activity did these actions generate? So sure enough, searching for logs, we find logs for them. So we see Keystone and Swift logs, including here's a sample Keystone log. Of course, we need the UUID of the project and the user, which we see the project is in blue as summit test and then in green as the summit test user. So we see a record of what we've done. So that's maybe interesting. It may not be clear how to use it. Let's try another example. So we can create a sample test volume in Cinder. So did that. Then we might delete it. Now this is an interesting thing because in the lab, sometimes we delete a sample Cinder volume and it may not completely delete. We've experienced this in our lab. So what more could we learn about the situation from the logs? So as you can see here, I had a sample test Cinder volume that I just created. Then I deleted it and it looks like it deleted. Let's see what the logs tell us about this. So we find records of the deletion and Cinder as well as in Nokia. And this one that's highlighted in yellow here, I think is interesting because it tells us not only that it was deleted, which would be interesting as a confirmation anyway, but it also tells us the exact command that it sent to our storage provider to complete the deletion. So it's telling us it sent Navisex CLI and it sent the command to destroy that particular Cinder volume. It gave an argument of forced detach. So if we were in a situation where we needed to troubleshoot a Cinder volume that hadn't completely deleted, this would be a really good starting point because we'd know the exact command that went to our storage provider. So this gives a little bit of the flavor for what logs can add to our foundation of service and assurance data that we gather from alarms and statistics. Can give us more detail and more color.