 Get there now. I did promise you all one more dramatic demo So to help me with that. I'm gonna welcome on the stage the newest member of the open stack foundation Ildiko Vonsha All right, what do you got? Well, thank you mark I got that I'm very excited to be here and be part of this amazing NFV demo for which we actually have a 4g mobile system running on top of open stack on these servers right next to me So many of the NFV use cases are coming from the telecom industry And I think I will not share any new information under the Sun if I tell you mark and the audience That the carrier grade requirements of this segment are very very strict So give you a little bit of an insight. Let me share you an example mark. You heard about the five nines, right? It's a lot of nines. I know that. Yep. That's exactly five Of those nines so feel all of you in who maybe never heard about it, which would be beard, but it can happen This actually means that the availability of a given service has to be above 99.999% Okay, so amazing. We have still the five nines that what that means So it is actually less than five and a half minutes per year So I I don't know about you, but personally for me it took more than this to have my coffee this morning So for me this five and a half minutes actually means at most one coffee Per year if I'm fast enough to finish that in time so that's just tough right and I guess I don't have to explain it to you how tough that is to build a system that is Actually able to fulfill all these strict requirements, but today we will Demonstrate it to you that open stack is capable of serving as a base of these systems The work has started about one one and a half year ago But we really couldn't have done it alone We are collaborating with opnfv Which is an open source community with the mission of building an end to end integrated Open source cloud platform to accelerate the adoption and evolution of NFV I'm proud to say that I'm involved in this community in a few projects for example opnfv doctor this project is about to handle fault management in the platform and They are using and also collaborating with several open stack projects for example AODH the alarming service Congress which provides policy enforcement Nova which I hope you all know opens that compute and we try with Raj which provides root cause analysis And without giving out too many hints I can tell you that with Raj will be in the spotlight just a few minutes from now and Looking over to mark. I think I should stop with the storytelling because this guy looks really eager and impatient to get into Some trouble. Am I right? Yeah, let's do something crazy Let's just start demoing things. Yeah, let's get the show started So it is my honor to welcome on stage the project lead of vitrage ifad effect and the project lead of opnfv doctor Riota Mibu all right, so Riota and ifat are gonna actually help us demonstrate what a phone call does look like on an open stack powered 4g mobile network So I'll deco why don't we do something you make a phone call? And I'll play the role of the real-life chaos monkey over here and see see what happens when I start going crazy Yeah, now you just made me scared of making the very first phone call live on main stage Hi Heather, how are you doing this morning? Oh Alright Mark so put those back in So you see all right So we all know in the real world cables get cut and there are chaos monkeys of various types running around data centers and digging up cables So Riota, why don't you tell us how you can actually solve this problem? Sure So let me explain the demo system first the phone calls go through these servers to launch in these server We have an ec commercial mobile core network of 4g running on top of open stack and it's very High rev sorry high availability requirement. It has redundant nicks and VMs For head over to initiate this failover. We need fast forward notification But it had been missing in open stack So when you remove the orange cable for the primary VMs the packet got dropped in even though we have backup VMs connected by the blue cable, so Yeah So we started OP NFV doctor project to develop fault management framework for higher availability and I think this project is a great example of collaboration of Among telco operators vendors and open source communities So the project Was initiated by NTT docomo and supported by several companies from the beginning And in the project we studied NFV industrial specification and requirement and developed solution architecture like this and We mapped functional blocks in the architecture to appropriate open stack services so congress inspect for event from infrastructure area and Nova neutron and Cinder need to change the status of the fault affected resources and AODH send a notification to the application manager so that it can perform switch over to backup VMs quickly but We still had few features missing in open stack and we received tremendous support from the open stack community For the development of these missing features these missing features are now included in Open stack Newton reviews. So let me enable the doctor features. All right, Ryota is gonna Bring the doctor to the house and see if maybe we can withstand the chaos monkey one more time This time with the the doctor project So now I'm creating one alarm in a of DH and seven policies in Congress and No, we are using bunny open stack, which means without any proprietary add-ons to the open stack It seems ready. All right, ill to go. Let's try this again And I'll try to behave myself, but no promises. Let's let's make one more live phone call from the stage Well, I trust doctor Not really you mark them Smart Yeah, this is me Heather. I'm so sorry so more of our chaos monkey just But now we have no going back man turned on so hopefully everything will go fine I can't leave they trusted me with these Are we are we able to mark proof the phone call? Yeah, the phone call is not totally mark proof the cables are cut to cables are cut and we can still hear you So thank you for supporting this demo. See you around this week and have a great time with open stack Bye Wow All right I just want to say I had the easiest job of anyone. I just had to but those scissors are pretty heavy Okay, so look it worked amazing. I'm excited, but I did notice some alarms went off here And I think this is the vitrage dashboard So if I why don't you tell us what what we're looking at up here? Thank you mark Open stack vitrage is a cool and innovative wood cause analysis service that helps the cloud administrator Understand and manage the faults of the cloud. I'll explain how it works with rush collect data from open stock or external data sources combines it in a topology graph and uses it to And uses it to show an holistic view of the cloud topology On the screen behind me, you can see vitrage entity graph in horizon on top You see the physical aero below is the virtual aero and on the bottom the application layer The great thing about this view is that it show you how these three layers connect to one another and how they affect one another When mark cut the cables New alarms appeared in the graph and some resources changed their color to red. I'll explain what happened vitrage received a notification from an external monitor that the host was down it then executed some pattern matching algorithm over the graph, which is really cool and Executed the wanted actions it notified Nova that the host was down and changed its state to error And it created additional arms on the instance and on the application Notifying the cloud administrator about these problems But this is not the only view available in vitrage alarms view you can see alarms from different sources like Nagios, Zabix or AODH that are currently active in the system You see here the alarm that was raised by the monitor and the two alarms that were raised by vitrage In a larger setup there could be a very long list of alarms And if you want to understand why there are so many alarms you can open the most important view the root cause analysis view Here you see a top-down causal relationship of the alarms in our case you can clearly understand that The application alarm was caused by an instance down alarm and the root cause for these alarms is that the host was down Now you have a full vision of the cloud apology the alarm that happened and the effect it had on the system. Thank you Thank you so much Amazing NFV team