 Hello everyone, welcome to today's LF networking webinar. Our topic today is O'Rorge deploys on app in production. Just a couple of housekeeping items before we get into introductions. All of the material shared today, the slides and a link to the recording will be available to anyone who's registered for this webinar. In the coming days, it will be emailed to you along with a one page detailed asset about the topic as well. That's going to be available online today and that will also be emailed to registrations registrants. And then we'd also like to note that in progress is a longer form case study that will be available within the next few weeks that goes to goes into even more detail. So instead for today's webinar, if you have any questions, please feel free to click on the Q&A tab in the bottom right of your screen and type your question at any time. We should have time for live questions towards the end of today's session. We do have panelists who are also happy to type in responses in real time in the Q&A tab. All right, let's go ahead and kick off our introductions. Next slide please. All right, so speaking with us today is Olivier Ogizu. He is with Orange Innovation Network Automation Project Manager. We have Mohamed Hassan. He is with Orange Egypt. He runs the telco cloud engineering and automation as a senior manager. And then our other two speakers. You can click to the next slide. Yes, thank you. Sorry, we've got Abdel Muhammad Saudi. He's the head of B2B Solutions for Orange Innovation in Egypt, and we have Mohamed Daoud. Also with Orange Egypt, telco cloud NFV DevOps integration. All right, I'm going to go ahead and kick it off to Olivier, who's going to get us started with today's discussion. Thank you. So welcome. And so we are going to present you, in fact, what we did using Orange to match the transport network automation. So we had several challenges to face when if we want to automate our transport network infrastructures at scale. So the first one, and maybe one of the most important one is to decouple the legacy IT application from the network because currently we have strong adherences and the network interfaces are outcoded in our IT. So this is a current strong impediment from our agility, TTM and effective cost reduction. The second challenge we have to face to is to unlock the vertical vendor silos with two axes. The first one is the network element management openness so to bring more openness to each network element considering the on the global way the transport infrastructure so it can be the IP networks but also the transmission networks like WDM, micro, etc. And we place a vendor of vertical management solution by open solution because more and more we need interoperability between the different network domain. So we need a global view of our network infrastructures. And then the network automation has become a business continuity challenge because we have less and less people in our operational teams and the network operations are getting more and more complex. And we have also more operations to perform and we must enforce and ensure our operational processes, our operational efficiency. So now the question is not do we need to automate or not, but how to perform this automation. So if we talked about network automation, so it's a long journey and we have to take into account where we are today and where we want to go tomorrow. So here you have a maturity evolution scale based on six levels and these levels are inherited by the team by some team from works so in appendix you will have the definition of all of these levels so but for the sake of the presentation. I would try to explain so the level zero everything is done manually so it's let's say the past legacy legacy system level five. We have a global automation system fully aware about all the networking domains and implemented everywhere so maybe it's a target we will never reach but it's a let's say a long term target what is interesting is to focus on the intermediate levels levels. And then you have assisted management, this is pieces of a technical automation performed in some location of the network so globally this is the situation today with level two, we are starting to implement some close loops we have some partially autonomous networks based on predefined events and predefined words and predefined actions on level three, we have for a specific networking domain, a more autonomy in the decision making and we can adapt the decision to the global awareness of the network domain. In level four, we have this awareness on a multi domain, let's say for instance you can drive your IP layer according to some events, some awareness of the transmission network and we introduce some more predictivity. And this is important to see this so of course we are targeting level three, or even level four, but as far as we are starting with level one. There is a big difference in the automation journey because at level one, you can achieve your automation with standard on projects based on single use cases basis, selecting different solutions for each use case. And with this you have no guarantee to be able to evolve to a higher level, because when you target automation from level three, this requires a global automation approach because you have this awareness requirement of your networking infrastructure. So, we need to have a disruptive transformation and to drive this transformation, we need not a theoretical framework but I need, at least we need some concept and functional architecture and so on. So our vision is target is that we must bring IT and network together to drive this transformation and we must move from a closed IT architecture to an open platform one accessible with open API. It's our chairman and CEO who said this, and we have selected the TM Forum Open Digital Architecture as a global framework to support this transformation. And in a nutshell, if you are not familiar with the Open Digital Architecture, the target is to transform the vertical and monolithical applications into modular components organized in horizontal layers. So today we have for instance the building system, the provisioning system, the service assurance system, etc., that are vertical applications and the idea is to bring this into some horizontal blocks. So the HRT management deals with the who, so the who can be the customers, the partners, the providers, the core commerce block deals with the what, so the products and the offers we are selling to our customers and the production blocks deal with how we can implement this into factories, into network factories. So the production block is really a key component for the network automation and now the question is how can we implement this production block. So Open Source will be the cornerstone of our transformation to build an open digital architecture compliant platform and we have selected on app. So we have many use cases, but we must consolidate these use cases in a single environment to have this multi-domain awareness we want to reach. So here for instance we have the IP infrastructure, the micro infrastructure, the optical infrastructure, the fixed access infrastructure and you see we have different kind of use cases such as 5G transport, IP management, service provisioning, IP backbone, IPWDM convergence, optical cross-domain and to end the scenarios, service provisioning for micro, etc. And we have selected on app components to build a common automation framework. So sharing the same infrastructure, sharing the same technical enablers and adding at the southbound of on app. Some domain specific components such as the Open Daylight, SDN controller, unseable cable, the C-square initial solution and we can add in the future other vendor solutions. So now to go into production, of course starting from the Open Source on app distribution, we have to package it. So what is the on app packaging by Orange. So we have selected some components from the on app platform. We have done them to answer the transport network automation. So we have started with prototyping activities based on the real use cases. So involving IP configuration changes so performed by C-square and unseable and we have worked on the integration of these components into on app. We have selected also we have made a design of the global architecture to select the mandatory components from the on app. But we also want to have a modular design because in the Orange Group we are operating in 26 countries with different situations, different kind of networks and the requirements at the deployment day is not exactly the same in each network in each country. So we had also some strong requirements so the platform must be installed offline. So without any internet connectivity in Orange Country's data center, why? Because this data center are very sensitive, they are hosting our network management applications and we cannot have a plain internet access. So this is very important to have this offline installation managed in a CICD way. So today we have a distribution of enabling zero touch installation on bar metal servers and we are working to perform this to have a flexible infrastructure. So from bar metal, VM or container as a service. What also we intend with hardening is to work on security to remove all the default password and to improve in fact all the operations of these components such as logs monitoring backup restore. And next year we plan also to work on the upgrade when you have lots of data in your components, you want to upgrade from one version of on app to another version. We must be able to upgrade, sorry, keeping all the data of course the operations data we already have. And what is important is we do not want to fork from on app. So we each time we modify something we upstream it to the to the community and our packaging. We intend to have a lot of tests running so we are reusing the community test we have specific test dedicated to the orange situation and our target is the one when we have a new version of on app. We have our orange packaging available within two or three weeks after the official release. So it is a simplified or functional architecture in the reality it's more complex but at the northbound we have a customer facing services API that are consumed by external applications or graphical user interfaces. And if it is the case, we have a first level focused on the operational processes. And if we reference the team from open digital architecture. This is the some functions of service order management your with order you have to take care to understand order on the generic way it's not only commercial order but it can be any kind of working order that can be sent to a network factory for instance so upgrade the capacity of the network to upgrade the network element in the firmware of a network element in the network or whatever you do the order name is quite generic. Then we have a resource order management functions or room function that is focused on the technical orchestration. And then we have a network mediation layer. So the target of the network mediation layer is to abstract the network specificities or the vendor specificities to the room. And we have a single source of trust so it's very important when if we are going toward this automation journey, the network is no more the source of trust, but it is the service and resource inventory that now will be the source of trust and here we model or we represent all the intense we want to deploy in the network and we link the resources or the service resources with the network resources with several levels of news. So what are the on that component used to perform this we have so the color code at the same so at the some level we have the service orchestrator that acts as a service order management function. So today we had to implement some orange specific workflows or pieces of workflows. And, of course, the target would be to try to generate this and to propose this workflow extension or workflow new new workflow pieces to upstream to get them upstream to the community. We are using the CDS or the CDS is the controller design studio and the CDS is acting as a room so resource order management function so kind of technical orchestration while the so is more business or operational oriented orchestration. The active and available inventory is a masterpiece of our distribution because it's the single source of trust and here we represent the service and resource inventory. So today, we had to add some specific schema extension to this database. And of course, it's something we would like to propose upstream to the community once we are sure about how to generate this and because we don't want to obtain something too specific to to orange. So we have replaced the SDNC by the standard distribution of open delight because we wanted to be flexible on the open delight distribution we we have we are using the open source distribution of open delight. And this open delight is here to integrate some network mediation components that have a net conflict interface. So this is the case, for instance, of the of the shelf. This is the C score in a so solution that is a high P configuration manager. So why we have replaced the SDNC so as I told it's to be able to to select any open delight distribution to have more flexibility on other version and patches not the way to have them integrated in the SDNC and at this stage, we do not need the specific functions of the SDNC such as such as digigraph or VNF a specific function for the kind of use cases we are deploying today. And what is very important last point is that we have a clear split between the business orchestration so some function and the technical orchestration and we just need to exchange some unique ID stored in the active enable enable in the inventory between the layers, whatever is the use case so we are we have totally standardize the interface between the SO and the CDS for any kind of use case. So we think it's a great improvement. So I let the floor to Mohammed Hassan to describe the opportunity journey. Hi everyone, this is Mohammed Hassan and this slide. I'd like to call it on and orchestration success cheat sheet so there is a normal way to say it and there is the interesting way to say so all of this slide talking about how to be successful in deploying the ONAB and orchestration in general and during this journey so I'm telling you our experience what we found what we will do to make it happen again in another affiliate site orange so first of all, as you know that now is not out of the book solution needs a lot of customization. Even you will not purchase or install on that you must install it to start learning program it's considered as a role model for the orchestrator and you if you installed in your lab you will learn a lot. A lot of 5G vendors will give you the orchestrators as a free of charge to be including the package of the 5G but also you will need ONAB to learn from it so please start to install ONAB in your lab and start playing with it. If you finally decided to go commercial orchestrator don't buy license please buy a scope from them, for example, zero touch provisioning for the new Mac and give it a duration for example three months over six. So by this way you will have a successful implementation for orchestration, most of the orchestration projects has no end and always to be extended and get delayed. Again please please please don't buy license buy a scope so this is very important for anyone who's dealing or managing an orchestration team or responsible about the orchestration. If your team responsible about automation orchestration, please start with 100% of your manpower using the focusing on the automation projects and start most simple use and important use cases. After 10 use cases of automation, start thinking about orchestration. Starting with orchestration within the same time with automation tasks will get things more fuzzy and very complicated so start with automation after a while you will start with orchestration. Next orchestration divide your forces. Two thirds would be of your manpower must focus on automation cases and one third in orchestration. Automation team will face more non technical issues implementing operation team mindset in the code and let them accept the results. Customer satisfaction is very important. You must set with the operation team and your product owners must daily set with the operation team teams to let them accept the result and start using the tool. The orchestration team will start to face more technical issues more than the automation to concentrate multiple technologies and devices at the same time. Automation team must be ahead of orchestration team so as I told you before. He will be the road for the orchestration guys. Try to complete your CI CD both cycles the code CI CD and configuration CI CD before the orchestration phase. Make sure that the developers follow the code disciplines. So this is very important to follow those steps. You must have a very good lab to test everything. Everything must be tested once and twice before the deployment. You must hire a good and dedicated testers for the code and the configurations and the scenarios. Try to develop good lab for your testing as I told you before. Try with engineering team to minimize the variety of devices of IOS. There is a huge IOS there in types of variety and this will make a lot of confusion and problems and automation and orchestration. So try to set with our engineering team and try to minimize the variety of devices. Try to squeeze a little bit and shortlist some devices. Front end development. It's very important as back end development. The user experience has a huge factor for project success and acceptance and let the guys use the tool. You can develop an ugly tool that no one will use it. So please spend some time with the front end guys. You have good interface, active interface that can handle the need. If you don't have source of truth, please start to use one as soon as possible. You have a lot of open source, source of truth in the market and you have great products in open source and try to log everything. Everything that includes batches and orders, batches and DO numbers, end of support, end of live dates or devices, log everything. The last most important recommendation from me to anyone working there in the deployment of orchestration by two large 60 inch screens and install them beside the CTO office and the CFO office. So on them, how amount of savings that you gained by deploying the automation of the station. That is very important to get your budget. Thank you. And I can pass to the next to Muhammad. Thank you for the interesting journey. Hello everyone. Now I will talk about the what is the change or enhancement that orchestration can do. Also, I will describe the difference between current status and orchestration mindset and mentality related to automation opportunity, but to be able to do this, I need to take about reuse case and service provider. For us, our use cases, it is a layer 3 VPN in Corin bless network. But generally, we generally work orders over the use cases, take some flow. So the current situation, we have some steps to deploy any work order, the service owner or as you can see in the slide the service owner or customer can ask. I will request a service, a new service, maybe it's a new or need to update something or delete the old service so he can ask the engineering and the planning team then planning team and engineering create work orders. Then forward the work order so operation. And the operation team can create and the blue. Sorry, it was a work order. At the end service owner can test and validate the request, but sometimes we have some issues. So the customer again after validation and service owner of the validation and testing, maybe he's facing some problems. We'll report again engineering and engineering will update some work orders and the operation operation team will troubleshoot and do and deploy the updates. As you can see, it's a long, long, long journey to without automation and without orchestration will take a lot of time creation of communication and delivery. And also, it may be have some troubleshooting and errors. But actually, after our experience with automation and orchestration, the process will be more easy in deployment, any end to end service is like a predefined template. Some parameters come from customers and some parameters come from automated resource management. And this is the one benefits of orchestration. You have automated source management tool inside orchestration itself. The orchestration will provide totally life cycle management for any network service. That means we have service service orchestration and also will provide a lot of accuracy, because we will avoid human error and save a lot of time. Also, it will enhance user experience for management and deployment. And as you can see in the right side, this flow of the creation of the work order, but in easy style by orchestration to mission and we'll go more deep in the next slide with Mr. Abdul-Main Soud. Thank you, Ahmad. So in this slide, we will try to show a quick overview of the architecture used in the use case with Orange Egypt. As you can see here, this is an overview of ONEP framework. And beside it, we have also NSO. If we start with the flow, we have first to design a workflow. And this will be agreed of course with the operator in the use case to see what is the use case we need to develop and what is the workflow that we will need to follow to achieve end-to-end instantiation of a service innocence. In this case, we designed this workflow and build it using Camunda Modeler, as you can see, so that it will run in the service orchestrator Camunda BPMN execution engine. Also, we will have to design a package in CDS blueprint to processor to be used to carry out the different technical steps needed to instantiate the service. And as you can see here, assuming that we are going to create a new service instance, we will have customer portal, graphical user interface, used by the ordering team or operations team to request a new service order. This portal will submit a REST API to SO API handler, which will run the custom workflow designed by Camunda. This workflow will manage some of the basic elements used in standard on-app workflows like creating a service innocence ID and AI, like updating the service request ID and the service request status, so that you have an idea about the service request. Is it now in progress or is it completed or is it filled? Also, the SO Camunda workflow will use AI as a single source of truth when trying to synchronize the elements that will be instantiated. Then the SO will contact CDS using the REST API to deploy and run the blueprint package. This CDS package will carry out different functions. One of them is communicate with AI if needed to pull some data or update some data. Also, in case of automatic resource allocation, like the generation of a new IP address subnet or whatever, CDS can integrate with Netbox to pull out a new resource, either an IP address or a VLAN or a VRF. Also, we can communicate with Vault to retrieve secrets that are encrypted in Vault. In case we are going to access systems like, for example, OpenDelight or maybe EWX or Ansible, we can use Vault to save some sensitive passwords. Finally, CDS will trigger the actual request to SDNC or OpenDelight using REST or RESTConf. Then OpenDelight will contact NSO using NetConf to trigger the instantiation of a new configuration. This configuration in our case is Layer 3 VPN VRF with static root subnet on the PECE link, root target import and export and so on. So this describes an end-to-end overview of the use case from a technical point of view and you can see the solution architecture. And there is also here in the slide, as you can see, an opportunity also to use Ansible, which is also integrated with CDS, part of the ONAP framework. Okay, thank you. So thank you, Abdel. So as a conclusion, what we could say is that ONAP is suitable not only to prototype 5G use cases, but also to support transport infrastructure automation with the production target. And I would say this is something very important. So maybe because when you want only to build a proof of concept, you don't need the same availability stability than in production. But we found into ONAP some components that are quite stable, quite straightforward to use and very useful. So that's very important. We have to find a balance between these mature use cases and components with more exploratory topics because when you deeper in production, you want everything to be to be stable, to have no regression from the previous version and you need a stability. And your concern is all the operational concern that you can have, such as lock management, memory consumption, CPUs, consumption stability, et cetera, and not maybe too many new features that bring something that are not interesting to your use case. So I guess that today there are few operators using ONAP in production, but if we were more and more to use ONAP in production, we will find a way in the community to find this balance. And also, we have seen that we have used specific workflows or we had to adapt some specific workflows and using specific workflows, we are not able to use some design part components such as the SDC, for instance, because the SDC is not compatible with the customized workflows. And I also mentioned some schema extension to the active enabled inventory. So we will have to find a way to talk about this to the community, to reach the upstream effort on this, but we didn't want to do it right now because we are running a lot and we do not pretend that the way we are doing it is the best way. So I guess the CCDPN could be the good place to talk about all of this and it's something that we forecast to do next year. So thanks a lot for attending this webinar and what we propose now is to open a session of questions and answers. Yeah, great. Thank you, Olivier. Thank you everyone for this great information. It does look like we have quite a few questions that have come in through both the chat and the Q&A window. So we'll just get started with the first one that came in from Carlos at 11.14 am. He's asking what CAAS do you currently use? Is it all open source or a specific distro? Also, what is your solution for bare metal orchestration? Okay, so I will try to answer to this question. I'm not the best orange expert to answer to this. So in a nutshell, the current deployment has been deployed in bare metal servers, so my colleagues from orange Egypt could elaborate on this if necessary. The CAAS plan is not within the lab or in the process of building at orange. For this kind of application, we are using our own CAAS based on some open source solutions. But the global, but we can have some differences between one network and another network. So I would say this would be studied the case by case according to the networking, to the networks where we would have to deploy in a CAAS. Okay, did anyone else have any of our panelists have anything else to add? Are you good with Olivier's response? Yes, and regarding the bare metal orchestration, we didn't start this phase yet. We are using step by step as I told on my slide. We started with this use case and we started to build over it. So we will try to get to rush to cover everything from the beginning. Okay. Great. Thank you. Next question is also from Carlos. Can you clarify you said don't buy a license via scope? What do you mean by that? A lot of vendors or some orchestrator, you will sell it with license, number of license. If you have 100 device, you will get 100 license and leave everything for you or to get someone else company to do the coding for you. So the best use case that we discovered if you want to orchestrate or automate thing, give it as a scope to the company or the vendor who sell you the software to orchestrate end to end. For example, zero touch board version for the server. For example, we are truly at 3DVN. It's the best way if you decided to buy a closed source orchestrator. Great. Thank you. Roland is asking, do the integrations between SOM, ROM and inventory follow open API TMS standards? Okay, that's a very good question. So I would say the target is yes and we will at this stage we rely on the current on app implementation. So we have some good news is that the service order API handler is compatible with the team from a service order. So here we are compatible. Of course, the specific payload of your specific service is specific, but all the wrappers of the request is standardized in the active and available inventory. Also, you have the team from information model or pieces of this information model, both for service modeling and resource modeling. And yes, we will try to go into this direction as far as possible. Yes, but I think with on app as it stands today, at least for SO and AI, we have a very good basis to start with. Great. Great. Thank you. Next question. Did you integrate just Cisco NSO or also some additional Cisco tools like IAP? Okay, so yes, we are using IAP in some limited deployments. But in fact, we have seen that the functions provided by IAP are redundant with the functions provided by the app. So our target is ready to we see really the value in Cisco NSO as the universal mediation layer for to manage configuration changes in IAP returns. So we worry focus on this. So we have just integrated Cisco NSO as Cisco tools. I've seen another question about the interface was it simple or complicated to integrate. So I will try to because it's related to try to answer this. So, and if we had a collaboration with Cisco to perform this integration so that the very beginning. Yes, we had a collaboration with Cisco. In fact, we have seen that there are several ways to to integrate the Cisco NSO into the on app framework. First of all, in the top of Cisco NSO, you have two kinds of interfaces. You have the network interface and we have a REST API, REST Conf API. And you can integrate in the several components that you can integrate directly in the SO, you can directly integrate in the CDS or you can directly integrate the SDN controller, and maybe you can find other ways. So we had a different proposal from Cisco to work on this. But at the end, we have selected a way that we think that is the most generic using the network of interface because if you use a network of interface, you we do not have to to wonder if one day Cisco because they have a major vision that changed the REST API or whatever. And with NetConf using a SDN controller, we can automatically compile at runtime the young models exposed by Cisco NSO and have it available to be consumed in the open delight SDN controller. So that's why we we met this kind of integration at the end. As far as we are talking about standardized protocols such as NetConf, Young and so on. The integration was quite straightforward. And we had nothing specific to be expected from Cisco to make this integration. Great. Thank you for that. Moving along. We've got someone asking what were the selection criteria slash dynamics of a wrong in selection of own app as an end to end orchestrator. Oh, that's a complicated question because today we are talking about transport networks. In fact, we would talk about so. Yes, there is a DNF department, 5G department. In some also it depends on the operational situation of each country because we have different operational models in some countries we are outsourcing the network operation. So in this case, it is more a matter of contracting with the company operating network in order. Another situation we are operating by our own resources. Everything so even if we just focus on transport and transmission networks. Quite complex. So what all I can say is that for this kind of scenarios, yes, we are promoting on app for other domains. It's still under assessment. And I cannot give you today. A definite position, let's say. That's fair. Thank you. Mohammed would like to know, do you use own up NBI for API integration. Do we use. One up NBI for API integration. I'm not pretty sure to have to understand the question. No, we don't use. We don't use NBI interface we use the rest call service instantiation rest call to the on app service orchestrator directly to trigger service creation request. Thank you. What about deletion of service instance. I mean, how can you manage to release the IP address from net box subnet. Does CDS execute some additional workflow to release some parameters from net box etc while deletion of instance. I think you are the best one to answer this question. We have two workflows in CDS package one to create resources and allocate them in a net box and update them in AI. And we have another workflow to delete the resource from net box and free up the resource location in the IOS so yes. Great thank you. How does CDS compose the CPE configuration and why doesn't NSO do this. If I may. CDS does not compile the CPE configuration. In our case it's the P E actually for the layer 3 VPN VRF use case the configuration is actually a template in NSO so NSO is actually having the configuration template CDS role is to drive NSO to deploy the configuration and provide NSO with the necessary parameters for the configuration template in NSO so actually the configuration template is in NSO CDS drives and provides parameters like IB addresses and VLANs and VRF names to NSO. Thank you. And maybe because the question was related to CPE because in the schema we are presenting CPE and in the schema we have uncivil playbooks. In some situations we have in other situations than the original one we also have a CPE to be managed and it's true that it's directly the CDS in this case that is managing the uncivil playbook. And why there is this difference is that the configuration sensitivity is not the same if we are talking about CPE on a technical point of view in P. We need really a state full management of the configuration changes because we have many changes. So that's why we need the state full engine provided by CSCO NSO. For CPE once we have the initial configuration we have not so many configuration changes and the CPE configuration is quite simple and we can have a state less configuration tool. This is what I mean if we have the service description we that is needed to be implemented in the CPE. This is enough to generate a full configuration of the CPE so that's why uncivil perfectly fits this kind of requirements. Great thank you these are really good questions I don't know if we're going to have time to get to all of them but we will see how many we can get through we are capturing them so if we don't have time to get to your question we'll see about potentially posting a Q&A doc or a follow up blog post. It looks like we have a couple of questions about Lighty.io. It was mentioned in the zoom chat that Lighty.io is a replacement of SDNC. Does that mean we can think of Lighty.o as an open daylight packaging distribution for own app. And also, was it the enterprise edition that was used or was it a community edition. Okay. So, yes, when I said that we are replacing the SDNC by standard distribution, yes it's true we are using the Lighty.io distribution of open daylight. Okay, and we are only using the community edition of Lighty.io it's not the specific enterprise distribution. So everything is public open source we have also upstream the way to build the health charts from a Lighty.io to be able to deploy it easily in the own app framework everything is available in the Lighty.io public GitLab. Great thank you. Moving along, was there any integration with server cloud infrastructure managers? Like Cloudify for example, no, we did not integrate with. Okay, Moodasar if you want to clarify what you're asking feel free to type it in the chat but we'll move on to the next question then. Did you use AAI as a single source of truth for inventory information and Cisco NSO databases for operational slash state info? Did you coordinate AAI and Cisco NSO DB? Yeah, so in fact, yes, when I said AAI is the single source of truth, yes, in fact, I should say AAI federates the different sources of truth but the root of the sources of truth is into AAI. Of course, in AAI you will never have all the configuration details that are written but only the parameters that are useful to generate this configuration and in Cisco NSO you will have of course all the configuration details because it's the scope of Cisco NSO to have this kind of information and in Cisco NSO we are not storing any operational data only configuration data and to make the link between the two it's based on the unique identifier generated by the active and available inventory and we are mapping the service instances configured by NSO with this unique identifiers in the NAI and just to give you an example, if you want to track a service creation for instance in all the systems so you have many components because you have ESO, you have active and available inventory, you have networks, you have CDS, you have Open Delight, you have NSO, you just filter your logs according to the service ID generated or stored in the active and available inventory, you will see what happens in each component and it is also stored in the NSO database. Wonderful, thank you. So Modisar has actually come back and provided some additional clarification on the previous question about integration with server cloud infrastructure managers. He was referring to Kubernetes or VMware as an example for element managers. The current deployment is using Kubernetes and it is orchestrated by Ansible Playbooks. So if by server cloud manager you mean something like Cloudify or other orchestration platforms that orchestrate the installation of Kubernetes right now we are using Kubernetes but we are not using the management orchestration software like Cloudify or us. Great, thank you for that. Dong is asking what I guess the end goal is for Orange when it comes to own app specifically something like closed loop. Is that on the radar? Oh, it's also a very good question. So we are actively working on closed loops on other scenarios, not for transport. But of course, we also want to implement closed loop for transport scenarios. But in fact, as far as the transport networks are concerned, if we want to talk about the closed loop, we will talk about telemetry. This is just a personal opinion. I'm not pretty convinced that the current DCI of own app is the best platform to manage telemetry at scale. But it's very good to have DCI for other just cases. So it's at this stage, it's more for our point of view, a lab topic rather than an operational topic, but it will be something next year. We will have to work on because it's very important if we want to reach a higher level of maturity in our automation transformation, of course, to have these closed loops and to try to figure out how to integrate some telemetry tools with the own app ecosystem in the same way we have done it with on simple playbooks to manage the IP or configuration changes for instance. But it's still an open question. Yeah, these are all really wonderful questions. Thank you everyone. Unfortunately, we are out of time for today's session as I mentioned we have captured all of the questions and we are recording the slides as well so what we'll do is we'll take the questions that haven't been answered yet and do some follow up to get that information out there. So stay tuned for that again anyone who's registered will receive a link to today's recording along with the slides and the one pager that was just published. And we are working on a longer form case study that will be published soon as well. So thanks for joining us today everyone have a good day. We'll see you soon on another webinar.