 Okay, so hello everybody So this session will be about open-stack Manila in a telco context. So I I am Marc Hodor Working at Deutsche telecom and I miss I'm an active contributor since the Havana release and mainly active in the QA and Storage sector so I have with me Thomas Lichtenstein from netapp and Christian five from SVA So the question is how how Manila fits in the telco cloud and Our idea was to have an evaluation How far the development of open-stack Manila is currently There and how we could use it in our production and production environment So we we were talking about the use case that we see for for shared file systems in the context of a telco we Automated Everything and build the test lab lab This is everything open-sourced so available for free and we will we will see and demo about the the automated test lab and an actual scenario test that we developed during our project So maybe a little To give give a little idea of how the team set up was so Deutsche telecom acted like a customer in that phase and usually if a customer wants to to onboard Let's say a storage system like from that up We will have an integrator in between so SVA is one of our integrators that we usually have in Deutsche telecom So we choose this kind of setup which which is a kind of traditional setup for building new services in inside of Insider in inside of an enterprise context so but open-stack Manila is something quite new and We didn't expect that it's stable enough to directly go into our production system. So We wanted to contribute our viewpoints directly to the community. So everybody of in our team directly contributed to To the community in the way he wanted to so we we deliver use cases and discuss them in the in the mailing list We identified Test cases and automated them and contribute them back to the community so and the the the main idea was at the end to have an idea whether we could use Manila in our cloud whether we could Design our architecture for the telco cloud that we have and shared file system storage in in our architecture So let's talk a bit about our use cases that we see and why Open-stack Manila is needed for us. So actually We have this this picture on the left just to summarize What kind of storage we usually have in the cloud? So if you're talking about open-stack you're you have block storage. So cinder is the the the manager of a block storage devices and So so with block block storage is nothing else as a hard drive for a virtual machine So you you attach it to a virtual machine So there is there's some drawbacks in doing something like that because you you as an infrastructure administrator Do not do you don't have any control about the file system that is Created in the virtual machine So something like snapshotting and so on is a bit more complicated if you do not have any control So you need you need if you want to do to create snapshot You need to control the virtual machine and the underlying infrastructure which makes administration effort quite Increased in administration effort so on the other hand we have object storage and Object storage is something new We usually have Swift or and storage with S3 interface The the usage of object is If you want to use object in your application you have to adapt the application so it's an restful API which is Scales a lot so it's I would say one of the future topics for for many telco applications That we see but they are there's a gap in between so shared file system are quite type of traditional storage and the the cool thing on shared file system storage is that The actual storage system has control about the file system snapshotting. It's a bit easier It's it's has less latency as much of the object storage so So this picture actually we created on Startup of our project and Manila was quite new and so we indicated in the open stack world that we have Cinder in place for block storage We have object stores with or any other storage that gives a swift and F3 interface and Manila was one of the missing pieces So Manila delivers an API to manage many other many types of Storage and this is the kind of thing that we need for our telco cloud So we are planning to to create a huge network of data centers across Germany and Europe and we for an NFV telco cloud we we have two types of Data centers that we plan we have front-end data centers, which will be close to the customer These are actually hosting the network functions. So network functions like maybe load balancing or firewalls and and Chaining them together Usually if you if you have this kind of context in a telco you have real-time requirements, for instance So these application Usually do not arise synchronously on to a storage But we have also back-end data centers and these back-end data centers will store the actual data for our data delivery For instance, so we do streaming and so on will be stored on these back-end data centers so our plan is to to spend this across Germany and Europe and One example for usage for for shared file system in this context will be our email service so we have the largest email platform in Europe and One idea would be that we store the mail header and bodies into the shared file system these because the the users of these shared files are of Mail browsers who usually are used to directly access the emails the email body and the header but When it comes to an attachment you can think about having the attachment in an object store and just stream it to the end customers So this is one of the ideas of the usage of an shared file system in our context so Basically now we are going a little bit deeper in what is what is Manila about And we then show what we are delivered to the community So I hand over to Thomas Thanks mark, so I'm Thomas Liechtenstein also. I'm working with NetApp. I'm a cloud architect based in Namia So to get everybody on the same page What we are talking about here today? So Manila is really The shared file service of OpenStack What does it mean? So shared file service means typically we're talking about NFS or NFS shares or SIFS shares But the concepts really apply in a very similar manner to HDFS for instance or other shared file Concepts that can also be addressed over Manila API control plane So why would you want to do that? So actually Back a few years ago. I think it was 2013. It struck NetApp that there was a you know this gap within the OpenStack Blocks or the OpenStack storage Story that there was actually a missing piece well like Mark explained so We asked our customers what kind you know how much of your capacity is actually deployed on shared file service And this number is surprisingly high. I don't know about you guys But we we see within the industry that actually when it comes to capacity around 60% or even more are stored on shared file services So it I think it's a no-brainer to bring this on into the cloud as well And quite recently, you know in the last year Microsoft with Azure file service introduced a similar concept and Service and now even you know a few weeks back Amazon falls you as well so We clearly see here OpenStack you know tapping into new ground obviously and I think Personally, obviously we have a heritage, you know, we can maybe come from object storage and block storage, but Shared files have been untapped in a way and Manila is really here to actually allow you to onboard Maybe homegrown solutions or legacy workloads, but also maybe new applications that were just not be able to build before that Right. So what could you use this for specifically? You know simple things like or centered things like home directories, of course But also big data might be just be a valid use case as well The good news is Manila is very similar Conceptually to Cinder. So just as Cinder allows you to Provision block storage Manila really allows you to Actually deploy shares, right? So speaking about the concepts So for one obviously there's there are the shares. So this can be any Protocol mostly that will be you know predominantly NFS or assist, but also the other protocols and Then we also need to because it's a shared environment need to think about access rules, right? so for instance for NFS you want to make you maybe and let your tenants define the You know access rules on an IP basis or if you have an active directory server deployed you might want to use SSS IDs We also need to think about how we actually tie the shares towards the neutral networks of the tenants or the Nova Networks of the tenants or The or other networking solutions that you may see in the future. So Manila is really Flexible in this regard. There's a notion of the share network, which is just solving that need There's more concepts. So if you are deploying Kerberos for instance or have other more advanced security services, this is also The be exposed can also be exposed through Manila or address through Manila This is nice property just as Cinder has snapshots Manila also allows you to do snapshots on shares and How this is actually implemented is actually done by the back end driver So this is the vendor specific implementation that you are pretty much well aware of from Cinder as well To this end Manila, which is very important is not in the data path It's a control plane for that for shared file services and it's also multiple back-end capable So you can run multiple back ends at the same time just as with Cinder So What I wanted to talk about now is actually well, what was the foundation of our work here? So when touch telecom invited us for this joint evaluation We quite early on decided that we wanted to do this evaluation in the open source community. So we actually built A fully automated test lab based on very common components that you are very well off. So for instance You are probably very familiar with death stack and uses maybe with vagrant. So we we're just using the same tools to deploy an open stack based environment where Manila was enabled and In addition to that in the same manner that you run the death stack in an automated fashion We actually added the back end configurations for using a generic driver, which is the reference implementation for Manila, but also Netapp Base back end using our simulator. So in the end of the day, this allows you to actually build and Actually verify what we are talking about today on your own and by just pulling this The source code of the or the configuration files for this automated lab and run this on your own laptop This was the foundation. I want to quickly give you an Head start on how this looked like by by showing you a demo It's a video where so it's on based on github so you can check this out and actually You know deploy this on your own. You can also use the generic drive implementation as a back end So in this case, we're using the net up back end and the familiar workflow is actually to you know clone it and then issue a vagrant up and You grab a coffee basically, but In this case, we will do a fast forward We will deploy service VM, which will configure the net up back end Just the court with some DHCP and networking configuration that we do in the background and Once that service VM is up. It's actually a Ubuntu based image that is just pulling You know package to package updates from from source It's also making use now of some Chef automation and as the next step once the service VM is up We are going to deploy a net up back end. So using our simulator. That's the net up cluster data on tap system as a virtual machine and Once the simulator is up and running so we can also orchestrate that through an IP a base protocol so through what we call the SAP calls This usually takes six minutes, but we we fast forwarding now It's actually kind of like the fastest death stack run you ever saw I guess So now it's the next thing we run the usual death stack Which means we are rolling out open stack where with the specific capability of and a manila being Enabled and also be configured to speak to the in this case to the net up back end Again, if you want to you know validate this on your own terms you're free to check this out as well, right? and With that I will turn it over to Mark again who will go into more details of what we did with this lab yeah, so The question is now we had we defined our use cases that we want to to see in Manila that it's really working and Now the question is how can we test this and so we created this automated test lab, and we could do anything manually but but now Since I'm heavily involved with the tempest team. I I Thought that it would be a great idea to to automate these kind of scenario tests So we discussed with the with the manila core team about that and this was actually something that is was missing in the QI QA process Upstream so when it comes to to quality assurance in an open stack modules You there's usually three kind of types that you see so you have usually unit tests that are just focused on Functional block in the code so these Functional blocks were executed, but it's not that manila itself is completely up and running It's just mocking everything around away, and you just have to test the functional blocks So this was already nearly with I think 90% coverage in place So we didn't care about that this one. So and then after this we usually have and Another approach to have API tests. So with that you really have a running manila back end and you're you're trying to Test certain kind of API functionality for instance creating a share is one API call And you just check whether this runs or not so but what what is missing with that is an end-to-end behavior So creating a share is nice But you have to attach the share to a ritual machine you have to mount the share in a ritual machine And it's right actual data on it then it's a real life and to end test So this is actually the thing that we did in the scenario tests. So The these tests are These tests run every code change in manila now. So they are introduced to to run In a non-voting job. So I think for liberty one. We will we will Activate them as voting that means nothing will come through the through the Through the gate that doesn't have running scenario tests. So these are the tests that we actually delivered So I just wanted to show the the actual code how such a test looked like and the actual Scenario tests are quite abstract and quite easy to read So we see here in the first line that the security group was created then as shared network So we already heard from Thomas about the concept of share networks So it creates it creates neutral networks and assigning these networks to to manila Then it creates an actual share boots an instance allows us access and then Tries to SHH individual machine and checks whether this works. So this is this is Let's say one of the the the easiest cases that we had so in the in the current code We have one high-end test with two virtual machines one virtual machine is writing data The other virtual machine is reading data. So this is the real end to end test for for let's say a basic manila check up so Yeah, so we want to to show you a bit how this works and we have a live demonstration how these scenario tests are actually look like So hand over to Christian. Thank you mark Yeah, let me just quickly introduce myself. My name is Christian fine. I'm with the SBA in Germany as a systems engineer for storage infrastructure and What I want to do now is just just try to map the codes in a bit that mark just showed in two way Let's say high-level picture to just explain what the demo will consist of and Yeah, we have our open-stack cloud with manila and neutron and Nova Probably there are other components involved as you know, but our test mainly mainly revolves around these guys and When we start our tempest test it will let's say create a base That means the share and therefore the neutral net the manila share network in the share server and after that it will boot an instance since all our tests work from a Let's say client perspective and this instance will then try to mount Yeah, the actual chair and if that succeeds it will after that unmount and clean up the whole environment So that is what what the demo will look like Let's just Look in it. Okay, so what we have here is a Yeah, naked. Let's say naked. Oh stack environment. There's no instances and no shares deployed at the moment and What we will do now is run the actual test So we'll advise Tempest to to continue so what we did is we we did some Yeah minor modifications to the test in order to enable a stepwise walkthrough normally this test would just run through without any manual Intervenation, but this is for for demo purposes And yeah first step is creating the share network as you can see in the background there's something going on in the topology and After that we create the actual chair since we now have our base our neutral net the manila share network and The next step is now to create a share You can see in the background the topology has built up and so this share server creation needs some some sort of some seconds We can just wait for that Okay, you see that the shares created and the next step is to to boot a instance since we need to to have a test candidate that does all the tests and Yeah, after we booted the instance We need to allow access for the share and also that we can access this instance with SSH and What we can do now is just have a look again So you see there is our instance in the in the correct networks And we can also have a look at the manila UI, which is some sort of the the manila GUI in in horizon and There we can see we have the share networks in place that are created by the tempest test and also the shares and if we drill into that we can also see that there is already a export location that is the actual mount path for the share and If we look a bit down then we can see that there is already a access rule defined which allows our instance to to mount the the actual chair Okay, so let's go back to our topology here and Continue with the test. So since the base is all there We have the instance we have to share and so on the next step is at least our first test So this is the verify SSH test. What this test does is it connects Onto the VM and tries to ping the export location with that We just try to find out if if our environment is is okay, and if the whole networking stuff is there in place and Tempest is Yeah works in a way that it that it retries to SSH onto the machine until everything is settled and as we can see This verify SSH did work so it was able to ping the share server and so the next step would be to to actually mount the share on on these instance and Yeah, you see it that that also worked So there's only the last step left that means unmounting our share And if you have a look at the background Tempest will now clean up the whole environment and delete the instance and the whole networking stuff and there will At the end nothing we left there and Yeah, what what tempest at the end does is it prints out a time? How long it how long it worked and what is the the overall status? So normally that should be okay, but could also be the case that is in error state, and then it would just print out Yeah, what what the actual error was Okay, so far with the demo I would know like to Yeah, to talk about the the evaluation work we did and what what were the findings so we called it what is working what is missing And before I go into that I just want to say what about the community so when we started with the Manila evaluation our Let's say first point of contact was actually the community and we use the tools like github and RSE and launchpad and Probably all you here are quite familiar with the tools And I do not need to to explain how they work since that would also take separate sessions But what I wanted to say or what I want to do is encourage everyone that wants to get in touch with Manila or OpenStack To participate in the community and then do not be afraid to open bugs in and in launchpad there will nobody fight you or so you if you if you do that and Even if you do not want to open bugs You can just ask questions in the IRC chat and what what we found out is that there was always helpful guys that that gave us information And yeah, so we like that pretty much Yeah, again what is working what is missing we Have a little yeah overview about that First of all this generic driver at the end provides storage at ease So to be honest if that would be standing on the right side, we would have at least big trouble there So but happy to have that on the left side We also have as you just saw indeed in the demo basic integration into horizon that means we can create and display shares It would be nice to have a more advanced integration But there is this nice Manila UI project and there's a lot of stuff going on So we will we are sure that this will be improved further And what we also have is some sort of basic error handling or protection against minor failures That means if we for example do not have the correct license in our system or have not enough back-end storage or whatsoever Manila would during the share create stay in a clear error state and not have some sort of fuzzy creating State that we need to clean up with whatever hacking. So that's quite nice. It would be Even nicer if we would have a more consistent error handling in order to maybe have the driver to give some sort of Yeah information back to Manila. What was the reason for the error? For example license not found or no space left Yeah, we would like to have that Same goes with documentation There is documentation available there on Manila wiki and also there's documentation in the code But yeah, if you compare it for example with Cinder, it's yes only little documentation available To continue Manila is at the moment aware of multiple storage back-ends, which is very nice You can use a mixed mixed storage strategy with the generic driver or with net up and other Windows drivers With that you can also make use of different storage classes That means you can provide shares from SSD storage or from zeta storage And if you have a nice storage box that is capable of thin provisioning or other fancy features you could also use the the share types or extra specs to to make use of that and Probably some of you are familiar with net up systems or cluster data on tap even there It is possible to use the advanced network features on this systems. That means having multiple star network interfaces That's quite nice and additionally to that you have the advanced security features means you can attach your system to a directory service What we would like to see at least from a customer point of view is the ability to migrate shares in and between different storage back-ends We know that this is a very hard job and it's not not easy to implement that But it would be nice to just have a talk about that and and maybe get the chance to to improve that or to to establish that Same as with resizing shares. So we would like to have resizing But yeah, again, we have multiple storage back-ends and yes, some of them are capable some not and I think there is just a design choice needed if or if not Yeah, that party vendor testing is also a interesting topic From what I know NetApp has I think a week ago established that so they have their the testing environment ready Hopefully the other vendors will try to catch up with that and then also implement that and Yeah integration into heat Would also be a nice feature and from what we know it will be available in Liberty milestone one Yeah, so when we Created the project or started with we we had some or Deutsche Telecom created use cases from from what they want to see from from Manila or how they would use it and I would like just to go through the use cases and Check if Manila is capable of fulfilling them So first one is quite easy create a share and use it in a tenant that that works actually We can also assign pre-existing shares to to other instances for example with just extending the security rule or the access rule we can Yeah consume Share or storage from even from instances that are not mediated by Nova We have the ability of cross tenant sharing in place we can at least Deactivate the share that means we did we detach the share and do not destroy the data So that makes sense if you for example from legal requirements need to keep the data for a year or a month And yeah, do not clean them up directly But at the end you can also delete the share means detach it and then destroy the data to free space What is not possible and you see it with screw driver We can actually not resize the share and we cannot Yeah Copy or my great existing shares at the moment, but all in all that's from my point of view a good average that Manila is Capable of and yeah with that. I would like to give back to Mark. Yeah Thank you Okay, so the main question of our project was is Manila ready for the telco cloud So it sounds like an easy question. It's it's not easy to answer that The maybe the answer is that it depends a bit so on for for the telco cloud or for enterprises It's quite important to get to get maintenance by an Distributor so what is currently missing for Manila is that an enterprise distribution? package Manila as Part of their product. So that's why we can't use it in the current state for our production the other thing that That needs to be considered when talking about Manila is The quality depends much on the underlying driver into implementation. So we see here a different Kind of quality from from vendors So if you're considering having Manila in your production, you have to have a look at the driver And and really test the things that you want to use there So in to give a little bit conclusion in a summary so overall Manila Exceeded our expectations. We see a really responsive community of many vendors Really important for us as telecom is that we are not in a silo and and locked for one solution so this is exactly what Manila gives us for for shared file systems, so Quite often where we discovered bugs there was already in solution in in the community Available and so this was really really great experience So further steps for for this effort is now we want to try so we have now built a completely Virtualized environment to test these things now We would like to see how it behaves on a real bare metal environment with a real open stack installation not based on dev stack So to be more productive like So another thing is establishing performance tests, so it's not only about performance of the storage It's also we can stress test Manila to see whether there are race conditions or not So these these two aspects are quite Important for us because we want to manage many multiple data centers And it could be that there's some kind of load and race conditions are usually something really bad Which happens quite often in an open stack Another point for sure we need Manila H a this is something where we currently not having a closer look But for sure, this is something that that needs to get let's say described If you want to use it in the production Another way what we are currently thinking is Improving this in our tests for sure. This is we have an initial set now from scenario tests But we need to improve that one important thing for us here is Having heat integrated tests with open stack Manila So thank you, and I think we have some time for questions if there are some Yeah, so this is something that is currently on the discussion how to to get let's say the location of the Share that is created into the virtual machine what is currently is it's a complete manual job So for the automated tests, it's quite easy because you create a share You have the information where the export location isn't you just give it Into your SSH command, but for it's a real-life thing You really need to to have something like whatever agent or other technology to to to mount it Yeah, but I think it's the same thing for Cinder because you have to mount your your volume that you attach And you have to know where it is and so on so Yes, yeah, this is I think one of the design session that we I think already had and Yeah, so this is something that is currently in the discussion how to do that Please use the microphone at the back of the room for questions, please. Yeah, all right So if there's a question or is someone running away? So you have a keen interest in Tempest so do I I was curious if there's a possibility to As you get closer to production if that if those real-world tests could be somehow merged back into Tempest so that vendors of the drivers have a better idea about Where's the bar that they need to hit so so the test that we actually see was just a bit Adapted but it was it's already available in the code and and for the third-party vendor tests They should they should use these tests as baseline to see whether it's actually working or not Correct, I was interested in I guess what you would like to do with Tempest next and how that might look Where you bring the the real-world production Meets Tempest. Yeah, I mean do you have ideas about that? So it's so much so our idea where we define this use cases and we are started to to create test cases From these use cases so and this is actually what we are trying to do I mean that the thing is you can't do a lot of heavily complex tests tests like having 10 VMs And so on in the gate because because this kind of test will take a long time and you cannot do that regularly so For sure we could define some additional tests for production environments In a rape or what in a rape or what not in the let's say actual active scenario tests Understood. Thank you. Okay Great. Thank you