 Okay So it's time to start this session. So good afternoon. Good evening everyone. So thank you for this Thank you for coming to this session. So the last session today So we are from an entity data system integrator company in Japan So last autumn so the last October we had OpenStack Summit in Japan So my home country and I spent a very exciting time with you So and I'm very happy to join again to the OpenStack Summit here in Austin So as mentioned in some keynote sessions like Jonas and one So now OpenStack is coming to the next stage to support diversity of IT systems which running on the OpenStack So today I will share you our experience in our Swift project So especially putting focus on the problem we faced So I'll talk about our real use cases and how we try to integrate Swift to Some customers especially legacy system And I'm happy if it can be some help to enhance the diversity of Swift and Swift users on the Swift community Okay, so let me start with the disclaimer of the presentation So I think it's very normal one conventional one. So in this presentation So we were going to provide the knowledge gained from our system integration project using Swift So we are going to share our experience of what we faced or what we learned in our presentation Hoping this is a help to some people who is now planning to use Swift into in their system So before talking detailed content, so let me shortly introduce ourselves So the three people or platform engineers and the data. So I'm Takashi Kazunami I'm I've worked in the Swift and OpenStack project entity data I'm Masahiko Nakagawa OpenStack engineer at MTD data I'm Masahiro Ikeeda. I worked for OpenStack Swift Docker container and IoT platform Okay, so as I talked in the beginning so entity data is one of the system into greater company in Japan So we we do not provide any service Using OpenStack, but we provide IT systems for our customers and also develop some features Which is required in our customers use cases. So we three belongs To our open source professional sector in entity data So working about something like cloud technology like OpenStack Swift, SwiftLock and Docker and so on and on the other hand We are also working about the data processing technology like PostgreSQL, Hadoop, Spark and blah blah blah and we are especially We are standing here Especially responsible for the cloud platform using OpenStack technologies and working to provide Private cloud by OpenStack and cloud storage using OpenStack Swift. So that's a reason why We are going to talk about our Swift project Okay, so here is the agenda for this presentation so first I will show to explain what is Swift and then the reason why we use Swift for our system integration And then so Masaki and Masahiro explain what kind of problems we face in the real Swift project and Introduce our approach in this in this project with some detailed information about the two real use cases and Finally, so I will summarize and share what we learned from these these cases Okay, so Swift so I cannot see Many so like is so bright so that I cannot see many the face of you But how many people know about Swift here? Yeah, I know you are very good and how many people using Swift or now provide have provided a system using Swift Okay So let me explain shortly about Swift So as you know what you may not know so OpenStack Swift is a part of OpenStack project and especially storage project and Swift realizes distributed object to switch like something like Amazon S3 So object storage is a new style storage with different interface from the conventional block storage or file system so Particularly so Swift provides less for API and the clients can upload data into storage as put request and download data into From the storage as get request over HTTP protocols So this rest API works on HTTP protocol And so Swift is often used at the archiving storage for the web contents like photo or video or backup data So Swift has many many good features, but I don't know I don't have so much time today. So I'm going to explain Some key these slinky features so the durability and the scalability and the openness So the first key feature of Swift is the durability So Swift creates some copies of the data in the storage cluster for example three copies it may create that three copies of data inside the cluster and This will distribute these copies over devices, nodes, racks and data centers So even if some parts of a switch cluster fails you can protect data from these failures and You can also continue to access to all data with remaining copies So in addition so Swift automatically detects disk defeats and here missing data copies and other device working properly and From greasy and haven't released. I think very long time has passed since this gets released So Swift gets global cluster feature. So Fritcher enables a geographically distributed storage cluster over multiple data centers to realize disaster recovery and I We are going to talk about this topic but the results are recovery is one of the topics most interesting to especially Japanese customers after our experience of big disasters like art cake tsunami and so and Second feature at the scalability So switch to distribute a state over multiple devices as I talked before and the thing you add the new devices It will balance data to the new node So we can enlarge capacity and improve performance of a swift cluster by adding a new swift nodes And we can extend the storage from small capacity like 10 terabytes to huge capacity like dozens of petabytes So I know there are many people who is running the modern But they came dozens of petabytes swift cluster that in addition So there are no limitations on the number of devices you can add so we can extend swift cluster very flexibly So you can add the capacity as much as you need when you need and you can adapt your storage to your What's the undictable market situation your undictable business situation with effective cost So the last one is the openness so Swift is the open source So Fritch works on the Python framework and you can drive it on the common Commodity IU servers and Linux So you do not need any special devices for swift to run swift And you can select cost effective hard wheels to construct huge storage in addition So you can flexibly mix mix up some types of services in a single cluster So you can add latest hardware to existing cluster consist of old servers So you can have the latest Servers when you extend and on the other hand you can remove all servers from your cluster when they get broken after the maintenance period So you can keep your switch to cluster for a very very long time regardless of the maintenance period of servers Which is which causes some problems for Previous or conventional systems if it's replacing old servers to new servers Okay, so now I've explained about three key features of swift So switch to realize a very durable and scalable switch and makes us free from any vendor lockings Okay, so Swift is great. So Swift is the best solution to realize durable scalable and open-storage platform but in many project especially in our many project many of our projects So we faced a difficulty when trying to integrate Swift to the system. So today we will show some example here So this slide shows what says hot open happens with us So one day sales guy comes asking me hey our customers is now planning the replacement of their existing Storage and looking for the way to save data their data from disasters. Do you have any good solutions? So I will answer to him. Okay, Swift is the best solution for that So it has very excellent global cost of future and they realize Geographically deep replicated storage as I mentioned So then the sales guys is going to Yes asking does it have NFS wise because it was something like file system interface No, we do not have any file system like interfaces So in many many cases our cost they are not so many Customers who are going to replace whole of their system So they are going to replace the stretch, but they would like to keep the existing system and such kind of Existing applications Does not Support the rest API and they are very very often require a conventional file system interfaces So this kind of interface program Often comes to us So Yeah, so this interfaces problem is a very big problem for us So search is a very very good solution to realize disaster recovery or massive scalability Low operation cost using a self healing feature for this failure But however as I mentioned so many customers don't change the applications. They are existing their Applications running in their systems today at the same time and these applications don't support the rest API in many cases Okay, so I'm so we are going to show some cases About these interface programs So We are going to explain to use cases. So Masaki and the Masahiro is going to you to explain to detailed use cases to In which we are tried to solve that interface problems Thank you. Takashi. I'm Masaki Nakagawa they are I'd like to introduce to use case in this presentation And the first case are using a swift as backup storage of legacy backup server and After me as a hero will Explain another case Okay, let's start a first case Using swift as backup storage of legacy backup server One day our customer came to us to talk about new project About new project overview Upgrading backup system for public agency this system backups Barrier size data for example user data user virtual machine image virtual machine host image Supporting contract of backup storage of this team is being expired. So user want to change storage to suit This project's goal is changing backup storage device user requirements Backup data size is from a hundred terabyte to one petabyte Next backup data is stored from much data centers These data centers are located far away each other Backup store the backup data should to be stored randomly Backup storage should to be placed in multi data center In addition backup system uses proprietary backup software And the user want strongly required required that Backup software is kept to use. So we should not change anything without storage From these requirements We define some points for design These are points From requirement backup data size is from hundreds terabyte to one petabyte this this scalable Next data is come from much data centers and this centers are located to far away each other This is this is much region Next backup data should to be stored randomly And the backup storage should to be placed in multi data center. This is global cluster We thought that these points are satisfied by Swiss So Swiss it's a wishing Swiss it's good for discuss the now Next we discussed We have discussed Swiss integrated system design This figure shows that system design overview, which is satisfying design points for convenience This figure set two sites Oh User system backup system backup server switched proxy and Swiss the storage is deployed each site Backup server gets back up data from user system Backup server put or get backup data to or flow each proxy Which deploy the same site? Swift proxy is set affinity configuration to operation Operate put or get data to save site to switch to storage and Swiss to storage is set region region configuration each site individually We thought that this design is enough for a requirement but But unfortunately this design was difficult to realize Because because of low compatibility of the backup server and the swift Backup server does not support swift API and This server requires the server specific virtual type device on block storage To use swift as backup storage We have to mount swift to local stress file local file system as a block storage Like this case we often received RFP RFP from customer that using swift as backup storage of backup software Which does not ready for a swift Use case of backup storage is sweet for swift But we have to give up to use swift swift because by legacy backup software To avoid such a unbearable bearable give up we should prepare something workaround As workaround we have Experimentally retried to mount swift to file system by using cloud fuse. I would like to introduce it Cloud fuse is OSS which enabled swift to mount as block storage to file system If you create delete update files Cloud fuse changes this operation to swift API request If you want to know about detail of cloud fuse, please Default Website in this case we set cloud fuse like this figure By this architecture we succeeded to mount swift and make backup server Make backup server with mount point So we try to backup process to get issue point of this architecture by trying We get two issues to proceed backup process The first is fail initializing virtual tape device Virtual tape device initializing process creates temporary file and rename it But cloud fuse does not support it support rename operation So to achieve this issue We need to improve cloud fuse or choose as a same component like cloud fuse or do workaround In this time we choose workaround create virtual tape device at other location and move it The second is swift does not support append object data Dueling backup process backup server append backup data to virtual tape device But swift does not support append operation To achieve this we use DL plug-in We can apply DL plug-in to Append operation By achieving these issues We have gotten one pass to backup But if using this architecture for a commas we need to more detail analyzing summary of case one Requirement we would like to use swift as back end storage of legacy backup system Issues backup software does not support swift API Second backup software needs to locate virtual tape device on block storage device required attempt mounting swift to file system by using cloud fuse Result we have got to succeed pass or backup workaround Avoid feature of initializing virtual tape device and using DL plug-in to support append operation That's all of case one and Next Guess two okay Second case is to use swift as storage for file server We are asked to renew file server which has access from users in geographically separated areas There are three requirements first users store data in local storage without any overhead with replication Second all users can share same files Third supports file set file-based protocols to write various size of files Let me show the details of requirements first requirement is that users store data in local storage without any replication Without any overhead of replication to keep the latency of access low Users are located geographically separated areas and users store in local storage So swift so storage needs to be located in geographically separated areas like this picture But it is required that all users can share the same files So it is needed to bundle local storages in one virtual storage If there is one virtual storage users can share same files After user in EU writes a file in local storage It is replicated to all other storages and users in North America can read the file Third requirement is to support file-based protocols such as NFS or SHIFTS to write various size of files The reason why customer requested is to reuse the existing many legacy applications and to avoid high-risk and high-cost developing new applications To use storage as a back-end of file server accessing small files in low latency is required These are the customers requirements From these requirements we compared three storage software first row expresses requirements second row expresses futures to realize requirements third to fifth rows are the name of comparison of software We chose safe cluster FS and Swift which have global cluster future This table shows Swift fulfills most requirements Additionally, since we have knowledge of Swift we planned to use Swift But Swift has two issues for requirements First issue is about file system interface All those users need to access by file-based protocols such as SHIFTS and NFS Swift supports only less API Second issue is about small file optimization To use as file server accessing small files in low latency is required But Swift is not good at processing many small files because Swift is not optimized for small files So how to solve the issues? This table shows issues and solutions second row of the table shows our solutions We solved first issue to integrate cloud storage gateway We solved second issue to add two futures data management with small block size and storage cache First solution about file system interface is to integrate cloud storage gateway This is a gateway software which translates could cloud storage APIs such as REST API and standard file-based protocols Then users access to the cloud storage gateway by SHIFTS or NFS The cloud storage gateways transforms user's request to REST API request After Swift responds the request the cloud storage gateway transforms response of REST API to SHIFTS or NFS response Users can access the data in Swift by file-based protocols This is a solution for file system interface issue Second solution about small file optimization and data management with small block size and storage cache Most cloud storage gateway has these two futures First is the future which manages data with small block size So the latency to access many small files randomly is optimized Second is storage cache which stores data which users are accessed If cloud storage gateway has the file user accesses in storage cache already It returns the file to the user directly Since there is no access to Swift the latency is very low So does combination of Swift and cloud storage gateway suit the requirements? Also cloud storage gateway solved the two issues It is important to confirm if cloud storage gateway has the future of clustering Because this future depends cloud storage gateway If a cloud storage gateway doesn't support the clustering future It distinguishes the benefit of global cluster which all users share same files We used for us cloud storage cache, which is one of cloud storage gateway It's proprietary software This has many futures Of course, it has the futures data management with small block size and storage cache Another notable future is loosely cluster Lusory cluster is the future to bundle file systems of for us cloud storage cache with virtual once file system Fitch enables to share same files in multiple locations in cloud storage gateway layer For us cloud storage cache in multiple locations replicate metadata respectively So our users in North America can read the file feature user in Europe creates Since for us cloud storage cache has clustering future the combination of Swift and for us cloud storage cache can realize every requirements So we use it This is a simple architecture Each location has swift and for us cloud storage cache Since there are clustered and replicate data all users can share same files We did performance test The table shows a result of compare comparing with the performance of ordinary file server I'm sorry that I can't share specific performance We got the result that the performance is enough good for file server when the protocol is NFS When the protocol is shifts the performance is not so good But we can use it as file server which that performance is not high priority I want to share three limitations to leverage the solution First is that the performance of shifts protocol isn't so good. I Think the result is that the performance is not so tuned so far Since cloud storage gateway developers tend to focus on flexibility scalability and availability As cloud storage gateway software market becomes majored this problem would be solved Second is that the performance becomes much worse when data is not on cache Of course, the reason is that latency increases by that of Swift Because cloud storage gateway gets data from Swift It is important to design size of storage cache carefully It is a key that to compare performance and cost efficiency while thinking the tendency of data usage Third is that there is a good and bad point of integration The good point is that we can get software specific future such as global cluster The bad point is that The availability becomes slower because possible failure increase while the number of components are increasing At last, let me suggest an idea The market size of cloud storage gateway is increasing rapidly The market size would increase almost 8,000 percent by this year if compared to the level of six years ago So I think the demand of the combination of Swift and cloud storage gateway is increasing too Although pro-fliatory software has many useful features. It is not optimized for Swift How about developing cloud storage gateway optimized for Swift? In my opinion, there is a certain demand summary of second use case This case is integrating Swift as back-end storage of file server Which has access from users in geographically separated areas Because Swift doesn't support file-based protocols and is not good at processing small files in low-rentency We joined together Swift and cloud storage gateway to realize requirements The performance is enough good for file server Since Swift is superior to other software in global cluster future To realize file server which has this feature has the future Swift and cloud storage gateway is very good combination Okay, thanks, Masahiro So we So these two guys explained about the real use cases and how we solve the problem we faced So we I'm going to summarize short read what we run from these two real use cases So in the case one so in the case one and we use the storage Swift at the storage of the existing backup solutions and the case two we used Swift at the back end the storage of the file-shelving service So both of the cases we use is something like a cloud gateway So that's the interface transition So in case one we use cloud fuse to convert the rest API to something like a fuse so Linux file system interface and in the case two we used for overseas It's a kind of proprietary software to combat the rest API to the Convention of Swift's NFS file-shelving Interface so other results so as a result over some so Projects so we we started to think about to the cloud storage is very good starting point to integrate Swift you into your Existing systems, but of course as Masahiro mentioned so it often put a remittance of such kind of a cloud gateway It put our limitation on the benefit of given by Swift for example So we put more layers on the top of the Swift so the the number of possible failure points is going to be bigger and bigger and Also on the out so we are now trying to use conventional application But on the other hand we if we can use cloud native application We can get all benefits of the Swift Just existing reality into this solution So our future reason so yeah I think the cloud storage is very good starting point But of course we know the that is the not so totally ideal point But I think so we think what is the most important is to defining It's defining and showing some successful something like a migration stories from the conventional conventional software with conventional storage to cloud native applications and Swift something like a new cloud technologies. So for example, so We should share about it. I think it's better to share the starting points So we can start with that something like cloud storage solutions to integrate Swift in existing applications and we also Should write some migration path to a wide you go so something like a cloud native solution So for example for for the backup solution dear some back already backup solutions Which can natively support Swift so at least so we can bring of At the first point that we can bring the some of the benefit of the cloud technologies and in the future We can get the all benefits of the quality technology after migrating to the totally cloud native applications And in addition so as I said you mentioned so they are We think that they are possible improvement about the clouds to educate we and we think the clouds to educate We it's very useful at the mic starting point of this kind of migration. So now now we are trying to some possible improve improvement about stability and usability and Of there are some other for the technologies and the conventional interfaces like NFS Swift to improve the scalability or availability. So I think we can integrate them to realize more one the more efficient System using cloud technologies Okay, that's all for our presentation. So I think we have some minutes still so Are there any questions? I think many people are Looking forward to go to the party Thank you for the great presentation. So Just a question so you Describe the The customer doesn't want make Doesn't want to change Their application at the same time but so you can provide you provided the solution to integrate No, we've out without application change with this so Industry I thought That is a midway of the integration. So I don't know what the actual customer requirement, but I wonder if In that this way we can migrate Whole application into the sift native after this migration ended Yeah, you're right. So For example in the backup Example so we can add the next bit we will replace the existing backups so to be used to The new backup so to be as which can support swift natively for example Amanda has something some support of the cloud storage interface to Add the back end switch and I think in the second Examples of file sharing example, so they are not using the sifts or NFS or something like conventional file sharing interface But I think they are not other ways to realize For sharing so something like a dropbox like interface. So we did not have to access to the sifts, but we can Add some verification from the we can put some verification between our local machine to Declare this and in that so now on cloud or some other Fatsies or dropbox like softwares can natively support space. I think that the Possible second steps or the last steps to the total a totally cloud native system Yeah, I'm looking forward to hear the successful story for that. Thank you You do any other question said that To simulate the append operation you need to enable DLO was this An option in cloud fuse or did you have to write something extra yourself to? Get that in I think we created a special middle view to into intercept the request to create the so intercept the request of the Creating a new big object to the creating Faces a spriti to the segment and create a DL. So we added a new new middle view to do it automatically because it is not so Easy to change the existing backup software. So we do it inside the swift Okay, so it's in swift and not in the Mounting of the file system Yeah, yeah Hi, quick one You showed that you actually have multi-cache. You have NFS Running between two cache US and Europe and at the same time at the back and you have Swift I wanted to know how you handle. Let's say if there are simultaneous access from NFS At both these sites or let's say SMB How does that play in? Yeah, so yeah, very good question. So the family put the cache and In the multi-cache so the collision of the update is very big problem. So in this case is we are the we use is Cards of strange gateways feature so it can it has for the detection feature of the college Updating collision. So so we can eat Raises our art when detecting that collision and Also speaking, we can fix it manually after detecting that So basically then eventually the last person wins kind of Is there any security issue facing all the kind of the how to say access rights or something like that? Yeah No, so yeah, I think that depends on the cloud storage gateway and This card stage gate. We have some let's say the control about Has enough access control to so it manages so they it can manage For the accessibility of the data. So they are not so problem in this cases But of course, they are not so many they are the cost of for example, the Crowdfuse does not deal with that kind of access control. Yeah, that depends on the software you use I think time is over. So if you if you have another question, please come to us