 All right, good afternoon everyone and thanks for coming here. We hope you enjoyed your coffee break so far Today we're going to talk about the Swift deployment at Turkstar Personally, I'm working on Swift for some time now and mostly upstream but I'm also have some DevOps background from my work before that and so in 2014 Turkstar reached out to me or to Red Hat especially and to discuss Swift and some Swift deployments That said today with me are Orhan and Dorok. Please introduce yourself Hello everyone. My name is Orhan Bikloulu. I'm working as a senior systems administrator For Turkstar for more than 10 years now Hi, my name is Dorokaksoy. I'm also working with Turkstar nearly five years and Responsible for the open-stack environments operations All right. So before we're going to talk about Swift itself we're going to have a short introduction to Turkstar and what Turkstar is doing and Yeah, I will tell you a little bit about it. Thank you I'll try to give a short introduction about Turkstar who we are and what we are doing Turkstar has actually been known as the leading mobile operator in Turkey and in the region but currently we are defining ourselves as a converged telecommunication and technology services provider and We had been founded and headquartered in Turkey, but we have group companies in nine countries and In these nine countries, we are providing our clients voice data television and video services on both mobile and fixed networks Currently we are serving more than 66 million customers in these nine countries and By the way, Turkstar is also list has been listed on New York Stock Exchange and birthday stumble since year 2000 and Is the only New York Stock Exchange listed company in Turkey, sorry too fast As a technology services provider, we are also providing cloud and data center services to our customers as well In June this year. We have We have opened Turkey's largest data center actually in gives a near Istanbul We are planning to provide all of our all kinds of class services in this data center to our clients Also, we would like to serve global internet companies as well as the Turkish public and private sector companies This data center is a state-of-the-art tier 3 certified data center it's spread over an area of 33,000 square meters and it has a white space area of 10,000 square meters We are also Building a similar data center in Ankara, which we will be using as our disaster recovery center And also we'll provide the same services there to our clients As part of our class services initiative we have created the product called life box Which is life box is a cloud is a public cloud storage offering for the end user where you can store and Share photos videos and all kind of documents The product actually has been called Akılı depot, which is the Turkish name for smart storage Kind of but we have recently rebranded it as As life box Because of our global initiatives and because of our operations in the other countries It's now called life box the product can be reached at my life box comm and It can also be used on mobile devices as well We have a client apps for iOS or an Android devices on the app stores Where you can get the app and start using our product As you know, there are many different public class storage offerings out there So the question is what was the motivation for took said to build such a service? as I've stated we are currently serving more than 66 million clients already and a great majority of these clients are mobile users and a large number of these mobile users are actually using smartphones Since smartphones are one of the primary reasons why people are producing so much data nowadays people need some Sort of some clouds Storage space to store their extra data because as you know the capacity of our devices is never enough We are taking too much pictures shooting videos producing a huge amount of data and our clients We're always complaining about the capacity of their devices as the Network operator and also the as the cloud services provider, which is the closest to our clients We provide we wanted to provide them our own solution where they can use Easily and we also wanted to provide them a solution at very competitive prices Is the very competitive provides prices we are Alright, let's Let me explain about how we started this project and how we implemented the solution actually we started this project in year 2014 and back then we Implemented a solution from one of our vendors. It was a legacy integrated solution this solution consists of large enterprise storage boxes and a gateway application in between which Gateway between the storage devices and the clients after a short while we decided to implement our own solution using open stack Swift in In 2015 we decided to implement the solution using open stack Swift But back then we had no previous experience with open stack or Swift So we decided to get some consulting from our vendor reted. That's when Christian came into picture He was he has been the console. He has been our swift consultant since then With his support we have designed and implemented our solution from scratch within a couple of months and Put it in production He actually he not only had to build design and build our infrastructure, but he also had to He also given guidance to our developers how we can actually implement the application using the swift API's Currently our product life box is actually running on open stack Swift and it Has over 3.3 petabytes of storage and we have more than 3 million uses Actually while Migrating from the legacy solution to the open stack Swift We did something different and we We wanted to keep our existing investment in those enterprise storage boxes and actually use this enterprise storage boxes as the Backend disks in our swift environment. I think Christian and Dorg will tell more about this But this is one of the differentiating factors of our swift implementation Okay, now I would like to give a short insight about the look and feel of the product here You can see the screenshots of our mobile client it's Using this client you can browse upload download and your pictures videos, etc. And It is a decent UI and also we have the this is the Screenshots for the desktop version where you can which you can access via a browser And use pretty much the same features All right, if we return back to open stack as I mentioned we have built our solution using open stack swift Open stack, but open stack is a very big framework of Very large ecosystem and there are many different components For this project. We have specifically used swift and keystone and that was enough for our project Now I would like to let Christian explain how we use swift to Create our product. Yeah, thanks, Ohan All right, so planning the swift deployment as on said I came into play at the end of 2014 and At that stage there was a premier previous solution already in place So we had to find a solution to make the migration as smooth as possible while being in production and So we had some discussions around this So first, let's have a short look why swift is actually good use case in this case or the good fit What we're storing is unstructured data, right? Your clients are don't Don't care about where you store the data. They don't need a POSIX file system, for example And and locking and all these mechanisms that come with it They just need for example and URL where they can access the data if if you are accessing the application on the mobile phone And you're viewing an image or photo. You just use URL at the end of the day So all the metadata that is attached for example to a photo the name of it where it has been recorded Who you share the photo or video with is actually start separately are not inside swift itself it's stored in a database as well as an elastic search for well searching and That makes it actually quite easy to implement this so swift itself is I think well known for its availability for its durability Scalability, but it's also a very good case a good fit in this case because of its flexibility as on said before There was a storage system in place already So there were quite a bunch of servers existing quite a bunch of disks that we're already paid for So we wanted to make to reuse them actually Another case or another important Thing that we do we while swift is a good fit is a failure resiliency So as an operator or do yeah as an operator of a large deployment like this You don't want to tell your customers. We're down for the next four hours or something like that your users are using the application all the time 24 7 basically and you try to minimize the downtime as far as possible and Last but not least it was very important to run some customized middleware on swift itself So we're going to have a look into that shortly But some of the actual part of the application is running inside the swift proxy itself so this made it much easier because we need less servers actually to Run the application because all the data can be for example modified inside the proxy when it's uploaded by the users and As all handset we are not running a complete open stack cloud including Nova and Cilometer for example on neutron. We're just running open stack swift together with keystone for authentication services so When you talk about Developing applications on top of open stack swift you need to say to keep a few things in mind actually You want to distribute your objects and your containers as far as possible It's probably not a very good idea to have a single account and to have a single container within swift and store billions of Objects from every user into this container at least not today. We have a few people working on that One of them is sitting there right next to him there and But right now it's a good idea to actually distribute it You also have to keep in mind that swift is an eventual consistent system That means for example when you upload a new object to swift and you have an outage on some part of your cluster That maybe your container listing is not Showing this new object right after the upload Because we keep a separate layer of database and an elastic search Separate from swift that was no problem in this case so when we plan to deployment actually we needed to estimate our well our growth actually and Need to choose our partition power wisely so the partition power is a is a setting within swift that is Responsible for well basically for the distribution of the data inside your cluster if you wanted to Get a few more details about that there will be another talk tomorrow afternoon where we are going to talk about this, but Basically, it's a very it's a fixed value and it's very important to to select a value that is nicer to small nor too big because otherwise you run into problems later on and Last but not least you need to know your failure domains So a failure domain might be for example on the smallest level a single server on a bit a little bit bigger level a rack for example or Finally a whole data center So when you take these into account you can build a cluster that can even tolerate Failing a complete data center for example in case you have two data centers, of course All right, let's talk a little bit about the architecture So when a user is Using the application the mobile phone application or the desktop application it and the user uploads a new file a new video or photo It gets sent to the Swift proxy and the Swift proxy itself talks to Keystone checks authentication Keystone itself is talking to my scale database and so on and so forth what happens afterwards after the Upload of this video for example has been successfully there's an event triggered directly from the Swift proxy using a custom middleware and it's sent to rabbit MQ so rabbit MQ is a message queuing system and Then it gets processed afterwards. So we have three actions So whenever you upload a new object you create on the system creates actually an entry in the Oracle database It creates another entry in the elastic search cluster and it also triggers the creation of previews for example Using image magic and ffm pack So let it's all of this is done asynchronously outside of the Swift cluster itself But of course the data that is generated here for example the thumbnail is sent back to the Swift proxy and Using the Swift proxy to store it actually replicated across the cluster now when a user finally Goes to the application and broses his files our searches for something it is directly talking to them to the application part and to the Oracle database and elastic search and Retrieving back URLs and these URLs are actually our syndicated URLs that you are using to access the data on Swift Alright, so Dorak is working on the operations team and he will tell you a little bit about well actually the deployment I tell the man should be before I'm responsible for the Turk cell opens take operations nearly for three years Let's have a look a little deep inside the infrastructure design of our project At the beginning we started using only eight physical servers On the beginning of the project there were three existing storage systems And we assign 13 Lunes so each one a size of 4 terabyte to each server. So Every region has nearly one under 150 terabyte data disks rings are using three regions One per store system every region has its own storage box Best practice says Swift has to have Three replicas so we made it like this our region one and region two is in Istanbul data centers and Region three is the Ankara data center One storage system with two servers are located in the remote data center Ankara So Initially we used write affinity To write only region one and region two so we expect a swift should move one replica to Ankara afterwards But after that when we growing The things are changed We need we need more capacity then we change some things in the infrastructure design like this We start using dedicated firewalls and dedicated load balancers because of high bandwidth Servers are a classic rec mount servers with two sockets from Different vendors the CPUs are version three and version four Intel zones Let's take a look a little bit network part We have basically we have two VLANs One VLAN for DMZ and for all connections and one private VLAN for only synchronization, you know the our sink Swift our sync data We have four 10 gigabit network interfaces for each server Also, we use bonding configuration for high availability After sometime we makes also upgrades in our storage systems Because of a capacity needed With that we add more servers to our cluster After that we started with eight servers, but now we have 33 servers for all regions 11 for each region each server now have Nearly 25 loans the capacity as a capacity of four terabytes. So we can say that Every server has a 100 terabyte data disks So when you look at the cluster region all regions have 1.1 petabyte data disks for all cluster We have 3.3 petabyte disk for Swift cluster Okay, let's have a look a closer at one note These were existing servers. So each note runs all services We have account server servers container and object also sweet proxy and the keystone Proxies also run a few will a few middlewares that we develop in-house a Sandic notification to rabbit MQ and Checking if required headers are sent along you see There was a mySQL we have not one we have two mySQL servers Working as a cluster active passive So finally when you see hold the picture we can say that nearly We are a 99% high-wavel cluster system. Okay deploying Swift. Yeah In our environment, yeah, we have a big enterprise environment when we install operating system using our automatic provisioning tools After that we install open stack packets from retat repos to servers Configure all parameters system network disks file systems keystones Swift everything then Adding a new servers to ring and copy the fresh link files to all servers after that we finally rebalance start the rebalance process But you have to be careful about rebalancing If you add too much new servers to the cluster at the same time Swift try to copy the data to new server So quickly so it's a normal habit for this but It's good for capacity and your person is your disk of persons will decrease But the problem is the existing production servers the existing Swift try to copy big data to empty Servers as quick as it can for example if trend of one production server lot nearly 20 or 30 percentage When you start to rebalance with so much this or new with new servers The trend will increase nearly 60 and 70 percent So it will be a problem for production environment or for and also for our customers but now we have that now how When what how much data we have to increase at what time so we can handle it After rebalancing we just monitoring the system. Okay, I told you before we Install and configure everything with manually Ansible we can talk about ends of a ansible is a great tool to manage big environments before ansible Installing servers packages configuring tuning everything's we made it manually Also after in the rebalance process we copy the swift files ring files and the other configuration items needed With our things and or SCP manually after ansible now everything is more clear clear and reliable Just you have to be careful About get the ansible files libraries and playbook up to date So adding servers or disk switch is much easier now Let's have a look at the monitoring part Yeah, we separate info and warlocks files for each services, you know Keystone object account and Other things starts the metric collected and visualized using graphana I will show you a screen shot about graphana and tell you after that and Swift we come to collect important metrics and trigger alarms for whole cluster Swift dispersion report It's very important after rebalancing to show what's going on or what's going on in our environment And we have some health check middlewares query by existing monitoring systems Yeah, this is our graphana a screen shot You can find lots of information about cluster. I mean whatever you want you can put in here For example, now you see in the free space of cluster Loves disk one loon in the cluster highest minimum one minute load incoming data transfer request per second I o weights and at the bottom is the CPU loads of the whole nodes I Told you first where Swift dispersion report Dispersion is torch test objects Distributed across the cluster All replicas are created by the tool Some replicas will be transferred after a rebalance and therefore Not available So your rate will vary between 66 percent and 100 percent If only one max one replica is missing and lastly Querying metrics directly from sweet Querying objects with our internal web services and lastly Swift recon replication command. Let us know the metrics about replication and what goes on going on Now I will give you words Chris again All right to take challenge. Thanks, Doug So when you operate a Swift cluster at that scale, especially with a few million users and hundreds thousands of users using it every day You will run into some challenges as dork said We increased the size of the cluster quite a bit in the meantime much earlier than we expected, which is actually good It's nice as an operator and as a company But we had to actually find a few workarounds. So initially we started with eight servers six actually in Istanbul to an anchor site as dork said and We had to add five new servers very quickly Well, actually if you have five servers to eight existing servers, you need to well Move about 40 percent of the existing data to the new servers to rebalance actually the cluster, right? So you want to have some of the existing data on the new servers and the old servers should have less storage used Um We had also as dork said write affinity enabled so write affinity basically was enabled because we wanted to ensure that all Data that is coming in Well, it's replicated three times But all the data that is coming in is initially written to the Istanbul site and the anchor site is only used Well as a last resort in case Istanbul gets lost or destroyed for example, as you know, Istanbul is for example Close to earthquakes might happen from time to time So you want to be sure that even if something happens on that side that you're still able to recover from that and have some data off So when you enable write affinity You write more data actually to your regions that you have in your cluster Let's assume you have three replicas and you have three well, yes three regions in our case And you enable write affinity to only write to two regions initially Actually, there are still three replicas written then but these three replicas are written to two zones to two regions So that means these two regions at least for some time need to hold more data Until this data is finally moved by the replicators to the third region so it requires more space and Depending on the speed of the replication process That might pile up actually especially if you have a lot of data coming in so Initially our growth was very fast and well It's still very fast, but we had a smaller cluster and we could barely keep up with adding new disk and new servers So what was very important in that case for us was to rent to run the replicators with an option called Handoffs first so handoffs first actually moves data That is not intended to be on these primary nodes to some to their actually primary nodes for example in their third region now Well, actually I was talking about like replication time and we had some troubles initially with a replication time It took it a lot of time. We are not talking about days here anymore and so we dig Into the reasons why why this happens? Well, the cluster growth was very high but Actually, we had also to tune of course the existing default settings for swift for example the replicator workers But on the one side you need to be to use a value that is not too high because otherwise it affects other services But you also need to use a value that is well fits your use case and your application speed So we had to tune a few things But at the end of the day this improved the situation a lot for us, but not totally so We had a look What is going as what else is going on inside of our cluster? so one of the issues we found actually is When we saw or when we looked at lower level IO statistics So when you start a lot of data on an existing one on in fire system you or your system actually keeps Some some I know cash entries and this and the amount of I know cash entries depends on the amount of storage that you attach to each server and Depending on this size your memory will memory usage will go up so What happened here is that the I know it cash Actually, or there were actually I know cash misses So for example when you dig deep into the directory structure of swift on the disk on the server itself And for example did an ls just to show which file like system there it took a lot of time to actually list the current directory and Of course this affects not only your ls operation, but also your application process So we had to tune this a little bit what helped a lot is the VFS cash pressure setting Which is by default set to 100 on the right at enterprise Linux We set it down to one that actually means that the kernel is Not expiring cash entries that fast you can't set it or what you can but you should never set it to zero actually because Some some people are smiling in the background You're actually it's very actually very likely actually that you run out of memory in this case There were some other issues that we had in the meantime for example one day we got reports from users that were not able to Skip parts of video files, so they wanted to view a longer video and They wanted to skip a little bit forward a few minutes and it started, but then it stopped suddenly And we're like okay. What's going on here? What what a swift doing right when when you stream our data Actually, it was not swift good for us We found the reason finally it was one of the load balancers That actually had a setting that affected this video streaming and especially when you skipped part of the data So it's for us. It was very important not only to look at swift itself when you run into some troubles But to the global picture Taking your load balancers or SSL terminators into account as well And other parts like keystone response times of crosses will affect your swift cluster as well It's a keystone is too Too slow to actually or well if it's slow then it would slow down swift as well And we had to take we had to work around this as well All right So that's that we love a lot of growth and Orhan is telling us a little bit about the future of this cluster and this product. Thank you, Christian As Christian and dark mentioned we have actually expanded our capacity during the project but And also as I said, we are currently have more than three million users now the growth was actually Higher than we initially expected We already Expanded our environment, but we are we are also we have also plans to expand our Server and storage capacity to five times. It's now by the end of next year of course, this will be bring the Challenges that Christian mentioned we will have to be very careful while doing this We will need to do this over time in order to not in order to move not move too much data at once It will so it will bring a lot of load We are also planning to upgrade our Operating system and open stack platform releases. That's one of the items on our list as well But we of course we will have to do this in production to And it says it's on challenges One other thing that we are playing for now is to introduce to x-tech In to our monitoring environment, so we will do better lock monitoring and analysis Which will help us as the operations guys We are also planning to change our infrastructure a little bit as Doruk mentioned currently We are running all swift and keystone services on all servers as our environment is growing. We are planning to separate those separate those serve these services and put Put them on their dedicated servers We will be installing a few dedicated keystone servers and we will also be Separating object storage nodes. We will have dedicated storage nodes Also, we are also Planning to use storage policies to keep the data balanced on our cluster Other than that This was our first project with open stack and with Swift, but We have already started a second project where we are implementing a second Swift cluster for one of our other services One of our other applications, which is called pip which is an instant messaging application from Turkcell and other service We are also we have also plans to use other open stack features since we are at telco We we are we have also plans regarding NFV or other use cases for open stack I think that's all from my part If You'll be happy if to take your questions if you have any Otherwise, thank you very much for listening to us Yeah, any questions Yeah, any questions John has a question. I see that Need a special introduction to Swift 101 maybe That's kidding If if you're going to open source the Android up Currently we don't have plans on that, but we'll see Actually, well Independent of this architecture. I think it's it's a little bit tough because you To run application like this. You need all the infrastructure, right? So you would require to open up Basically everything to everyone and otherwise it will be tough to use it. Yeah That's true. I think In this case, it would be difficult because you're not only talking to Swift You're also talking to a different application You need actually the second application for example to know Which object to retrieve or where to send the new object to so it's not in the native So the application the mobile application is not talking directly native to the native Swift interface Well, it's talking to the native interface, but it's not using it in this way Yeah Yeah, Matthew, do you have a mic there? It was working I don't think so working. All right. I repeat your question Yes So Okay So just repeat the question to be recorded another question is Actually, we're using Middleware in Swift to send out notifications to rabbit MQ and what happens if for example the message queue is full or rabbit MQ is not available Sure. I oh, okay So in this case, there's a there's another background process that ensures that every object that was sent some day Sometime ago it actually really has an entry in the database and tables. So that will take some time Actually so far as far as I know there were no real problems using this approach But just to be sure that everything is running fine. There's a periodic job on the user's container. So every user has its own container and Checking if the last time stamp of the container matches with what we got recorded in the database actually and if there's a mismatch then just using this command to see Okay, what what is here and what do we have in the database? Kind of that yeah You mean stuff? So with stuff so yeah in this case it was very important for us to reuse existing storage system and Swift offered a little bit more flexibility in this case and What is even more important in this case is that we have the Swift proxy which Allows us to run customized middlewares So when you when you look at the source code of the Swift proxy server, it's written in Python Seth and especially in this case is the rados gateway is as far as I remember written in C++ So if you have your developers already using Python for other stuff and not working on C++ It's much easier or it was in this case much easier for them to adopt this This use case Yeah, no more questions. Thanks for attending this talk