 My name is Akhil Malik and I work as a site reliability engineer with LinkedIn and with me I have Veera Bahu who works as staff software engineer with LinkedIn. Today we will be talking a bit about Python at LinkedIn and how we use Python to run our systems at a very large scale. So as part of this talk we will be covering essentially two case studies. So one of the case studies will be presented by Veera Bahu and the next one will be by me. These are the two topics for those case studies. So just to give you a brief introduction, LinkedIn has been growing at an enormous pace for quite some time now and as you can imagine it is not really possible to scale up teams linearly with the growth rate we are having. So it is important that while building automation systems around the needs of such a growing stack, you need to choose your ecosystem very carefully. And as part of that LinkedIn has been using Python to scale its stack to a very large scale for quite some time and as part of that today we will be covering two case studies. So for the first case studies I will invite Veera Bahu to start with this talk. This case study is about scaling LinkedIn infrastructure for infrastructure monitoring using Python. Veera Bahu. Yeah. Thanks Akhil. So yeah before starting with let me ask how many of you are from infrastructure management? And you do infrastructure. Okay. Yeah I saw a few hands so I mean if you want. So why do we need to have automation in monitoring and infrastructure? So for example take your home network. Do you need a system or automation to monitor the network? You don't need. But how about a startup that has like 50 to 100 employees? Yes of course you need some systems but you can fairly manage without automations. But how about an infrastructure like LinkedIn where the data centers are distributed across the globe and it has few thousands of devices interconnecting few tens of thousands of servers and each serving real time traffic. So any glitch in this infrastructure is going to impact the user that's you all and then it's going to directly impact the revenue. So everybody is in agreement that we need to have monitoring for this infrastructure. But can we do this manually? No it's next to impossible to do this manually. Even if you try to do a bigger percentage let's say 20 percentage of it manually that's going to stop your employees from innovating. So we need people to innovate and we need systems to do all the mundane jobs that is there. So that gives rise to our in from management suit. So what is this in from management suit? So it's nothing but a group of systems which is going to manage our infrastructure. The first and the foremost is our asset management which is the source of truth for everything that needs to be managed. It goes from the network device to the cards, the ports in the device to the servers, the server interfaces, the VMs and whatnot the VCs in our meeting rooms, the phones in our meeting rooms what not everything that needs to be managed is going to be the asset management. So that is the one point shop for what is there in LinkedIn. If I want to know at any point of time that's the one point shop. The second is monitoring, alerting and visualization and configuration. These things I will dig deeper into it as I go through. The last part is analytics. We gather like huge metrics of data that we will see in the further slides. So what is, hello, yeah. So what are the, can you guys hear me? Yes. So what are the, what are, what is the useful data that we derived out of the huge information that we gather from the infrastructure. That's all about analytics but I'm not going to touch on analytics but because that's a topic of its own and the way that we do is, we don't use much of Python here, Python there so we use Kafka and our adobe systems to do that. So I'm not going to touch on that. So what are the different architectures that we have in our infra management suite? So it varies from a web interface to a CLI and asynchronous services and what not. But the good part of this is everything is exposed, I mean whatever the data that it has in infrastructure management application and also what are the services that it provides is exposed through the rest interface. So it's easy to build some specific automation on top of it and everything is a multicolo. So my multicolo, I mean that it is, disaster recovery is built within the application. So if one data center is going down, we don't have to worry. The other data centers are monitored from the other locations that we have, right? So let's come to the monitoring. So we have discussed that we have tens and thousands of servers interconnect by thousands of devices and we need to monitor every metric in the system which includes and but not, I mean, but it's not complete. It's a CPU metric, the system metrics, memory and each Java process, how much JVM it consumes, the traffic statistics, what is the in and out of every device that we have in the network and so on. So this easily gives rise to a billion metrics that we need to monitor. And can we monitor for an hour? Can we monitor this billion metrics for an hour? It's going to be of no use because when we see a spike in the data, then we need to know what has happened, what has happened during that period. So this billion metric is actually going to give us insight and to analyze the root cause for an issue. So this has to be monitored in as granular as possible. So today we have monitoring billion metrics at a per minute interval. So how we do that? The system that we have is built purely on Python and certain libraries that we use for huge specific purposes. So let me get into the, I mean, like an eagle's eye view of architecture of this monitoring applications monitoring system that we have. So it's like, there's a master node, which is going to talk to the end nodes, which could be on devices, VMs or servers, whatnot. And it knows what to monitor and how to monitor. So what to monitor is going to get from the asset management and how to monitor is its own intelligence. So this information it gathers and then it's going to distribute to N number of clients. This N number of clients is going to use certain libraries for example, PySNMP or the asynchronous request to collect the information from the nodes and stores it in different data stores. One predominantly used data store is round robin database. So how many of you have used PySNMP before? Okay, there's only one guy, he's from my team. So the point is, okay, so if you are using, if you are going to use SNMP monitoring, it's basically PySNMP is the library that you, I would recommend you to look on. It has like, it supports all different protocols from SNMP get to SNMP bulk get and it supports asynchronous version of it. That's one, that's the reason we are able to write a simple Python system which can scale to billion metrics per minute. We are not waiting for any IODAR. So that gives an idea of how we monitor a billion metric, but how about a negative event that is happening in the system or negative event happening in the infrastructure. This needs to be intimated to the user or the, not the user, infrastructure engineers as early as possible so that they could analyze, found out the root cause and remediate it. So today we have a system that is built using G event. Can you guys guess the other two? Right, so it's Rabi Tumque and Celery. So what it does is we listen to, we listen for the events. So every device, every server is configured to send something that is called a syslog. So we listen to those syslog and it is not restricted only to syslog. There are also other applications which sends the information in different ways. Say it could be an API call that it makes to the rest interface that we add. And these are collected and put into Rabi Tumque. There is a set of Celery workers which is running, which is processing the events on this queue and what it does is it filters those things which are just an information and classifies the other set of events and sends it to the infrastructure engineers. The good point of it is today we are managing around 100 events per second. But tomorrow if it scales up to, for example, 1000 events per second, then it could be two things, one is scaling up only the Celery workers. Might be that might solve the problem or say if you go to 10,000 per second, then it could be scaling up the number of hosts that is listening for the event. So this simplistic system is able to handle and it's able to handle hundreds of events and solve the purpose of complete monitoring and all that. Got it? Okay, this is another unique thing that we add. So what we started with is we add this monitoring solution, but still certain services are going down. But our monitoring application is not reflecting that, why it is? So it could be due to, there is globally there are network changes happening. So it could be due to that a particular service is not reachable from a particular location. So we have users sitting across the globe, we have offices across the globe. So it is very much mandate to monitor each and every service from each and every location, I mean as much metrics as you could do. So that's what I call it as distributor monitoring. So most of you would have used Twisted. So the Twisted Internet Library has something called as Reactor, which is going to help you execute a command in a synchronous way. So what we did, we used that, we deployed agents across the globe. We sent commands that needs to be executed, and those commands are executed by the agents and the results are sent back. Which is again feedback to our monitoring system and follows the same process that we discussed earlier, right? Right, so we have billion metrics, okay? If I ask the infrastructure engineer, go look into billion metrics, they are died. They can easily do the manual management for that same. So we need some visualization tools. So here I'm discussing about only very few of the visualization or very customized visualization tools that we have. So this is one where I bet you guys if you have Flask, Jinja and some JavaScript libraries call it like D3 or jQuery libraries. You can visualize any data in an user friendly way and the code can be maintainable and readable. That's what we do and it is sustaining us for till now. It has not given us a problem. So we have a combination of Flask plus Jinja and we have few JavaScript libraries on top of it. And the second point that I would like to highlight here is we have a library which is built, which is a wrapper around the Flask. This gives us a unified view across the different system that we built across LinkedIn. So it doesn't matter if I built the system from sitting in India or if another guy sitting in Dublin or if another guy sitting in Mountain View building the system, it's going to have a unique look and feel, right? I'm not talking about LinkedIn.com. This I'm talking about the applications that we internally developed for infrastructure management. Configuration, so I mean think about this is like if you want to generate a configuration for one vendor, call it Cisco. It starts from Cisco 3K to Nexus 7K and Cisco is well known for its acquisition. So the commands are not going to be same for the same purpose. It's going to differ from each application to other. So how do we generate configuration for this? So that's a very big challenge for the engineers who is going to work on it. He has to develop the domain knowledge. So at the same time, we don't want to burden him with the scope for landing a language also. And at the same time, we need a readable code, a testable code and a sustainable code. So which is the language of, I mean which can unanimously everybody can choose. It's simply Python. So Python is the language that we choose and for that. Our configuration system you can use able to view the config, backup the config, audit the configuration, and generate the configuration. You can go to the next. So generate is actually a multi-step process. So how does it work is an user can raise a request. Say he wants to deploy an Accola or he wants to deploy a VLAN. So this system is going to intelligently identify what are the systems that I need to deploy this and what is the configuration that I need to deploy. And that is identified and it generates the config. That's the first step. The second review process, it could be certain infrastructure engineer can intervene and he can see whether the configuration generated or he can validate the configuration. Once that is done you can validate the, I mean the current state is validator and the configuration is deployed. So the point is this is the workflow is simply impact is a complicated workflow which I put a very, very short time of it but this complete workflow is also very lean implemented in a lean way from using Python, using it's a plain Python system. So I've been working with Java for like say eight years. So when I moved to Python, what are the, what is the thing that I liked about Python? The first thing is the REPL interface. So which Java lacks it. So if I want to test something, I need to get into Java class and then write few lines and then test it. Here if I want to test even a simple function, I will get into the REPL and then see is it my expectation is right. And I bet you guys, you guys can win fabulous prizes from LinkedIn booth by solving the quizzes using the REPL interface. It's just a minute time to do that. And the other things are same. And it also helps us to do faster iterations since we follow Ajay earlier. It helps us to do really a faster iteration with Python. Go to the next. So what's the take away? If I want to group everything in a single thing, if you are working in a team or if you are starting up a team which is very small, let's say two to three, two to four peoples and you are building an application for a bigger problems and different, you're solving a bigger problem and building different applications that have different architectures, then we bet you you can choose Python. It's not going to fool you, all right? I think I completed my things. So if you have questions, you can shoot at the end and I will over to Akhil to discuss about the next system that we have. Thanks, Veerabahu. It is really difficult to put these numbers in perspective. We are talking about trillion messages that are flowing through our messaging systems. We are talking about 70, 80,000 servers that we are monitoring and we're talking about like million matrix per second. Thanks. So as part of the next topic, we are going to talk about something that I developed sometime back. It is essentially a torrent cluster which runs entirely within Python ecosystem. So just to start with, these are some of the topics that I'll be covering. I'll be talking about the problem we had. I'll be talking about the obvious solution we came up with. Then I'll talk about 20,000 mile overview of the architecture and I'll have enough code samples to keep this interesting. So let's move it up. So if any of you guys who has worked in operations, DevOps knows how frustrating it is to use our sync to transfer a large amount of data. At LinkedIn, we have close to 500 services. Each service has its own application cache that might be as big as 20 to 50 gigabytes in size and all of these services run across close to 70,000 servers. So you can imagine if I had to build up, let's say a new data center and I had to transfer cache from one data center to another data center, how huge the task will be if we were to do it using SCP or our sync. Now we are talking here about terabytes of data. So how do you actually fix this problem by staying within Python ecosystem but still be able to do it using a relatively well-known solution? So the most obvious solution was using Torrent Protocol to actually achieve this transfer. Now if you look at it, the idea behind designing this system was to abstract functionality at each level by still staying within the Python ecosystem. So at every server, we basically have a Torrent client running which is in turn orchestrated by SaltStack and which is in turn exposed to the user by our Python applications. So this is the overview of the application that we have designed. So if you guys are not familiar with SaltStack, SaltStack in a nutshell is essentially a remote execution framework. You basically write a piece of Python code and your Salt Master will be essentially able to run it across all the available minions. So this is the part where basically SaltStack runs off. So these are essentially our list of 70,000 servers. We basically have pushed our Salt modules to each of these servers and we basically invoke all of these, we invoke Torrent Client and all of these servers using our Salt Master. So this is a Salt Master part of it. But when you actually submit a job using Salt Master, you cannot just leave it running. You need some sort of coordination, some sort of monitoring and that's where Celery comes in. Celery is essentially a Python-based remote task queue. And to actually abstract all of that from the end user, you essentially need some sort of API or library to actually take care of all of that. So in the next few slides, I'll be talking about all these components in some detail. So to start with, our Torrent Client is essentially based on something called LibTorrent. LibTorrent is essentially a C++ library, but it has very, very easy to use Python bindings. Just to give you a perspective, this essentially is a full-fledged Torrent Client. So all the downloads that you do using your Torrent Client, Azurius, BitTorrent, or any of them can be done using just this much code. You basically start listening on a particular port. You add your Torrent file and you're done. That's all. So basically, you can put this code inside your Python file and have it treated as a Salt module. So if you do that, you can essentially remotely invoke a Torrent Client on a server without ever logging on to it. So just to show you how, this is how our typical Salt topology looks like. Salt typically works, or our Salt is typically deployed in a master Minion mode. So you can have multiple masters and you can have multiple Minions. And this is basically how the distribution looks like. You basically invoke a task using one main master and that will essentially distribute the task to its submasters and that will essentially invoke your code on those Minions. So let's talk about Salt. So writing Salt code is actually very easy. It is just Python code. If you look at it, this is my basic Salt module. It is essentially normal Python function that can be invoked using a normal Salt call. So if you think about it, I can simply put my Torrent Client function within Torrent Client code within this function and it will be remotely invoked sitting on some Salt master and you can essentially have a Torrent cluster of close to 70,000 servers running up in a Jiffy. Now, having just a Torrent Client running remotely isn't enough. You obviously need to have some sort of real-time communication going on between your Torrent Client and the user. And that's where the Salt's event system comes into picture. So Salt's event system is nothing but a stream of messages that passes between the master and the Minion and it is completely bidirectional. So the idea is that you can essentially pass any message from master to Minion, Minion to master using this simple piece of code. The idea is that you basically invoke Salt's inbuilt function event.send and you can basically send any data through it. So the idea is that once your Torrent Client is invoked, the library actually exposes certain functionalities to have real-time view into the current transfer. So for example, what is the current transfer rate? What is the progress that the download has been completed to? How many peers are there in the swarm and so on? So you can basically use Salt's event system to actually communicate all of this data back from Minions to the master in real-time. And obviously, Salt exposes a REST API. So you don't really have to go to a Salt master, invoke some Salt command and actually do all of this manually. You can have REST API, you can have REST or even Python API calls made to Salt master setting remotely anywhere. And you can hook up Salt to your internal authentication system like LDAP or AD, which is what we have done. So essentially to invoke any Salt module, basically this is what you need to do. If you're using Python, you can simply invoke it directly using this. So in my case, it could be something like Salt module dot my function and then your arguments. Or if you're invoking that using your REST API, you can simply call it up like this. So all you need to do is essentially specify the module name, the function name and the arguments. It is just like invoking Python function only remotely. That's all. Now that we have actually abstracted the functionality of a torrent line wrapped around by Salt stack using REST APIs, the next part is exposing that to the end user. Obviously, end user isn't really educated enough to log into a Salt master and run the Salt command or make the API call to actually trigger a complete transfer. So that's where Flask comes in. Now, if you guys know, Flask is basically a very minimal web framework. The idea is that you define your web API endpoints and it can simply execute Python function that is wrapped around within those endpoints. Now, if you think about it, if I simply put the Salt invocation code within one of these endpoints, I can invoke Salt function simply by a REST call. And obviously, I can wrap around much more functionality. But at a minimum, this is all you need to actually invoke Salt module on all of those hosts by a simple REST call. Now, making a REST call is all fine. But as I talked about, we are transferring close to terabytes of data. It means that however fast your transfer pipelines are, it is not going to finish up in a few seconds. So you obviously need to make it asynchronous and you obviously need to make it real time as well. So how do you do it? That's where Celery comes into picture. Celery, as I mentioned, is essentially a remote task queue. You basically submit a task and the corresponding Python code is executed. Just like Salt, Celery also executes any Python code. So if you look at it, a simple task can be as simple as this. So all you need to do is call that task with an added decorator of delay and specify the arguments. Now, if you think about it, I can again wrap the Salt API call within this function. And that will, in turn, invoke my Salt module on all those hosts. But all of this will again happen in the background. It means that I am able to expose this sort. I am able to expose Celery to my Flask application. My Celery, in turn, calls my Salt. And the entire system works asynchronously even though the information is coming from minions in real time. So as I said, till now, we have been able to execute any arbitrary Python function on all the remote servers. We have wrapped that around Salt to have orchestration. We have wrapped that within Celery to make it asynchronous. And we have exposed that using Flask to actually build a web app around all of this. Now, the next thing is a user obviously wants to have those nice progress bars. So how do you actually achieve that? And that's where things like, that's where web sockets come in. So as I mentioned, we can actually transfer real time data back from our minions to our application in real time using this sort of code. Now, all you need to do is expose one more Flask end point, which basically starts a web socket for your client. So any browser or any command line REST client can simply open a web socket connection with an application with that specific transfer ID, unique transfer ID, and we'll be able to listen to the status of the transfer in real time. So if you have looked at any of those torrent clients, but torrent deluge or azureus, they give you nice progress bars. They give you nice transfer rates. They give you a number of peers and so on. So basically using web sockets Flask, we are able to build that sort of interface in a browser. But that manages an entire torrent cluster. There is actually one more alternative to building real time applications. So instead of using web sockets, web sockets is a slightly new thing. Not every browser supports that. And it is a bit complicated to implement that using Flask if you're using HTTP-based workers. And running both web sockets and HTTP requests in a single instance of Gunicorn can be a bit complicated. So the slightly old school solution to that problem is basically using servers and events. Servers and events are nothing but a tech stream that is based on HTTP protocol. So the information that I showed you that we're salt minion is essentially pushing back information in real time back to my application can be again pushed back to the end user using servers and events. And once you push that, any client can simply start listening to those events using this simple piece of code. And once you do for event in event stream, you will be able to basically pass that data in real time. So I have been continuously talking about what sort of data are we going to measure in real time. This is a data that we're actually looking to measure in real time. So the LibTorrent library actually gives you this sort of data. It actually exposes this sort of data using very simple functions. So in real time, you can actually check on what is the progress in terms of percentage, what is the current download rate, what is the upload rate, what is the number of peers in your swarm, what is the number of seeds in your swarm, and so on. So essentially, you can wrap this code around your salt event bus. And salt event bus will basically push this data to your salary worker in real time. And salary worker can expose this data to your Flask application. And using either web sockets or servers and events, you can actually expose this data in real time to the end user. Now putting it all back together, I know I went through a lot of things in a very short amount of time. So this is a diagram I sent in the, this was basically my first architecture diagram. So if you look at it, at the salt minion side, I've covered a few things. One is writing a full-fledged Torrent client. Number two is exposing the data given by a Torrent client in real time. And this is where basically that real time data that goes from salt minions back to our application. Apart from that, I talked about exposing that salt function or salt module through my salt master. And that's where salt master comes into the picture. Salt master exposes REST and Python-based APIs. So you can invoke them. After that, our salary comes into picture. Salary is what basically invokes all of this for us. It is like a gatekeeper for us. So you can have your REST API, you can have your salt REST API code written within your salary worker. And that will essentially take care of coordinating invocation and orchestration of your entire Torrent client. And apart from that, your worker can also listen to the events coming in real time from salt minions. Now that we have covered till now, the next part is obviously web app. You essentially create a few end points. A simple end point can be to actually submit your transfers. Another simple end point can be to actually have a pullback URL where you can go back to the URL and it basically gives you real time status. You can have more end points for your web sockets. You can have more end points for your servers and events as well. So if you look at it, with at most five or six or seven REST end points exposed by our flask, we have been able to essentially create a Torrent cluster out of close to 70, 80,000 servers. And if you have actually looked at the numbers that your Torrent client on your desktop gives you, 70 to 80 peers is actually a very good number to have a very good distribution in your swarm. So imagine how scalable the system can be with close to 70 to 80,000 servers. So just to give you a perspective, sometime back, this library was recorded transferring at 10 gigabits per second cross-colow without any interruptions. So that really pissed off our network team. They actually tried to shut this library down. They basically cut the link. And even after they cut the link and resumed it back again, the library itself resumed again. So if you think about it, not only is this very resilient, it is also pause enabled by default. And you don't really have to worry about how fast it is going to transfer and how reliable it is. You just need to write five or six lines of Python code to actually invoke a Torrent client. And it does all of this for you. So these are some of the things that we have used to actually build this project. So just a disclaimer, something similar to this was built by Twitter sometime back. It is called Murder. You can read about it more there. So I think that instead of Solstack, that uses a different orchestration framework. And it uses both Python and Ruby to actually implement all of this. And these are some of the libraries that we have used in Python to build this entire thing. So that's about it from my side. Any questions for either of the topics? You're basically looking at the real time data coming back from the minions to my application and how that is visualized. So it is essentially up to the user how many peers and how many servers he or she wants to use to submit the transfer. And basically based on that, we generate a complete manifest of all the servers in that. And because every single server is sending real time data back to our application, it means that you can simply pass the data to have a per server view of your real time data. So basically you will have some source nodes. You will have some proxy nodes. And you will have some destination nodes. So you already know about that. So once you start listening on the Solstack Event Bus, you can actually, Solstack Event Bus passes message per server. So it means that it is duty of my, it is duty of my salary worker to figure out which server is sending what data. And once that is done, you can basically compose the entire data into a single manifest and display it to your user. So it can be a simple table. It can be a data table. Basically it depends on you. So basically what we do is we display a list of active transfers. You click on the unique link. It basically expands and that shows your current progress for every single server. Oh, so LinkedIn has been an evangelist of Solstack for quite some time. So some of the top five contributors to Solstack project are actually LinkedIn employees. And till some time back, we had one of the biggest deployments of Solstack anywhere. So if you look at it, I think the last publicly available data was close to 70, 75,000 servers managed by Solstack. So if you already have Solstack in your environment, it doesn't make much sense to deploy a puppet or something like that just for this application. How is the server send request better than socket? Oh, server send event isn't better than web sockets. But it is more compatible with older browsers. So server send event essentially is your HTTP request. But web socket essentially upgrades your HTTP socket to a web socket. And that is much more lightweight. And also server send event isn't technically two-way. Web socket is two-way. So you can write a generic Python decorator to handle authentication for both the endpoints. So basically our web application acts as an authentication proxy for our entire back end system. You were asking? Yes. There are multiple. So at this point, I'm using Tornado. So is there any equivalent for, you know, we have SSH key pairs to authenticate communication between the client and servers? Is there any equivalent you are using? Can you just repeat that question again? With SSH, we are using a key pairs to authenticate between the server and the client. OK. So are you using any equivalent? In this system? Yes. So basically our salt masters are tied to our enterprise authentication, which is basically LDAP or AD. And because our application acts like a gatekeeper, so multiple options of either authenticating using either AD or LDAP, or using API keys for that. It is, at this point, very pluggable. You can very easily replace salt with puppet or Ansible or anything. Yes. So basically all you need to do to protect your API endpoints in Flask is to have a decorator. And that decorator can use any of your authentication systems. Yes, they do. So I don't really have any experience working with salt on Windows. I have some experience working with puppet on Windows. And that wasn't very nice. So I'm assuming it is pretty similar experience on salt as well. I don't think we do that. For network devices, we don't use salt. So that's the configuration system that I was discussing about. So that doesn't use salt. It uses the, I mean, each say, for example, Arista provides an API to deploy your configuration. Cisco provides a different mechanism to deploy the configuration. So it's basically could be a combination of API and Rancid is what we use for the network side of the things. I mean, if the Arista device is going to provide the REST APIs, yes, then we use it. It's up to the vendors. Cisco doesn't, Cisco, let's say, in XS3K is not going to provide you any. It's providing, yes. So it depends on. So that's what I was discussing about the first step in this is to understand the user request and generate the deployment. I mean, generate the configuration. So that is going to decide whether I need to go through the REST interface or do I need to use Rancid or whatnot. I mean, it depends on the vendor and the models in the system. So what is your preferred interface, like SNMP, REST API? See, basically, it is not the preferred interface of a safe. For example, if an Arista device is going to provide both Rancid and API, then we obviously go with the API because that's more flexible to that. But if Nexus 3K is going to provide me only Rancid and it doesn't provide me any API, then I don't have a choice, I have to go to Nexus 3K. I mean, through Nexus 3K, I have to go to Rancid. So what's the idea behind this is we use the interface which is more resilient, right? And this interface can able to understand API, Rancid, and I think that's it. OK, so if you take any APIs, so what are your basic expectations with an API like? Expectations, in the sense, can you elaborate on that? Basic API, it should be push model or pull model? No, I mean, devices can't be a pull model. Everything is on about production devices. So production devices can't or can't pull models, correct? So it can't pull the configuration. So what we have, we have to go through a change management and during that time slot, we have to deploy. So it should be a push model. So basically the procedure that we follow is a push model. So we don't have a pull model from the device. So just to add to that, and slightly out of context, there's something called salt runners, which is basically Python code executed on the salt masters. So you can leverage that functionality of salt to do all of this and abstract that from the end user. We, I mean, it's an SSH. It's not a telnet-based. It's a SSH-based and key-based authentication mechanism. Security for what sort of data? Right, so the data that we are essentially sharing between servers is our application cache. So by and large, that should be pretty safe within one single data center. But this library has an option to enable SSL for transfer as well. So it is just a simple flag. And it will enable SSL transfers for you. All right, thank you guys.