 V svoj svoj državnji stok imam, da imaš skalabilitivne opcije v manguzaim, zato da je bilo svoje systeme, kaj je nekaj preformant, in nekaj nekaj trafičnih. Svoje vrste nekaj prav me. Proste, da sem glasben, svoje vrste vrste vrste vrste vrste vrste vrste vrste vrste vrste vrste vrste vrste vrste. Ilko stvo se mi pričašnjo postupno software ingenjeri z Erlanga Soliušnji. Jen ni se spravil po 6 jah in lepoca izpronovili уana z pijeljama vse predmišljanovi noh razgledanji, bila prv vse za manga ZJem in XMPP. I čistim sem zelo zelo zelo in struckovil v zelo vrečenju z manga ZJem. Zelo sem naredil po prvno v Marat어주 na aprilu, da sem namžet po zamršarenju. Evno iskode, bojo v svojih manju. Zelo, kako je v Mojcu? If you know XMPP, then you may know that XMPP, that MANGUS is most of all XMPP server, it's written in Erlang. But these days, MANGUSaM, it's not only XMPP server, we rather call it a platform because we also have komponenti. We are writing them in Alex here. They are standalone components for push notifications. This is Mongoose push. Open source standalone component. You can take and use without Mongoose. If you want to integrate with APNs or FCM. Also we have Mongoose eyes. This is standard server to set up your audio video course. This is also written in Alex here. And you can use it just alone if you need. We also contribute to XMPP client libraries for Android, that's SMAC, and for iOS, that's XMPP framework. And because of all these parts and contributions, we call Mongoose now a platform. Because we are not focused only on the XMPP server, but we are adding components around it to provide better service for our users and customers. And getting back to the point of my presentation today. What I'm going to say next may be a bit of disappointing for some one of you, but for me, I think that's actually quite good. Every Mongoose installation I was involved was unique. This means that the users were behaving a bit different. There were different machines, so the hardware was different. Every installation was unique. And having that said, I mean that there is no silver bullet, which you can use to scale every messaging system. There are, of course, some patterns, which I will describe. There are some things you have to have in mind when scaling the system. But every installation will be unique and maybe a bit different. So you better monitor your service and see how it behaves so that you are prepared and know when to scale up. When thinking about scalability, we need to consider, first of all, the machine power, which is CPU, memory, IO bandwidth, and not only. We, these days, have mostly mobile clients in our services, and they are frequently disconnecting and reconnecting to the server, which has its impact on the server. And they are completely different than web clients where we have a persistent connection for a long time. So the scalability options is a bit different for web clients, or desktop clients, than for mobile clients. Also, the features which are enabled and used on the server have very big impact on the performance and scalability. In the simplest case, we have only one-to-one chat, which is, I can say, the easiest to scale, but then we can have group chat, we can have many presences because the roster is big. There can be many pub sub nodes, which also makes the scalability harder and needs to be properly low tested and monitored. And also the database usage. If the database is slow, our performance is also getting down, so we may need to shard users or scale the database appropriately. And talking about scalability, we can also describe these three limitations. About database I already told. In most cases, the first limitation is memory. The memory is our limit to how many connected devices users we can have on a single node. Then this is CPU. If we have enough memory, but our users are very talkative, then we are using almost all CPU power, for example, and this can be our limit. Here are the three steps I talked about you can have in when scaling mangos. First, you can start with single node, then we go with the cluster of nodes, but sometimes when the user base and the popularity of the service is big enough, the cluster may not handle the traffic as we want and we have to go beyond the cluster. All right, starting small with single node. Since mangos.m is written in Erlang, we model all our users' devices around processes. This means that every connected device has two or three processes depending on the encryption if it is terminated on mangos or somewhere before mangos. And thanks to Erlang VM scheduler, all the cores on your machine are utilized equally, so all the users are spread on all cores of the machine. And when putting more resources to the single machine, we can have better capacity, we can feed more and more users on the single node. The single node installation has its advantages. First of all, it is very easy to get started. We provide Docker images for every pull request. So even if you want to have a new feature which is still under construction, but you want to test it, you can take the Docker image for this pull request, start the container and try the new feature or build your application quickly and start just this one node. Single node is good for small installations when we have little users. It is easy to manage, easy to monitor because we have only one node to look after. And it is also easier to debug when some things go wrong with your client code or with the server code, you have only one node to trace. But this has two major disadvantages. First of all, this is single point of failure and we never recommend to go on production with single node even if you have little users and you will feed all your users in single node. This is still single point of failure and in case when the node is restarted, you will not have any backup. And also, which I think is maybe not that obvious, single node installation, especially on staging on pre-prod environments makes things harder to prepare for production. Of course, I don't know if there is any staging or pre-prod environment which can simulate the production traffic, probably not, but when you have your staging with more than one node, you learn how to monitor, how to debug the system which consists of more than one node. In this case, you have users connected to many different nodes and you need to learn how to debug things and it's better to do it on staging than on production when you realize that something may go wrong. And going beyond single node, there is cluster of nodes. In this case, users or devices can be connected to any of these nodes and messages will be routed between these nodes so they reach the destination even if the user is connected to some other node than the device from which the message is sent. And to build the MongoZIM cluster, we leverage the airline distribution layer which basically gives us fully connected network of nodes. So every node in the cluster knows about every other node in the cluster so the routing is very efficient. When we have a message to user connected to other node, we know exactly where the user is so we can send the message directly to this node where the TCP connection is. And the information about users is shared between users, between nodes. In fact, it's replicated thanks to Minitia database and tables kept in RAM. When new session appears in one of the nodes, it is quickly replicated to all other nodes so that we are always up to date with the session information. Considering MongoZIM cluster installation and preparing for failures, we need to make some room on every node. For example, if we know that we have 16 gigabytes of RAM, then we should have enough left memory to handle a crashed node or when we, for example, want to restart the node to upgrade the system, we need to have the capacity on the remaining nodes to handle the traffic which was on the just stopped node. And one more thing, on production installations, all the persistent data, it's better to be kept in external database like MySQL or Postgres, React even, if you want. Because keeping persistent data in MySQL doesn't make clustering easier. In some cases, the node may take too much time to restart because it has to sync all the data from disks on different machines. That's why we do not recommend to have persistent data in Minitia. All the RAM tables are fine, they are replicated just without issues. So this can be easily used. As you may expect, cluster installations have their limitations. First limitation is size of the cluster. Since we have fully connected network of nodes, this is on one hand a benefit when we have up to 10 nodes, but when we go beyond around 15 nodes, the fully connected network may start to be a bottleneck when it comes to replicating data to all other nodes in the cluster. And when we have clusters big like that, we may start thinking about going beyond a single cluster. Also another limitation may be database. We may be serving the user's fine, routing messages without much latency, but the database may be just too pushed to its limits, and maybe we should start thinking about sharding the users between many databases. Also, another limitation may be a latency. In single cluster we have the cluster in one data center, for example, in Europe. But the latency for European users is fine. They connect in milliseconds, but when we have users in USA, for example, or in Asia, then it may take a bit longer to connect to the cluster. And then we have to go beyond the cluster. In this case we form different clusters, possibly on different continents if you want to reduce latency. But what should be used to communicate between these clusters? And as in many cases in computer science it depends. It depends on two major things. First of all, if we can shard, divide users between these two or three data centers, and if we need to allow users to connect to any data center, for example, to have lower latency. If we want to shard users between different data centers, we can simply use XMPP federation. This works similarly to e-mail. You may have e-mail accounts at Gmail, at Hotmail, and you can send e-mails between these two e-mail providers. And this is very similar with XMPP federation. When you have different domains, you can address messages to users connected to these different domains. But in this case, users has to be bind to a data center. If we register a user on a given data center, it will always have to connect to this one and it will never be moved to closer location. There is another option. If we need to allow users to connect to any cluster in the database, we can use a new feature of MongoZM, which is geodistribution. This is a federation, but a bit smarter. In this case, all the data centers or the clusters serve the same domain, but the session and routing information is replicated between data centers using Redis with dynamite from Netflix replication layer. This is already in MongoZM master branch, which you can take and test. We still consider this experimental as it works only for the basic features like message routing, session management, some MOOC services and external components. Persistent data is not yet replicated, so your roster or vCards will not be easily replicated between clusters, but we are working on this and we want to release MongoZM 3.0 around mid-April and this geodistribution will be the main feature of the next release. That would be all I wanted to say. Thank you. We'll never stop supporting the XMPP federation. OK, so you're suggesting to use only this geodistribution instead of the regular clustering? Not suggesting, but asking. No, I... Yeah, the difference is you can have efficient Erlang clusters in one data center when nodes are close to each other, but when you have to form cluster between continents, Erlang distribution layer is not fit for that, so we will still have these two...