 So, well, thank you for coming. Hola, everyone. Did you notice that there are free talks at the same time we were distributed in name? The Arup-item folks have some sense of humor, for sure. Well, first of all, I would like to introduce myself, as it was already did. So I'm a software developer and a maintenance guy. It's focused telecom Poland. My main toys are G event, Twisted and Celery. What we are doing at the company is to providing a business-to-business telecommunication solutions. Right? We host teleconferences. We sell Pepe Xs, but our main product is a contact center. So nowadays, you have multiple ways to communicate with each other. There are social media. There are SMS text messages. Well, there are emails and there are phone calls. So a contact center is a way to unify all these, to gain a much better contact with your customer. So, throughout this talk, I will show you the principles of implementing distributed logs in Redis using Python and show the usage in a real-life application. So does anyone know or does anybody doesn't know what Redis is? Okay, there are some hands, so I will quickly refer to that. The Redis is an OSQL database. It's basic key value storage, but the key can be only a string, but the value can be more complicated data structure. For the sake of this talk, most important are string and list. They are pretty much like Python counterparts. And also Redis has a simple support for a message passing. So first of all, I will introduce some case study. Then I will show you how to implement correctly distributed binary semaphore, so-called mutex, and later on semaphore. And finally, I will give you some tips and there will be some time for questions. So first of all, this is a loose definition that I came up with a few days ago. The Gido on his talk yesterday gave a great example of locking and keeping safe the reference counters. The other case would be that we have a short database connection between threads in Python, and we would not like to other thread sending some SQL comments while the other thread was trying to carry out the transaction. So this is basically what lock is, it's to protect a shared resource. So the case study will concern a contact center. Hope nobody expected that. So in the contact center, work people, of course, and we refer to them as agents. So an agent can answer or originate a phone call, email messages, text messages, or carry out chat conversations, and this kind of stuff. Well, and my advantage of contact center is that it automates all of this kind of work. So it keeps the agents busy. So they are not looking at the floor or sailing, okay. But to be more familiar, I have replaced agent with giant W that stands for a worker, and we are tasks. So you could say that this looks like a task queue. Who doesn't know any task queue solutions like Celery, RQ, Germans? Are there anyone who hasn't heard any one of them? No, it's good. So basically, you know the concept, and this is the problem we are trying to solve. Yeah, it looks similar, you know the stuff. You can apply a working solution, because it will be great, right? Wrong. There are a couple of things that you have to worry about. First of all, a worker is an autonomous creature, and it can choose whether to work or not. He can simply log out of the system, and he will not be available anymore. And this is a normal situation. This is not a failure of system or some kind. You can't force somebody to stay logged in and work, at least not in Poland. The tasks must be prioritized, and this is a heavily desired feature from our clients. So yeah, I don't know if any other distributed task queue is like Celery support this. As far as I know, in Celery, you can partially achieve this by routing to different queues, and that's all. So it's also not sufficient. And the last, mostly mind-blowing, is that one worker can handle multiple tasks simultaneously. And from, I don't mean concurrently, but simultaneously, so in parallel, at least from a point of view of task manager. So this will be viewed that way. Of course, Q1 cannot carry out two phone conversations at the same time. But why not write an email and speak to somebody? That's fine. So our solution depicted as a whole looks for some centralized task manager, and it's written using Twisted. And the Redis will serve as a logs server and will hold them all. So the first thing I would like to say is about binary semaphore. It has only two states. It can be locked or unlocked. To lock it, I will refer to this as acquiring and to unlocking as releasing. This is the same as the standard Python library. Who has a background with multiprocessing or multithreading library? Yeah, that's quite a lot. And who was on the Monday talk on this? Yeah, that's quite a few things. So, and it will be stored as a Redis string, and all calls to acquire it or release will be non-blocking from Python. So the first use case is about keeping state of a worker, whether he's busy or idle. So meaning that lock is released, it's the same as idle and so on. And the lock will be acquired when task manager will assign a task, and it will be released afterwards. So it's over. And the second case is about preventing race conditions. The one I mentioned that a worker would like to log out sometimes, and I don't know, for a break or something, that he can do it using this mechanism because it can acquire a lock before a task manager do. Or it will simply do not release it after finishing its previous task. Okay. So since this is stored in this Redis string, we would use two Redis comments, a set to store a lock, to set the string, and a get to retrieve it. What we need is atomicity. Every single comment of Redis is an atomic one. So no one else can disrupt your data. So the first draft, we have a Chrome disk, something like that. We're getting the lock, checking if we can lock it actually, and then do it. But there's a problem, because between get and set, there is some instruction running, and this as a whole is not atomic. So someone else might have changed this in the meantime, and this is a problem. And it's not acceptable then. So we'll introduce some kind of Redis transactions. This is not the real transaction in a sense of SQL databases, but it allows you to execute a bunch of comments in atomic ways. So we issue the comment watch, and giving the key name. So this will be carefully watched during the transaction. And finally, if we cannot do anything, then we unwatch it. So this connection will be free of side effects later. The next step will be to introduce the transaction denotions, multi and exec. These actually will denote a transaction, or the things that should be done conditionally, even if a watched key has not been changed. So I guess that's quite simple stuff. Okay, the important part that task manager sometimes would like to know whether the state changed. When we lock the worker, he will find it eventually, but when we do not, when we are freeing it, it must know when the worker is freed. So we can actually perform some action, like, for example, look for the next task for it. And to receive the notifications, we issue a subscribe comment. But to be more precise, on the Python side, so this is a task manager, as I stated is written in Twisted. There is some kind of magic, like ready subscriber will handle the subscribe and actions for them for you. And this is the event driven. You see the very ugly method name, message received, which is written camel case, and by overloading this method, you actually define the behavior for this. So the second part will be about distributed semaphore. So the semaphore is different from lock in that way, that it maintains an internal counter. So it means that it can be released or acquired many times. If we define a semaphore with counter two, we can acquire it twice and also release it twice. But the original counter cannot be exceeded by further calls. And this will be stored as a list and the calls to it will be blocking. So the use case is that we can handle multiple tasks at the same time. As I stated before, the usual lock is not sufficient and the use of a bunch of binary semaphores would not be sufficient, too. So this is similar to string operations, but now we are using our push, which stands for right push, so it will append an element to a list, and the error pop, which stands for blocking right pop. So it will pop an element from the list. And our push will serve as a release operation and blocking right pop as an acquiring operation. So when there's nothing there, and it means that semaphore has been acquired maximum number of times, the command will block until there's something there. In the Python part, we do not change anything because it reacts only on events. Eventually you can implement your checking whether the semaphore is free or not. And there are cases when you might need a state of semaphore after changing it. You could use previously showed a multi-exec block, but that's not always the case. The alternative is to write a simple Lua script. You see that it's very much like Python in this case, except this word local, so it means it's allocating a local variable, so it will be garbage collected after this block. And this script only releases our lock and return a length that we have left. Okay, but there's a warning. Multi-exec blocks and Lua scripts all are atomic in Redis. The problem is that blocking invocations in atomic way has not sensed this time. So it would mean that the Redis instance has to be blocked forever to serve this one. So making bear-pop inside multi-exec will return you a nil, so it's none in Python client. And invoking such a Lua script, Redis will respond to you with an error. Alternatively you can use an air-pop, so it's non-blocking feature. It will return you either nil if lock cannot be acquired or a value if it has. Some final remarks before you rush and implement these solutions in your software. Please care about starting conditions. Make sure that the locks are properly instantiated before using them. And study carefully control flow in your application to not introduce too many locks or too less. And also please work up a restoring state procedure. Something goes wrong. There might be cases when the state in Redis is not finally consistent. So you might want to fix that some way. For example, we persist a state in SQL database as some kind of history of activity of given worker, so we can restore a state properly after some kind of failure. So well, that's all for me. And now it's time for some questions, if you have some, of course. Thank you. Hello. Have you considered using setNX for the lock, the setNX command? You mean setNX, right? Yes, yes. For lock. Yes. Yeah, we have read about this, and this is described also on Redis documentation as far as I know. But well, this is an expiring command as far as I know, yeah? You can use it as a non-expiring command. You don't need to set an expiring. Yeah, but somehow it didn't fit our needs. I can't remember now why, but no, it wasn't sufficient for this kind of locking. Yeah, but that's an alternative for sure. Okay. Have you tried to look at other implementations besides the built-in lock implementation in a Redis client? You mean besides Redis, right? Yeah, besides the official Redis client. Well, ideally, it would be to write your own locks server in Python, but the case is that some code base we have and the most is written in PHP, so it has to be something outside. Ah, I see. Okay. Hi. When you're talking about distributed semaphores and locks, are we talking about, so are the locks in one location, so in one Redis instance, and the workers can be distributed, or are the locks in semaphores actually distributed? Yeah, that's a good question, and I hope that someone asked that. Well, basically, we use a single Redis instance for serving our locks due to different limitations. We do not need more resources, but distribution you can achieve is by either sharding Redis instances, and this is done automatically by using Redis clusters starting from Redis 3.0, or replication, which is also in this kind. But when I prepared this, I meant distribution by, yeah, the workers are distributed, and these are different kind of resources that might achieve a lock, sorry, acquire. Does this answer your question? Okay. Any more questions? Have you tried, have you thought about using ZooKeeper, because I'm not entirely familiar with it, but my colleagues at work use it, and it's supposedly that, which is synchronizing few distributed instances of anything to only get one resource at a time. Well, we didn't, and I have read about something like ZooKeeper just yesterday on OREALLYSTAND, but I will surely consider that. So thanks for a suggestion, but no, I don't have any experience with that. Me too, but it seems like a good idea. Yeah, if it would be simpler or better in some case, why not replace? Yeah, because it's used supposedly to, you know, select the name node in a Hadoop cluster, for example. So this is also a really distributed system, and only you have to think about it and synchronize that, so ZooKeeper is supposedly for that. Yeah, I will investigate this for sure, and if I came up with something interesting, I will surely give a talk at the next occasion, not this year I think. Okay. Thank you. Any more questions? Well, I do have one there. Have you actually ever seen anyone taking on a phone and writing an email at the same time? Yeah, myself. Do you manage to actually make sense? Well, you know, there are some studies that shows that a human that is carrying out a few simultaneous tasks is completely reduced by its brain capabilities. So there was a comparison in the book I read that he behaves as if he had a brain of a six years old child. So, you know, I don't find personally that multiple tasks at the same time are a super killing feature, but that's what the business wants. Yeah, yeah, because they respond with some delay, so yeah. Maybe. So, any more questions? I guess we could then thank you again, Sebastian.