 Thank you. Sorry for the inconvenience. Welcome to this session. Honestly, I was not expecting such a packed room for this, but we have a very long title, but basically what I want to tell you is that we need to delegate, not only in Drupal, but also in everywhere. So firstly, let me introduce myself briefly. My name is Pablo López. I'm Drupal developer at Lulabot. Here you have my handle for Drupal.org and some other social network. So what is this session about? So basically when we are processing a web request, some tasks might be time-consuming or resource-intensive. And performing those tasks directly can slow down the server time response and impacting the user experience. So if we try to move this task to background processes, we can leverage those additional resources that can be used to manage more requests at the same time or any other tasks. Besides that, at the user traffic increase, it can become crucial to have quick response times to give room for more requests and also to avoid possible concurrency issues. So it's also not only for that, it's also for maintaining the whole system stability. So I think the main idea I would like you to have after this session is what is the sentence about by delegating secondary tasks to background processes, we can enhance efficiency, responsiveness, and scalability of our web application. Also ensuring that we'll have a better performance and a seamless user interaction, that's at the end what we all need. How we can decide which tasks are secondary enough to be moved to background? Well, there is not a size fits all solution. It depends on each situation. In this case, I selected these five categories, but maybe you can have others that works better for you or not. So the idea is we need to think whether the task is critical or not. It's something that if we are not doing it now, it will affect some other parts of the platform in general. Also whether the tasks are simple complex. Sometimes we can do a lot of simple tasks. We don't need to postpone them because they are simple enough that they are not adding that overhead by subversa. We also need to detect where our bottlenecks are. We can have our bottlenecks because we are doing very complex calculations like generating reports or whatever, or whether the problem is because we are accessing to the third party APIs or the database or file system a lot of times during the request. Also whether the task is urgent or not. And probably the most important is whether this task has a user direct impact or not. Based on that, we just have a series of good candidates that could be moved to background. So email or post notification, that's something that happened very often. File processing or image processing, that's something that Drupal Core already does. Search indexing, PDF generation, third party generation or even collect payments for e-commerce site. But also it depends a lot on the context. For instance, if we are running an e-commerce site that is selling books and we are selling best servers, it's not super critical to check the stock on the fly every time because that's usually those are books that are available in any warehouse. But if we are selling like first edition of a Shakespeare novel, there are only one or two in the alone in all the world. So we need to be very clear that we have stocking off to sell it because also it's quite expensive. Or other things like post notifications can be trivial and be postponed if we are sending, I don't know, supermarket offers for the week. But if we are waiting for a heard surgery and I'm the surgeon, I need to send the post notification. I need to be sure that that notification is sent now and is received by the patient. And well, so once this is some theory, let me now present you a personal project. This is a personal project that I think this is going to make me millionaire so I will retire from Drupal with it. So basically this is a social network that it has a feature that everyone wants. It allows you to broadcast messages to all users from a public phone who wouldn't like to receive random emails at any time from arbitrary people. So basically it's pretty simple. So we are providing this form. It can be done just a few lines of code and anyone can access to this form. You can submit the form and any user that is registered in the platform will receive an email with this text. So well, that's a good idea. Well, let's see. And here is the code. Here is the code. So we have, this is the submit callback for this form. You can see that it can be done pretty straightforward. Just a sick line of code that will make me millionaire. So basically we are collecting the data from the form. We are calling our super service that iterates over all the users in the system and sends the email with the data that we collected from the form. And we print this nice message that, okay, we have sent all the messages. So apparently everything is quite good. But well, let's see what's happening under the hood. So this is a diagram I think everybody is familiar with. So we are submitting the form. We are processing the request and we are sending a response to the user. If we dive into that blue section, we can think that we are doing like multiple tasks across the system. So we are receiving the request. We are processing task one, task two, task three and we are sending the response. We can map these diamonds, sorry, with these three operations that we are doing here. Collecting the data, processing the data and printing the message. So at the beginning everything is okay. We have done our load test locally with a few users. That's a good idea. We are validating our idea. But now it's time to send this to production. What's happening? We are starting to get people. We are receiving a good number of people that likes to be annoyed with emails every day, many times. So the thing is that the initial task that is collecting the data from the form is not being affected because it's always the same. Receiving the, sending the message is, sorry, printing the thank you message is always the same. But what's happening? Processing the image and sending the image. As the user is increasing, this task is starting to take more and more time. We are starting to receive some notifications on, okay, we are struggling. This form is taking too much time. Sometimes we are getting errors. But at this point we say, okay, let's increase a bit the PHP memory and let's see what happens. But after a few more users, we are starting to succumb to success. So we have too many users. We have a single server with that amount of memory and we are not able to manage the request from the user because it's not possible. So I made some numbers with this. So you can see that when we have one user, the request to the form, it takes nothing, 10 users, 100 users, 1k user. But when we arrive to 10k user, it takes like four minutes to process the form. That's not acceptable. If we are talking about memory or initial server had only 128 megabytes of memory. So it worked well for 1, 100, 1,000 subscribers. But when we arrived to 10k, we had to increase the memory because we were spending like almost 200 megabytes per request. And that's too much. So if we grow, that's not scalable. So just do provide tools to try to solve this bottleneck. Yes, of course. I will not be here otherwise. But we need to think about what kind of, where is the bottleneck? We need to analyze the situation because in this case, for instance, we can try to, the first thing to improve the code, to try to reduce redundancy, to try to add cache layers, to try to access directly to the database instead to go through the entity API to collect the emails. But in the end, the bottleneck is sending the email. That's something that we, well, we can do things, but in this example, we cannot improve. So we need to think about how can we achieve this? How can we delegate this email processing to have a better responsive and scalability for the system? So here in Drupal, we have like three different approaches to achieve that. So we have batch API, Chrome API, and QAPI. Any of them are modern. All of them have been in Drupal for a very long time. But sometimes we are not thinking about them when we are working in a project because, no, that's something that might happen in the future or project growth. So let's talk firstly about batch API. I think we all have seen this progress bar here in the rebuilding permission, running update PHP or any other beautiful components, for instance. So basically, what batch API does is to try to keep resources under control. So it adds, that's progress bar that, well, we are having that progress bar that takes care that you are not having to mount, that you are not running out of memory. But in the end, you are having that progress bar that are not executing the task in an actual background because it's happening in the main thread of the server and also takes control of the user roles. Well, if you can run batch API also through DRAS, but if you are going there, you are not having that experience of having a better website because you are in that progress bar. And also, that's something that is quite important. We are not sure whether the operation is finished or not because at some point, when the progress bar is there, the user could just, okay, I'm bored, I'm opening any social network or whatever in my browser. So the task was stopped at 50%. So sadly, half of our subscribers will not receive that email. So from my perspective, I think batch API is more intended for administrative tasks. So because administrators now that they have to wait until the task is done. But well, it's also good to see how it works. So if we go back to our diagram, we are collecting the data from the form, we are printing the message, but instead of having a huge diamond in the middle, we are splitting the task in small batches that are run one after the other until we finish. So we are not taking all the resources that we did in the past. So here is the code, how we solve this solution and we move it to batch API. So basically, everything is happening in the definition of this array. Since Drupal 8.6, there is a batch builder class, but what basically it contains is a wrapper for this array. So the key part of this array, these operations, where you define the different chunks of data or chunks of operations that we are going to execute. And the finished callback, that is what is happening after everything is processed. You have the title, the message or message, some other properties that you can add. And here is where we are creating our operations. So we are loading all the users in the system and we are dividing them in groups of 50 users. So if we have 10k, we will create like 200 operations. So we are going to do the same task 200 times and we are using the different name and message. And we are just calling to bot set and only with this call to this function, the batch, we are delegating the whole batch to the batch API and they will take care of everything for us. So here we have this init batch function where you can define some variables that's completely optional. And here we have the process items that basically we are doing exactly the same that we were doing before. The only thing is that instead of iterating across all the users, we are just processing 50 of them. And finally, when everything is done, we are sending the same message. So basically it's almost the same, but let's see how the things change. If we talk about time, well, we are almost in four minutes again. Even a bit more because the batch itself is having some small overhead. But well, if we talk about memory, things are much better now. We are more or less stable and we are using about like five megabytes of memory instead of 200. So we are not having the best user experience, but at least we are not burning our servers. This is batch API. And let's go for Chrome API. Chrome API is the way to delegate administrative or scheduled task that needs to be run in the background and is based on hookron. So that's something that you implement hookron in your module and any time the Chrome controller or the Chrome command is queued, all the tasks are all the hooks are invoked and all the tasks are execute. We have some limitations here. So Chrome tasks may not all have the same requirements. I mean, sometimes we have some administrative or background tasks that need to be done like one every hour. Sometimes some others needs to be done like once a day. So it's not easy to have that granular execution time. And it's also hard to have a clear overview of the tasks that have been run. So if you check the log after Chrome has been run, you know, some messages hookron for module A has been executed and it took two seconds. Module B, it took three seconds. But you don't know exactly which tasks have been executing in any of them. And also if something fails, the whole thing are stopped and you can have a problem with that. That leads us to this other problem that could be the overlap. Imagine that we have a lot of tasks that have been executed as part of Chrome and Chrome is taking like 10 minutes. And we have one of our Chrome tasks need to be executed every minute. Even if we schedule a Chrome to be run every minute, even that we cannot run Chrome while we are running Chrome, that task is not going to be executed every minute. It's going to be executed every 10 minutes because the rest of the Chrome tasks my task is not aware of are not capable to be finished on time. And also that's something that we found the hard way. You can have timeout issues. We had a hosting provider where we were running Chrome every hour. It took, in our test, it was taking like six or seven minutes. And after a while we found that we had a lot of tasks that had not been executed. The problem was that the hosting provider was killing the Chrome processes after five minutes of execution. So we need to be clear and be aware of that kind of things that can happen and silently fail. There is also an initiative to replace Chrome with a more modern approach in core. That's something that people are working on and hopefully will improve this. So let's see this with a brief example. So here is my social network Chrome implementation. Here we have this maintenance task that can be executed two or three, five days a day. And that's enough. But now we had a great idea. We want to send an e-mail to every user, besides all the I have already been saving, seeing them good morning. So we are starting to run in Chrome every hour. That is affecting this task because this task instead of once or twice a day is going to be executed 24 times a day. But that's not a big problem so far. But well, if the hour is between eight and nine, we are saying good morning to everyone. But what if we have another team member that creates another Chrome that needs to be executed every five minutes? And I'm not aware of. Our users are going to receive this e-mail not once, but maybe five or six times a day. So with Chrome tabs, we need to be careful when we define the Chrome hooks. And how can we solve this? We have country modules for that. I talk, there are many, but I will talk about two. We have the simple Chrome module that just provides a UI to show all the different Chrome jobs that you have in the system. You can add them some way to execute one after the other. And also, that's something very important. You can add different time intervals for each Chrome defined by each module. So that adds some kind of priority that it's necessary. And also give access to queues and allows to define plugins. So the problem we had before, instead of having the two tasks, the maintenance task, at the good morning task, at the same Chrome with that extra logic to execute them depending on the hour, we can just split this in two plugins, one to be executed every hour, one to be executed once a day, and the problem would be solved. And we also have ultimate Chrome. This is a much more flexible and complete module. It provides basically the same features that I mentioned before, plus allow to configure it to use any callback to be a Chrome task. So anything that you already have could be converted automatically to a Chrome task. It allows to have parallel execution and also add support for custom plugins for triggers, runners, launchers, et cetera. So for instance, instead of just checking the hour, you can check if it's 8 a.m. and whatever, whether it's raining or anything that you can to trigger the task. So that makes it much more simple. And if you don't have access or you don't want to use those country modules, you can try to achieve this playing with Chrome tab. Chrome tab, that's something that is, it requires some knowledge, but well, it's something that I guess is affordable by most of developers. But we have the problem that not all the hosting providers may give you access to Chrome tab. And sometimes if you are managing a huge platform that is using different servers and so on, you may need or multisite, you may need to have all your Chrome tab files synchronized. And that could be a bit problematic. That works. So basically, we have this Chrome tab that runs the regular Chrome every hour. But we have extra tasks that like doing this indexation of categories list. So you can just execute this just command from the Chrome tab. So we don't need to take care of generating or draws command, sorry, or Chrome. Something similar, we can use drash php eval to execute any custom service we already have or any custom faxion we have available in Drupal site. So that's fine. And also, we can run Qs. For instance, in this case, every two minutes. So that's something that will help. And yeah, we don't have some number for here. And let's move to QAPI. I think QAPI is the key part of this regarding the examples I'm showing. And QAPI basically is it offloads the load from the main thread and throw it to Sanborn. Basically, instead of doing things, you are telling someone to do the things on your behalf. This is something that is not new. This is something that is not only for Drupal. There are lots of third party services that provide Qs like SQS or RabbitMQ. There are, sorry, Q implementation, plugins for Laravel, for symphony. So that's something that is pretty common. Basically, when you are running your task, we are back to our example. We have two initial diamonds. But the thing in the central diamond, instead of sending all those emails, we are just telling the Q, okay, you need to send one email to these people. You need to send one email to this one. And that's all. And we can continue. Someone will take care of it. But the user experience is not being affected because of that. Because also, the user doesn't need, in this example, to send their emails now. So working with Qs is something that is not quite hard. In general, in Drupal, we have like four concepts, four views in general. So we have this Q service that is calling to the Q factory. You pass the name of the Q and your Q factory relies on a Q backend. Drupal provides two Q backends that are the database Q, that is the default one. And the memory backend that basically is on the right. And you can also define other backends to integrate it with third party, like, well, as mentioned, RabbitMQ or SQS. And you can define your own backends for your needs. Maybe you can use a different database table than the Q table that is used by default or whatever. So we already have the Q. And once we have the Q itself, we can just create items for the Q. It's pretty straightforward. We have the Q, you can pass the data. And that's all. You are telling the Q that, okay, we have this thing, process it whenever you want. And at some point, we want to process the Q. What we have to do is to claim an item. When we are using the default Drupal Qs, it's a traditional first team, first talk Q. So the first item that was added to the Q, it was returned. And you can start to process. You are doing your things with your item. And if everything goes well, we are deleting from the Q, we don't need any more. And if something goes wrong, we can just work with it. We can just release it to put it back into the Q to be processed again. We can delete it or we can move it to a secondary Q for retry or whatever. So here is a simple example. And let's see how we did it, the implementation with C.Q, C.high. So that's pretty straightforward. So basically, we have the same submit callback. The only difference is that we are loading this Q, the say hi greetings Q. And once we collect the data, we are iterating over the different users and we are adding items to the Q. Here I'm using an object. That's a personal personal feeling that it simplifies to process the Q because you have an object that you can use and stream and arrive or whatever. There are no restrictions. And finally, we are printing the message as we were. So well, pretty straightforward, but well, life isn't all roses. What happens when we are creating the Q? At some point, we need to process the Q. If we are not responsive processing the Q, we can have huge Qs and in the end, we are having the same problem that we had in the past but in another place because sending 1,000 emails is going to take four minutes even if it's running the front end or even if it's running in the back end. So we have some risk with Qs. First of all is the scalability and resource management. We need to think about how, when and by who are running our Qs. And also we need to think about the complexity of the different tasks because it's pretty straightforward to send the things to the Q, but we need to think about the complexity of the task, how we are going to react. If anything happens or something unexpected happens, are we going to redry? Are we going to put it back to the Q? Are we going to discard? Are we going to throw an exception? That's something extra things we need to think about and also prioritization. So by default, as mentioned, Drupal Qs are first team, first out. If you want to set priorities, there are different approaches like creating two different Qs depending on the priority. There was a Q priority module in the past in Drupal 7 but it's not there anymore. So something that we need to take into account. So scaling the Q management system efficiently, it takes some time as the load grows and that's why Drupal is providing the concept of Q workers because sometimes we are trying to do most of the time the same things with our Qs. So Q workers are automating the generic Q processing operations. So when you're creating Q workers, they are integrated out of the box with Chrome. So anytime you run Chrome, your Qs are being executed. It takes care of taking the Q, it takes care of whether the Q is suspended or not and the only thing you need to take care of is to process your items. Also incorporate some additional features for Q management that are not there by default and what could be pretty interesting. So modules in core that these are media to generate thumbnail images or locale to import different translations. So that's something that is already tested. So here is how we implemented it in say.hi. All we need to implement our Q worker as a plugin that is auto discovered in the Q worker folder and with this annotation where the magic happens here because this Q worker ID is the same ID we have in our Q. So automatically it connects the Q worker with the Q because they have the same ID. And basically the only thing that we need to do is to implement this process item method. That if we run through Chrome, if we run directly through DRAS, the Q, it invokes or Q workers iterates over them and the only thing we need to take into account is to process the item. And well, I also forgot to mention that this is quite important. This is where we are telling for how much time this Q will be running to avoid to have a timeout issue. So when we are invoking the Q worker from Chrome or from DRAS, it will be iterating over the Q during 60 seconds. After 60 seconds, even if it has not finished, it will stop to avoid to have timeout issues. So that's why that's quite important because also we'll need to help us for that that we'll need to take into account also to check if for Q grows much than expected and we are not processing as much items as necessary to keep it to leave it empty. So the thing is that we have these special exceptions provided. So if we want something goes wrong or whatever and we want to delay a specific Q item while we are processing it, we can throw this exception where we pass the seconds, we can postpone it. This is an at least. As I mentioned before, this Q will be being executed for 60 seconds. So if we are at the beginning of the time of the processing and we delay it for 10 seconds, it will probably be re-executed again during the same Chrome run. But if we are by the end, these 10 seconds will not be able to be run and it will need to wait until the next Chrome execution. If we just want to update the item back to the Q to be reprocessed immediately, we can just throw the Q exception and if we just suspend the Q, so something won't happen or anything happened with this item and we don't want to process any more items from this Q, we can just throw this suspend Q exception and that should be all. Chrome or Dress will pass to the next Q and forget about it. And finally, the only thing that we need to do is we are back to the exactly to the same function that we had at the beginning. We are loading the user and we are calling our service that sends the greetings to the user with this name and that message. And that's all. So basically, we are telling the Q worker what it has to do for ourselves while processing the Q. And let's talk about some country modules that we have related to Qs. So we have Q UI. This module is quite helpful because it helps to run the Q. So you have an overview of all the Qs and give you a UI to say run this Q and also allows you to decide for how much time the Q will be run. But if you already are using simple Chrome or ultimate Chrome, this is a bit redundant because that's a feature that is already provided for those modules. We have modules that provide external Qs. We have been talking about creating Qs and processing the Qs in Drupal. But Qs are also a very good excuse and a very good way to connect Drupal with third-party systems. Because, for instance, we can, in this example, we could send our emails to a Q in whatever of these services that is being not executed by Drupal by another service that sends email in a much more performant way than Drupal does. And we don't need to take care about the resources in our server or whatever because anything is happening outside. And also we have this module that is Q unique. Sometimes when we are adding too many items to the Qs, we are adding them twice before they are processing. Imagine that we are indexing notes and we are, during an absurd period of time, we are saving the same note or editing the same note 10 times. So that item would be added to the regular Q 10 times and would be indexed later 10 times. And that's not necessary because at the time that would be indexed only ones need to be. So Q unique adds a hash to the table and when you are trying to add in an item that is already there, it's rejected and it's not, you are avoiding to have de-applicated items in your Q. And we also have the Warmer module. This is a module that is based on Qs but it's not for processing Qs. Basically Warmer module is a very helpful one to repopulate or cache after the cache is cleared. So it creates a Q by default. It has multiple plugins but by default it provides plugins to re-cache or to warm the cache for specific entities or specific page. So after you clear cache it creates a Q that it's being processed and it loads your pages or entities. So the first user that access your page or access your API, if you are creating a de-coupled site, will not need to wait because the cache is cold. So the items will be already cached by default. And here we have some numbers. Well, we are having some progress. We are still seeing some peak when we are increasing but if you see here, we are talking about one second compared to four minutes when we have 10k users. So that's a pretty good improvement. And also if we talk about memory, we are having like four megabytes of memory consumed when we have 10k users. If we compare this with the original data, you can see how the normal implementation and the batch API-based implementations are taking about four minutes to be executed. And here we are talking about one second. So that's a huge improvement for user experience. Our users will be much happier. And also if we are talking about memory, we have the normal implementation that it takes like 200 megabytes and the batch and Q implementation that are taking around five megabytes. So not only our users will be happy, our servers will be happier. Our servers will be capable to manage much more requests in parallel. And well, so these are the main takeaways I'd like you to have from this talk, is that analyzing your tasks and the things that you have to do is something crucial. It's something that we need to have into account before the problems happen. But at the same time, we need to have an aesthetic balance. Adding the skills, as you can see, is not solving the problems. It's just moving the problems somewhere else that are easier to solve and is not affecting to the end user. But if we move to these kind of approaches, we need to think about how we are going to handle the things that we delegate to this background process or secondary task. But on the other hand, it's an scalability advantage. We all need to have it into account. And when we are working on our size, try to have it in mind that it could happen in the future. And the most important thing is that we are here to enhance the user satisfaction. This is a way to try to make our users happier. And well, finally, personal reflection that sometimes we are trying to do too many things, but the fastest task is the task that is not done. So, well, let's try to think about it and to have it in mind when we are implementing our website. And well, that's all I was going to talk about. Thank you so much for your time. I appreciate your patience. And here you have the slides. And here you have, in this repository, you have their say.high social network code. Please don't take my ideas. I want to be a millionaire. And well, if you have any questions, that's your time. So someone is asking, where did you time that one second? I assume not everything is processed after every queue run then. So who made this question maybe? Well, the thing, yeah, after what I was measuring, it was the time that the user experienced when they submit the form. So when we are creating the queue, we are the user experience is that it takes like one second to get the response. The emails are not being sent in one second. The emails are being sent later in the Chrome runs or whatever part you are using to run your queue. Any other questions, please? Okay. Well, thank you so much for your time.