 First of all, thank you to everybody who is here now, thanks for joining the Meetup today. So today is the March Meetup. So I'll be going through like, okay, so what's going to happen today? We will have one talk by Ted on event-driven notifications in Rails. And after that, we will be spending some time just in a breakout room, just to get to know each other, just for 10 minutes, just a very quick one. And then a short section, if anybody has any job shout-out, like want to hire or want to find job, et cetera, like both ways, feel free to say something then. And then we should, if everything is on time, we should end in one hour, 8.30. So again, shout-out to Michael and engineers SG. Thank you for hosting today's Meetup. As usual, thank you for hosting us. Then for those who don't know that we have a Telegram group, recently there has been some people who has been asking me how to join, apparently the QR code on Ruby S3 is broken. So yeah, this is the QR code to join the Ruby S3 Telegram. So yeah, Ted, are you ready? We can, yeah, let's start with Ted's talk today. Okay, let's see if I can share with you. See it working? Yep, I can see it. Okay, excellent. All right, hello. My name is Ted. Unless you haven't met me before. And today will really be a sort of case study of how we have used event listeners to implement our notification system in Ruby on Rails. Before I start, I should say that I work for a company named Ascenda and in fact, I'm in our office right now and at Ascenda, we do a lot of Ruby, really. We have quite a few applications. Some are Rails, some are Hanami, some are pure Ruby. And we have one application that is written in Elixir as well. And we're more or less always hiring Ruby developers. So if you're interested in coming and working with us or even working on my team, then feel free to reach out to me on the Ruby SG Telegram chat or you can check out our website AscendaLoyalty.com where you will also see some of the job descriptions there. All right, so I'm going to start with some background for what we're doing. And this will serve as sort of the answer to the question why for what I'm going to show you next. So what you need to know is the application that I am specifically working on is a sort of orchestration app that coordinates a lot of the work that is done by our other Ruby applications and it interfaces with all the frontends and also it interfaces with our partners and vendors through JSON APIs. So you can think of our whole ecosystem as sort of macro services. We have five or six Ruby applications that work together to accomplish what we do, which is loyalty points related and the application we're talking about today is sort of the hub in the middle of the wheel that does a lot of orchestration of what's happening in the other apps. All right, one of the things that we do quite a bit of in this application is send notifications to end users. End user meaning that you are a customer of a bank or an airline and you are either earning points or you are spending points in one of our marketplaces. So when you do that we will send you all sorts of notifications sometimes by email, sometimes by SMS, sometimes push notification if you're using a banking app for example, sometimes through some sort of widget. So this is a recurring task that sort of finds its way into most of the workflows in this orchestration application. So what we did initially was we built a simple notification system because we figured this is something that we're going to need to do quite a bit. And one of the things we considered was we don't want any unexpected errors in the notification code to break the flow for the user. Because notifications aren't really that critical and if the user was about to spend some points which translates to spend some money then we definitely don't want to prevent that from happening because our notification fails. So we built this simple notification system that has two steps, a triggering of notification and then delivery of said notification. And if you wonder why one trigger will or could lead to multiple deliveries it's because we can deliver the same notification through any number of channels. So you could receive the notification both by email and push notification, for example. So that's why the triggering of a notification might lead to multiple notifications being delivered. So looking at the first step of triggering the notification what we do is we create some notification records in the database. These records will have some information like which template are we going to use, which is the user we want to deliver it to. And what is the data we need to populate the template. So our templates are using liquid text as a templating language so we interpolate some data into there. And then the triggering of the notification schedules some background jobs, any number of background jobs depending on how many channels we want to deliver to. So here is a heavily redacted version of the trigger notification class. And you can see, notably, that it takes in a user, an event, and some data. And for each channel we try to create the notification record and schedule the job. And then we have the trigger notification. And then we have the trigger notification schedule the job. And this trigger notification class does not contain any code that can unexpectedly fail. So this is quite safe. It fulfills our requirement of notifications, not breaking things. So far so good. This is how you would call trigger notification from anywhere inside the application. So in this example, we are triggering the checkout completed notification for a certain user. And take note of the third argument here, which is the hash. It's the hash named data. And note how we are sort of digging into the transaction to pick out bits and pieces. And these bits and pieces are what will be interpolated into the template string when we render the actual notification template. So in this example, I have actually taken out a lot of code so that it's easier to digest. But this kind of data hash will normally have something like six to seven keys. And the values have some varying degree of complexity when it comes to picking them out of the domain objects that we're dealing with. But remember this hash because it's a key to what we'll be doing next. Then once we trigger the notification, it will schedule some background jobs. What these jobs will do is they will render the templates and they will try to deliver the notifications using the correct channel. So delivery could be send a mail using our own Ascenda SMTP system, send a mail using the partners system, send a push notification through the partners API, send an SMS through the partners API. We have all sorts of channels implemented here. And here is a redacted version of what the job looks like. So we have some retry logic here. And other than that, it's fairly straightforward. We try to find the notification again. And we have some deliver notification method which figures out how to deliver it based on the channel and tries to do that. So far so good. We are quite happy with our little notification system. It's quite flexible. It's easy to trigger notifications. And it's easy to add new notification templates. And because, of course, the delivery is done in background jobs, we don't really care if that fails or not because it's not going to error out the whole application. Unfortunately, there was something that we didn't quite foresee here. There's a hidden step almost. It turns out that a lot of times there's a lot of code involved in preparing the notification data that needs to be stored in the notification record in the database in order to eventually render the template for the notification. And this was sort of an unforeseen thing. But it led us to the problem that we're trying to solve using events. And this is that any bugs in this code that sort of cherry picks data out of objects and formats it somehow and puts that into a hash will basically blow up the application. Which is really not good if the customer is trying to do a checkout, i.e. they are committed to spending some points and we error out because of some notification code issue. It's really bad because that notification we could have gotten away with not sending it really. And if we just error out uncontrollably, then this order is going to end up in some undefined state and it's going to require manual intervention to fix it most likely. So I did some checking in our existing order flows and it turns out that this type of code that just prepares the data for triggering the notification is usually somewhere between a quarter and one third of the total length of code. So there is quite a substantial bit of the order flow that is just preparing data for notifications. And that means it's a pretty big footprint for bugs as well. So another way to visualize it is like this. We started out with our notification system in the green rectangle which is known to be safe. There can't be any errors in trigger notification and if there are errors in deliver notification job then that's fine. It can just retry itself until it eventually fails permanently. But it turned out there was this hidden step where we prepare the templates data that is not safe and it will cause exactly the kind of failure that we try to avoid. And that kind of failure did happen once or twice. So I decided to try to fix that. So one of the solutions we came up with, we considered a few of them, is to turn the service that needs to trigger the event into or trigger a notification rather into an event source or a subject if you're looking at the formal definition of the observer pattern. And then we let other parts of our call base subscribe to this service. And importantly, any callbacks that are subscribed will automatically handle errors in those callbacks. So the errors won't bubble up into the main flow. Then what we hope to end up with is something like this. So on the right hand side you see what we previously had. It's the notification system. But now we have some callback that has been registered with our, say our checkout class. And that one is automatically error handled. It is wrapped in an error handling wrapper when we register it, which we will see how it works soon. And then the event source being any class that can trigger events like a checkout class will call all the callbacks that are registered with it on some certain condition, like if it was successful or if it's had an error. So this is the state that we want to get to. So some of the pros of this solution is it solves the original problem, which was one of robustness. It turns out we can integrate this quite neatly with some of our existing systems. A positive side effect is it makes business logic code more focused. So it turns out that once you take out these 25 to 40 percent of code that do nothing but cherry pick and format data for notifications out of like a checkout class, that class becomes much more focused and it's a lot easier to see what's actually going on with this checkout. So it becomes a little bit less clear that it does send a notification, but at the same time we don't put all the nitty gritty details of sending that notification inside. Another nice thing is because the registering of callbacks can be done from anywhere basically, we can organize the registration of those listeners in any way we want. So if we want to optimize the code base for discoverability in some certain way, then we can play around with how we put those pieces of code in our directories and files. And another nice side effect is we can actually use these events for unrelated things, but that also should never make the application error out. So things like logging and instrumentation, like having this inside the checkout class is not really critical. Actually, it's detrimental and we also don't want errors in here to do anything bad to the actual business logic. But of course, everything is trade-offs, right? So it's important to consider the cons of the solution as well. So the big one, which is like a general critique of the observer pattern, which is rather obvious, is that the cause and the effect becomes obscured here. If you look at the checkout class now, there is no notion whatsoever about triggering notifications. So if you unwittingly just stumble on the code base and then you are tasked with doing something notification-related and you try to go into the checkout class to look for it, it won't be there. This is sort of a minor thing because we are using this for notifications across the board and we have a lot of them. So you only need to learn once that okay, notifications are sent through event callbacks and then you know where to look for them. And because of how we organize the files, you can also fussy search for notification and you will instantly find where the notifications are being triggered. But yeah, this is the biggest downside. Like if this is not such a cross-cutting thing, if you are trying to fix a single point in your code, then maybe you wouldn't go with events, maybe you'll go with something else. Another con is errors inside the actual preparing of the template data will now get swallowed up, unhandled. So it's actually harder to catch errors in preparing notification data in production and I will go through in the third part on enhancements how we can fix this. There are also some possible possible issues with lazy loading. So because you put the registering of the event callback in some arbitrary file, if that file is not loaded when you want the event or the notification to trigger, nothing will happen because no callbacks have been registered because the file hasn't been loaded. This isn't something I have experienced myself. It could theoretically happen in development if you have lazy loading enabled, but there are some ways to get around that, like Rails lets you specify directories to get eager loaded. So you should be able to fix that relatively easy. But important to note that before you get super excited about using events for something, just make sure that you have the right problem to match it to. Finally, before we go into the implementation, I set some success criteria just for the heck of it. We want primarily the robustness to go up and the way I will know this is I will never again see in app signal that check out failed because we failed to prepare some notification data. Hopefully, our confidence will go up overall. I think our confidence will go up when it comes to the critical parts, which is doing the actual business logic, even if our confidence gets slightly lower in the triggering of notifications, which I think is a fair trade-off. And ideally, we should see no change or a little bit of an increase in productivity. There might be a tiny increase because we have removed a lot of the noise in the relevant business logic, but that might be offset by someone being confused by the whole event listener concept. Okay, I think I'm done with the why and the what. So part two will be the how. So this will show you some diagrams and some Ruby code that will hopefully explain how this works. A lot of the code here is redacted, not because it's confidential, but because I don't want to overwhelm you with things that are unrelated to the stuff I'm going to show. But it's actually not that far off in terms of complexity. So I think you can appreciate that it works anyway. Okay, let's take a look at a typical check out or order flow. The user wants to spend his hard-earned points on something. He goes to the front end to pick something from the marketplace, and then he decides to check out. This application that I'm working on will receive the request from the front end. It will do a number of things depending on what you order. A lot of checks are done before we can carry on. And the way you can read this diagram is the green arrow means that the step was successful, and we can proceed to the next step. The red arrows represent edge cases, which means that there was an error in the check out, but it was not something exceptional. So we handled it gracefully. This could include you don't have enough points in your account to buy the thing. The thing went out of stock before you hit the check out button and all sorts of other things. If the check out is successful, we move into the fulfillment phase, which means we prepare to order this thing from the supplier. This is the simplest step. It basically just runs a background job that places the order with the supplier's API. And eventually we will hear back from the supplier, and that usually happens through a webhook. And in that webhook, they will tell us whether the order was successful or not. So again, there are some possibilities that we will exit the flow unsuccessfully here. And when that happens, usually we trigger a bunch of other steps like refunding of your points and sending notifications. All right. This is how it used to work. This is where we call trigger notification directly, and the part of the code that prepares notification data sits inside these services. So the checkout service has 25% of its code dedicated to crunching this data. Fulfillment has 40%. Webhook processor has 35%. And all that code has the chance of having a bug in it. With Ruby, it's usually the no method error. We try to access some attribute that was nil. And so it will just explode, and this order flow will be left in some weird state. And the front end will probably get some sort of unexpected error. So this is the old solution. You will see the familiar trigger notification method here. It's the same piece of code that we looked at earlier, but out of context. And you can also see some simple DSLs that we're using in this checkout class, like specifying the options, which are like keyword arguments, and also specifying what result key will be there if this service succeeds. So if you check the call method from top to bottom, you will see a guard close there. And this is basically one of those red arrows that says if the price check fails, then we will just return the result of the price check, which will be an error result. And the call sites can handle that. If all the checks pass, we will schedule the fulfillment, and we will trigger a notification. So this is one place where you could potentially end up with the code failing, because preparing of this hash that is the third argument to trigger notification had some bad code in it. Instead, what we want to do is in the new solution, we want to emit an event. And instead of having all this cherry picked and hyper formatted data sent to the callbacks, we will pick some bigger objects, just the objects themselves. So in this example, you can see the trigger event is now gone from our call method. Instead, we have a callback at the top that says just before we return success from this service, we will emit an event. And its name will be success. And its data will be a transaction. And this before success callback is evaluated in the context of the instance. So that is how we can call redemption transaction there and actually have it return something that is an instance method of this. So this callback at the top before success, this is not part of the event system. It is a feature of the code that lets you call success at the bottom with some result. That was there before we even started this work. So what will happen when we call emit? Well, the before success is called in the context of the instance. So we know it is an instance method. And it is defined in the event source mixing. And it is a deceptively simple method. It just goes through a list of registered listeners. A listener is a callback wrapped in an error handler. And one by one, it will invoke them. So one thing to take note of here is this happens in the same thread, in the same call stack synchronously. So even though you have this event system that can automatically swallow errors for you, you should not put some long running task in this event listener. What you could do is you could have the event listener schedule a background job. That would be a better use of it. Otherwise, your main thread is going to end up spending a lot of time in here. And the whole request is going to take a lot longer than expected, which might be a bit hard to debug because we might not know exactly what listeners are there at the time. This is the implementation of the emit instance method. Basically, we just create a qualified event name. So it's prefixed with the name of the class, which for us will be the checkout class. And this is so that if you have the same event emitted from multiple classes, they won't be confused with each other. We turn that into an event object. And then we look at event listeners defined in event source. And event listeners is just a hash with a key, which is the event name, and the value, which is a listener, which is a block wrapped in an error handler. So we look at all the event listeners for this event, and we just call them with the event as the argument. Event is a very simple class. It's just a wrapper for the event name and data. And we have some open struct like shortcuts so that you can you don't have to type so much. And also, instead of doing hash access that can silently return nil if the key doesn't exist, we get proper errors if you call some method on data that doesn't exist. All right, so we know how to trigger events. Now we need to register a listener that can be called when the event is triggered. And for that, we have a class method called on and you just pass it an event name success, which you will recognize from earlier when we decided to emit the event success. And it will take a block, and the block will basically be whatever you want to happen when this event is triggered. And here, again, now you see the trigger notification call. This used to be inside the checkout class. Now instead, it is registered from outside. And you can really organize this any way you like. If you notice here, you just need to reference the class and call on. In our case, we said, okay, let's introduce this concept called notifiers since it fits with how Rails likes to classify things. And also notifier should make it pretty clear that this thing sends notifications down the line. And it works quite well with fuzzy finding. If you type notify, you will find these files and you can figure out where notifications are sent. It also lets us organize the event listeners in neat ways. So our order flows tend to have this checkout fulfillment completion three step approach. So what we do is we just put all the event listeners for the whole flow in one file. So then if you want to answer the question, hey, what notifications do we send for points transfers? Then you can just look in that file and you can see, oh, and check out, we send this on fulfillment, we send that. And if the thing completes successfully, we will send this final email. But this could really be organized any way that works for you. This is the implementation of the on class method, even simpler than the emit method. It has the event name. It has an optional keyword argument that allows you to override the error handling. So in case you're debugging something, you can register this with race errors true. And it will help you by actually racing the error instead of just silently failing. And of course, it takes the block, which is the actual callback. So the only thing we do is we wrap this block in a listener. And we push that listener on to the event listeners hash, where the key is the name of this class, plus the event name. And as mentioned earlier, a listener is just a block wrapped in error handling, basically. And this is how errors are not raised by default. When you call emit and it executes all these listeners, you can see that the error handling is in there. Unless you've passed race errors true, then it will raise an error. So if you look at the whole implementation of event source, it's only really the event listeners hash, which is omitted here. And one class method on, and one instance method emit. And this is the whole implementation of the event system right now. So a very simple code, very easy to understand, but quite a powerful concept. Now that was quite a deep dive into the implementation itself. So in this third part, I'll just talk about some enhancements that I'm already working on. And these are addressing some of the shortcomings I mentioned earlier. And they're also simplifying the use case for our application. So one of the things I want to do is for our particular service classes, I might want to define some default event data that is always included and then allow the addition of custom event data. So if you take a look at the checkout class that we used this in earlier, you will see that the event data is the same as the data we return if the whole thing is successful. And it turns out this is not that incidental in maybe nine out of 10 cases. The data we need for the notification is the same as the data we return in the success call. Or at least it can be reached from that entry point. So what I want to do is instead of explicitly calling this before success emit this, I might just have the base class do that automatically. And the data will then be whatever the data is for success. But in that one out of 10 cases where we need more data, then we need some way of specifying that right. So then we will just add an event data method that you can pass some symbols to which will represent instance methods. Or potentially we could allow a block here as well. Another nice thing about this is we can make it even safer in that if this instance method is missing, for example, we could just ignore that as well and have an even more robust implementation. The next thing I want to do is a configurable error handler. So I've already implemented this actually. And it addresses one of the problems I mentioned earlier that because listeners just swallow their errors, we wouldn't know if the preparation of some notification has failed. So I just decided to add the option to define your own error handler, which is just anything responding to call. So in production, we can report to app signal. And in development, we can just re-race the error so that we can have it instantly debuggable. The last thing I want to do is maybe a bit more unobvious. So if we look again at our checkout class, we have this edge case, which is that if the price check fails, we return the result of the price check. This will not trigger a callback in the service. So there's an error method that you can call if the error is in this class. And that will invoke the before error callback, which means we can emit an event from that and listen to it if we want to send notifications. But because this price check is its own service, it is going to emit the event with that class name. So if we wanted to listen to notifications from checkout, we would have to do it in a very indirect way where we actually register the event listener on the price check service class, which is a bit indirect and maybe not immediately obvious. Not to mention, it would be really nice if we could just say checkout before error emit this event, even if the error is coming from an inside service class. Yes, so those are the three enhancements I'm currently working on. And that's all I had for this presentation. I have no idea how long I've been going, so I guess if there are any questions, we can take those now. If anyone is still here, I'm still in full screen mode, so I'm not entirely sure. Anybody has any questions? Okay, I'm going to stop sharing. I guess not. I guess since Michael is here, if there is anything you want to revisit, then there will be a video? Yes, there will be a video, there will be a recording. Okay, oh, there is a question by L3 if I'm pronouncing your name correctly. While you're working on rebuilding this feature, what was your greatest challenge while using Ruby? That's an interesting question. Okay, the biggest challenge is making an informed decision where to put this event listener's data. So if you recall, the event listener's is a hash that has keys that is a class name, dot event name, and then it has an array of listeners. And then you need to ask, where does this hash go? Do I do a class instance variable in each of the service classes? Or do I put this one registry inside the event source mixing or what? And then it's very easy to get lost in the object model of Ruby with class instance variables and single-ton classes and all that. So it's relatively easy to do, but it's hard to do in a deliberate or informed way that you feel like you have made the right decision, if that makes sense. Anybody else have any questions? Yeah, thank you, Ted, for your sharing today. Thank you.