 My name is Michel, and in reading back on the description for this talk, they thought about people must be thinking about what kind of talk is this going to be, and what kind of words is there to talk about, that's specific to payments on October 10th. And also, if you read the description, probably some of you might wonder about whether I'm going to complain about some aspect or another of rails, kind of the kind of complaints often hear from people that have not yet achieved the state of nirvana of two rails last time. So I definitely have not achieved that, and I will complain a little bit, I think. But I hope that you agree with me by the time I'm done that some of the straight-off that I'm going to present are worth making for any of the million payments in the world. So I'm going to tell you about my experience in building, or more accurately, participating in the building of a few different payments for several large companies. Most recently at Airbnb, I've tried to take some of those, some of the experiences and turn them into best practices, and try to share a few of those with you. And yeah, even though I might complain a little bit, I think I started my journey not really thinking that rail was the right tool. And I have to admit that I now am confident that this is the right thing and that using rail for payments is doable with self-accompaniment that I'm going to talk about. So I want to start over a decade ago when I joined what is now the largest successful payments company, which is then a startup that has just been acquired by one of Silicon Valley's successful companies. That startup was in the middle of Oxford, and it was hiring lots of engineers to help it sustain its growth and create the foundation for the future. And pretty quickly after joining, it became clear to me that there's tons and tons of work not just because there's lots of features to be implemented, but because folks there just did not have the time, they didn't have the inclination to focus on long-term thinking. It's not that they were smart or had the wrong motivations. They were just focusing on survival and growth. And what it felt like to me was coming in to find that your roommates have been partying all night and they've left the house in complete disarmament. They emptied the fridge, they left a mess in the kitchen, and tons of medication. And if you go around, by the way, you might find fewer examples from the Airbnb's past. I think that's a topic for another day. And now, you know, the kitchen needs to turn out a lot of gourmet meals for a very long time. So it's a little bit of a problem. And that really brings me to the next point, which is any company, any tech company, no matter which phase it is in, can and should think about quality. It should think about reliability, destability, maintainability, and those are all things I'm going to touch on later on in my talk. And I think it's especially true for companies or organizations within companies that are doing business. Because people have kind of this unique special relationship with their heart and the cash. You know, they might, people might tolerate if such results are at a lot of date or if some social media site is not 100% caught up. Usually people just don't know this. You know, people notice really quickly and get really upset if something happens with their money or it's a misunderstanding. So, you know, it's a little bit, in many cases it is possible to iterate on our product and gradually weed out all the problems by, you know, gradually exposing its to them. And you know, it's something that we often do. But with payments it's a little bit more difficult. And that space is exactly where the anchors, you know, meet the hipsters where the payment requirements collide with the Ruby and Rails way. So let's start by talking about audit trails. Audit trails is, you know, one of the most basic things, any organization that's dealing with payments needs to orient. And you know, it's a very easy concept for every change in, let's say, business balance, you have to keep track of who made that change, why they made it, and on behalf of whom. Could be the user interacting with the site. It could be a bank returning or completing a transaction to the user's question. It could be a customer service agent on the phone with the user. Those are all the general reasons why a user's balance will change, but they all need to be on it. So, you know, it's some simple item that has a few active record callbacks and you know, you're done, right? I think, you know, not exactly. So, I mean, in theory it's not going to work. But think about it. You have to be able to assure, to guarantee enough that you can stand in front of a judge in the court or you can stand in front of an auditor that's trying to look at your books. You have to guarantee that 100% of those changes are audited and that there is no other change that can somehow outweigh. So, you know, maybe the idea of active record callbacks is not that hot, I think. I mean, how can you deal with the potential for really any part of the code to call, you know, some update to all, or who will be entertained, or just turn off the callbacks, whether or not because whatever developer had some issues with it, they needed to learn. So, you know, this is, of course, possible in other languages as well. I mean, Java, SQL stuff, whatever the language you use, there is a way to bypass the protection provided by the language and the framework. But Ruby is special because what's so good about it is also, in this case, it's bad because it has weaker protections about scopes and basically any code can call any code, almost always. And that's great in most cases. But in this case, it actually makes it kind of hard to find and eliminate unnecessary calls or unwanted calls to your call. So, just to reiterate, if you look at what you get if you have active record, you get a lot of stuff just by virtue of declaring a class as an active record base. And, you know, a lot of that stuff is really useful in most of the types. But, I mean, if you have that class or if you think about that class and go back to the transaction, it's possible. And I'm sure people can come up with a few ways that describe, too, of bypassing or monitoring as if they were using active record. So, if they're being a big part of our course, it's important to use database triggers. We basically don't worry about the changes at the active record level. And we let the database do the hard work. The problem is database triggers creates other problems, right? I mean, it's hard to manage, and, otherwise, it has triggers. It's hard to migrate data. It becomes, you know, an operational data. So, we're experimenting with a few applications and solutions where we're trying to see if we can just for the payments related, the payment tables, have a little bit more stringent access model and still allow the rest of the application to use active record and all these benefits. So, I'll just give you a quick snippet here of, you know, one of the things we're experimenting with. Basically, creating this protected access gem that we're going to be using. And it's not exactly a record in the sense that it looks and feels like a little bit like active record. But what it does in Germany is it hides the actual active record model inside the class in a private domain. And that means that anyone outside of this particular class cannot see, well, people have to work harder if they want to see the active record model. So, you know, I'm sure some of you are thinking, well, you know, what's the big deal? If you don't use active record, you don't use active record, but then you don't have the benefits of active record. And I think what we're trying to achieve here is this middle ground where you can still expose active records methods or you can expose methods that make limitations to the underlying table, but you have to do it explicitly. You have to add another method to your building that access model. And that means, you know, or the negative side, it's more of work, right? You don't just get it for free. But on the positive side, it has to be coded. So someone has to think about it and actually add it. And then after it's coded, it has to be coded. So someone else needs to actually agree that this is a good idea. And, you know, a discussion can be had about the pros and cons and, you know, there's no accidental messy run with any of these tables. So, you know, we sacrifice speed and ease of use, but we get in return a much more explicit way of interacting with our database and we can ensure through protected access that everything is audited as well. So the next thing I wanted to talk about is early prediction errors. And, you know, I'll explain why I have this tiny spare wheel here. Because I think, well, first of all, let me just make sure that you understand what I'm referring to when I'm saying errors. I don't mean the errors that happen when the user interacts with the system. So, you know, if a user is typing a normal critical number, that's not the error I care about for the purposes of this explosion. I mean, this is like a normal flow. It's an error, but, you know, the application needs to deal with it as part of its normal processing. The errors I'm concerned with here are the kind that your application can continue to use. So it's kind of like a flat tire, right? You're driving on the road, there's a flat tire, and you can't move, right? So, you know, you have... you're expecting that, you know, a form within a credit card field and it doesn't contain it. Or you're expecting that the database is going to be in a certain state and it's not in that state, right? There's nothing you can do. It's about someone who messed up at some point and someone needs to fix it. And this is another thing where we're taking a different approach than what might be the standard... you know, the standard approach often done with Rails is we'd like to make sure that there's a flat tire we stop and we call the tow truck, or we call some things, someone can fix it. There's an excellent book, The Pragmatic Programmer. I don't know if anyone of you knows it. It's kind of old, it's from 2009, but it's still very, very good. And this is how they talk, right? Their program is the overlay. And it's very true. So, you know, how does that work? At the edge of the system, every, you know, the start of the process of entry, or... Well, okay. At the edge of the system, when APIs come in, when calls from clients come in, primary validation is what we have. And that causes programs to terminate early if there's an error. Also, when there are a variance in the code, places in the code where you put a comment, this should never happen. So, if you have a comment, replace it with a fail and, you know, race of exception. Because if it should never happen, but it actually has, you know, something is really, really wrong, and you don't want to keep limping along on this tiny, strict tire because, you know, it's going to potentially damage your car. You might, you know, write something wrong to the database, you might miss some response. For the user, it's better to throw a 500, you have a few users who suffer, but at least someone will know that there's a problem and you can fix it. So, you know, parameter validation sounds good in theory, but you know, it can very, very quickly deteriorate into something that you can really, really mess with. And I can almost see those, you know, thought people's, you know, it's like, yuck. This is like a job, you know, it's terrible. But, you know, we've created a little BSL that makes it unbelievable, this agreement. And, you know, you can define a method and use the BSL, there's a convention of, we use the convention of passing options as a made hash with all parameters and then you can easily assert what's there and what's not there. This also solves other problems. For example, if someone makes a typo and, you know, instead of confirmation code, they miss my confirmation code, this will complain about it too. So, the BSL also validates that stuff that's not expected is not there. And once you have this validate method, you can just write it, you put it somewhere, like at the end of your file or at the beginning of it, whatever, put it somewhere and you know what to look at it. Right, so the main, the means of your work doesn't have to be muddy with validation. So, all right, we'll talk about protecting the database from other changes and keeping an audit trail and parameter validation. But still, all too often, this is at least what I feel like when I deal with production issues. It happens, you know, everywhere I go, it's always like this. And I think it begs the question, you know, what's missing and what else is going wrong? And I think, one of the things that we try to think about when we write code and evaluate code is something called the law of the reader. And I'm not going to go into the details of what each line and your means. I think that the high level is that you should not reach out into things that are not directly accessible to you. So if you are in a particular class and you're trying to do something, let's take an example. If you're in a class that's you try to apply a coupon. Notice that there's coupon.department.try name. And by the way, this is inspired by Avli Green's excellent blog post about this. You should look it up. It's a good post. And I tried to simplify it a little bit here. And maybe the examples are a little contrived, but bear with me I'll try to walk you through it. So the coupon.department is a direct access to coupon. You're just reaching into coupon and saying, okay, we're in the department. But now you're making an assumption. You're saying, oh, I'm not sure coupon has an apartment, but if it does I'm going to go and reach into the need of it. So your code is now dependent on two hops away from this class. And let's assume you're doing that in several places in this particular class of this particular object. Now, what if the coupon were to change how it encodes the department? You actually don't care about the department. What you're trying to do is to access the department name. But now you're actually carrying the department because you're saying, give me the department. And then you're saying, okay, if the department exists give me the name. So a slightly better way of doing this is to have the knowledge of how given a coupon is the the department provided and you can use it to get that. So I just realized that I don't have the name here. I should fix that. But I think the key concept here as follows. It's that instead of dereferencing coupon into the department, you have a method that does that for you and you segregate that moment into the department and you don't have to worry about it and the rest of the code. An even better way would be to have the department be have this code maybe part of coupon. And then you don't have to worry about it here. But that's a seven issue. So I'm going to focus on what you're going to do here. So and I think it's important to not confuse it with just chaining. It doesn't mean just because there's the law of commuter doesn't mean that chaining is bad. The way to differentiate is if you're chaining something that is the same type of object. For example here, it's string, right? You're taking a string, returning a string, doing something to it, it's still a string so this is just convenience. You're just replacing consecutive parameter assignments instead of having a local variable assigning to it again and again. You're just chaining, right? That's okay. You're not making any more assumptions about string. You're not taking a couple of years to how string is. I mean, long chains might be hard to read at the least of the hell if not so they don't contribute to dislocating the rules. I want to talk about another thing and that's testing. I'm not going to go into testing frameworks. This thing is better or worse than another. I just want to propose this idea to you that I think, you know, a lot of people have very different opinions on somethings. It gets into religious words. I'll just focus on the payments in this case. The idea is small is beautiful and when it comes to testing really, really, really trying to have your methods and your classes as small as you can. It's like, again, trying to find a hole in your bicycle. You go step by step. You need to cover the whole thing. And each time you look at one spot until you find a bubble that tells you that the air is leaking. And, you know, if you have large files particularly large models it's really, really hard to test them. And because it's really hard to test them they're often not tested well. So the coverage of the test is insufficient. Which means that you're always thinking about this field of view. You're suspecting that there's a leak somewhere but you're losing it. You're not sure until you realize that there's a flaw. So, you know, being unsure about your tests is really bad. You want to know that your model is tested and it's tested well. And that, you know, if it's 3,000 microns long you can't judge, right? For 3 years. How can you guarantee that there's no off by 1 cent in that entire file? In that one scenario that's an edge case? You can. So, having smaller files especially smaller models is a really important thing in this scenario. And I know that a lot of people object to having anemic models that may have more functionality in models. That may be true in some cases. I'm not saying it's across the board but I mean, what I am saying is that if you want to have a model that's really, really well tested a good starting point is to have it do as little as possible and segregate or move the heavy lifting somewhere else. And one thing we do and it may be for moving the heavy lifting somewhere else is use what we call service objects. Service objects are used to communicate with service inside our infrastructure but also sometimes they're used inside an application to define API so we want to have a clear definition of an API between the pieces of code. Excuse me. But I mean, a service object is, I mean, it's a fancy name all it is is a perform method which you see here at the bottom wrapped with some standard logging and exception handling and it's just it can be called only once and after it's called it has to be thrown away it returns its results in a hash that you can inspect but it's basically a glorifying method but why is it good? Why do we like it? Well, first of all we have this concept of doing the validation so I already talked about the validation framework we have we usually put off the validation code in validation we call it from the initial integration code and we're done we know now inside the perform method that the parameters are what we expect we also often avoid having any real functionality in the perform method it only calls out to other small methods that are defined within the service object and they're all private so we are sure it's only the service object and the perform method is very small and it has a few calls about the methods, that's all so it's really easy if you open up one of those files A, the name is really descriptive because that thing does work it's almost like a method A, the name of the file so it's really easy also if you want to look inside you don't have to bother with the validation and the initialization, you don't care about this you can go straight to the perform and understand what the file is doing so as we have more and more of those it becomes clear that those are much easier to test they're smaller typically and they have well defined dependencies so the initialization you can see here lists up all the dependencies of the service object and that also makes it easier to test if you want to test it, you just need to provide these parameters and that's it and that way we can avoid unnecessary use of for example factory girl or database level access we have dependencies here we can mock them up and that's it another thing we use, and this I think is outside of the specifics of payments, it's a good practice in general, is the ABC complexity filter so ABC complexity is essentially assignment, branch, and collision it does some math with them it measures how many assignments how many branches, how many conditionals it takes the square of them and then square with the final reason so it ends up being the number and bigger the number the more complex your code and the defaults which are pretty good are set at 15, so 15 is considered the card, below 15 is an okay a good method above 15 is too big so let's just again look at a very simple example here's a piece of code and it takes you a second to read through it and understand what it tells and here's another piece of code that does the same thing except it's ABC complexity is much less, it's almost half of the first example so what happened here well the ABC complexity if we go back to the previous example you can see there's a lot of the references here users do references do references for parameters messages last, sent by there's a lot of the references here and there's an if and else so I guess it's also kind of the density compared to this example where we just extract the input which is what we care about we do a quick check to see if we actually need to deal with it and if not we're speech-able and then all that's left is to actually consider looking at this code is much easier to understand what does it's much easier to see the validation of the source, so pretty validation it's checking edge conditions at the beginning and then you're left with the actual work so the ABC complexity doesn't tell you to do this but if your method is too complex by simplifying it you will end up moving towards this direction and of course it's up to each developer and the conventions of whatever organization you're in to decide how exactly to do that and then one other thing I wanted to mention again it's a derivative concept I don't think it's specific to labels but I think it's very, very important is the don't repeat yourself and again this was introduced to the same Bokeh refer to the programming program and those three words single and ambiguous and authoritative are we in the P here so every piece of knowledge be it in code or in data I will argue though this rule doesn't speak single and data should have a single and ambiguous and authoritative representation within your system and I think a way I like to think about it is those labels that you find on the backs of computers and P and G everywhere if you look at the UL label on the left on the left this label is actually your way of going to the computer manufacturer and talking about the mechanical properties of the computer it's constructed and what's the size and thermal requirements it also talks about electrical stuff it talks about the computer being able to protect itself from problems in the network electrical grid two problems of its own there's a lot of stuff if you read the specification it's lots and lots and lots of pages instead of going and saying that every time, being this of every time you go and check is there a UL local and if there is you know that this authoritative single and ambiguous definition in this background is actually implemented in this problem of course this is a metaphor for a different world but if we go back to the example we had from earlier this is an example of true because there's one place where we actually extract the department from a coupon and we will longer care about it outside of this method the knowledge is in here and that's it if we ever change it we change this method and the rest of the code we need to care about so a few parting thoughts as I've indicated earlier I don't think all of the stuff that I've presented here is relevant for all of the kinds of applications you can collaborate but it definitely is in my opinion important for opinions because of the unique environments of pages and I'd like to also finish with a meta quote I think metaphors and maybe like my world like my metaphoric world it's all tires but metaphors are in general really really important when we think about software because the world actually has gone through the design of systems before the computers and people have thought through those things and have created solutions that are very useful as metaphors and we can use them and if we find the right metaphor we can sometimes make our lives a lot easier in how we design our code and how we learn from other people's mistakes even if they are in a different line of work even if they don't write code even if their decision was made hundreds of years ago so with that if there's any questions then I have to take them I noticed that near the beginning when you inserted protected class for active method active record inside of your class do you use a lot called in-circle only? so you're trying to keep an atomic database inside your database are you using just a regular database where do you use a special atomic database? so that's a good catch only is another gem that we're using which is very very basic it basically uses active records existing mechanisms to declare that active record model is in the room so once you can only knew it and once you save it you cannot make any changes and that's very useful when we're talking about audit trails because even though an object might be new for example if you have a transaction a transaction might transition from pending to success or failure or the amount might change because of itself so the object itself might be new but the role in the database is new so every time you create a new or if you mutate the object in this case in particular case it's a new case we actually create a new role in the database so we're using MySQL at the moment we have no concrete plans to move away from it for payment but what we are experimenting with is different ways of keeping the audit trail and the history of the patients so the way I just described where every time there's information on the object it's a little bit difficult it has problems so we're also experimenting with having an archive table basically a site table so if you have transactions you have transactions archived and mutations and transactions actually happen but every time there's a mutation a role gets written to the archive so that in that case and that makes it easy to use active records straight out of the box just a quick follow up what a lot of people follow you from creating a basic payment system what kinds of models do you use that kind of that kind of audit trail like your orders or just your payment transactions so that's a very good question I think the high level I have a high level answer to this would be as little as possible so I mean messing with audit trails and all this history stuff is complicated and it's painful and generally people hate it it's so convenient to use rails and this just makes it not convenient so we try to use it as little as possible but so if you look at some of the stuff that we're doing transactions are beautiful, it's not only audited but balances for example are not because balances can be computed from transactions so there's no need to make them audited and all that stuff because it actually will create problems if you have two things recording the same data now we have a sync problem and they will always happen so it's better to not choose the one that is the source of the data, in this case transactions and the balances are almost considered like a cache if you wanted to if you really wanted to you cannot have balances and every time you can do a sum of all the transactions of course that's not performed but conceptually speaking balances are you can create different transactions so the high level is as little as possible only the things that you do and I always think about the auditors and auditors come in what are they going to be looking at what are the questions that they're going to be asking and if it involves some of those tables then we lock them down yes so to be perfectly honest we are currently evaluating some of those gyms as I said we're still experimenting with this this is all stuff that we started doing for the month's sake I don't know enough to answer this in a more detailed way but I think it's important to understand that this is in a more detailed way but what I will say is that what we mean is something that generates C and generates data structures that are accessible by C I think some of those gyms audit trail, there's another one I forget its name use different mechanisms to keep history of the data which might be I know it started for us because there are downstream consumers of that data and they rely on the data internally to be accessible to use I'll just mention since we're talking about alternatives we are also actively exploring using non-sequel mechanisms to propagate changes so instead of writing a row into a paragraph table and relying on that for guidance from changes we might use the system like Kafka to propagate basically Kafka is a reliable, resilient deliver deliver at least once guaranteed system so it's like a message key but very robust in the city instead of writing a row to a database we might publish a row to Kafka and then have a consumer on the other end write a row to a consumer database so we're experimenting with that as well yes Do you have some kind of technical excavation process for things that come in from customers that are involved? That's a very good question but I think it's a generic system it's not specific to payments but in general we have all the standard mechanisms so basically if code throws an exception in production it gets collected and it's visible to us in the backend we have all kinds of metrics in order so I think as code will happen we get alerted to if there's a new kind of exception that started to be thrown or an exception that's been thrown consistently someone will go take a look at it and have a look at it I don't know if that answers your question If I can keep on going a little bit I'm thinking specifically of something like a say that even a customer is really in a state of payment processors not any games so for whatever reason that got passed it'll keep you in control so how strict are you about that? I think this is a little bit outside of the area where you are operating there is a customer service operation actually a pretty large one and within it there's a small group that is focused on on the payments so those people typically end up dealing with customers that have questions or issues with the payments I think generally speaking the philosophy of Airbnb and I'm not that this is not the payment specific generally speaking the philosophy of Airbnb the whole travel experience for users as pleasurable as possible and we try very hard to assume that our users are well-leaning and if they call then there's a real problem and deal their side it still that doesn't mean how that doesn't mean that we solve disputes there's a group of people right? Thanks a lot