 We hosted in Grab Taxi, and so our program today is we're going to have Kari from Grab Taxi over there. Kari? Right? And then we're going to talk about Go in the context of Grab Taxi, and then we have Valentin, which I think he... Okay. So, and then he'll talk about his experience of trying Go with a coding challenge that was online. So, without further ado, then Kari? So, thank you. Thank you guys for coming and for letting me speak to you. I also like to say I'm quite impressed with the number of people. I think the last Go Meetup I saw in Malaysia was about six people and it was really kind of sad. Supplies to say that it wasn't the second one that I saw. So, on the other hand, you guys look quite strong and people seem quite keen, which to me is very exciting because I mean, personally, I'm very excited about Go. It's my new favorite language. I'm sure that'll change at some point, but for the moment it's true. I guess I should say also that when I was first suggested to talk to you today, I was asked, as you mentioned, to talk about how using Go in Grab Taxi. What was your past favorite language? I guess I'll probably have to say Java. Please don't think less of me for that. But, yeah, strangely, I think whatever I'm doing at the time ends up being my favorite language because there was a time when I absolutely hated Java, so it's really getting off topic. So, yes, I was supposed to speak to you about how we're using Go in Grab Taxi. And while I'm happy to do that, it felt a bit self-serving. If you want to know about that, by all means grab me afterwards, or grab one of the other, the Grab Taxi Gopher's around. We'll be more than happy to share with you. That said, I thought it might be more interesting if I share with you some code and a couple of at least one neat trick that we use that we're very happy about. And hopefully it's as useful to you as it is to us. So, to dive into it, the trick here is dependency injection and actual fact dependency injection in terms of testing. And before you get all like, that's got to be a crazy boring topic. Hopefully it's not too bad. If nothing else, bear with me a little bit and see if it helps. Okay, so to give you a little bit of context, I assume we have a function, pretty standard function, that pulls a list of passengers from the database. Pretty straightforward, it's just going to hit the database, it's going to perform some kind of query, then it's going to type the rows object and convert it back into structs. I'm sure you've all written this kind of code many times already. Question then is how do you test such a function? So short answer, it's easy to test the happy path as in the correct operations. The downside though is you have to actually insert the data into the MySQL or whatever database is your preference, right? For those that have done a bit of that will immediately go, hang on, that's crazy, not only is it going to be slow because you're worrying about external resource, okay, the database, but it's also going to break. As soon as there's any data in the database other than this, that's it, test is done, it's broken, useless, right? On the other hand, if you want to test the errors, you just can't do it unless you find some funky way of like turning off the database or having it return errors and that's just a rabbit hole you just don't want to get into, right? Traditional solution to this is where we started, dependency injection. Injecting the database resource so that you can then mock it out or replace it with something else, right? If we dig down into that, it might then change the function signature to look something like this. Pass in the database object, but otherwise everything else is the same. You're still hitting the database, you're still converting the results. Where that gets us is the ability to do this. A happy path test is blown out, it's really quite big, but it does allow you to test in very fine detail. Now, I'm actually using a tool here called SQL Mock. If you Google for that, it should show up as probably the first result. Go SQL Mock. Actually, this tool, we do use it and I actually recommend it. However, I recommend it with a proviso. You don't want to use it too often because as you can see, it's a lot of code. More code means more to maintain, means more of everything basically, right? So if you really do need to test the database, by all means use a tool like this and approach like this will totally work. What is also given is now we can actually test the errors. We can test no data, we can test connection down, we can test all kinds of other error conditions. So we can feel more confident in our function and that our function is going to gracefully handle errors. However, there is a better way which hopefully I'll show you in a minute. Just to quickly summarize the quote unquote traditional, normally you're going to pass in the resource or dependency inject resource. A lot of the time, the thing that you pass in will be an interface. In good ghost style, you'll be trying to pass in the minimum interface you can. In this case, there is no interface. The database is a struct, not an interface and it's not a small one. So then becomes the downside of this kind of approach is that then becomes expensive to maintain. You can, as I said, use mock objects and as a result of mock objects, then you've got all the mock generator tools which will help reduce that burden, but it still ends up to be a lot of copy. So last point there, in fact, the one I skipped over which is one that took me, basically I had to get out of Java to learn the last question, the last point which was that the knowledge shared between the caller and the caller is quite great. As in, if you want to test this function that we had, the caller or the outside could be an external package, has to know that you're using a database, has to know what the configuration of that database is, and how to create it and pass it in. In the Java world, we had all kinds of tools for auto-wiring this stuff and annotations and all kinds of clever stuff. Go, we don't have that. And in a way, I guess I'm kind of hoping we never had that. While I do miss my annotations, all the other heavy weight stuff and the 30-second startup and the long compile cycles and all the other features of Java, I'd rather do without. So, going back forward, so just to prove my point about how expensive it is to maintain this mock object, this is the database object you'd have to pass it. Now, if you're writing this by hand, that's 10 functions. Even with stubbed implementations, that's 100 lines of code, more. You do that, how often do you have to do that before it becomes like, I don't want to do this, I'm just going to skip that test and hide it. Thankfully, we found a better way. So, if you consider the function I had before was private, if we then make it a public function, then the issue I mentioned before about knowledge leaking out from the package to the column and the cost of that becomes much more apparent. In the approach I'm trying to show you here, the resultant code would look something like the first one. In the traditional direct dependency injection approach, it looks much like the second one. Now, I'm hoping you're all with me in blacking the first one for the lack of, like, complexity. But, I mean, each to their own, right? We can argue about that after. So, if I dig down into that, this is, I skipped out some code. How I can then, okay, so, sorry, let me just say, then the question becomes, how can I test the function that calls this function without actually interacting with the database? Right? If I have a function that depends on this list function, instinctually I need to have data in the database, otherwise that test isn't going to work. This is where the trick comes in, and when my realization, when I move from Java, which obviously doesn't have functions as variables or as first class citizens, this was a mind-blowing kind of moment. In production, this code will run as if that second thing here, you give me an arrow, this bar list passengers are for in production that looks like a function. It doesn't look like a variable. As far as the rest of the code path goes, it's a function. But, because it's a variable of type function, we can replace it. We can replace it in production if we wish, but in this use case, we can replace it in testing. So, I can completely obliterate the need for the database by replacing that function with another function that returns whatever data I like. Let me show you that. So, okay, hang on. Yes, okay, so forgive me, it's a little bit... This one's a little bit complicated, and I'm hoping that once you see the trick once, the complexity drops away. But, for the moment, if you have a look at the middle section where it says what calls to the data package, that is how I'm replacing that function I had with another function which returns what I wanted to return. Right? What I should have mentioned earlier is, so, something here. It's going to feel a little weird at first. What I've done is the bar list passengers are for is actually just a wrapper around the call to the other package. Right? And this is intentional. It seems like duplication. I'm sorry, it is. But it's intentional duplication so that I can keep the scope of the knowledge inside this package. Or if you like, I can keep the scope of the damage inside this package. Right? There is actually no damage, but if you're a bit of an OCD like I am with my code quality, it starts to feel like damage. Right? So, back to this. The part that we skipped over, defer a funk. It looks disgusting. Actually, all I'm doing is setting, using defer, to replace the function that I overwrote. So, when this test is done, the function goes back to what it was before it started. But this test actually has no knowledge of how that function is implemented. So, I'm not replacing it with a new function. I'm using defer as, in my opinion, a fantastic feature that this, let's say line four, or it says curly bracket, this passenger 04, is evaluated when it's executed rather than when the defer fires to copy that function so that when that defer actually runs, I've got the old function in that memory, if you will. Now, you can further make this whole defer mock thing a lot less disgusting by having functions, so you can just say defer function, and the function we often write is restore function x. That at least drops off those first three lines and makes me feel a little cleaner, not so dirty. But you cannot skip over the middle part because that's the important part. If I was to show you this one, for example, this is how I would test the database error of no works. So, same defer to restore different mock implementation of that function. And I hope you see that you have sort of unlimited power there. You can make it panic, you can make it return anything you like. And it's all self-contained in the test, so the test is collected and sort of easy to understand. You're not pulling logic from all over the place, and perhaps if you're a TDD purist, you're not pulling from external resources, so you don't have that cost. Good question. How do you say, like, you know, during all these development, some of the stuff in the database also changed? And now here, because you're using code to mock the database, there will have to be some discrepancy between your code. Your code would pass, but actually one wrong production could have been disordered. Okay, so... Yes, good. Thank you for reminding me of that. I believe the question is, what happens if the data in the database or, I'm guessing, the data structure changes and therefore breaks production but not the test? In this case, I would defer it in the sense that in this case, this test won't help you. I would hope you would have tests in the cold package, which would do the verification. And that would call out this guy and say, this guy wouldn't pass? No, I think, from my mind, I would have the SQL lock that I mentioned in the traditional section. I would have it actually responding as a database, which, in fact, probably makes no difference. Yeah, because the sliver changes might not help the... Yeah, absolutely right. We... The only way we kind of account for that is to have more UAT or end-to-end fact tests or slope tests, if you will. Those are expensive and they kind of break a lot of the rules. But I find it valuable, at least for that kind of stuff, just sanity check, but I can tell you we've made that mistake a couple of times. It happens. There are ways to deal with it, but the problem you've got there is you have to make sure the database... You don't write too many tests against the database, because then the size of the database itself drags the performance down, but you also want to have enough to know that it's the same. Otherwise, you'll deploy the production and the whole world comes crashing down. You got to balance it. Thank you. In fact, I used test skip a lot for those kind of tests. So my day-to-day, like IntelliJ or command line run, it does automatically skip all the database required tests. But I do run that in the build server. I see. No problem. In this particular example, I mean, not sure... I mean, you could. Maybe I was trying to break away from writing Java code in Go, which I've been accused of doing. And I have done. I will openly admit it. This feels more Go-esque. That certainly sounds like it would work too, but I don't know what to do with it. That certainly sounds like it would work too. Yeah, this is a very functional programming approach. Yeah, and if I wasn't know a guy before, maybe I would... You could blame it on that, but for the moment, it's just a neat trick. I won't say there's that much. Anyway, to the previous question, what was the problem? Database code breaking or the data structure that's written and changes? When you iterate pretty quickly, you often will change the schema like you change the code schema, but the database... The data structure that comes back, you expected to change it, but you didn't update your test. It's similar to when you change the structure in your sandbox and the test work, but you forget to do it in production during the deploy cycle or before. It's the same kind of error. You just have to do enough to make yourself happy about it. So just write the database test, I mean, look, I'm a big fan of smoke tests if I can pull them off, but I only want a very small number because they do write dirty database and there's all kinds of other expenses. Okay, so to sum up on that side things, I would say, look, there's no... a big plus one for me, there's no impact on normal operations. It's not test induced damage for those that listen to DHH and some of those guys. It's much simpler for my mind at least than just straight up mocking. It feels to me like it decouples my packages and my layers a lot cleaner and that I would reiterate that point of having the function call a wrapper function and then through. That sort of keeps that decision in the calling layer and so the underlying external resource database or REST or whatever it is doesn't have to be aware of what you're doing or not. On the downside the restore function is kind of disgusting. It looks bad, like I said, you can pay that cost down by making a function that adds some cost, it's not free and perhaps the worst one we've come across is concurrent. When you run go test it doesn't seem to run it concurrently now maybe I'm missing a flag to pull that off but any attempt that we've done to make them concurrent has highlighted this problem and one of our engineers who's probably hiding the back over there actually found that there was a way around that I think for me for my mind there was two parts to that problem. Most of the time you're going to mock out inside the package in a consistent manner so again if you're using that local mock it is less of a problem but if you're using a function that has input parameters in our case we were testing a lot of booking related stuff, we can actually make this mock function just a little bit smarter so it said if the booking code is this return the mock, if it's anything else just return the old function. I've effectively put a bit more smarts in there to make concurrency more of a thing and in actual fact I hope I haven't sped through but there was kind of a reason why I sped through firstly I'd say thank you is there any other questions on this otherwise if I have time I have one other trick I'd really like to share. How are we doing? Questions before we just touch on since you are doing dependency injection you already mentioned that you run it currently other people can modify your export function so do you think it's a pretty good enough issue or what you guys are doing now is it just a tradition of this is a relatively well not a relatively new trick but it's not one we've used everywhere. The concurrency issue took a long time for us to surface. It was only when we were our build server was taking longer than we were patient enough to wait for that we tried to run them concurrently. If we hadn't done that I wouldn't have even mentioned it because I wouldn't have noticed it. While I find the solution that the gentleman wrote very clever I kind of don't like it because it still feels not very robust at this point I just want to say your mileage may vary I know that's a cop out I'm sorry for that but I would say we should use this trick quite a bit and I haven't really understood that back. Because you still got the boost of somebody else in production by injecting something I mean I was going to make a joke but the most serious answer is I trust the guys I work with I don't Yeah I guess I've been trying to be paranoid against protecting myself against myself or myself against the idiots I work with but eventually I'm like actually they're not idiots perhaps I should just trust them and myself or if I hurt myself it's my own damn fault so you know I think that's a non-requirement. I think the problem with concurrent testing is because your function variable is a variable and concurrently requires me only Absolutely right I mean in a CDD pure sense they will say your test must be concurrent that's one of the commandments I never really I mean I'm a big fan of those gentlemen but I don't take everything they say to be gospel Like I said if my tests weren't fast weren't slowing down I wouldn't have even Very very good Actually not means the test must be concurrent it must be independent Yes, yeah absolutely The acronym is like fitness or something Thus Yeah and one of them is independent and idempotent It's all kind of like Yeah and I try to follow the rules as much as I can but sometimes I just like screw the rules It's gonna cost me too much I'll pay the price later As you can tell he works for me he's happy I'm saying screw the rules I still have time right So next trick So I'm assuming everyone's written any kind of goal and I know myself I've written a lot of this the code looks like that and by that I mean a whole lot of if ever it was not null right This is goal style What if I told you you can write goal like that It's not in silence so I mean it's irrelevant the point is I'm still having these functions could still return errors or could still generate an error right but I don't have all that boil it for me Are you using null as an error value? No No Are you doing decorators? No I'm not sure You basically wrap these things around with something and it's a common code I think that's similar to option from Scala right? That's probably a Scala story from here The short answer is no I missed out the key important point here Pretty much that So this was inspired by the clean code book from Bob Martin if anybody knows who that is and obviously that's a Java book but the same thing seems to apply here If I zoom out from this what you can do in your sub functions is instead of having them return an error you have them panic and the input of that panic will be the error that you want to return right Now it doesn't look so great here because I've probably not saved any lines but when the function starts to get quite long or you do some cleverness or you don't do quite so much with the recover part the benefits of readability and nothing else are fantastic or if you're anything like me I kind of the recover seems to disappear I don't see it I write it and then it disappears all I see is the business logic part which to me is like I'm sorry a lot sexier Can you function dependent error error is here for example you've got two or three functions here which one then could have different error messages right by this method you can check the error messages at the corner function so let me paraphrase I think your question is will the errors in the sub functions still be obvious to the return so with that as the question yes absolutely and I think in this case that disgusting looking recover at the top actually what it's doing is saying if the error is a run time error as in an actual panic just re-panic you don't actually have to do that and in a lot of my character doesn't actually re-panic I just catch it because in a lot of cases I want to catch that any random panic still have somehow missed whereas the second part the cast the result of the cast or when the cast is successful I'm actually grabbing back the error which was the value of the panic and returning that to the function so the net result to my public consumers is actually identical all I've done is changed the code around to hopefully look better I'm a bit lost a built-in function recovery is a built-in function I'm not familiar with that assuming I don't explain it very well there's a go blog article called panic defer recover which tries to explain this if this example feels a little contrived obviously it is but there are a couple of real life usages you can look at the marshaling JSON function actually uses this to unravel the recursion so as you can guess JSON parsing is recursive instead of return return return they just panic and it just says screw all that jump to the top so the logic is a little bit cleaner beyond that it's also used in go linked or go imports I have to get back to that it is actual go code I was referencing surprisingly key points here I have notes here so that I remember following the go recommendations rules if you like you'd never let your panic outside your package right you could totally just make it panic and not catch it but that's not very friendly you've just crashed whoever is calling you the whole point of the panic here is not to actually panic but to just clean up the code so by catching the error and rethrowing it as far as anyone knows you've written the whole area because not no they get a nicely formatted error and the exact same error that they would otherwise get at unravels recursion which we already mentioned you don't have to re panic which I also mentioned and ok so another thing for those that haven't seen panic or use this sort of thing panic doesn't kill defer so any defer stack that you have it will unravel it it doesn't have to be the first defer it doesn't have to be the last defer it doesn't matter it's a stack for those not familiar so if you make multiple defers it's a first in last out stack so all your resource cleanup and other stuff you would normally do in defer still happens you don't lose any of that sexiness that's all I have that's a very neat trick thanks for sharing