 Hi, I'm Chang from Red and I'm here to share a few of our humble experiences while we're doing build testing. I know you guys are pretty much really familiar with unit testing and why should we do it? But I want to say something about it because strictly jumping into a cold makes me feel like stupid. So why do we do unit tests? Is it really slowing us down? I think it depends on the team size and the mutual trust between team members. So for example, I used to be working in this game where everybody is like a really mature engineering. And everybody mutually trusts each other. They trust the guy's implementation so that they feel that oh it's really slowing us down when we're trying to do this initial implementation with this MVP so that we impress people so that they can put money in our project. In that case, the unit test is not needed for the MVP product. Where in the long run, you still need a unit test, otherwise your code will be simply rotting. People can't read it, people can't maintain it. The other thing is that tests is really documentation for your project. So if I write a big library, I convince people to use, I will always have them pay to look at my test cases. So these are the examples you can how to use my library. So the second thing is that does test coverage mean anything? The general answer is that my opinion, my personal opinion, it should be always above 60%. But is it meaningful to reach 100% test coverage? It's okay, but if you're trying to reach a higher coverage by writing flaky tests, that's a big no. So that's one of our experience that we have this CI server that failed basically at one of the two summits. Sometimes it's because the test is just wrong, sometimes it's because it's flaky. It's using external resources that depends on the database. It reads readies, but this value might be available at test time so that your test is flaky. It depends on something, it depends on something might not be available. But unit testing is done as a work group. It shows that we love the code, but it shows nothing more. So except unit testing, you need a UAT, you need integration tests, you need stress tests, you need small tests. All these are needed for your project to succeed, for your product to go to production. Okay, so the key to not write flaky tests to me in our experience is that you should try to mock. We need to mock. So what that seems we need to mock? HGP calls, Mexico readies calls or whatever database or cache layer you're using. Also people tends to provide SDK to you and this SDK writes their implementation. You're not really testing the other people's code, you're trying to test your code. So mock them. In Golang specifically, if you're testing against a package, we shouldn't be testing anything outside this package. So anything with a package name dot something, this indicates that it's an external package. We shouldn't be testing other people's implementation. But anything within the package should be tested. I think that's a rule of thumb. Then we can write code in many ways. Whether we can mock easily sometimes shows that we can change our implementation easily. So let's say I'm trying to read database. I'm trying to read SQL, raw SQL directly and calling Golang SQL implementation to read data from Mexico or write to that. It shows that if I want to change my mind and I don't want to use a Mexico anymore, I want to go for no SQL. I can't do that easily because my code is flooded with all these raw implementations without proper writing. So without proper writing, you can't mock and you can't change your mind. You can't change your implementation. So other things, if we can mock easily, your logic is likely to be loosely coupled. So example, a very simple test, a very naive simple test. So let me explain the implementation, which is on the left side. Get from TV is something, so it's a key and we're supposed to return a spring. Or if there's error, we can error. So suppose there's another sub-party package called dv. And in that package, there's a get method. What we're really doing is that try to use that function. And if that function returns an error, then we return error. If not, we simply append an underscore OK to the value retrieved. So a naive way to test it, of course, is just call the function and assume that with this test as key, then result will be text as well. And so that the result should be text underscore OK. So we test it. But this is apparently very depending on your dv. What if for the key test, the value is not test, then you're fucked. So by the way, on the left side is also a test. It's also the test, but it's not implementation. The implementation stays the same. So the shorter path from dv, we're really trying to mock away the dv get function here. Before we can mock it, we have to make sure we restore it. Because after this test, we may be running some other test code. Then you simply can't replace the function pointer without storing it. Restoring it, which will break your other test pages. Then this is a very bad example for concurrent tests, because you are mocking the function pointer. Other tests may be mocking the function pointer. And if two go routine testing together, you basically fuck again. But you generally get the idea. Use a default call to restart function pointer back and then mock your function pointer to the test. This is better than the previous one, because it doesn't depend on what key is stored in the dv. I can always make sure if I'm the only tester doing this test here, it will always pass. So apparently that's just a happy path. So what if we want to try some corner cases, when the function actually returns an error. So you have TTT, you have table driven test. You basically release your test cases and test them in a loop. So that's nothing fancy. And even better way, dv.get is a SAP package. And the get function is really a global function that many tests may be assessing at the same time. So if we mock it, replace the function pointer at runtime, it might break other tests. So how to make sure we are not doing something so in the wrong way. So we have a local wrapper called dv.get, which is starting with a small list. That means it's within the package. If it's within the package, I can mock it. I can mock it and I'm sure that if any test running on another package is not going to see this function pointer. That's why they have no chance to mock them. That's why changing this function pointer doesn't really destroy anything on the other packages. And usually if you don't run low test based concurrency level in your own package, all the tests are supposed to be running one by one. So it should be good. It should be good. Then of course on the left side is the implementation. It's the updated implementation. On the right side is the TTT. This time we're still doing the same thing. We make a deferred call to replace the function pointer back. And of course you can come out some test libraries and do this. You don't have to write this ugly code every time. Maybe it's just a one line call. Then with the TTT, we're actually mocking the private function in our own package. It's a bit better. It carries a lot of problems, but it's not the best way because what if you have multiple places assessing a private package or external resources? They basically have to create a local wrapper for each and single one of them. Then your code looks like how the shape or what sort of code is that. Also you have to make multiple deferred calls in any of your test cases. So that's not something we mock. So the ultimate way is ultimate to our experience. It's not to be corrected. There must be a better way to do a lot of things. But this right now, the state of our graph, we're basically trying to set up a context. So for each function that does something need to be tested, the context will always be the first parameter, first input parameter to this function. So what is within the context? Usually you will have a setup. You can run this in your package in it. You can do some pre-setup code. You can even do some setup in your text. So that's not the thing I'm trying to cover here today. But basically you have a way to initialize the context. Let's say I have a DB helper with a lot of DB function calls. I'm going to set up a context. The context really is just a hash map. So within the hash map I have a DB helper struct. So this struct is very special because its member is a function pointer. Its member is a function pointer. It means that you can replace it. But you can also get your own very specific copy of this DB helper. And it's in the context, it's in the hash map. So when you test the next logical thing to do is that you set up a fake context. So in the fake context you can have a fake DB helper. So for the fake DB helper you can always the real DB helper will test. But you're supposed to replace the DB helper to get DB member which is a function pointer. Let's replace that. And then you can basically do whatever you want. You don't have to defer because this is your own very private object when you're testing the function. So you can see it here. We're making a context here. So this context is just within this test function. No other test function or whatever production call is going to see this variable. As far as it's safe to replace it. That's why it's trans safe. So I feel this is the ultimate way. For example, you know, web text is basically doing, please. Yes. Can you give us an example? Let me give you an example. So why context is helpful in business logic development. So for example, web is basically a red hazing bag. Right. We have a booking service as the service sitting internally processing passengers request and try to find a matching driver. Then make it happen. Make the red happen. And you know, picking on drop off. There are so many things happening in the whole lifetime of a booking. So all these things can be stored in the booking context. Right. And also the helper functions. Suppose this booking, you know, I need to flush to DB. So I need to flush to my heart storage, but the data become cold. I need to flush that to cold storage. I need to read from these, these storiders. I need to, you know, send a message to driver. I need to send a message to passenger. All these are helper functions. So they might not be available until test time. Right. When you run your code on CI, when you make a submission, the key here is to mock them. So in the booking context, there will always be a helper. So helper has many external functions that can be mocked at view when, you know, you're trying to set up a text. Also, we are storing whatever we are storing in the memory, putting this into the head map. So I hope that answers your question. We are actually dumping other things in the context. So the good thing about this is, the good thing about this is that we are, we are really free to replace them at any time. So that's why it's trade safe. It's very good. Please, Michael. Yeah. How do you compare this one, this way of writing the function with dependency injection? Right. For example, in this one, you have, you set up the context object, right? However, you rely on another package, like a DB, right? But instead of passing the string of a context on the function get from DB, you're passing the interface of the package. Like in the fact, in the case of the package, we have a getDB method, right? So you're passing here, then later you can mock it. That's a very good thought. So we went down that path one time. Let's say this DB package, we have an interface that defines all the methods we need, right? It starts the next logical step is that we can have a mock implementation of the DB, right? But what if the DB has like 20 methods? You only need, you only care about one of them in your test. You always have to mock all of them to call yourself an implementation of that interface. No, I don't think so. For example, here we need the getDB method, right? So in the test find, I will create a new, for example, a new truck. Yeah. The truck has a getDB method. So here in the implementation, instead of depend on the DB object, I have a like getDB adapter. So on the test, I create a new object, we also have getDB. Yep. So I only have to define the interface for that method only. I think I got what you mean. So with some manipulation or techniques, you can get away with implementing all the other nettings, only implementing that. But it's mock up to me, right? What if you can simply replace one function pointer? So compare, there were some design decisions here. I didn't want to go into that, but let me just briefly say what I think. Go-la interface is not costly. You're passing an interface around, you cast the interface back to your strut. Actually multiple things are happening. It's not like in C++, what would you call it? Dynamic cost. It's not like that. So go-la interface type actually stores one type and one pointer to the actual data. So casting that to your strut is not free. If you have this DB helper, it's just a pointer. Passing it around is quite free. It's like passing an integer around. And replacing the pointer in the object is also sort of low cost to me. That's why for the simplicity for the low cost thing, I choose this way to inject the dependency. But as a third party package provider, I tends to provide this sort of mock at page level so that you don't have to write everything in the DB helper and put it in your context. You can simply like if I'm the writer of this DB package, I'm going to come up with this new DB helper sort of helper method. So this app method will return this helper with a list of a function pointer definition each pointing to their default implementation. But you're afraid to change one or two or three of them at any time. It feels like it's more flexible. Alright, so I think the interesting discussion over there, here I think you're sacrificing some type safety to use like it. Exactly, so I was hoping nobody was pointing that out. Yes, you basically lose your protection over these around time. So if you don't agree on this, we have a big group of programmer who doesn't understand this principle and they come here, hey, I can change this, I'm at review. So this one definitely needs some communication and discipline and agreement. But my point is that this is the way we find it's very efficient doing concurrent testing. So test us how the people in the team that get this concept. So since this opportunity, I'm going to put this slide here. We are still extensively hearing. We are so short-hand of people and we are so enthusiastic in goal. Please, if you're a go-round programmer, you want to help us to build our pipeline to make our stack better, or you're interested to do something meaningful for the 0.6 billion people in Southeast Asia to make their life easier by providing a better way of transportation, please join us. You have questions? I might not have answers. Any more questions? How long will your stack run? My whole stack? Do you have all the tests? That's a question very tough to answer. We have so many tests in the system. So one of the tests is UAT. It's user acceptance test. It's very important to our service that we have to simulate other scenarios, possible scenarios generated by user that what if the user make a booking and there's no driver or not. What if a user makes a booking and driver leaves the booking and cancel? We go through all of this. So it can take from minutes to 20 minutes. It depends. So we have a smart detector because in grad, all the goal programmers are sharing one and only one repository. For the sake of everybody owns the code, everybody can jump in and see what other people are doing. And you literally can submit this to any branch, to any of the service. So you only need the engine also of the team to be a code reviewer. So to do this, we actually implement some kind of smart detector with the lines of change you just made. How many projects are actually affected? Then we're going to test against all these and make sure you're not breaking anything. So once those are satisfied, your teams are really trying to bring on UATs. Please. A bit of a shift. What is your testing framework, not even the testing infrastructure, the CD pipelines that you are using? For example, Jenkins, Go CDS by ThoughtWorks, or ThoughtWorks by Pivotal. So the short answer is Jenkins. The long answer is that we have been travelling from Traveys to, you know, drawn dial, from drawn dial to Go CD, from Go CD to Jenkins. So each of them we learn about less. Either it's too complex, too slow, or too costly. Then we set up our own Jenkins, our own, you know, Amazon Cloud. So our Jenkins, right now, is the state of art. It has this master slave structure. It can, you know, spawn more slaves on demand. And it is pretty smart. We are pretty satisfied with it. We're still trying to improve it. But, you know, testing framework level, we are not using anything. We are not using anything. We are using a CD file, yes. But that's just for a super convenient assert statement. So we can assert this and assert that in our test cases. We don't have to write out if this and stop testing. It's sort of some rubbish book. But, you know, for user acceptance tests, we actually try to, you know, come out, come out our own, you know, test random framework, like, simulate other scenarios and generate other outputs and compare are they the same where you start and it will be. So, yeah. Are your tests written in R-spec style or J-Unit style? What is that? Looks like J-Unit style. Yeah, J-Unit table. Okay, it looks like J-Unit style. I didn't know, so you say. It looks like there. I think yes. Well, it's the whole table J-Unit style. Okay. It looks like your tests are also table tests. So, you pump your test factor through and check all in the window. Yep, yep. So, table test is a must. And please. Instead of doing a mock in the test for the DB connection, there's very new reason why you're going to have the developer install the DB in your local environment. No, we do that, we do that. So, no. This is the context of the J-Unit testing. So, the goal of J-Unit testing is the package level. So, in the package, I'm really trying to test my implementation against, you know, if the DB method is retending me something or not retending me something, it's all the statements and I try to cover them. But in reality, you know, in the DB package, of course, I'm not going to set up a Mexico mock and try to, you know, intercept SQL while doing all the fine things. I will go straight to integration tests. I'll be writing. I'm trying to store this data in Mexico. Then I'm going to read it. Are they the same? I'm going to update it. I'm going to load it again. Are they the same? So, I do a lot of integration tests in the DB package. For the user of the DB package, they can be pretty confident that this thing will either work or, you know, it will serve me a proper error so that I only need to handle my IP path and the part that handles the error. Yeah, but for your UAT and integration tests, you point to a real database, right? Of course. So, for the UAT, why is it so slow? It takes, like, 20 seconds. It's because we are reading and writing. Real data. 20 seconds is so fast. Learn to use other languages. Learn and some other things. You know, 20 seconds is the best time. If you do this, not even proper way, you're building yourself to take part of it. Yeah. Please, sir. Like, for example, you have a metal. We use usually DB package. And the DB package, you know, the R&D or R&D, they have, like, all of them, have a metal, like, from where I see it, where I wear it, and where I eat it, and where I need to build a query. Yeah. My SQL. So, basically, the metal itself is only contained in another metal. Like, for the DB package. Yeah. Yeah. So, how would you test that one? We're not glad. But we agree that we... we don't actually run another metal. We mark the external. But all the metal, everything, all the live code in the metal is only code in another metal. Yep. So, how would you test that to make sure that the final query is correct? If I understand your question correctly, it involves some code-level separation. So, you have business logics that needs to assess data, that needs to read or rep to the data. Right? So, these things, I usually rep them in the package itself, like, it's DB Assizer or whatever. They take care of the database and the cache layer at the same time, or differently. But anyway, it's correct in the package. So, in that package, in that package, you can actually do some integration tests, because it's supposed to do that. Otherwise, you're just confusing yourself from this from and this to what is it really doing? Marking doesn't make any sense there. But to the upper level, to the, you know, if we layer the code in this way, then to the top of the code is the logic in the rotation part, then it should be pretty transparent. Also, I can change my mind at any time, right? So, this DB Assizer I'm reading from this cache and this Mexico, right? I can change to elastic search at any time. So, part of this is very important that you arrows and dependencies are one of your own. This one doesn't depend backwards. Therefore, as long as you know that you've tested in the one every level coming up, then you don't have to worry about... No, for example, if I use the third part of this, like for example, I want to use a package from a Github repo. I need to import it in our project. And the thing is like, for example, the goal or M, I have found method to get data from database, I use the goal or M to build a query. And at the end, I have the dd.get to return the data, actually run the query and return the data. But the thing is, in my whole function, it's only the code of another library, to the third-party library only. So, basically, because we know the third-party library, so how do you write it in depth or not? That's the thing. That would be in your integration test. So what you do is you write the unit tests for things, assuming that it returns correctly for everything else. What you've done is you built your interface around the external party. So, if you test everything around that, and you test all your stuff through the integration test, then you have a good coverage and you have a good coverage of your tests coming in and you test going on. So, yes, I don't think the integration test is meant to be replaced by unit testing. You definitely need an integration test. The thing is, in my point of view, if we can, let's try to campaign the integration test to a certain specific package or a few of them, so that the logic to the logic is pretty clean. I don't need to do integration tests in my business logic package. I can rely on those special packages. I can run them on CI. If there's change, if not, just assume that the integration test that's been done on this part has been changed. Let's just verify the unit testing which would be super quick. Any more questions? All right, thank you, guys. Please. If you're interested in grab this, let's have a chat. Grab a beer. A beer, we can sit in it. Thank you, chance. If you would like to join grab, you're going to tell me. That's why I also program.