 Good evening everyone. So first up my name is Brega. As Winston already introduced, I work for Luxulat. So we're a e-commerce company. It's a cosmetics and beauty products. And I'm going to speak a little bit about the things we learned, observed from IT audit. So it's something we earlier in the year went through. So the whole objective of this talk is to kind of share what we went through. Maybe we learned a few things, some things we observed, some things we're like just more learning for us. So we want to share it with the community. So nothing super technical, it's kind of a bit abstract. So if you're looking for something a bit hardcore, forgive me, sorry. But that's kind of the theme. OK, let me just get rid of the menu bar. No, still sticking around. View? Ah, crap. OK. OK. Thanks. Yeah, so, OK. Sorry about that. So first up, what's basically an IT audit? Or what is any audit for that matter? So it's just an independent entity outside of the two parties involved who come and who evaluate independently that whatever is presented is good. It's in a good state, et cetera. And in case of IT audit, specifically auditors who come on board who basically check that whatever you presented as your IT infrastructure, your IT assets, goods, scalable, whatever they decide, business metrics that they want to look at and as stated. And it's also usually a part of a due diligence process. So basically when one company acquires another, they just want to go through this process to see whatever is stated, be it revenue, be it IT, is pretty good. So that's the official interpretation, but the unofficial one usually is lots of, lots of questions, things that you actually make a decision on and you wouldn't even think that you'd have to justify it. So you'd probably have to go to someone and have to justify it like one example writers, right? You choose to host on Herocone. You probably have your developer-based justifications. And then you have to go and say, hey, okay, why did you not choose, I don't know, say AWS? Why did you not choose X, Y, Z? What is your motivation? And it goes on and on and on. So that's one more way to kind of look at this one. And why am I speaking about this? Again, something Vincent broke a little bit. So my company, Luxola, got acquired by Sephora. So Sephora is a cosmetics retailer owned by the LBMH group. So they acquired us earlier this year. So we had to go through this elaborate due diligence process. And that's kind of where the entire experience comes from. So first up, right? So the first thing that got looked at from us is the IT architecture, right? So what basically is an IT architecture? I mean, it could be a few different things. It could have a few different interpretations. Like it could be your application topography, right? Whenever a company is small, you're working on one primary product. That's your application. But as in when you grow, you evolve into having, you know, either multiple services, multiple products. But what are these? And also, once you have multiple products and multiple services, how do they interact with one another? What kind of data flows do you have? What kind of interfaces do you have? What is basically the relationship map between the different services you've got? Also, when you go applications, every one of them have a hardware stack, a software stack. What is your stack like, right? In Rails, you know, you have this famous DHH, and then the custom stack. So I think you definitely know what this one is. Also, what is your build and release process? So how many times do you guys deploy in a day? How do you package your code? How do you build it? How do you release it? Where do you release it as well? And finally, the hosting and the deployment topology. So how do you actually deploy? Do you use Chef? Do you use something like Git and Heroku to deploy? Again, that's one more thing that was being asked at. That was something that was curious. So in case of lock-stroller, right? So our application topology kind of very roughly looks like this. So we have the e-commerce engine. So that's the first one we started building like four years back. And that's our primary product of sorts. So it handles a whole bunch of stuff like, you know, the registration login, checkout payments, navigation search, et cetera, et cetera. So anything you would probably typically look at from an e-commerce engine of sorts. But also I think besides what the user sees, you also have the backend systems and the CMSs that come into play. So anything and everything can be customized from the backend because we have a staff of about 100 people. So these people interact with the systems on a daily basis to manage the systems. So whatever tools that's needed for those people to do, they work efficiently. Once we got that ticked, so then last two are completely different products. So we have our own warehouse management system and our inventory management system. These are products we build because we do our own fulfillment. So if you place an order on Luxular, it goes through the e-commerce engine. It flows down to the order management system and finally into the warehouse management system. So this is again one more application that we built ourselves. So that became a part for topology of sorts. And finally payments, right? I mean, if you are in this region of the world, payments is like a very interesting space. I think there's a whole bunch of alternative payment types besides the credit cards. So you go to Malaysia, you got one trade, you got a completely different trend. And if you're focusing on the ASEAN region, then it's a pretty complex tool in itself. And then you got your APIs, right? So there's a whole bunch of APIs. Your application is going to be dealing with be it your exchange rates, be it your... If your warehouse management system deals with the logistics APIs, there's your MailChimp, Mantral, your SendGrid, whatever it is, your communication-based APIs. So all the APIs kind of get bunched together one place. And finally your analytics. So if you're doing any kind of BI reporting, your data warehouse, your Google Analytics, so these get bunched into one bucket as well. So the one important thing kind of was a key takeaway for us, from us, for this was like a... These things, right? I think I'm not going to detail of every one of those. I present the last one for sake of time. But this is something all of you guys know by your head. All of your developers, you work on it, one way or the other, and you have it in your head. But we never bother to document it, put it in a place. So if you do it, it's usually out of date, right? So you basically draw just rough diagrams like this. You draw some squares and you leave it elsewhere. But make an effort to put all the collective intelligence. Basically, probably the senior members in your team are probably going to know this. They're going to be working on their respective components. But put it in a piece of document, share it in whatever is your favorite tool, be it your GitHub wiki or private wiki or whatever is your poison, right? Try to document it. Try to put it on a piece of paper or a piece of document. The other thing we kind of found that was pretty good out of this was that once we started doing this, onboarding your developers was really easy, because once we bought them onboard, we had a document that could give them to them and say, hey, okay, look, this is kind of how our systems work. This is what we have. So I think you have your tests and that give you a level of documentation, et cetera. But that's way more fine-grained. You want the bigger picture. And as in many systems, become complex. It's important to bring these layers outside of your tests. And I think that's one thing we kind of took away from this first part of the exercise. Next up, develop and practices. So you buy a piece of software. You want to know that it's actually well-made. It's not going to break one month from now. It's not going to break when your traffic quadruples, et cetera. So a lot of focus goes into how this was actually made. So how do we actually develop? So I think this was one of the easier things for us to explain. Because when you're going to develop, make sure that you cover a couple of the basics. So whether you do agile, whether you do waterfall, make sure everyone in the team understands what it means and what the implications of one or the other. But also, I think how do you care for code quality and other metrics really matter? I think one of the things we actually already were practicing before the audit was it's Ruby. So you don't try test, then people are going to look down on you. So you do test for your own goodness sake. It's a very, very testing for one community. So I think we are one of them as well. So we had testings kind of pretty much nailed down. So we had RSpec going. We have unit tests. We have a couple of integration tests, which actually we really like. So we have Selenium covering the key flows in a e-commerce engine, like the sign in the registration, the checkout. Because if it breaks in any one of the builds, then our app is technically broken. So we also had some sanity and smoke tests. So basically, we are deployed in Heroku. So all the tests are first in CI. So CI automatically runs all the tests. And once it's good, it pushes to Heroku app. That's our staging application. And then we start running some sanity specs. So Selenium starts pointing to here and starts doing it because this is more closer to the production environment than your local browser. Also, we had in particular to code quality, we had a couple of things that was going for us. Maybe these are things you guys can look up as well. So first, we had code climate. So code climate covers quite a few stuff. So if you guys don't know what it is, it's basically a hosted code review solution for Ruby and Rails applications. But I think they've moved on to other languages as well, but started out from Ruby and Rails. So it gives you a whole bunch of metrics around how complex is your code, it grades your code. Any time there's a security issue or any time there is a bad styling issue, it points it out. You can actually have a unified voice across your entire code base. So that's one thing we had in place. We also had a tool to measure our coverage. So that's one more thing we keep in mind because it goes back to testing. How much coverage do we have on board here? Also, I think we use Hound. So again, one more styling tool from there. Yeah, so lastly, documentation. I think that flows back to the previous point. Yeah, your tests are a level of documentation. I think if you write your tests and you put in your described blocks that give you a finer detail, but also I think have another layer of documentation that's between the previous one, which is your architectural level documentation and your tests. So maybe you write a new module, you write a new, I don't know, save product. Just again, put in a more developery words what it means purely for the purpose of yourself in the future because you tend to forget things and also for other developers. We found that pretty incredibly valuable. We had it missing for some pieces, and that's our learning because for those that were missing was the hardest pieces to get new developers on board, or the hardest ones to refactor as well because it was all floating in the head rather than having it in one place. All right, so the next one's a slightly bit of a controversial topic. So one of the things they look for is technology ownership. So if you claim your technology is one of the key strengths, it's like ultimately who wants the technology becomes this question, right? And then especially if people are not open to this community, they have this averseness to open source. So I think we had to kind of win that over a little bit because the moment people say open source and they say GPL, lawyers in particular freak out like crazy. So the most important thing here is, I think if you're any good with your words, you can probably convince half the people that it's not as bad as you imagined, right? But what actually really wins over people is that, okay, yes, we use open source, but we know what we're doing and we know what are the license that goes into our project. And we have some process in place to monitor it because that was the key thing for us. And that's going to be the key thing for you guys as well because anytime you add a gem, anytime you use anything from there, you are adding a new dependency, you are likely or not bringing in new license. So I'm not saying don't do that, but just try to get an understanding of what you're actually doing or what kind of license you're bringing in. So understand what's the difference between your Apache versus MIT versus LGPL versus your GPL license. There's a whole bunch of others actually, but try to understand what the implications are without going into the complete details of it. So one of the nice things we did was this really nice software called License Finder that's from Pivotal Labs. So basically this is a tool that pulls up all those dependencies from your gem spec and gives you this gigantic list. But that's pretty useless, right? If you have, in our case, I probably have about 300 over gems. The list is pretty useless. But the useful part is you can actually whitelist and blacklist saying, okay, I'm never gonna, for example, take a GPL in my project. Or I'm always gonna approve only MIT and Apache 2 and nothing else, right? So you can potentially do it. And what's even better is that it's tied to our CI, so no one monitors it or CI does it for us. So anytime one of our developers adds a new code and puts it in some place, and if it breaks a dependency, a build is gonna fail and it's gonna flag everyone. So this is, I think, one of the really nice things that came out for us may be something for you to explore. I have a quick question. Does this prove like transitive dependencies? Because with GPL, I think for us, we didn't use GPL or wherever dependencies we had GPL, we either rewrote or use some other dependencies. Actually, I think even for LGPL, it was kind of like a gray area for us, actually. So we don't, but yeah, I think that's kind of what it boils down to, essentially. All right, so the next one is security. So obviously, the application you buy needs to be secure. So just make sure you do the same things, right? So just understand how you store your sessions and what the implications are. Do you store your session in your cookie? Do you store it in your database? Do you store it in, I don't know, Redis and Uncached? Which is probably not a wise idea, but you store it in elsewhere, right? And if you put it in one place, what are the implications? If you put it in a cookie, can you put in sensitive data in there? Likely not, but depends on if it's encrypted or in case of Rails, it's usually just tampering production rather than complete production. So understand how session management works and what you're doing when you actually do a cookie-based session management, but also passwords. I have a bit of a run, but maybe I'll just stay a bit polite on this one. So please, please always, you know, hash your passwords, don't store them in plain text because I had this really bitter experience from a local company. They're a huge local real estate company. I shall not name the name, but I created an account with them and then I got this really nice, hey, welcome, and then here's your password. I was like, wow, okay. Now your first completely screwed up my password, which I probably might or might not have used elsewhere, but also, on the funny side, at least they don't need to ever send a forgot password email as all you need to do is just, you know, search for your welcome email and you have your password in plain text. So please don't do any of those. Denial of service vulnerabilities. With Ruby, your symbols never garbage collected. There's also a bunch of other gotchas. Don't create any records without sufficient protection in place. Like if you have registration players, bots can potentially attack it. And if it's an expensive process, you're probably running into a divorce. And if there's any process that creates your symbols and never, at least with the current Ruby, I think with the newer ones, it's going to be garbage collected, but at least for now, if there are edge cases which leads to this, keep an eye out for them. Now when it comes to hosting, yeah, with AWS in particular, use IAMs. So basically it's very simply put. If you guys are not aware, it's like a per user based security rather than a master security, right? And if you have root keys and if you have master ones, get rid of them pretty much ASAP because if they get compromised, your entire account is screwed. So make sure you create new IAM users for your production servers, your staging servers, your developers. Put them in groups. Try to make sure you have that covered. And lastly, I think a couple of the things I touched on breakman and code climate. Basically these are hosted tools, but also dev board, which is by our own and only Winston. So the thing with open source and having multiple dependencies is that you have lots of gems and lots of dependencies, right? And there are vulnerabilities being reported like, I don't know, every day, every week, probably even more frequent than that. How do you even stay on top of it, right? And if I'm probably a hacker, I want to hack somebody else, I'm going to look out for these vulnerabilities and it's very easy to tap on. So it's important to stay ahead of the game because if you have 300 over dependencies, you can't potentially upgrade them every single day or whatever. So Winston draws a really nice board that basically goes through your dependencies and automatically sends in a pull request especially even the security base with the latest version so you kind of get covered. So one more thing for you guys to consider. PCI compliance. So this is one of a big topic for us. I'm not going to jump into too much detail for this, but I think Dinesh Raju from Referral Candy, I think a couple of months back he did a really nice presentation on this full-on. So maybe if you want full details, you can kind of go back. But first thing, never ever store credit cards in your database. It's really, really too risky and you open up to lots of attacks and a lot of compliance requirements. If you do it, don't ever do it. It's not worth it. Also, I think what we had to go through is we were already compliance so basically it's just a very short just of how we got compliant, right? So the way PCI works is they first determine two levels. So first is how do you handle your card data? So do you store it in your database? Do you use a hosted party page? Or do you use transparent redirect or iframing, et cetera? So how do you handle card data? It determines your one level and the next level is based on the number of transactions handled. So if you handle millions of transactions, it's a different level of compliance requirement versus several thousands. So once you know your levels, the first thing is the SAQ, it's a self-assessment questionnaire. So basically a couple of compliance requirements which you have to go on and answer. It's not just a question of answering because you're going to certify and say, okay, hold to this or make sure we put in measures to hold on to those as well. And the next one is a pen test, penetration test. So this again depends on the level of your PCI, but more likely than not, if you're doing over 50,000 transactions a year, I believe you need this. So you need to get a third-party penetration test, security tester to actually come, ping your app and to basically say, okay, it's secure and safe and certify that. There are a couple of vendors who are PCI approved. So make sure you find one of those and have them scan your app. And this needs to be quarterly as well. It's not a one-off. So it needs to happen four times a year. So once you have these two combined, then finally you get your compliance. You can present it to your banks, put it on your website, et cetera. Last one is on-site audit. This is for the last guys who have huge amount of transactions or who are stupid enough to print these on a piece of paper or elsewhere. So I think these guys usually have on-site audit and people come in and really make life hard for them. Last is scalability. So I think this is, I think Board Winston touched on a bit as well. So Rails can scale. So I think first thing you guys need to do is anytime you have peak periods, so just note down the metrics for those. I think for us, we have to go back to two or three years. And the metrics we have to get were like stuff like, you know, what are response rates where, what is our RPM, how many users do we have on board? And if you use Neuralek or something else, what is your appdex course, right? So it can also be a business level metrics. How many orders per minute do you fulfill and what is your system capable of? And people don't usually document those, but try to have, I don't know, maybe have a hand up on these things. Next up, I think it's very important to have, you know, a controlled environment to be able to test traffic, right? Without that, you're most likely guessing they're not, because if you're handling, I don't know, there's a lot of questions. And the question is, can you handle 3,000? It's going to be a wild guess saying, oh, yes, we can. Because your bottlenecks can be really, really wild. So it can be your database connections. It can be one of your dependencies. Your bread is might blow out. It can be a memcache. It might be some missing index that is only frequently hit when the traffic goes really, really high. So it's very important to identify what your bottlenecks are. So try to do a load test and try to do them on a regular basis. I think for our case, we're probably doing it way more frequently than we were before. So make sure you give sufficient care to this as well. This was something we're kind of completely ignoring. But there are also a couple of other tools you can use, like Apache Bench and Jmeter. I think Jmeter is from Java world, but these are pretty standard. So one more thing you guys can look for. Yeah, so I think that's it from me. Thank you so much. And yes, we're hiring as well. So if you want to work in a company which uses Rails, which gives you, which does a lot of these cool stuff, but also gives a stability of enterprise, yeah, come speak to me. Thanks. I think we don't really take a super fine look at it, but I think we use Debo to it. I think we're using it not in all our products, at least for the moment. Just so I think it's purely for security perspective than who is the contributor and what the contribution is going. But for the moment, no, we don't. I think we're more focused on the security side of things and see if we are covered. Just now you mentioned about the penetration test. You have to do it like four times a year in order to get it certified. Does that mean that you already know you need to prepare for this? That's why you have to do it like four times before your acquisition? I think we were already certified before. It's also a really unique case. Ideally, we don't have to do it, but we're working with a Thailand bank who are making, in my opinion, unreasonable demands for it. But then again, it turned out to be good when we actually got acquired because we were already doing it. So I think it covered us a bit. Yeah, like the PCI test, are you using like a vendor like Komodo or something? Yeah, I think we use a vendor called Security Metrics. So again, it has to be a third-party vendor who is approved, but yeah, we do use a vendor. On your production? Yes. So how long did the whole IT audit process take? It was actually close to a month. I think there's a whole bunch of things actually skipped through. There are stuff that goes along like they look at your org structure, what's the scalability like, they look at what's your budgets, what is the strategic plan you have for IT investments, et cetera, et cetera. So things that I kind of have to skip at this point in time. So I try to keep it more focused on a bit, more on technology than this. Yeah. Any other question? So for the stuff that you've portrayed on it, was it like, just, was there a lot of whole shit in this trip? Or was there a lot of, oh, we didn't know about this yet? I think it's actually a fine mix of both to be fair. I think especially on like, attached the bigger challenge for me was to actually speak with a bunch of auditors who are very used to working with enterprise systems and who are used to very waterfall model and deploy a couple of times a year. And then I have to go tell them say, okay, we deploy multiple times a day. And they're like, what? So that's the first reaction. So and then next question becomes if you deploy, then what happens to your stability? So how do you even test? How it tests you to automate it? And then the question becomes, okay, how is it automated? And what do you actually do for automation? But the bigger challenge was more along the lines of explaining and taking our point and our viewpoint to them. But also there are a couple of points, the raise, which are valid concerns, which I think we were running at a breakneck pace and things we had obviously ignored. And we willfully ignored some of them. Some of them were definitely off shit moments. So yeah. I think the safety of fit usually goes along the lines of when we actually insert, right? So I think when we add a new dependency, that's kind of we see what the library offers and we actually validate if it's something we can potentially do it ourselves. Is it worth injecting another unknown dependency? Because I think we've grown to a point where I think we have like I said, 300 over gems. And that's like quite a lot of dependencies already. And I think we're breaking down our main app into a couple of smaller modules already. But if any such check happens, that usually happens when we actually add a new one rather than on an ongoing basis, right? When we do it on an ongoing basis, the focus is more just that the one we use is secure rather than or has the latest version rather than. So you have some sort of like the code to be shared in the same? Yeah, I think there's a few. I think the code climate is one such. It does more static code analysis, gives you some stuff. Breakman is pretty good. If you haven't used it, just try it out. I think that's one more thing. It's really, really good. No. Thank you. Thank you for that.