 Wow, that's a male, jeez, all right. This is gonna be a long night, guys. Apparently, I'm the only one here. So, my name is Justin, I'm from Glova Pass. I apologize that you have to listen to me for the entire night because yeah, that's the way it's gonna be. So, what is this talk? This talk is basically about two years of Rails in Glova Pass, but it's more about how we as an organization have come up the mistakes that we've made and I want to give it back to you guys and expose our mistakes to you, only so that you guys can one learn, make decisions about how you guys affect your own organizations and take a little bit back at how we do things. All right, so who am I? I'm 34. My name is Justin Lilly, CTO of Glova Pass. Born and raised in Seattle, I've been doing startups since I was 26 years old, moved to Hong Kong around 29 because I was tired of my life in Seattle, and then came over to Singapore to do Glova Pass with Jeff and Rob, who are the founders. And I like to think that these two pictures kind of represent who I am. On one side, I like to do serious things and pretend that I'm like this guy who's got it all together, but realistically inside, I'm like just it's a party going on in my face, something like that. Okay, cool, so before we really dive into this talk, what is this talk about? It's our mistakes and so I apologize if this talk is a little bit chaotic. And it's supposed to be like that because a startup life is chaotic and that's how our journey's really actually been. And I want you to agree or disagree with some of the choices that we've made and that's kind of the point that maybe some of them you like, maybe some of them you make you angry, but if nothing else, it's our journey and that's what I want to share with you guys. So what is Glova Pass? At the heart of Glova Pass, for those of you who don't know, we are a network of boxing, spinning, yoga studios, pole dancing for some maybe, I don't mean that. And you pay a subscription anywhere from $100 to 190 sing. Depends which market you're in and what deal you got, so being real. But you book a class and then you go to attend that class so you go to the yoga studio and you go in and we facilitate the lesson creation all the way to the booking. So for us, Glova Pass was founded on March 15th, launched a couple of weeks thereafter, not March 15th, March 2015. And the first city was Singapore, soon after it was Bangkok to buy Hong Kong. Today we're in nine cities, we've opened up more but we've also had to close down some cities. Always a sad time when we have to do stuff like that. But the cool stuff is we have two concepts which is called Glova Labs and still boxing. They're inside the basement of the OUE building in downtown. And you can see a picture over here, this is our boxing concept. And so for Glova Pass, this is kind of our heart and it's being able to facilitate people having healthier lives and being able to exercise better and doing that across the world. It doesn't matter where you are in Southeast Asia, you're able to go and visit a studio that is part of our network. And so who's the group behind it? We call ourselves Dev Awesome. Okay, but we've been on rails for about two and a half years. We've, through those nine cities, we're about 1200 studios strong. The cool number for me is this one. It's 910,000 completed bookings. So this is 910,000 visits people have made to various lessons in studios. And we're really proud of this because of all the health that we're, you know, living healthier lives. And I think some of the Dev team has actually started, you know, to get in on that a little bit. And of course we've done 1.9 million lessons, so not every lesson has a booking in it. And in the past year we've had four nines of uptime, which is pretty good for a startup. I don't know, okay, no one else is interested. Okay, cool. So that's the people, that's Glova Pass and the people behind it who run the technical side. So I'd like to run you through a timeline of how we came as a development side. And then what I'll do is I'll show you some very exposing and specific examples of the atrocities that we've committed against the code base. All right, here we go. Okay, so first commit. I got to actually write initial commit of GlovaPass.com. Initial commit of the blog, anyone, no? Okay, so the goal was literally anything to get it out. That was as fast as we can. Build whatever, do whatever it takes. So what did I do? I was the only guy in the beginning. So what did I do? I put it on Digital Ocean as a one server machine running passenger MySQL on the same server. Single point of failure, right? And it was fast because it's all in the same machine and runs Capistrano 4.2 Rails, a decent test coverage, which was the thing because by yourself you kinda wanna make sure that the stuff that you built isn't gonna break later. At least that's how it started in the beginning. You can kinda see where my voice is going with this. But a couple months later, as I said before, we totaled into four cities, Singapore, Bangkok, Dubai, Hong Kong. We got translations. Here's an embarrassing picture of myself. As you can tell, I like to do these kinda things to my own person. This is me getting married. And so, of course being the only guy, and I'm born in Seattle, I had to go back to Seattle to get married. Well, part of the wedding, right, was in Seattle. And so what did that mean? That meant the only tech guy was on a plane for a while and asleep in not the same time zone. And so I had to teach the founders how to log into Digital Ocean and click the reboot button, but not the destroy button, right? And I don't know if you guys know, but this is two years ago and you could destroy the servers and there was no confirmation screen when you hit the destroy button. So you could do a destructive action and bring everything to a screen, anyways, yeah. So, and you know, the early days are the early days. So, CSVs were cool up until they weren't, and then they would bring the whole website down. And so enters in, delayed job, and a second server, putting it into the background. And, of course, speed is another thing that happens. Feature changes come down the pipe like crazy. By this time we're in four cities so we're getting people in the door. And so what's the first thing that dies is testing. And so feature changes didn't result in any good new tests. And so we're just flying by wire in the seat of our pants. Six months later, it became unsustainable for myself personally. So enter Marcos, James, and Leon. Marcos and James are here. Thank you guys for supporting me in my trying times. But we also added another three, technically four cities. Sorry, that should be a four. Shanghai, Melbourne, Melbourne, Sydney, and Taipei. And some of the cool things was we got off a delayed job, went into sidekick, one of the caveats, of course, was that if you run delayed sidekick through the active job interface, sometimes it can pick up the job faster than the actual commit has been transacted. And so the transaction has been committed is what I wanted to say. But what that meant was the background worker was picking up the job and there was no object there. It would go, where is this thing? It's not here. It would try again in one second and it would actually be there. So that was fun to figure out what's going on. We took our first foray into React, building the studio dashboard. So this is the dashboard that all our partner studios log into. And then we actually started trying to put some redundancy into the system so that we wouldn't have a single plan of failure. Got two HA proxy servers, three front end to back end. And our database was now run on RDS. What is it? The people who read into this will notice that this is possibly still in digital ocean, which is what it was. And our database was in AWS. And the reason why we did that because we were too cheap to pay for the entire stack of AWS. Part of the problems that we ran into was every now and then at lunch the site would go down for five to 10 minutes, seemingly randomly. And we found out within a day or so that basically we were running a backup job. It was a script that I downloaded. It's my fault, I apologize. Is that would do a full table lock when the background job ran. So it would lock the whole database out and nothing could be written. And of course all the servers are like, well I'm just sitting here waiting and then the site would go down. But these are the kind of things that happened in an early stage, super, super early stage startup. Taking us to a little bit about July. So we ran into that exact era that I was talking about where digital ocean and AWS, we had when you put your database in one location and put your front end servers in another location. Up until that point, all of our requests between the front end servers and the back end servers were running less than one millisecond full round trip time. It was great. But then one day we're all at the gym actually and our webs, we get Pingdom alerts, we get New Relic alerts, we're like, oh crap, the website's down. And we started investigating and it turns out the link went to from less than one millisecond to about a hundred millisecond round trip time. So 20 calls all of a sudden starts taking two seconds and that compounds really fast. And then so what we did was we did emergency migration from AWS over to digital ocean to co-locate the database to our front end servers. And then we moved it all slowly back over to AWS. And we decided, you know what, it's probably time to start paying this actual fee of getting the nice multi-zone easy failover stuff like that. So this is a lot of DevOps stuff, but it's kind of the stuff that is real when you're doing a startup. And of course, now that we have more people being able to be on the team, we get better test coverage, which was a benefit as we start to grow out. Okay, so everyone doing okay so far? All right, so here are some of the exposing things that I want you guys to be able to see what's going on in the background. So in the early days, sorry this is a little hard to see. I don't know if we can make this better by any means. This is the booking controller. And you will be able to see it in the slides hopefully afterwards, but the booking controller, this is creating the booking. So I go, I find a lesson that I want to join and my account, I just hit book and it hits this endpoint. So here's an example of just getting it done. We check for if the user is active. If they're not active, we send them someplace else. Okay, cool. Then the next thing we do is we try to update the user for some unknown reason. We check to see if the lesson is present and if the lesson is bookable. So the lesson's not bookable. We send you someplace else again. And then gets to the really fun parts is we get the, we get, we try to see if you've had a previous booking with this particular lesson. And instead of creating a new booking every time, we reuse that old booking. Yeah, and so this is, this is how I built it back in the day, two years ago. I'm not saying this is the best architecture to do things in. It's probably one of the worst, but it is what actually happened in some of the stuff that we did back in the day. And of course we now have to fight a lot of this particular code these days. Another thing that we did was we launched our mobile app. March 2016, we did our first 4A into React Native, which was really fun. Great project. Our second try into React. And this time what we did was instead of dealing with writing modular code and doing it dry, let's just copy and paste the code and make the modifications because we don't want to mess with the website, but we want to get the mobile app up and running as fast as possible. So we just copied the pasted in the code and then would make the modifications we needed. And as you can see, one of the things that results out of it over time is you try to build on top of this code base and eventually you can't build on top of it, so you have to make a dot one and then a two and this just continues to compound. Here's kind of the picture that explains the different versions that we've built. You can see this is the version one folder. It's this long effectively. This is version one dot one. It's only this long and version two is even shorter. So what this is effectively saying that we started building and then we wanted to make a better API but we couldn't have the full time to be able to convert the entire API over. So what this causes is your mobile app, our mobile app actually for different actions asks for different endpoints on different versions and that's problematic. Okay, cool. So, and then anytime you deal with a project last a particular amount of time is you'd end up dealing with some of the code base that you've taken in and I wanna say this first and foremost, I'm not trying to knock the maintainers of these gems. I just wanna explain some of the things that we've run into and these are our difficulties. They're not difficulties that I think are across the board and I think some of these gems you definitely should use them but these are the problems that we ran into. Active Admin is a very divisive thing. People love it and hate it all at the same time because it's a DSL that you have to learn and it's a complicated DSL but once you've learned it you can crank things out like crazy. You just want it. They need a whole page into an object, business needs that, okay, ship it to them in less than a day kind of deal. We actually did a foray into using Administrate and the problem, Administrate's beautiful and for developers it's great. It feels good and makes you feel like you're in Rails the difficulty is that to build anything simple like filtering, you have to build the whole thing from the ground up and it takes a long time whereas it's already done in Active Admin you just do filter something and you've got it basically. Next one, Access Tagable is really cool up until you try to join across anywhere like say a couple thousand rows and you make those joins it starts getting slow really fast. Access Tagable has a caching system built into it which works great until you need translations and then the caching system breaks pretty fast because it wasn't really built for that and that brings us to the next one which is our Globalize. This one Globalize is cool for translations and us being in Southeast Asia we all have to deal with translations I imagine and it's not really ready for Rails 5 it's kind of in beta too and it's been like that since July and so we're still kind of waiting for that to come along so we can upgrade our old code base to Rails 5. We think they're pretty much ready but we're gonna have to do a lot of testing to make sure that the beta 2 works. For our mobile app, Devise has a tokenization system and they actually pulled it out of the original device code base and what came out of that was device token auth and there's a couple of issues with sessions in it specifically we had to store all of our sessions in Redis you can't store it on the user it creates a bunch of 500 errors at least for our code base it did and so this is dangerous because that means Redis is now mission critical if Redis goes down we lose everything in terms of the user session so we have to re-log in and they get angry and stuff like that so we'd like to remove our dependency off of Redis being mission critical that'd be nice one day. Friendly ID is something good for SEO of course at some point and we run into slug collisions all the time so you have to work around that especially when for us we have lessons that are named the same a lot because it's the same class yoga 60 minutes at this studio but you know it's compounded over like 25 times kind of deal so we run into a lot of slug collisions and we've actually had to deal with how do we name things friendly yet at the same time unique. All right, cool so diving even slightly deeper actually pretty deep into poor architectural decisions so these are problems that I created for our team two years ago couple of them are we don't reuse old booking objects I mentioned the booking one before but specifically also what's killing us today is the subscription object and when you subscribe and then you cancel out and then you resubscribe we use the old object we don't create a new object for you when we should have been doing that part of the reason why we did it in the beginning was because of fraud people like to resubscribe a bunch of times trying to get better deals I don't blame you I'm just saying it's something the company would like to avoid and so it's kind of shooting us in the foot today as far as doing business analytics booking has a lack of a state machine if you look at the table it's just a bunch of flags completed, no showed, late canceled notified canceled normally and a bunch of different mutually exclusive states that for some reason I didn't think to use a state machine back in the day and because the booking object is touches so many different things it's too much of a nightmare for us to be able to bring it up into a state machine so we just deal with the problems of having it and of course if you get two flags of conflict then you don't know what state the booking is in and it's the reconcile at all poor naming I think every developer deals with this in general but I'd like to explain one of ours we have a couple tables, objects billing, billing charges billing item pricing so the billing is just the user's billing the billing charges is how we charge the user if they go happen to no show or late cancel okay they're kind of related, not so bad billing item pricing is how we our studio relationships and have nothing to do with the user I don't know why I named it that but now I have this confusing naming every time I onboard someone I'm like I'm sorry just they're not the same thing they're not related, I apologize all right, how you guys doing? here is probably the most damning one the one of the, I'll just go into it okay, here we go the where our referral objects has a generate token so I want to refer you we generate your token for you maybe you can refer someone nicely okay so let's walk through this code we generate the referral token we have a loop the token is based off of your user's first name Loves Guava, so Justin Loves Guava and a random digit and we assume that there's not gonna be a thousand justins together cool everything seems to be good you need to be unique of course so we loop until we've if we hit any collisions and we keep trying again on a random number generator here we go so we changed our flow of sign up so that we only ask for your email address we wanted to make it as easy for you to sign up as possible so we stopped asking for your name and a bunch of information so now it just says someone, nothing Loves Guava, random digit and we ran out really fast that happened in August 2016 so what does a classic speed developer do? let's just add two zeros to that and then we'll just ship it and we'll come back and fix it you can see the date is September 2017 a couple months ago we were like why people can't sign up why haven't all our registrations people aren't, and we're like, oh my God this is so embarrassing so yes so these are some of the problems that we created for ourselves and the way it looks for our particular product but Enter Guava Connects about a year ago we recognized an opportunity to go to studios and help them manage their studio with a SaaS product and so for us as a dev team we're like, sweet, this is awesome we get to build it right this is our chance to do it and this is the dashboard it's not pretty from a front end but it's fast and it works quite well and we started using web hooks and whisper, strong testing and we I started taking a step back so I clearly didn't have a hand in a lot of the creating new problems and guys who are smarter than me now build these things and so it was really awesome for us to be able to do this and this is our third attempt at React React version 15 and we spent about a year on it and recently we helped manage about 110,000 US dollars across 41 studios actually four to seven of them were the ones who were actually doing this volume the other ones were just studios that we moved over but as anything in a startup while we as a dev team loved it and we're really proud of it the sad thing is that two weeks ago we effectively put this project on permanent hold and for the dev team it was devastating it's really hard and you know we had to have long conversations about why we didn't why this was important and it really came down to the that we couldn't afford the risk as a company that this brought in because to be able to continue development would have cost another six months of X number of devs and a sales team to be able to do it our sales cycle was too long we weren't converting enough studios while these numbers look amazing it sounded really cool when we took a cold hard look at them it just put too much risk on our company and so collectively we said you know this is this is what it means to be a startup and we have to bite the bullet and it's tough and it's really difficult but we had to do it and so I I bring this slide in and it's like why talk about these things why why do we do this this is startup life and we fight the dirty fight and I want everyone to recognize that and understand that as startup people as people who are in Ruby part of the reason why Ruby is so amazing is because it allows us to iterate quickly it allows us to build features very nicely and we have to balance between speed and sustainability clearly for us we can we can afford a certain level of risk that maybe larger teams or more mission critical teams can't afford and our guava connects our studio management dashboard represented that that a risk a version that we wanted to take we wanted to build a system that was bulletproof we wanted to build a system that people could rely on and not deal with all these hidden bugs that would creep up in the middle of the in the middle of the year from code that you built you know way long ago but if nothing else I think the biggest thing is as startup community we have to remember that these are decisive decisions that we have to make and and even if you come from a larger organization that doesn't consider it a self-start-up anymore let's say these rules still apply in the sense that you can't you can't not move if you stay still you're stagnant and you will lose and so for us we we do these things and we bite the bullet when it gets tough and so at the end of the day um going more philosophically uh these are the pillars of dev awesome learning transparency and team learning is because if you ever stop learning you're starting to burn out it's one of the first first signs of burnout um and this whole room represents people who want to learn and that's awesome and I I congratulate all you guys for being here and taking that step but not everyone is is uh at the at the same level and so it's our job to influence others to be able to come learn and and join and make ourselves better um and in Guava Pass that's what we do we go into React we go into React Native we do it over again and we bring all those learnings that we that we have into our other products right and this is how we make ourselves better because I I tell the the team all the time that we might realistically I'm not going to be here when I'm old like 80 years old and so that means we're not going to be together forever right and so while the time that we are together how can we make ourselves the best that we can be together and part of that is transparency and transparency is really big because so many times people talk about organizations that you don't know what's happening you don't know why you're doing things and it feels you know terrible and for us I hope that this presentation if nothing else has been transparent into the organization that Dev Awesome is and that we do make mistakes and that we do fight and we do you know have to do dirty things but if nothing else it allows you to make the best decision so we I give you the power to make the best decisions and I want everyone else on the team to give each other the power to do that and that happens through transparency because if you can make your own best decisions then you can make the best decisions for the team which then affect the company and of course team and I have very particular opinions about team and I think it shows it's shown through a lot of times but I think one of the key things that we try to do is wins are on the individual losses are on the team so wins are celebrated individually Marcos does something amazing James something Kerri who's here as well like you guys nail that out of the park that's celebrated all together but if someone screws up the code someone breaks something something the website goes down landing pages we lose thousands of dollars that's not on the individual that's on the team as a collective whole and I want you guys to be able to understand that that's really important for our organizations if you're a leader if you're an individual it's on everyone to be able to protect the team collectively and so that leads us into the mission statement Tony Shea from Zappos the pursuit of happiness talks about a mission statement this is a block of text you don't have to read it the main ideas behind it is that everyone in our in the dev awesome organization affects this list it goes on for another page or so and you affect your personal the way you conduct yourself individually with dev awesome and with a company down below and each person who comes in writes into this and then we sign it well we'll sign it soon I keep forgetting that last bit but the main thing is that yeah it's this is important because if nothing else it's about the community and so why why why are we here and ruby enables guava pass and I love ruby as a language I really do and max is a very opinionated he makes ruby enjoyable right that's the thing he always talks about and by extension that's what we need to do for the community if the language is enjoyable then we can also enjoy each other we're not complaining all the time and it's really important that we make the community powerful and we empower each other and so hopefully to take something that you you like or you hate but take it with you and please make your own judgments and we've made our decisions and we continue to fight on and I hope you can learn something from from it take it back to your organization and do the same so I that's that's all that and I don't want to be pure doom and gloom so I do want to talk about 2018 and the future and all the fun things that we're trying to do so guava pass as a whole is actually growing faster than ever in the past couple of months we've we've grown I think by 25 percent alone and all the learnings that we have from guava connects that really awesome product that we built from the ground up correctly we bring it right back into the main code base and we start fixing all the problems and that's what December into January represents for the dev team going into 2018 we're building all our cool functions we're going mobile first for ourselves and then social features the community outside of the dev team is actually really strong when you talk about it from like events that the outside team hold different things that they they hold community events they hold all sorts of different like get-togethers for this and here's a picture of us at our still boxing and so the offline needs to start coming into the online so that we can help retention inside the company and this is one of our goals for 2018 some of the cool things that we we do want to build and so all that together maybe you like some things that I said enough that you are interested in joining so of course we're hiring for 2018 hopefully I didn't scare all of you away but I hope you guys all walked away with something cool and so we're looking for senior rails engineers with reacts we are completely willing to teach it because we all taught ourselves it so that's not really a problem and we've been doing it for a couple years now and we think we're okay at it so yeah I think if nothing else thank you for listening to my spiel for about half an hour and yeah any questions I think Janice Marcus do you guys want to try to answer that Karee okay I'll give it a shot just like anything it is it is nebulous and it's a definition that you you define together and for us it really meant heavy testing component making sure that our our code base is it wasn't full td yet t t t d yet t d d yeah sorry but but every every feature that released needed to be a certain level of testing and everything that we fixed also had to contain a test as well as we needed any time we saw a code smell we would start doing rewrites as soon as possible instead of letting it live we have a lot of td's in the old code base yeah you guys do retrospectives and how is that involved so scrum wasn't actually introduced until about November 2016 so about a year ago and it was actually introduced because business had such very low visibility into what dev was doing there's always complaints that you guys just ship things and we don't know why or how they got shipped and so you know the stuff that they requested and the stuff that was built was always different so we started introducing scrum pretty hardcore and guava connect was that first foray into doing proper proper scrum and doing the retro script retrospectives and so we've had like an example of them we just go through the things that we did poorly and then any interpersonal issues we also bring up during that time it's a bonding time it's a complaint time it's everything time and sometimes they last an hour sometimes they last three yeah do you say automated too for checking your code like code climates or something like that rooble cop we do rooble cop we used to do code climate rooble cop does most of our linting for us these days how about joseph yeah yeah that one's yes I don't remember exactly because my atom's just set up and it just goes all right any other questions cool thank you guys