 Twitter is probably one of the largest companies running Ruby in production today. And to give you guys some scope of how large we are right now, we've got about 110 million tweets coming in every day to the service. And because of the one to many nature of the relationship with Twitter, there's some multiple of this number actually being delivered every day. The record so far for tweets per second is about 6,939 tweets per second. They came in on New Year's of this year. They actually overlaid. You can't really see it too well because of the projector, but that's an overlay of a map of the world showing some of the epicenters on New Year's of where a large person tweet were coming through. That's pretty cool. We can check it out on the blog. Now, as far as my team put things in scope there, about 40% of those tweets coming in every day are coming in through a mobile origin. Now, it could be coming in through, say, a native app like Twitter for iPhone. It could be coming in through the mobile website, mobile.twitter.com, or it could be coming in through our SMS service, which is huge, especially in countries where smartphones are large. So today, what I'm going to cover is... I'm going to go on the architecture of Twitter in general so you can see where mobile fits into the big picture. And then I'm going to show some case studies from the mobile team, just three areas where we're currently using Ruby, because our team especially loves Ruby and we use it whenever possible. And they're pretty varying areas. And of the three, we're probably going to dive into most detail on the mobile website because I assume most of the people here are familiar with using Ruby and Rails for writing web apps, so we'll probably get the most of that. We'll wrap it up by showing some of the best practices that I think will be co-opted from Ruby early in development of the company. I think that Ruby was a major influence in the engineering best practices. And we have some time, hopefully, at the end to answer any questions people have. Cool. So let's just get started by showing exactly the architecture and where mobile fits in. So let's say you pop up your favorite client, and you type hello world, and you hit tweet. It goes into this big chart, which we're actually going to break down into three chunks. So initially, you load up your client, you hit tweet, and it's sent to one of our frontends. If you're on twitter.com to our web frontend, which is running Rails, or to api.twitter.com if you're on a native app, which is also running a Rails frontend. Or if you're on mobile.twitter.com, that's actually a 100% pure API client. So it's actually routing itself on top of api.twitter.com, so it's just a nice layer of course. So then your tweet goes into the core code that handles all the business rules, who's following who, you know, all the lookups on the accounts, everything is in one relatively large code base right now, it's being split out into separate service, a service oriented architecture. And right now, we're still in the process of decoupling all the frontend code from the core code, but this is the eventual architecture that we're striving toward. And I won't go into detail about the services in this presentation that make up all the core code, but we have for instance our social graph service, which is powered by Flock, our open source graph database. Our user service is sitting on MySQL, we have a Geo service, Timelines, and if anyone's interested in finding out where, I'll be glad to talk in more detail there. Now this part of the graph covers the request response cycle. So you hit tweet, your tweet was routed to the main code, and then we verified that we actually wrote your tweet somewhere. It isn't actually delivered to users immediately. In the context of a request response, we just returned you a 200, and really we're dropping your tweet into a queue, so that we can handle it using the asynchronous pipeline. And the reason here is mostly performance and scalability. Basically we have a big queue full of tweets and a bunch of demons on the other side of this queue popping tweets off one at a time and doing with them what they need to do. So for instance, in the example of sending a tweet, we have the timeline demon, or the timeline fan of demon. So there's a demon there that's just popping tweets off the queue. It's making a lookup in our social graph to see, okay, who's following this user, and then it's delivering the tweet into all of those followers timelines. We have other demons which we'll get into like the SMS and push demon, which pops tweets off the queue and checks, okay, does this user have a device attached to his account, or is someone who is following this user, and they have a device attached to their account. We also have the streaming API consumer. So if you're using the fire hose, it's popping tweets off and fanning it out to fire hose consumers or user stream consumers. These are all just examples of three demons. We have tons of them doing all sorts of stuff. That's it in a nutshell. So we already touched a couple spots where the mobile team plays a role, like at the very top, the mobile website, or as I just showed a second ago, that SMS stuff. Who here has ever had to like interoperate with an SMS carrier or an operator? Okay, when I say that now, how many people here actually have to connect directly to the carrier? So you're not actually using Twilio, you're not using a third party. Okay, a smaller amount of people. And for those of you who aren't familiar, which I would say the vast majority of people here, it's a lot of work to turn on a carrier insofar as it isn't just you point it to a phone and you send a text message. We actually have to first strike deals with each carrier everywhere in the world. So in the US, AT&T, Verizon, T-Mobile. But there's literally hundreds of carriers in the world. And for each of them, we have to set up first a business arrangement saying, okay, here's what you can and can't do with users. And then we actually have to configure that carrier. So such as each country or each carrier may have a different shortcode. In the US, it's just 4044, but it could be whatever's available in that country. We have to internationalize all the commands and we have to also manage the disconnects file. So when you turn off your phone service with your carrier, that carrier has to write down your phone number. And every so often, they upload to us a list of phone numbers that are disconnected now and no longer wear their service so we can purge them from our database because they may not have shut off their Twitter account before closing down our account. And the reason they do that is down the lines that they may reallocate that phone number. And if you forget to turn off your Twitter account, someone else is going to start getting your direct messages, which could be pretty awkward. So we have to arrange how do we get these phone numbers and start purging our database. Lots of stuff like that. So as for the code base, it's actually about into two parts. There's the web service portion, which is getting incoming SMSs from the carriers and handling all the business logic around it. And then there's the outgoing code, which is actually a set of demons that are calling out to the carriers. In some cases, using like a SOAP-like interface. Some of them are using persistent connections. Again, it could be, you name it, and some carriers doing some completely different protocol for delivering SMSs to them. So the main code, as I said, handles a lot of product logic. In the case of the demons, some of the product logic, when we pop a tweet off the queue saying this is supposed to go out to a user, we check, okay, is this device asleep? And you can actually set, I don't want to receive SMSs like from midnight to 6 a.m., so your phone doesn't ring. Or there's any number of other configuration things that you have to perform at this layer before we push it off to the carrier. When we get an incoming SMS, we expose an endpoint so they can make a restful call just to post the incoming SMS message. And incoming SMSs aren't necessarily just tweets. You can actually issue text commands like follow, block, and all those commands. So we have to have that layer on top of it and know how to respond inside this app. The app itself is a Rails 2.3 app right now. The logic is only about 2,500 lines of code. And it's actually a 1 to 1.5 code to test ratio. So there's actually more tests than there is code. So what's an example, though, going back of some of the demons that we write to connect to these carriers? Well, for instance, when we wanted to turn on Apple push notifications, all we had to do is subclass our demon base class and override the method and put all the logic in there. We pretty much refactored all of our demon code into one set of poor classes that you had across the entire company. And the big advantage there is that our ops team now knows how to monitor and spin up new instances of these demons whenever we need to. So it's all pretty much just one code base that's shared among all the Ruby-based demons throughout the company. And as for the logic that's fit inside of that one method that you override in subclass, it's about 300 lines of code which handles both calling out to Apple and receiving the disconnects from Apple when people uninstall the iPhone app. It also has a number of sanity checks. So in a large distributed system, anything that can go wrong will go wrong. Among other things, we've had situations where plenty of duplicate SMSes have come in that they expect to send up to users. So you might receive, let's say, like five SMSes from the same user, who knows what happened. Maybe a demon was turned off at the wrong time. But when you're dealing literally with hundreds of demons at once, anything that can go wrong will go wrong. So if you put in sanity checks, I like to check against duplicates because in the U.S. you're charged per SMS you receive. And lots of sanity checks like that. We also had to put in the payload formatting for Apple's payload that they expect from us. And we also do the auditing so we can keep numbers on which carriers are receiving more SMSes lately. For instance, Cairo is pretty big lately and we're actually seeing graphs of what's going on there. It's pretty obvious. That's the SMS service in a nutshell. So we're actually going to switch gears. And that last example was mostly about how we use Ruby as a web service and how we're using demons. This probably is more relevant to people which is how we wrote a web front end using Rails for the mobile website. So what's nice about the mobile website? I think it's a great example of an agile product. And it was developed with two engineers and it completed about 51 days from the first commit to the public preview. And inside those 51 days we actually switched over the stack from the existing web apps that were running at Twitter which were previously running Mongrel and we moved it over to Unicorn. This was actually the first Unicorn app that was running in production at Twitter. And then actually shortly before production deployment we switched over one more time to Rainbows because of concurrency reasons we'll get to in a little bit. So unfortunately, it's nice to use something new and higher performance but there's a lot of operational overhead so you have to rewrite your public configs, you have to work with ops to see if they can update their monoconfig and how do we handle when this shiny new thing runs into this weird situation you haven't run into before. So we had to pay the penalty so actually pouring things over. Also in the 51 days of developing the app we had to develop a REST client from the ground up and we'll go into the details behind this client in a minute and why we didn't use existing clients out there and some of the design decisions but in those 51 days we did not sacrifice code quality. It's maintaining a one to one code to test ratio or code, code to test ratio and also the big takeaway with this project is that we were developing a framework for connecting to the API. So I don't think that you need to be a really advanced Ruby hacker to get a lot out of Ruby and it's more important to develop tools so that more casual Ruby users can be really productive without having to understand the details of the metaprogramming we'll get into. So actually what do I mean by that? So when we're developing our client we wanted to develop a domain-specific language around connecting to the Twitter API. The goal being that when API comes to us and says we have a new API endpoint it has this awesome cool new feature when can you ship this? We want to be able to spin up a new endpoint in a matter of minutes describing how to connect to it from all the user logic. So here's an example of our RESTful client making a call to the home timeline and current user is just an instance of the current user based on their cookie and we're making an API call there to fetch their home timeline it's pretty straightforward. This method is generated dynamically by making this call inside the class that will generate the finder. And this is an example of the domain-specific language we created. We're saying wire to this API path the home timeline API path connect to that endpoint you're going to take the JSON that you get back from the endpoint and you're going to map it to the tweet class and the name of the method is going to be find home timeline so it's as home timeline and collection equals true means this is actually an array of objects it's not just one instance of an object and you're not going to be quizzed on this but it's really easy to even now just glancing at it it makes a lot of sense exactly what it's doing and you know if you spend five minutes learning this domain-specific language you can quickly define your whole API in about one page and this is cutting back down hundreds, maybe thousands of lines of boilerplate code and the less fobbing and pasting you have to do the less chance you're going to make an error so this is described in the entire API that we're connecting to this only describes though half the challenge which is actually connecting to the endpoint how do we now map the API to friendly user or friendly Ruby objects so you have business logic wrapped inside of there well the solution is a domain-specific language and here we wrote an objectification module which also allows us to have access to a lot of class methods to describe how we're mapping keys inside the JSON back on the attributes on top of the Ruby object so here we're saying in exposes we're going to list off a bunch of attributes that we want accessors built into the object around the hash keys and below that we can also define custom behavior for certain hash keys so for instance if people here aren't familiar with the Twitter API we return for tweets a tweet object and embedded inside of that object is a sub-hash that describes the user so here under that objectifies line we say take the user key and take that hash and pass it to the user class which will turn the objectified with a similar DSL like this inside that class we can also pass custom parsing so in the example of created app as a hash key we just want to take the value we get back and we pass the lambda to say alright now parse that using time.parse so it's pretty flexible no matter what API throws at it we can pretty succinctly describe how we should parse it and turn it into a real object here's a fun topic at least a couple of years ago I remember a lot of people were asking about this I don't know if it's still big in the Ruby community scaling it's actually not that difficult I think for cq about operations in Ruby to go after a lot of low hanging fruit one example would be doing research about the libraries that you're using and I'm surprised by the number of people who use the JSON C library when the Agile C library for parsing JSON is literally an order of magnitude faster in the benchmarks we were performing and they actually have backward compatibility with the existing JSON gem so literally dropping in two lines into your code will give you JSON parsing an order of magnitude faster than what you had before again just doing research and looking to see what other libraries are out there that may be more suited for what you need to accomplish on the mobile website because we can't use recapture for our sign up captures we have to generate our own captures from scratch for low end devices and I think everyone hopefully at one time or others have run into manipulating images on Ruby and maybe they've used our magic or other libraries and I'm surprised by how people have actually seen GD2 and this is a cool image library it was actually extracted from PHP and it's now there's wrappers available for pretty much all major languages and it's at least twice as fast as far as generating images compared to image magic or at least scaling images and we have to run into any kind of memory leaks or any of the other problems in the past our magic has been afflicted with and we actually wrote a very simple wrapper around it so that you don't have to worry too much about memory allocation it's mostly abstract away from you and again this was only like a project that took a couple days to throw together the image generation portion so one other way that you can watch CPU on your front ends is actually being aware of object allocations and there's a misconception carrying over and over again that because you were using a garbage collector you don't have to worry about memory management that's all handled for you and you know it's talking to a really talented engineer from Google I asked him what about these systems like in Java that have a garbage collector can you really tolerate garbage collection pauses when you're talking about the super high scale systems and he said well actually yeah the thing is Java programmers that there's memory management still involved and they're still just is conservative about allocating objects and how we're reusing objects and you're still working understanding when you're allocating frame objects it's just you aren't doing it by hand and I think that you don't have to go in there and really drill in deep to see where I'm allocating objects but you just have to approach it from the mind am I being conservative with generating a lot of temporary strings you know what's the difference between saying plus string versus embedding one string and the other how many temporary strings are you generating things like that as long as you're mindful of it I think there's huge gains but we've talked a lot about front-end CPU but I think that it's kind of a red herring when you're writing a web app I think that the first way that you can scale your app is being aware of concurrency so you're actually spending your time in a request so let's say you've optimized your app so that you can parse JSON and perform all your business logic and spit back or render template in 10 milliseconds that's awesome what if you're spending 100 milliseconds waiting on API or you're waiting for your database and in that 100 milliseconds you're just sitting there twiddling your thumbs doing nothing the end user that's 100 milliseconds wasted your server is going to reach capacity even though it's not actually peaked in CPU it's important to actually go into your app and see how is most of my request time being spent and it turns out in the app when we're looking at it most of our time was IO bound depending on the load to the API we could see load times over 100 milliseconds even worse and it's amazing how much you can accomplish in 100 milliseconds you just batch up your requests and you could accomplish 10 requests in that 100 milliseconds while you're mostly twiddling your thumbs you now have 10 times more capacity that's a hand-waving math so Ruby unfortunately has a lot of people have complained about its concurrency model there's a great great great article concurrency in Ruby is a myth you should all google it so when you're calling out to C code your threads pause and lots of fun things like that and I won't go into too much detail but we researched different solutions and we settled on for connecting to our API we're using a library called Titus and among other things that this library gives us for making restful requests one, it's a lip curl binding so it's very performant specifically it's a lip curl multi binding and especially interesting is that it respects the global interpreter lock and it will not pause all of your green threads while it's waiting on responses from API so what we ended up settling on is we built our restful client around Titus and then we turned on multi-threading mode in Rails and we switched over to Rainbows and we're using green threads so one caveat is that it's very difficult to work with concurrency in general there's different paradigms then driven programming there's functional programming there's different ways of thinking about handling concurrency and in the interest of keeping things simple especially for developers who are used to more synchronous programming we wrote again a very simple I don't even know if you'd call it a domain specific language around issuing concurrent requests where basically when we issue each of these finds we're not blocking we're actually returning a proxy object and it's not until we call the fire request method that we drop down to Titus batch those three rest calls into one lip curl multi call and we suspend the current thread in Rails and while suspended other threads are still running and receiving calls from incoming requests so and then when this resumes those three objects will be filled into the actual responses from API this is actually a pretty simple way of thinking about concurrency rather than throwing lots of lambdas everywhere and worrying about syncing and all that sort of thing switching gears one more time to probably the least conventional way that we're using Ruby on the mobile team is as a testing harness and I don't know if anyone here has ever tried testing on Coco or on iPhone because I don't think it's nearly as mature as the Ruby testing community there's so many great frameworks available and great tools and one solution would be to import a bunch of these tools over to Coco that'd be cool or be really lazy and just run Ruby and it turns out that the core code that's running on the iPhone, iPad app and the Mac app is actually just vanilla Coco it's just straight up regular Objective-C and it's not dependent on iOS frameworks for all the core business logic like how do we handle user accounts how do we tweet that sort of thing and just so happens that Mac Ruby plays very well with Coco and it has a nice one-to-one mapping so you can treat Coco objects just like Ruby objects and matter of fact in the latest version of Mac Ruby they even have built-in support for I believe it uses mini-test so it was pretty easy to develop a test suite and treat these Coco objects just as regular old Ruby objects and slap on a simple UI using interface builder. It only took about an afternoon and we're working to actually integrate this inside of continuous integration so whenever we push a new update forget it'll automatically run on the continuous integration server the best thing I've heard about company culture is that it's like cement and while the cement's wet it's very easy to sculpt and change but as soon as the cement's dry it's not going anywhere and the way I interpreted it that was about general company culture but the way I also interpreted it was about engineering and we were very lucky the company early on had very enthusiastic Ruby developers and with the Ruby community had a very strong culture of certain best practices such as automated testing and agile methodology and also with the help of Pivotal Labs they drove home a lot of the agile techniques I think that a lot of the philosophies of Ruby are more important than the language itself I think one of the great things about Ruby is the community and we're all here to get stuff done and we care a lot about the product and I think that's influenced our team so this is just an example again from the mobile team each team might be different but we believe in having small agile teams almost like a start-up and on our particular team we have one product manager one designer and three developers all working on the iPhone and iPad apps and having worked at different companies which have a massive scale of users I've never been on such a small team that's had such a large reach for the app that they're working on I'm especially excited that we understand you don't throw people at a problem also it encourages communication when you're that small of a group I can just turn over to the guys next to me and ask them questions it's very easy to communicate among our small team we also practice iterative development so in this team every week we have an iteration planning meeting and with the help of Pivotal Tracker how many people here use Pivotal Tracker they're all awesome and if you aren't using it go try and account right now I think it's still free but it's totally worth everything it's a great agile tool and using Pivotal Tracker we write out the user stories and these are the features that we want to accomplish this week or we write out the bugs that we want to fix and then we can score them drag and drop them this is probably boring for people who already use Tracker but based on how we score these stories based on the points that are left so we're always fluctuating the release date and we aren't trying some rigid waterfall approach and it's really exciting so one last part about agile development is that we believe in continuously shipping code on the website on twitter.com and on the mobile site we are deploying every day on twitter.com desktop actually there's an average of 3 deploys every day I think the record has been about 7 deploys in one day and inside each of those deploys are between 10 and 30 feature branches going out so this requires a lot of process for this to work for instance our rigorous test suite which alone would take an hour to run if you were to try to run our test suite but we have it parallelized so it takes about 2 minutes because it's going to take 2 minutes of testing machines we have a number of processes in place around releasing products we canary deploys so they only go to a subset of the servers and we monitor metrics before we spread them out to the whole fleet but we have a lot of processes in place that ensure that we don't run into any problems with this but we believe it's all totally worth it to be able to write a feature in the morning get a code review and ship it that afternoon native clients are a little more difficult for continuous deployment because of the nature of the beast one compromise we've had is we do weekly releases internally to employees it's nice on the iPhone if you have an enterprise account you can do over the year updates so every week now after iteration planning we say alright let's cut that release and we make a build and we push that to employees on their iPhone so that we're constantly getting feedback so hopefully if I've done my job you're really excited about the company you don't know how you can work there and actually the second way though honestly one of the most important things about your engineering organization is who you have on your team and in our mobile team we care about a few things I don't think it's worth the bullet point but we obviously care about people who technically know what they're doing but beyond that we care about people who are passionate about the product and you can tell when someone uses the product every day and they really get it what we're trying to do and we really like those people we watch for open source contributors if you have GitHub on your resume you've got your foot in the door we love seeing open source and even if you don't do open source in a similar vein if you're working on personal projects you've got a really cool start up and you're just trying out to see where it goes doing cool weekend projects if we see that, that's a big flag that this person is probably passionate about what they do and finally I think that this is the first company where I've been at where culture is one of the major points people try to drive home when they're hiring we will gladly turn away an applicant who is outstanding technically but it's clear that they wouldn't click with the team we care very much about being surrounded by people who again are passionate about the product and really excited to work with but just in general people you want to hang around with that you like being around positive and a lot of fun cool so actually that is all the stuff I wanted to cover so that we could leave some time at the end to answer any kind of questions you'll have I have a question about what I believe is a new bug on the mobile site let me just say that I haven't text the mobile site in seven months I'm now working on the iPhone so it's been around for about seven months I have a mobile site I send a user there you might have to tweet this button and send them to mobile twitter.com status equals if they're logged in everything's great but if they're not it has to be logged in and then redirects them to the mobile twitter home and loses the status and there's no way for them to get back to my site without going back through like free changes just now I'm really lost okay I probably know what's wrong with you you've created a ticket for me thank you find me out the words and I'll help see if it's actually being worked on okay thanks please do a full stack mobile dev on your team what do you mean okay do we do a full stack interface versus core data or things like that sorry interface builder versus core data okay so do we do full stack mobile development do we use interface builder do we use core data is that a question do you split your team up and do you separate sub parts oh is a person a specialist in one particular area so in particular on the iPhone app and actually this is true of all of I think our developers we believe in generalists so in fact as I said until seven months ago I was working in the mobile website and they switched me over to the iPhone and we actually have another developer who's working in the mobile site who's been working on Android so I believe we certainly want to avoid the bus danger where one guy gets hit by a bus and they contain all that knowledge and we're kind of screwed so in general we try to rotate people around and I think that we all have at least some knowledge of the different areas but for instance I've been lately working on a feature that involves using SQLite and because I come from a web development background I'm able to write SQL pretty well so they're like alright why don't you work on that one thing that we didn't cover is and this goes into the generalist aspect is that in order to ship features we believe in code reviews so before you can merge your code into the main code base you have to upload a diff to a review board and a teammate has to write ship it on and part of it is also part of it so that you get two eyes on the same code so you aren't making an obvious mistake but indirectly it's educating other people on what's going on in the code so they kind of have to know what's going on in order to give you a ship it With respect to code reviews your significant testing setup how do you deal with the taxation involved in time So your question is in relation to all the process code reviews and testing how do we deal with taxation and time and I think that one it's dependent on the team so some teams are more reverse about unit testing than others so we to a degree leave it a little to the discretion of the team as a company we strongly encourage unit testing but I think that what's important is that these best practices have tools built around them so there's minimal taxation so actually when we are ready to push code to the main repository to touch twitter.com desktop we have a script called make review and it's an actual it'll prompt you each step and it will once you've finished each step it will upload your diff it will create a a ticket and JIRA for you and it'll automatically I think it'll actually CC your entire team for the code review absolutely everything except for clicking the publish button to actually send out the email so I think that the important thing is one just building all the tools around the process otherwise you know unless you can ingrain the habit people aren't going to do it and you know make it easy for everyone I think it's the key thing how do you manage to get the test suite to finish in two minutes as opposed to one hour right so I didn't go into too much detail we built a system that parallelizes the test suite so it'll divide it among a number of servers in our fleet so it'll evenly divide it so instead of whatever one server it's ten servers the math there doesn't work out probably just using numbers but it parallelizes the suite and it has so an interesting set of challenges because you know your certain test if you aren't careful your test suite may not be optimized to be touching the same resource you could be clobbering over each other so our internals we have this team focused on working on all the details and then when you want to run this automated test suite there's a utility in our bin directory that's called swoop let me say swoop test and it'll give you a nice progress in two minutes later you'll tell you if there are any errors so I'll go you first you talked about having small agile teams do you do air programming and the stand-ups I mentioned small agile teams that use stand-ups and we practice company-wide little agile on a per team basis some of them are much more into pair programming on the mobile team for a long time we were doing pair programming and lately we haven't been doing it as much shame on us but other teams they still embrace pair programming and it really works out for them so it's really at the discretion of the team and we highly encourage it yeah so it's a team by team thing how do you do authorization for mobile plans how do we do authorization for mobile clients what authorization do you mean? we're actually so on mobile.tv.com how do we do authorization so you hop on the mobile site again we're playing by the same rules as everyone else and we're using Excel when we first launched the app we weren't using Excel and it ended up being a lot more trouble than what was worth to try to use some internal system so we just ended up moving everyone over to Excel so there was one pipeline with the same team working on it on the API end what's Excel? so Excel is hopefully everyone's played with OAuth which is you click the link saying log me to the site and I give you a pop-up and you have to click the link there so Excel allows the provider or the third party app to just give a username and password input which then makes a call to our API or in terms of the OAuth token and the use case for that especially on native clients like the iPhone it can seem a little weird to pop up a web view and then putting your username and password there and the web view disappears and also on really really low end devices like feature phones and pop-up web view so in certain situations but it requires special privilege you'd have to talk to API about what is the break-up of the users between mobile.com I don't know that off the top of my head sorry, what's the break-up of users on the mobile website to the native app sorry do we do manual QA and all of us exist in two video continuous shipping strategy? do we do manual QA we don't do any insofar as we don't have a team of people clicking on buttons to make sure yes however what we try to do though is instead all the employees for starters you own your product and there's a danger if you do have a QA team is you can fall into the pattern of alright let's throw this over the wall and let that handle it especially on the iPhone app there's a certain level ownership among all the people on the team where at all times for instance we're running the latest current build so as soon as we run into a bug we're fixing it and we're expected to do all responsible QA ourselves hopefully they'll I made a testing system around it so it doesn't become so repetitive and then we have a roll-up strategy of rolling out to all employees in the company and we give them tools to report bugs and then we also have a system kind of decider so when we're rolling out a new feature in production we may roll it up to zero users and then we roll it up to 1% of users which in part is to see how does this affect system stability and also you know is there an issue in general with users using this feature so we believe very much in slow, deliberate roll-out of features and also personally taking charge of QA and having personal accountability around the code we write efficiently to changing needs changing demands team members whatever can really influence how your team are acting how do you ensure that these tools don't lock you in their process that make you less efficient and how do you allow them or have them allow you to experiment okay so the question is how do we make sure that the tools if I understand art having too much control on how we do things like we're not being locked into one regimen based on the tools is that correct and I think that again it goes back to each team has a certain level of leeway as to how they get things done so actually the mobile team is one of the most evangelical people to use a pimple tracker we use it for all of our iteration planning however for actual ticket management we use JIRA when we're actually pushing things out to production so that there's that level of accountability and tracking so everything up until we create the JIRA ticket it's up to us just make sure we have a JIRA ticket so everyone can sync together and I think that part of empowering people is giving them that flexibility to do things however best works for them and the results speak for yourself that said company-wide we're trying to get slightly more regimented like moving toward you know the mature development style like at a google where we're trying to push toward new things people might not be as familiar with especially from a start environment for instance we've been active especially in the last year on encouraging design reviews on major new features so you get feedback from your peers before you even lie down your first line of code it's probably overkill if you're making a copy change on the website but if you're developing a new feature it really is a huge advantage to just spend a lunch with your peers and show people what you're working on um there's a cell phone I think so we're trying to push people toward newer processes they might not be familiar with and there's a little to the degree of regiment there but it's a balancing act you guys use the iPhone UI animation what kind of herbs we've looked at it but we haven't invested the questions we're using the iPhone UI integration testing toolkit and I think that was relatively recently released and I saw some info about it but we haven't invested time to fully build up and to end testing on the iPhone but eventually we're going to do it so I think I think we'll just wrap things up right now and if anyone has any questions please find me I'll be happy to answer them I'd like to actually thank JR and Kobe everyone behind this group that's been really exciting thank you for having me