 Live from San Francisco, California, extracting the signal from the noise, it's theCUBE, covering DockerCon 2015, brought to you by SiliconANGLE Media, with special thanks to Docker. Now your hosts, John Furrier and Jeff Frick. Okay, welcome back everyone. We are live in San Francisco for DockerCon 2015 SiliconANGLES, theCUBE, our flagship program. We go out to the events and expect to see the noise. I'm John Furrier, my co-host Jeff Frick. Our next guest is Simon Elkinson. Welcome to theCUBE. Did I get that right? No. Elkinson. Okay, so it's, how do you say it? Simon S. Gilson. Okay, there it is. It's hard, it's hard. From Shopify, welcome to theCUBE. Thank you, thank you. So we were talking before you came on. You guys are in full scale production, a little bit disappointed with the keynote because you're already out front. You're in the bleeding edge when you're running a production. So tell us about the size. Because large scale is what cloud's all about. So I really want to drill into what large scale means to you. So we've been running our main application in Docker for about a year. And I'm not talking about like the small services that don't matter too much. I'm talking about the main, main application. And yeah, so we've been doing that since DockerCon North America last year. We run in about three to 500 servers and we own our own hardware running in two data centers. We process around 300 million unique views per month. We were running at somewhat of a medium scale in terms of like our amount of visitors aggregated. It would be up there with like eBay, Apple, and so on. So big transactional system. Absolutely, absolutely. So talk about the Docker situation. I'll see how are you using it? How long did it take? What's some of the scar tissue that you've learned? Share with us. Yeah, why did you go so early? I mean, people are saying it's still early days and you're already a year into it. Lots of scar tissue. So we actually started looking at this about two years ago. So that was very, very beginning, just essentially just after Solomon started talking about it. So we sat down a team. Our CEO said, okay, we need to get on top of this. We need to understand what this is, whether this is something we need or not. So we investigated this in, that must have been 13, but it was too early. So we worked on other things and we started taking up this effort again in January of 14. We looked at MISA's, we looked at Corewise, we looked at all of these platforms as a service and at the same time we're iterating on our app, it takes a lot of effort to take an application that's 10 years old and try to put it in containers. I feel like there's not a lot of talk about that here either that putting an application in a container is much more than just putting it in a container and putting it on a pass. These applications are logged into files, which is not really something that worked great with the container environment. Their secret model might be quite off from what works with containers and there's tons of other things you have to do to put an application into a container. And so we were doing that, we were looking at Corewise, we were looking at MISA's and we actually had MISA's in production for a while in 14, start of 14, doing some of our non-production workloads or at least non-production critical workloads. But it was just, it was too fragile. There were too many moving components. What was some of the nuances that made it hard for the big app to go in the container? You mean specific? So we're talking about taking an application into a container, what was hard about that? So we can talk about at least logging. So logging is a big one. What most people do is they log to files. And logging to files doesn't really work very well in a containerized environment. If you just log to a file inside a container, you essentially have to log rotate your containers because they will start growing and so on. Or you run lock, rotate inside of your containers, which is super awkward. And another thing you have to change is how you do secrets. What we did in the past was use something like Chef, put it on your nodes and link it into your application at runtime when it boots and you deploy it. But it doesn't really work either. It's a very, the state of the secrets is not tied to the revision of the application. You need to tie it to, right? Like sometimes somehow put it into the image. And that's really the effort and which is somewhat application specific that needs to take all this state that was in a bunch of different places before and put it into the container image. So secrets, logging is a little bit of a special case. Another one would be something like your application assets, JavaScript, CSS, and there are a bunch of other things depending on your application. So secrets was another big one. We wrote our own thing. We call EJSON where we could encrypt JSON inside of our application repositories. And then we decrypted when we start the containers. So we could tie that into the container image. So logging, secrets, we're the big ones. We have to do assets. There's a bunch of things about just, you have an application that is used to running with a bunch of services on a host. Are you going to take these services into other containers and let them communicate with each other? Or are you going to communicate with them directly on the host? Are you going to remove them? What are you going to do about all these dependencies? So a lot of that is application specific. There's like a bunch of add cases that I don't think are super interesting around like going from Unix domain sockets to TCP sockets or accepting connections, things like that. It's not, it's more technical, but yeah. What did you learn? I mean, I'm sorry, you just shared a little bit of what you learned. What's the benefits? I'll see you slogged it out. You guys had an app, you retooling. You motivated. What were some of the outcomes that you guys saw? I almost want to talk about the work, like the downsides first because it's a little more chronological. So we tried to meet us. It's a therapy session. You can get it all out. Thank you, thank you. Lie down on the couch. Roll in the couch. Just go negative first. Thank you, so many scars. So I'll talk a bit a little bit about the scars. So we had some scars coming out of this CoreOS MISA situation. Fortunately, we spent a lot of time modernizing our application, right? Our application was actually running in containers. So we decided, okay, instead of trying to run it on MISA or CoreOS, which were still somewhat new at the point and even Docker wasn't stable at that point. So running all these fragile components on top of each other was just too risky. So we decided, okay, well let's just ship containers. Only thing we ship that changes. Everything else stays the same. Chef stays the same. We still use like upstart and run it and these kinds of things. Everything else stays the same. No other new components. So we rolled that out summer last year and everything was terrible. Our ops team hated us for a couple of months and there were fires all the time. We essentially spent six months just taking out fires. And our production infrastructure was in a state that was actually worse than what we had before. And we kept telling ourselves, well, at some point we'll get this stable. We'll get, there's all these like file system issues with better FS, AUFS, all this stuff that all these like super new things that Docker builds on top of the registry, all this stuff, we have to make it stable. The builder was really bad, all this stuff. And so six months later we, it was finally at a good state. And this is when we started being able to harvest some of the fruit of, we've been working on this at that point for 12 months, right? Like 12 months we've worked on this project to get Docker production. And after 12 months, we were finally able to start working on some of the benefits of Docker, was harvest some of these values of having this immutable image of Shopify and so on. So in January, February, March, we started working on our new deploys. Our deploys in the past taken 10, 15 minutes. We deploy about 12 times a day. And our deploy time went down to three minutes. We can now deploy across two, 300 servers in three minutes, which is amazing. Our CI, we're working on a new CI system. And a new CI system is going from about 12, 15 minutes down to around five. And this is for a large, large monolithic application with several hundred thousands lines of Ruby code. One of the biggest Rails apps in the world. And now we can run our tests in five minutes. We can ship code to production in three minutes. It's amazing. You can put code into master and you can have it in production eight minutes later. It's completely tested and in production reliably. What about the show here? So that's good stuff. So you guys really worked hard, burned the midnight oil, drinking a lot of beer, getting that done. But now at the show here, it just seems to be catching up to what you're doing. So compare and contrast the shows, progress in the industry with kind of where it should be in your mind or where you guys are. So we can talk a bit about how the keynote is related to what we're doing. So the keynote yesterday was a lot about how Docker is trying to decompose. They built somewhat of a monolithic engine and now they're trying to take all these things out. Solomon said at DockerCon Amsterdam in December last year that Docker needs to be this interface to everything, right? It has, they have created an interface. They've made people agree that he said this again here and reiterated it. He's created this interface for people to agree on something. And the things that he made them agree on at this point has been the container images. And here they've really taken that batteries included but swappable, mansion to the extreme. They're talking about swappable logging, swappable networking, discovery, all these things. That's really exciting for us. What people have been doing in the past year and what we've been doing is somewhat sketchy solutions on top of this engine because we couldn't change it. We don't want to run our own custom one. So finally we're able to be blessed with our custom solutions and get them into it. Instead of us having to write all these custom engine X things, custom things to get our logs to Splunk, all these custom modules that work with Docker but outside of Docker, we can now, the vendors will now write them for us. Splunk will write their own one for logging engine X will write something for theirs and all these vendors will now compete to write the best ones. That's really exciting for us. Being able to actually hook into that. And from the keynote this morning that was about the enterprise offerings like an enterprise registry, things like that. And honestly nothing there is going to help us. Which I think is, that's not great. Like if we could pay ourselves to not have to have expensive infrastructure developers to work on these problems, that would be much preferred. But unfortunately none of the offerings this morning were that exciting to us. What about hardware? You know we're seeing a world now where the software is the center of the action. That's where the innovation is. But you guys have a lot of servers and the servers in the data center. Do you actually care which person you supply or you go to or should the systems be engineered for the software? Because the DevOps ethos is, hey I'm pushing updates, just run for me, it's an engine. At some point there's an end to end argument. I want to optimize hardware. Now I don't want to actually go and configure it. That's what the abstraction layer might do. But there's a lot of hardware vendors out there. We're an HP shop, we're an IBM shop, we're a Dell shop, we're Oracle shop, these kinds of things. I am not, so I work more on the software side of it. So I'm not very involved with how we buy hardware and so on. I know we buy a bunch from Dell, Supermicro, our switches are mostly Arista, things like that. But it is a huge burden to own hardware. So we recently, this year, we set up our own data center and the iteration cycle on ordering your own hardware, getting it approved by finance, all this. Provisioned. Provisioning, it's, it doesn't. It's a new, it's an annoyance, really. It's an annoyance, yeah. And it slows everyone down, right? Like someone said, I can't remember, I think it was Adrian who said that if you give developers an API, they'll automate it, right? Which is, that's true, any infrastructure developer, if you give them an easy HTTP API, they'll automate it, right? And we can't really do that. Like we write these scripts on top of IPMI and things like that. It's not as bad as it used to be, but it's still not, it's not quite an API. And now we have to build that. So we are, right now, we are looking at Amazon, we're looking at Google Cloud, we're looking at all these different things that are now popping up and maturing. Because if we don't have to build it, well, that's better. But we did move very, very early on. You've got to have a lot of transactions. So storage has to be a big part of that. So low latency database is another interesting piece. Who do you guys use as a database? So we run our own databases and we have some very, very large, very large servers. We are sharded so we can horizontally scale them. But the problem is we have this problem of shops that have explosive sales, right? So on top of the hour, they release a new product and we have tons of people coming in. Like that's, that's, that's auto scaling right there. You need auto scaling and doing, right? Yeah, so right now what we have is just, you know, we're just over provision, which is, I mean, if you own your own hardware, that's about as cheap as auto scaling, except you don't have to do software part of it. But for the databases, that's not, if you're running a relational database, you can't really horizontal, you can't scale that like that, right? You just need really big hardware. And most, most of the vendors don't really support that. Like Amazon are only so big. We could, we could charge, so we had more charge, but that also has challenges and it's not something we've done before. That's probably what we're going to do moving forward, but we need, we need some kind of transition. And this transitional aspect of everything is not really something I feel that the vendors understand and even the Docker community understands. They're just like everyone or Docker managed to do development and CI really well. And now they're all talking about pass, which is over here at the other extreme. But in between, there's this area that they haven't quite figured out, which is Docker in production. That's what they're trying to figure out here. But I think that their take on Docker in production is a little bit too extreme. They want people to run on a pass straight away. It's not how it works. That's not how companies deliver software, at least not the ones who are not doing everything for greenfield deployment. But for companies like us that want to evolve our infrastructure into something more mature, we need that transition. We can't just go from one to another. That's what we tried in 14. We tried to deploy Coro and Mizzas, right? So I want to see more vendors doing these transitional solutions, right? And in terms of hardware that maps onto the problem where you want some of these cloud providers to give you a really, really big instance so you can run this and over the years you will make that bit smaller instances. I think everyone can agree that that's a better way of doing it, but they need that transition, right? And I can't adapt their entire stack at once. I need to take in a component at a time, get comfortable with it. I'll take another one and slowly my stack will turn into that vendor stack. And I don't feel many of them have actually realized that that's, yeah, or maybe they're just going out to the enterprise greenfield market. So question to have for you as we get down to the final minutes here is how would you peg your evolution with Docker with the rest of the industry? Because it's still very young. So it's probably a handful of folks that are in your category. Is there a number, an order of magnitude 10, 22, 5, 50? I mean, who's at the level? I mean, have you had to kind of do a bell curve? Where's the market? I'm getting the sense to you that there's a lot of fast followers to some early adopters, but we still haven't got a figure on who the early adopters are and what stage they're at. Yeah, this is tough. This is a fairer possession right here, right? And I look for other people and talk about all the challenges we've had in production that they're really hard to find. When it gets down to it, and in many of the talks here too, people talk about all the stuff they're doing with Docker, and then someone raises their hand and says, are you running it in production for your main application? And they say, yeah, not really, right? It's mostly for these smaller applications, right? So I have a really, really hard time finding users like us who are using it for their main, like their main money maker. Right, right, yeah. I can't even give you any names, to be honest. I don't know. So you're the first one surfing this wave, right? It's a big wave. People see this coming. No brainer, great for development. Right. What advice would you give the folks that are behind you? Hey, longboard? Hey, get on the longboard? I mean, you've got some experience now. You slog through it. What's the advice? I think my advice right now is that Docker solves real problems in development and for CI environments. They still need to figure out how to build images more effectively for larger applications. Or if you do have a large applications, maybe you don't even get anything out of CI and development using Docker there. But for a lot of cases for development in CI, I feel like the pros are starting to outweigh the cons. You can do, you can actually, now you have an easy way to write integration tests between different applications. That's been almost impossible before. That's amazing. You can have different versions of programming languages and testing among these applications and do so that things, when you're iterating on APIs in one application, the other applications, things start failing, right? But production, that's not something I would recommend yet for most people, I feel. And if they do go put their toe in the water for production, size of team, any kind of organizational and or skill set dynamics that you'd recommend? I can't say, I can't say. I think people should, I think a lot of companies should start investing in development and CI and see, at least sit down, prototype, spend a couple of weeks, see, could just take us anywhere, yeah. And then once they get a lot of value out of that, they'll, the time will have come where Docker and production is becoming more common and now they have the experience to do it. And then they can iterate faster and they can service all the developers. Okay, final question, take a step back, go to the balcony, look down on the stage, as they say, for the folks that aren't here, what's going on in DockerCon? What's the, what's going on in this community, this industry, what's your take? As someone who's out there, forging the path for others, what's going on? I feel that what's going on in this DockerCon and more so, I didn't attend North American DockerCon last year, so I don't quite know the vibe there. But at Amsterdam, people were quite excited. I think it was still, you know, top of the hype cycle. And I feel like now at this DockerCon, people have calmed down a little. They're starting to ask the good questions of, are you really running this in production? Like, is this really hard works? People are asking really good questions in this session. If you go into questions, really, really good questions are being asked because people have now been thinking about this for a while. So people are getting more skeptical of what's out there. They're- Sicken their teeth into it. They're starting, yeah, yeah. People are starting to experiment with it. They have somewhat of a knowledge. It's not this completely unknown unicorn anymore. People have somewhat of an idea of the problems it's trying to solve. And especially around production is where people are asking the questions now. And that's exciting to me as someone who's been who at DockerCon, Amsterdam spoke last year about all the challenges Docker and being Docker into production. We, yeah, how we, what we faced from that. Simon, thanks so much for sharing the story. I love the, love the talk about the scars. Because that's really the learnings that were magnified. And also the outcomes, fantastic. Three minute deployments, that's pretty massive. Huge numbers in scale. You call it mini scale, 300 million, it's pretty damn big. Congratulations. Thank you. Thanks for coming on theCUBE, really appreciate it. We're live here at DockerCon 2015. It's theCUBE. We'll be right back with more at this short break.