 Good afternoon. Thanks for coming out. I hope you're having a good conference. This user story session is Ancestries, OpenStack, Docker, and Kubernetes journey. My name is Al Walter. I'm an infrastructure architect. There's a group of us that are working on bringing OpenStack, Docker, and Kubernetes to the infrastructure to bring some agility. So I wanted to take a second and introduce Ancestry. Ancestry is a technology company with a human mission to discover, preserve, and share family history. We're putting history in the hands of everyone by providing the most comprehensive resource of discovering and sharing what and who led to you. We're proud of the work that we're doing there. For years, many had had an interest in family history, but it's time consuming, costs a good bit of money, and collaboration is a key component to making the discovery and the connections in your family tree. Ancestry built the world's largest database of digitized, fully indexed content, all of the records that you would need to do your family research. This was done nearly 20 years ago, developing a more feature-rich site. Perhaps on the TV ads, you've seen the shaky leaf, kind of a big part of Ancestry for the hints. And finally, it's become apparent this last year that the Genomics project is making great inroads to help people discover their ancestry. We have a large amount of content created by our customers. Our users create family trees where they add information about their families and their ancestors. They then create associations between the people and their families and the historical data that they find. They also upload stories and photos about their family and their ancestors. We have over 70 million family trees containing 6 billion people with records, stories, photographs associated with those records. Our users both provide us with data and help us create structure to the data, making connections between them. I was mentioning the DNA portion. Launched in 2012, Ancestry DNA emerged as the largest growing consumer genetic family history site. Using Ancestry DNA customers and cover ethnic backgrounds, they can find distant cousins and make more connections in their family history. It's pretty simple. You get a DNA kit. It's a simple saliva swab. And the data scientists at Ancestry process that on a big Hadoop cluster and give you back the data. It's pretty simple. What they do is find the genetic markers to find your distant cousins. And they create what they call DNA circles. And whenever you submit your DNA to be processed, then we find the DNA circles that perhaps you should be included with. It doesn't really matter if you have gone back far in your family history or have a family tree created, there's still connections being made with the size of the database the way it is. I guess I skipped ahead on that one. All right, so the Ancestry introduction out of the way. Let's talk a little bit about the technology. Being around for 20 years, we have a history of some products, legacy infrastructure, trying to figure out the best way to say it, but the cost quite a bit. And it's not very flexible, bringing new features and functions to market. So earlier this year, RCTO announced a strategy at Ancestry to change that. And some of the parts of that strategy is moving our old .NET code towards Java. We're converting everything over to Linux, starting to use CentOS for some of the container stuff. We do have a DR facility in AWS and building our private cloud with OpenStack. Developers are very excited about containers with Docker and the orchestration with Kubernetes. So this is the part, where did we start down this road? I think one of the things that's interesting about this user story session is we're still pretty early on. Some of them that I've gone to see have years of the trial and error and can talk about those experiences. This is pretty early on for us. So I'm going to introduce Jordan Nilsson. He's also on the architect team, and he's going to talk a little bit more about how we got started. All right, thank you guys. Thanks, Al. So you can imagine the surprise of developers that are coding in C-Sharp when our CEO got up and said, we're transitioning to Java and go. They were just like, what? So it was a huge, huge transition. But it's been awesome because it started from the top down. Your CTO comes out and says, this is the way we're going. Well, that's the way you're going. And so that's been very, very good. So where did we start? So give me five minutes. I want to start because really we wouldn't be into OpenStack if it wasn't for where we started with storage. This is really what kicked it off. And this is really where we saved a lot of money and where things became very actually really cool, I think. So a few years ago, we started looking at our storage platform. And we said, man, it is expensive. So basically, we store our images or our digitized content on NAS platforms, or have been. And so what's happened was we realized it's expensive. There's these licenses that we license, a feature per terabyte. So by the time you look at it, you're paying as much in terabyte costs as you paid for the hardware. And it's just we didn't like that. We didn't like that at all. And there were some other ones. We had some NAS platforms that didn't have any APIs. We had some that had APIs. And it's like, how do you work with that? So I'm not going to go through all these, but you get the point. So we came up with a list of requirements. I'm not going to go again through all these because we're kind of limited on time. We are really going to kill you by PowerPoint. We've got a lot of slides, but I think it's really actually pretty good. So open source, RESTful API, commodity-based hardware. This is where we wanted to go. So what did we choose? We chose Swift. And this really ties into Docker and stuff when you talk about storing images, Docker images, somewhere. Where do they land? I'll talk about that in a minute. But we chose Swift. And how did we implement Swift? So the digitized content that I mentioned, we partnered with a company called Swiftstack. And what Swiftstack does is they let you run commodity-based hardware. They let you deploy your own Linux. But then they offer support and a controller-based system for managing Swift. And it's been very good. I actually have the CEO of Swift here. Hi, Joe. So sorry, I had to call you. They've been very great to work with. And we've actually moved all of, well, sorry, it's about 90% of our data, image data, image content, to from, I'm not going to say the vendor, because I don't think that's really fair, from that to Swift. So if you go to ancestry.com and you pull up an image, most likely, 90% of the time, that is coming from Swift. So that's where we started. And what's awesome is we saved a lot of money. And the company was like, wow, this is great, right? Let's do this with everything. And so that's where OpenSec came and started opening things up. And honestly, I think this was a piece of the part of the strategy where our CEO says, hey, let's go to Linux, because we saw such a big win here. So I'm going to skip a couple of slides here. I wanted to actually just show you the hardware that we're using. We're a genealogy company. We're not getting billions of transactions a second. It's genealogy. But so we were able to go very dense. And what we do is we have per rack. I'll show you the rack design here in just a minute. Per rack we have a Dell. We use Dell. We have a Dell R620 SAS attached to a J-Bod that has 84 drives in it. We've got four of those in a rack. Raw with six terabyte drives in these guys down here. That's about two petabytes in a single rack. So that's great density. If you're looking for density and you don't need enormous performance, this is a great solution. And we worked with the Swift stat guys. We worked with Redapt, who is our integrator. We just kind of wheel in the rack, and we just provision it and go for it. Accountant container, we did 100 gig SSD drives. So this was the one that cost a little bit more because you have to have low performance on the SQLite databases with Swift. And then the proxy nodes, similar except only two OS drives, and that's your entry point into Swift. That's your API, right? So initial, the first few racks we got in, this is a picture of that. And as we were kind of starting migrating, and that's kind of what they look like. So here's your four enclosures, the drives that are attached, to accountant container to proxy. And we do that, we just cookie cutter them and shove them in, and that's when you get an image from Ancestry. That's where it's coming from. OK, so sorry, that is really where things started. Now, what's next? And that comes back to the strategy that Al talked about, that our CTO came down. And so what is the first part of that strategy? Well, the first part is CoreOS. We've decided to standardize on the operating system CoreOS. Who's used CoreOS in here? Goodbye, anybody? A few of you, OK? Can be a little crazy at first because there's no package management, a lot of the commands aren't there. It's a very, very stripped down version, lightweight Linux version. And it's really designed to run containers. That's what it is. One of the interesting things is when we installed CoreOS on a box, we had our security team run a scan against it. And I think about every OS installed by default has vulnerabilities. Well, they ran a scan against this, installed straight from default, no vulnerabilities whatsoever. So security guys were just jumping up and down. They're like, this is awesome. And it's designed to run Docker, but can also run Rocket if you choose to run Rocket containers. Or and, sorry, not or, and Kubernetes. So CoreOS actually deploys Kubernetes within a Docker container. So it's deployed in Docker containers. I'll talk about that in a minute. And the great thing about it is they're always at the latest kernel. And they're always updating. So if you run CoreOS, you don't have to do this. But the idea is that your OS is going to update and it's going to reboot. And if you're running Kubernetes, which we'll talk about in a minute as well, things will move around, but everything will still be fine. And then they also provide OpenStack images, some QCAT2 images that you can load straight into OpenStack. And that's very nice of the CoreOS guys. I'm not going to spend much time here because everybody knows what Docker is. But just two things, our developers love, I mean, absolutely love Docker. They just are going crazy about Docker. And they're building lots of applications for Docker. And what's great about it is you can, like it says there, deploy, ship, do all that. It's portable. And then they do run in isolation. Now there's some debate here on, do you run those in VMs? That kind of defeats the purpose. Do you run it on bare metal? Well, there's some security risks. What do you do? That's kind of the hard questions right now. So let me show you our Docker architecture. And we'll talk about more in this. And we go through Kubernetes. We'll include that in here as well. So we start with a base CoreOS image. Our developers build a Docker file, which is from a Docker file, a Docker image is created. And we've been focusing on images such as, so initially, we focused on images like Ubuntu and CentOS. But that's not a good idea. At least we don't think it's a good idea. So what we do is we pick an image like Alpine Linux, which is very, very lightweight. Or we pick a node. I just wanted to get some of the information here. A node.js, very small node.js image. And so we have a node.js app image that we use. We have Alpine. We're working on the Java one. That's a little more interesting. We haven't quite got that one yet. Anyway, so we get the image. And then what we're doing is we're pushing those images into, I think this, we used to call it Quay for the longest time. But I don't think that's right. I think it's called Key or K, right? Key? OK. So awesome. See, I knew Lachlan would. See, that's why I brought Lachlan to this session. So thanks, Lachlan. So we upload those to Key, which is CoreOS's private registry. And you can use a Docker private registry. There's no problem with doing that. The great thing about Key is it integrates with OpenStack Swift. So within Key, we provide an account and a container. And the images get, when they do a Docker push, get pushed into Swift. And there's also MySQL database handling some of the persistent data there as well. And then obviously we do a Docker push or a Docker run and things deploy. Now this is all fine and great, right? But it's really hard to manage something still like this. So even though we can ship and port all of this, there's really no orchestration around any of it, right? Or if there is, it becomes very hard to manage, especially as you scale. And so these were some of the questions that we started asking ourselves. How do we schedule it? How do we, what's the health of it? What's happening in my containers? Where are they? That kind of thing, right? And that's, this is actually my favorite slide. This next slide is the animated gift cats. So we've got to get these containers under control. There's got to be some sort of orchestration. So I'll let that go through one time. So that brings up Kubernetes. And we put a stake in the ground and we said, we're going forward with Kubernetes. And we hope it's the right direction, right? We had a few choices. There's Docker Swarm. There's Mesos, which the Magnum project is integrating with. We're not actually doing the Magnum project. It's still pretty early on. But anyway, great orchestration. So I wanted to run through just the architecture of Kubernetes really quickly. And then also then talk about how we're actually deploying an app. And I've actually grabbed an example app to our environment via Kubernetes, OK? Sorry, I'm not used to talking this much. So first component is SED. It's a key value store. And that's the stateful. That's something you definitely don't want to lose. That keeps all your state for Kubernetes. We deploy the outside of Kubernetes. And we deploy it on fast SSD disks. You need fairly fast disks for that. So separated. I know some distributions, they'll put that right on the API server. We didn't want to do that because if we have to do work on SED, it could affect the API server and vice versa. So there's that. And then you have the API server. I didn't really carve it out. But you've got the API server, the scheduler, and the controller manager all run on the API server. Or on the master server is what it's called. So you have that server. You usually have two of them. And the only issue, and it's not if it's really an issue, is the scheduler and the controller manager can only run it one time on one box. So you need to have a process in CoreOS. There is a container called the PodMaster that if it actually dies, the scheduler and the controller manager will start on the other master. So it's not too hard. There's some good documentation out there if you deploy that way. And then on the nodes, they used to call them minions. They could call them nodes. I still like minions, so I'm going to use minions. There's the Kubelet, which is the guy that kind of watches the master and makes sure that all the pods are running on that box. And if not, restart them. I'll talk about pods in a minute. And then you have the proxy, which essentially routes traffic to your pods. And last but not least is, this should say API and kube control up here. We're actually, our developers are not integrating with the API, even though kube control calls the API. Each developer is using kube control to deploy their containers to our kube Kubernetes cluster. OK. OK. So the concept of pods. So the smallest unit for Kubernetes to manage is a pod. So you never manage a container directly. It's always a pod. And the few things just about pods is they share a same network namespace. So if you have multiple pods in a container, or excuse me, containers in a pod, they talk to each other via local host. If you're on a Docker PS, there's a pods container. And your containers in that pod join into that pods container. So they share the same network namespace. And then every pod gets a routable IP, which is a little different from Docker, at least Docker in the 1.8, in that you get a 1.7.2 address. And it's a NAT. So one other thing is Kubernetes is a declarative language. So you actually feed it a YAML or a JSON. And then you kind of forget about it. And it goes out and does what it needs to do. Schedules pods, and if one dies, starts it up. And I'll show you an example of some JSON and YAML files here in just a second. OK. So then there's the concept of labels that are attached to objects in Kubernetes. And these labels really are used in two locations. They're used on replication controllers, which we'll talk about next. And then they're used on services. So this will make more. But really it's a key value pair that's associated with a pod. And you can query that, and you can find out a whole bunch of information about your pods. OK. So a replication controller does three things, really. One, it makes sure that if you have multiple replicas or copies of your pod, that those are always running in your Kubernetes cluster. Two, it defines, so that we talked about the pod, it defines a pod template inside of your YAML or JSON file. And three, it adds label selectors, which are responsible for maintaining the pod within the replication controller. OK. I know, sorry, this is a whole bunch of information at once. And I've heard you've probably seen this before during the conference. So here's an example of an actual, excuse me, of an actual ancestor deployment config that we did just, I don't know, a few, I don't know, three or four weeks ago. So let me just run through real quick. So define replication controller. You give it a name, obviously. You define the number of replicas that you want. You can adjust that to however many you want. And then I'll come back to the selector in just a second. Then here is your entire pod template. So what this is going to do is here's your labels that it defines right here. And then it defines the name of your container. And then this is, in this case, this is the key registry to pull it from, which is an internal private Docker registry. It is a Docker registry, but it's from CoreOS. So it goes and grabs the image, pulls it down. And there you go. And that starts your pod in your Kubernetes cluster. And then it listens on port 80. Now you'll notice that the selector, the name, and the pod, and the deployment in the initial need to match, actually, they have to match where you can't create those, because that's how it knows to manage those pods. So you deploy your pod. Everything's good. But now you've got to get traffic to that pod somehow. So then there's the concepts of a service, which is basically a generic load balancer within Kubernetes. And what it does is it uses the selectors. And there's a different selector within the service that basically says, if these selectors match the label in the pod definition, route traffic to that pod. I hope that makes sense if it doesn't come up after. So one thing we ran into, and this is kind of interesting, is the service also gets a routable IP and a port. But the problem with that is it's usually not a routable IP. And so you're like, well, if you're in the cluster, you're fine. But what if, in our case, we have all these services outside that need to talk to a service inside Kubernetes? How do you get traffic to that? And that was our, for the longest time, we kind of beat our heads like, how do you do this? So I'll tell you. So here I put these kind of side by side. So here's the replication controller that we talked about a second ago. Over here on the right is the service that we define. So we're defining a service. And the key here is you'll notice that the selector status active matches somewhere here, the label in the pod status active. So when a request comes in on, actually, let me do this one. Let me back up just real quickly. Sorry, this is kind of hard to explain in a quick session. So trying to think the best way to do this. So essentially, what will happen is the selector will balance across a set of pods if these labels match. So if the selector status active, status active, it'll route traffic to those pods. So one thing, if you need to get traffic into Kubernetes though, there is a declaration called node port. So what happens there is you define a high number port. And then instead of using that service IP that you got when you defined the service, you actually use the Minion IPs. So you hit a Minion IP or a node IP on that high number port, and that will redirect you to your app. That's how it works to get traffic outside into Kubernetes. If that doesn't make sense or come up after, we can walk through that real quickly. So anyway, you define a replication controller. You define a service. So let's review the steps real quickly. The replication controller, label, replica, image, image location, and then a service. And if you need to communicate via in cluster, you can use that cluster IP. If you need to do outside in, then you use node port on a high number port. That's essentially how you get data traffic into Kubernetes. So how are we doing this today and how will this look? So we do use F5 load balancers. So what happens is you create a VIP, and that VIP list is on port 80. So you hit that address. You add as pool members. So the VIP list is on 80. You add the pool members. They listen on that high number port that we specified before. And then data is load balanced across your pods. Anything else there? I think we're good there. OK, so how are we doing this with OpenStack and everything? We're doing this by, number one, creating an OpenStack VM for the devs to deploy to. The devs get a VM in OpenStack. And then on that VM, they then go through the process of creating a Docker file, like we talked before. They pick a base image and do all that, and then that pushes to the registry. Then what they do is they install kube control on the VM. And that's what they use. So in Kubernetes, you'll say kube control. Create dash f and then your YAML file. And it will read that YAML file and deploy it to the cluster. So we install kube control on the instance. Create the necessary YAML files. And then we use, so this doesn't cover some of the developer stuff, like Jenkins and get, pushing your code to get. We don't need a different developer here to talk to some of that. But then we do create a build.sh script, which basically compiles the code, builds a Docker image, and pushes it to the registry. So that's essentially what we do today. So anyway, that's a whole bunch real quickly. So if you have any questions after on the Kubernetes or any of that, come and talk to us. I'm going to hand this over to Al for the last part on OpenStack. So just real quickly to finish up, OpenStack is helping us a lot with these processes. For Ancestry, some of the reasons that we're looking at OpenStack is the features and functionality that we want to provide our customers. We want to bring that to market a lot faster. Another reason is to give the development teams the self-service portal to spin up those resources that they need instead of waiting in the ticketing system in six weeks to get hardware and the traditional stuff. Again, it reduces the provisioning time from months and weeks down to minutes, really. We had a, right before we flew out, we had a developer that was, a sent an email and he was very timid, kind of. He says, is there something I want to test? Is there any way I can get 20 boxes? And I don't know if it took you 10 minutes. And he had everything he needed. He had his scripts to go ahead and deploy it. And he had his full working cluster in no time. And he's singing praises up to our boss. Yeah, so that was great. So again, we're early on in the OpenStack journey. I was talking about some of the things that we use it for. We're running it on CentOS, using most of the core components of OpenStack. We do for networking. As I've talked to people nine, 12 months ago when we got started, or when I got started in OpenStack and coming from a network background, I had major concerns about the network node for Neutron. So one of the things we did is partner with MediCura for this SDN, the overlay to help with that. And that's been a critical part of us getting to where we are today. Thanks, Cynthia. It's been really good. Right now, we're not using the cinder. For the things, we're using it for very ephemeral. It can just go away, rebuild, do what you want. If you need to store data, it's being stored off somewhere else. Similarly to the Swift stack, it's very similar hardware except for the big deep storage. Did want to talk a little bit more about MediCura, Medanet, this illustrator. I don't know if any of you had a chance to hear Adam just a couple of hours ago. He had his Medanet 101 session. If you're interested in this, that'd be something to go look at up on the YouTube channel. I'm not gonna get much deeper into that here. But the ability to scale that out and distribute my gateways in and out of the OpenStack environment was critical to me. And not rely on that node port. Node port. Oh yeah, sorry. We threw this in here because early on we, before Kubernetes came along, we looked at Nova Docker with an OpenStack. But we've since gone away from this. This is just kind of one of the options that we looked at. And it's still there, you can still do it. And it deploys Docker via Nova commands. And it works, but it doesn't have some of the feature-rich stuff that the Kubernetes has. So Jordan. On this one too, where were we going with this? Just to show the ease of laying down that CoreOS in Kubernetes. Right, so yeah. So this essentially is, I think six steps. There's two slides here that we, this is deploying a Kubernetes cluster with like six commands in OpenStack. So we get the image, we deploy the image to Glance, the Kukau CoreOS image. And then hit that next slide. And then we run three cloud config files. The first one's LCD. The second one's a master server. And the third is a Minion or a node server. So with three commands, we can have a Kubernetes cluster spun up in minutes, right? I don't know how long it takes for sure, but not very long. And the great part about the CoreOS within OpenStack is the boot times are like seconds. Like it really takes like a second to like boot the OS. It's really great. The one, oh yeah, sorry, go ahead. No, you're fine. Some of these examples were things that you had thrown in as you're working with CoreOS. And this is just a sample cloud config file to spin up CoreOS. Now, one of the things that OpenStack is helping us with, besides the developer environment for them to work on, is as we're learning CoreOS, as you're tweaking the cloud init files and cloud config and waiting for post on bare metal, it can make for a long day as you get that refined. On OpenStack, it takes a minute or two when you find out if you're right or wrong. Change it and build it back up again. So I think that's awesome. Going forward, we're hoping for the maturity of both Ironic and Magnum, given that our development team is very set on containers and moving forward with that. And they wanna do all that on bare metal. That really ends the content that we have for today. If you have any questions, I think we have a few minutes left. Yeah. So we are just about, when it comes to the containers in Kubernetes, next month, we're looking to take some live traffic in that environment. When it comes to OpenStack, yes, it's not serving live traffic for the Ancestry site yet. It's more for development to have an environment sandbox to play in. Yeah, so we actually started very similar to the Lithium guys here. We started with five pilot teams that really wanted to hop on the bandwagon and get their service out there first. So they hopped on, took in a service that they knew we needed to deploy, right? And we gave them, one of the nice things is we gave them an environment within OpenStack of Kubernetes environment that they could deploy and test. But end goal right now is running CoreOS bare metal with Kubernetes laid on top. So the idea, anyway, I don't know if that answers your question or not, but yeah, we really started with really some small teams and no persistent data, not a lot of complexity, right? Something that could, we're kind of pushing persistent data until that matures a little more down the road. But yeah. Not really vital sections of our customer's experience. Yeah. Start small. Kind of just work our way up. And then the idea is to have these pilot teams turn around and train more teams and then those teams turn around and just we hope it turns into this ripple effect, so. So go ahead for the video though, just rephrase the question. Oh yeah, yeah. So I might have to have you repeat that one more time. Do you want to come up to the mic? Yeah, please. Come up to the mic, then I'll have to repeat it. For external network access you said you use a high number port and you low balance, how do you deal with adding or removing minions? And if you have multiple Kubernetes clusters, how do you have a permanent load balancer then you just publish the port to your F5 or whatever? How does that work? Do you want to talk to that? So we're, right now it's still manual with what we're entering into our F5 load balancers. The Kubernetes is going to be one cluster so we'll let Kubernetes handle the assignment of that high port. But yes, right now it's still a manual entry into the F5. And then also it's a static route entry to have traffic routed to that, you know the slash 24 that it assigns to the minion. He updates the static route. So we're still a little bit of stuff we have to do manually because yeah, because of the way we're doing it. So hoping for some dynamic routing out of the whatever Kubernetes is using. Yeah, the interesting thing with Laughlin and these guys is Juniper's kind of has come up with a solution where they're doing that for them. I believe, I don't want to speak for you but I believe that something like that. So that's pretty cool. Yeah, the CoreOS image within OpenStack or? Oops, that one? It's about 100 meg. Yeah, about 100 megabytes for the CoreOS image. And then our other images like the Docker images that we deploy, they're really, really small like 20 megs or something like that. So we're trying to get small as possible as fast as possible. Did that answer your question? Sorry, I'm not sure I understood the question there. Yeah, we just put the image into glance and then into glance and then just when we do a nova create, it just pulls that image and launches that image. And it's about 100 meg image so it's really small. I don't hope that answered your, if not come up after and we can talk through it tomorrow. Anyone else? Yeah. So we haven't been working on it. Our developers have. I think they have something. They may even have something by now. I'm happy to go back and ask them and if you wanna exchange some information, I can send that to you. Yeah, yeah. Yeah, so node and go, I think we've got those kind of under control. We have a couple apps and go and a couple in node but the Java one was a little, I think a little trickier for the, so these guys have a lot of experience with that too. So. You're doing the support channel with all of us? Yeah, one thing actually I recommend, I know we're probably close to out of time is Kubernetes has a Slack channel. I may just load the Kubernetes Slack channel by announcing this, I hope they don't mind, but we're all the Google developers and everybody that's running Kubernetes hops on and it's just question after question and we've got a ton of stuff figured out by just being on the Slack channel. So I definitely recommend, I think it's like Kubernetes at Slack or something, so I wish I had that up there. Any others, are you good? Thank you very much. All right, thanks guys.