 Hello and welcome everybody to this OpenShift Commons briefing. We're a little time zone challenge today. I'm doing this live from Bruno in the Czech Republic. So I apologize if my sound quality might not be perfect, but I'm in red head quarters here and heading off to DEF CON check Republic. There's about 2,000 people signed up to come to this conference tomorrow. So I'm so excited to be here, but I'm even more psyched to have my good friend, Mark Lamarine giving us this presentation today on integrating OpenShift on OpenStack. He's done a lot of great things for us in the past on the OpenShift team and he's going to explain the basics of what our integration story is and then we'll have Q&A afterwards. If you have questions put them into the chat room and then we can, if he pauses and takes a breath, I may read off the question, but we're going to try and save most of the questions for the end and we should be ready to rock and roll. So Mark, if you would introduce yourself and take it away. All right. You can see my name is Mark Lamarine. I'm a software developer here at Red Hat and I've been working with some people recently on integration on OpenShift and OpenStack. You can find me just so people know. I have my Mark Lama is my moniker to Gare and you can find me pretty much everywhere under that name. Just so you know, I'm working in a group at Red Hat called the Systems Engineering End-to-End Integration Team and our actual mandate is to work across the business units trying to identify places where our various products or components can work better together. And we actually have a group of people whose job it is to do that and to work with the various product teams and make sure that they keep talking as well. In this case, my actual task, the reason I'm involved specifically in the OpenShift on OpenStack is that I'm going to be writing a reference implementation document which describes a single layout. I'm going to, you know, pick a set of parameters and create one as an example. That paper's due sometime this spring, hopefully early, but maybe a little later. And in that capacity, I've been working with the OpenShift on OpenStack Engineering Team and as well with the OpenShift engineers and the OpenStack engineers to try and unify all of our efforts in this area. So sometimes I find myself in learning mode, sometimes I find myself in teaching mode, sometimes I find myself in collaboration and management mode, with all of those things with the goal being to produce something that works, that combines the components in the best possible way. I'm assuming that everybody here is familiar with, certainly with OpenShift, but at least to some degree with cloud services like OpenStack. If you're familiar with OpenStack, then you're going to be aware that it basically provides virtualized machines and network software to find components that 10 years ago or less would have been purely done on bare metal and would have been managed completely. And if you look at OpenShift, you could say, why are you even, what is there to integrate? It runs fine on bare metal. Why wouldn't it run fine in a service like this? And the answer is, it really does. But it turns out that there are some of the virtualized services that OpenStack can offer that OpenShift can take advantage of if it's aware that they're there, and a certain amount of awareness is necessary to take the best advantage of them. Today, I'm going to be talking about four areas, installation, networking, storage, and identity management. These are the areas where it looks like, or where it seems obvious that integration, tidy integration, would be the most useful. And it turns out that in some of the cases, your intuition is right. And in others, it's not so much yet. And we'll get into that in a minute. The three service areas, installation, I'm going to leave aside for a moment because that's purely something that OpenStack does. But as far as the integration services, networking, storage, and identity management are the areas where it seems most likely that some kind of communication between the two services makes sense. And when I say that, what I really mean, when I'm talking about integration, what I'm talking about is a situation where, for some reason, OpenShift actually needs to be aware in some way that it's running on OpenStack when it's configured or during operations, or cases where OpenStack needs to be aware, and this is less an issue, but OpenStack needs to be aware that it's working with OpenShift. There have been a couple of cases where there are features we want in OpenStack that OpenShift could use. That's not generally the case. More we're focusing on how to configure OpenShift so that it can take advantage of things specifically and that it actually has to know that it's dealing with OpenStack. I'll get into how each of these, how the integration happens in a minute, but just as a general thing for the networking, OpenStack has a couple of different software-defined network mechanisms, OVS and GRE, which normally are under the hood for systems that are working on top. We're looking at whether or not and how the networking could be improved for OpenShift by taking advantage of elements of neutron. In the block storage area, it's a little more obvious. When you're running OpenShift on bare metal, you probably have some kind of shared backing storage, some kind of shared block storage, so that your components have persistence and, if necessary, have shared storage. In OpenStack, the service that provides block storage is Cinder. Now, something to be aware of as Cinder is really kind of an abstraction layer, so Cinder could have behind it a number of different storage mechanisms, and I'll cover those in a little more detail later as well. The identity management component is a part where you're going to each of these services has their own user base, and what you really want is for, ideally, for the users of both where they need to use both to have the same username, password system, and ideally single sign-on, but that's not something that we're actually working on at this point. Right now, we're looking at merely making sure that they have the same credentials, the same user, the same password, so that for two reasons. First, so that the users have simplicity and second, so that the administrators of OpenShift and OpenStack, if they need to identify a user, it's the same identifier on both systems, and that they don't need to be managed separately. Now, back to the installation piece. This does assume, I'm talking primarily about installation of OpenShift on OpenStack, which means that I'm assuming that you have a working OpenStack environment with heat enabled. Assume that it has, we're working with OSP7 now. OSP8 is released, I'm not sure the timeline, I didn't check that, but we're not developing against OSP8 yet. I don't think there'll be any upgrade issues, but you never know. The assumption is that Cinder is there and that Cinder is backed by some kind of network file service. And this is, as I was saying, Cinder can serve in OpenStack from a number of backing services, whether they're network services like SAP and Gluster and NFS, or iSCSI, which is also a network file service, but which actually attaches at a block layer rather than being a service above. And it assumes, and we'll talk about this later, this setting assumes that the Keystone OpenStack service is backed by an LDAP service. So the Keystone is not using its own internal database for user management. That's going to give you the best set of capabilities, the best management tools, because LDAP has well-known management tools. And if you're using an LDAP service in general, as long as the schemas match for authentication, you should be able to bind both of them to the same LDAP service. OpenShift has a set of requirements to run, which come outside of the installation as well. In this case, the assumption is that you're going to have either pre-allocated or at least the capability of having a floating IP from OpenStack, which is going to be used for inbound connections from the OpenShift customers. It also needs to be published using a wildcard DNS record. People who've deployed OpenShift 3 are going to be familiar with this, so it shouldn't be, I'm not going to spend a lot of time on it. A lot of what I'm going to be talking about, my slides are actually a little out of order, so I may end up jumping a little bit as I go. Much of what I'm going to be talking about is the heat stack deployment, because the goal here is to have a heat template which encapsulates all of these pieces, so that when you're doing the deployment, taking advantage of the OpenStack components, you don't need to manipulate those components directly. You should be able to pass all of the parameters that you want directly to the heat stack, and it should do all the setup for you. The heat stack itself, again, I'm assuming that you're familiar with heat. The heat stack takes, oops, I have a typo there, two dots. The heat stack takes a set of parameters which customize your deployment, and those are provided through a YAML file, which is provided at the heat invocation. I'll show an example of that in a moment. What you're seeing on this slide are the standard parameters that you'd need to use to run the OpenShift heat stack. With these parameters, you'd get a naive installation. You can tweak some of them to your own, you can select weather image, but if you use just these parameters, you should get an OpenShift service that doesn't take any particular advantage of OpenStack resources. I'm going to be covering the things you can add to that in a moment. As I said, I think I'm going to jump around a little bit because I want to get to the invocation. If you were going to use that file, it'd be saved as nf.yaml. I don't think I can highlight that. This is an invocation and a couple of samples for the heat stack create. It creates a stack called OSC, has a timeout of, I think, 180 minutes. The parameters that you saw in the first parameter slide there are in the nf.yaml file. You notice that the minus e indicates that those are things to replace environment variables or parameters. The second argument I'm using in this case because I'm building a fairly small, I'm not building an HA. I'll get in a little bit to some stock variables that you can use stock environment variables that are provided. This nf.single, I'll show you in a couple of minutes, it basically says don't create a load balancer. Just mock up the load balancer. This is disabling load balancing. The final line of the invocation, the minus f option, is the entry point to the stack. That file is the actual entry point to the template. I'm not going to get into a lot of details of what's inside those. I will point you at the github for those. Once you start the stack, the stack create is asynchronous. The second invocation shows you how to see that your stack is started. In this case, I pulled this up this morning. It claims that the stack creation is complete. You can see how long it took to execute. You'd see create and process in the creation time if that weren't true. These are actually two different stacks. Great. The final invocation there, I'm not going to show because it's a large output, but it would show the current status of the stack as it's executing and what resources are part of it. I want to go back. This shows that as the stack is running, as the stack is executing, this is a create sample from what you'd see in Horizon as the stack's creating. You can see the components being created and the grayed out one there in the middle is a set of nodes being created. When you run the stock open shift heat stack, you'll see something like this. You'll see an additional network and router and you'll see the nodes that the stack will create. Some of those will be open shift master hosts. If you're doing HA, you'll have multiple ones of those and then you'll definitely have multiple compute nodes. Again, here's a list of the hosts that are created by the stack. Again, I made this snapshot this morning rather than last night after a run through. You can see it's not in progress, but this gives you an idea of what you'll see. In this case, you'll see an internal name server and a single master node or master host and two node hosts for open shift. Let me just check. That's what I wanted to get into. I wanted to start talking about the various integration points before I try to show what the invocations will look like. For networking, open shift has a component called the open shift SDN. What the open shift SDN service does, it's a standard open shift component and it provides the communications between the nodes, the compute nodes, and also between the containers and pods that are hosted on them. By default, if you turn on the open shift SDN, I believe what you get is an open V-switch network underneath Flannel. If you just say yes, I believe you get an OVS network. If you say Flannel, if you specify that the open shift SDN should be Flannel, it creates a Flannel network using the host gateway mode. The host gateway mode for Flannel is, Flannel actually uses OVS underneath it as well and Neutron uses OVS underneath it in most common installations. There's some question about the performance of a virtualized network layered over the top of a second virtualized network. It works. I don't have numbers. I'm hoping to get this set up and hand a couple of identical configurations over to some of our performance people who can compare the actual network performance between what we call a vert on vert network or one using Flannel with the host gateway mode. The host gateway mode is much simpler. The routing is essentially a lookup table, so there's very little routing logic being executed as a packet is received and encapsulated. The disadvantage from a generalized network standpoint is that the host gateway mode requires a single broadcast domain for all the participants where an OVS network is complete networking and can manage routing as well as switching. The integration here, these are the options we have set up right now for open shift on OpenStack using the heat templates. There is the possibility, we looked at the possibility of trying to use Neutron directly and various things led us to not seeing that as the best option. There is an option to Neutron which is going to be supposedly available in the next release called VlanAware VMs. When that service is available in OpenStack, we might reconsider what the recommended option is because that will allow us to add isolation to the networks that is not there yet or is not very strong yet. So while this flannel with the host gateway mode is the current recommended configuration, we're going to continue to look at other options. Now let me see if I can scan back. This one I actually have, this is a fairly simple, I was showing you the parameters. If you want to enable OpenShift SDN with flannel, I hate kind of saying simply do things, but in this case it really is. You add a single argument to those environments and set the OpenShift SDN parameter to the word flannel. This will cause the heat stack to pass down the arguments necessary to implement flannel across the various nodes. I'm going to back up once more because I wanted to show here something that I skipped in the installation part was that we're implementing this as heat only in the sense that the person doing the installation only needs to feed arguments to heat. They don't need to mess around with any other services, but one of the features of heat is that it's very good, it's designed for managing OpenStack and it has mechanisms for passing arguments down to run on the VMs. We're using Ansible as the definition for how to do the actual VM configuration and we're using the OpenShift Ansible playbooks which are being built by the OpenShift team. This is one of the cases where I think there have been a couple of instances where the developers of the OpenShift on OpenStack template have talked to the OpenShift Ansible people and said we would like this feature added to OpenShift Ansible and I'm pretty sure the Cinder is one of the areas where they've done that. When you're executing the stack it's important to be aware that heat is creating the infrastructure in OpenStack and then it's invoking Ansible on the VMs so that Ansible will continue the process of doing the configuration. Both of these are areas where people could potentially contribute and certainly can look and see what the behaviors are for the services. In this case the Ansible playbook that you would run, if you were manually running this Ansible playbook you would specify in the Ansible parameters that you give it, please create me an OpenShift SDN and do it using Flannel and those arguments are merely passed through to the Ansible playbooks. Some of that is true in others as well. Networking is one of the cases where we looked at making direct use of Neutron but so far using Flannel or a service like that for OpenShift has appeared to be a better option. The second area that I've talked about is persistent storage and block storage. Again the goal is to control it all using heat so there are parameters that I'll have to look up in this case that you can pass, tell it, give an argument that says enable Cinder. You also need to give in this case a set of open stack credentials and I'll have to add these to the slides later. I didn't manage to get them all in but if you're going to enable Cinder in OpenShift what this does, let me go cover the use cases. There's three use cases that we've identified that you'd want to use Cinder and for OpenShift. The first is that each of the nodes has a set of Docker containers and rather than having that storage allocated from the VM's ephemeral storage you could potentially want to allocate that from the varlib Docker storage, the actual container storage from some block storage system. That would allow you to rebind it later to another instance if you were going to share that information. I'm not entirely sure about that as a desired use case but at least the capability is there to use additional storage or to use permanent storage, persistent storage instead of the ephemeral storage in the VM. The second one is more important because you definitely want persistent storage for it. OpenShift can provide its own internal registry and I think it requires it for most operation. When it does builds it will place any new images you create into the OpenShift registry and that registry is actually run inside the OpenShift network so that it's available to the nodes when they start up new containers. You definitely want persistent storage for those final images and intermediate images. When you specify that you want to have an OpenShift registry and you specify that Cinder should be enabled the OpenShift registry will be created and it will use Cinder storage and that storage will be automatically created. The Cinder volume will be created by the stack and it will be mounted onto the registry host and it will be bound into the registry service. That's a second case second case where Cinder offers a benefit over a naive installation. The final one is actually for OpenShift applications. This one's less automatic right now because OpenShift doesn't currently have the means for a user to create volumes in underlying services. Any persistent volumes that OpenShift is aware of have to be created before they're provided. Now the way this works is that and this is why we'll talk about later the unified identification identity management will be useful because the user will log into their OpenSack account and they can create a Cinder volume there and they can retrieve the Cinder volume ID. That will you know OpenSack creates the volume, provides the ID. When the user goes to create a new service in OpenShift they're going to provide that volume ID and specify that the volume the exterior volume to be mounted is a Cinder volume. Underneath Kubernetes will receive this and Kubernetes can interact with Cinder on each of the nodes. Kubernetes will mount the specified volume onto the node wherever the container is starting and then also bind that into the container when it starts. The last bullet here is a place where there's documentation for the user behaviors that the users have to manage to enable storage in their containers. The best of my knowledge is I don't know of any plans at this point to make OpenShift aware of underlying ephemeral or excuse me underlying storage services but I wouldn't count that out. This is not part of our focus right now. This is going to be the behavior as it stands. There are options to pass in. I don't have them in the slides right now. I'm going to be adding them when this is done. Right now the Cinder behavior is not committed to the master branch in the OpenShift on OpenStack and it may be shifting a little which is part of the reason I didn't include it but I'll be including those in the slides and the links at the bottom will lead to the GitHub repo where the source code is. We talked about common identity a moment ago. If a Cinder user is creating a volume and then mounting that into their OpenShift containers it would be nice if those resources were identified using the same name and ideally using the same username and password and even more ideally managed in a single location so that the people managing the user base for the two services can do that in a single place. OpenShift and OpenStack through the use of an LDAP database can share user information. It turns out that OpenShift can actually use the Keystone internal database but that's not really a desirable mechanism because Keystone has its own units of user management. It would mean that the OpenStack administrators would have to know about the OpenShift users and manage those users as well and there may be users who don't need that and so putting that on the OpenStack administrators to manage that user database probably isn't the best choice. The ideal solution is to have Keystone use an external LDAP database. That way OpenShift which uses right now uses an Apache server as its front end both for the web interface and for the the REST interface. That Apache server can also bind using the standard mod-off-ldap-x module to the same LDAP server with the same credentials and then both Keystone and OpenShift can use the same user database. At this point they're still just using the same user database. I believe that the access policies still have to be managed on either side but that makes sense because the OpenStack and OpenShift administrators are going to be able to set their usage policies independently of each other. Now all they need is the presence of a user to be able to do that that policy management. Again I don't think I have, you know, see unfortunately I didn't manage to get in here. This is where I was going to put the the actual parameters and I will put them in the slides so that you can see what they are and have pointers to them as well. So the point here is that I'd really like you to take a look at some of the stuff. Try it out, tell us what's working, what's not working, what you'd like to see. The GitHub repo is public. You can configure your own, you can do a stack create and watch what happens. Feel free to, you know, put in issues and pull requests. At the bottom, finally here's the locations of the OpenShift on OpenStack GitHub and the OpenShift Ansible playbooks. There's also references here to configuring and using Cinder on OpenShift when they're not integrated. These should be included in the heat templates when they're done and I will add links to these various references at the end. And at this point I guess I'll open things up for questions. Yeah well thank you very much. I know that it's a lot to cover and Judd's asking that that he didn't hear much about DNS and how is naming handled and is anything automated? The stack as it stands right now creates a DNS server which is used primarily for the demos. DNS is a publication service so for the heat stack to modify external DNS isn't really, doesn't really make sense. That was why earlier on I noted that one of the requirements is that you have or are able to get the wildcard DNS for the inbound router. Scott's got something there. Scott just made a note. So if you're going to deploy this for real, the way OpenShift 3 manages the IPs is that it has its own DNS. It needs to be, that DNS needs to be delegated or actually I don't think it's delegated. I think the wildcard DNS will point to a single A record so that all IP addresses which OpenShift provides will be pointed to that A record which is then a router, a load balancer which will accept the packets and behind that is a router that will distribute them to the containers. That's, it's been a while since I looked at the DNS, the actual DNS setup for OpenShift 3 so I'm ready to stand corrected if that's the case but other than establishing a place where the OpenShift can add records using DNS mask to the existing service the outbound publication is still not automated. So both Judd and Scott, if you unmute yourself you can ask questions. Some of them to be aware of is that there's really two issues here. One is that OpenShift needs to be able to add records so that the applications it has will be available will be published. But DNS is hierarchical and DNS requires the cooperation of someone upstream to publish things. So that wildcard record still has to be established by someone with the authority to do that. If you just create an OpenShift service and say here's my fully qualified DNS name for the root of my service but you don't have an agreement with someone to publish that you're still not going to get it and that's that's a feature of DNS that's a fact of DNS. Using the wildcard makes that a little bit easier because I believe that it allows the creation of a single record once as opposed to delegation and then updating which requires continual updating each time. In OpenShift 2 each time you created an application it required updating a DNS service the actual zone records. I don't think that's true with OpenShift 3. Yeah it's not Mark. So for OpenShift v3 we just need to create a wildcard DNS entry in the zone file for the zone that covers OpenShift and then any application that gets created will dynamically prepend its host name for the app to that to that record and be resolvable. Hey Jed I'm curious as to what you mean by can EtsyD or or Consul perform these tasks here is that related to your DNS question what are you referring to there? Yeah one of the great features of Consul is that it's also a DNS server so as you're creating nodes whether they be containers or virtual machines the really the state machine the great database distributed database brains of the whole orchestration operation also manages all the naming and is listening on 53 UDP and all that and so you could potentially instead of using DNS mask or what have you and the potential points of failure with DNS mask or a bind server just leverage what you got in EtsyD or Consul to provide that all the naming necessary within the wildcard. Yeah so we haven't taken a look at that yet but I think at the end of the day what the requirements are is for us to be able to update some DNS service with the dynamic host names that are being generated for the VMs that are deployed onto the OpenStack environment. Now we can we can do a heat stack query after the environment has been deployed and pull certain parameters out of that based on preconfigured values from the stack that was created and if if you want to inject those into your Consul DNS service or into your bind you know dDNS update mechanisms in your production network then that's something that could certainly be considered and evaluated but but right now we haven't taken a look at whether or not EtsyD or Consul can look at that however I'll add it to the to-do list to take a look at later thanks. I might just save you a big step you know and you're there is a nice distributed presentation. Absolutely thanks. All right are there any other questions on the on the call? Add them into the chat we can do them. Yeah I have a quick question you know in terms of ETCDE services from OpenShift so how it is going to coordinate with the DNS you know from the OpenStack so what is the exact correlation with that when any container spins up in OpenShift? So that's a rash correct? Arvin yeah Arvin. Okay Arvin sorry you were really breaking up there and I couldn't quite make out the question I was asking because I wonder if it corresponded to the question in the chat window. No my question is related to the ETCDE services which will be used by OpenShift how it is going to coordinate with the DNS services at OpenStack layer when any container spins up. Sorry Diane could you could you make that perhaps you could just type that into the chat channel and I'll try to answer it. Yeah sure. And then while you're typing that I'm going to take a look at the other question in the chat so if someone might get OSP running by COLA would it be possible to deploy the OpenShift environment on top of this environment? Right now as of the 311 release for OpenShift we do have support for deploying OpenShift in containers and in fact we have been working on a feature via this heat stack a deployment that will allow you to deploy OpenShift in containers on top of an atomic enterprise host on top of the OpenStack environment. So provided your you know COLA is kind of orthogonal to that we we do have OpenShift running in containers that can take advantage of our entire you know atomic kind of ecosystem if you will for containers which includes the atomic host itself. So I don't see why running OpenShift 7 by COLA would prevent you from launching the heat stack in this manner right now I think it would work fine. Okay and we're just waiting for Arvind to type his question into the chat here we go how XED service coordinate with DNS services on OpenStack. How does XED service coordinate with DNS services on OpenStack? Right now they're functioning as independent entities and the DNS services or or XED services with an OpenShift are just used for the OpenShift requirements for keeping state on the OpenShift side. They're not currently correlated to the DNS services on OpenStack right now. These are all great questions and I'm really pleased with the conversation. Are there any others out there? Absolutely and and if you guys have any feedback or want to discuss these let's uh let's please do so this is this is great stuff Diane thanks for coordinating. All right well thank you very much Scott for jumping in there and Mark for giving us oops there's one more question here I'm going to sneak it in for Judd is is LDAP block? Yeah if I don't have an LDAP back end for my for for Keystone I can still deploy right none of none of our deployments are have LDAP back ends that we know of. Absolutely you can still know you're yes you can do what you want to do the you can also I believe still unify the the um the authentication the identity management but your capabilities are going to be more limited using Keystone than it would be if both were using LDAP. Sure your capabilities outside for things other than OpenShift on OpenStack. And Judd in addition to that like if you're just deploying OpenShift on top of take any provider for example and you're not doing any LDAP integration you can still log into that OpenShift host and configure it to authenticate however you want whether that's using wide open for proof of concept purposes or whether or not you're setting up ht password authentication on it nothing prevents you from doing any of that we're just trying to make integrating into your existing LDAP infrastructure easier if you do have it. One of the reasons I love Red Hat. Thank you. I sold a lot of LDAP and I built some big LDAP systems so yay. Good LDAP awesome all right and pause again anyone else have any other questions all righty then going going once going twice there's real easy ways to get a hold of mark on gmail github free node um IRC channel whoops there comes another one all right here it comes I knew they would sneak one in. Arvind is asking will C groups at OpenStack have any impact with C groups at nodes running on OpenShift? I have to think about that I I don't think from the standpoint the the the C groups of the VM where OpenShift is running should be distinct from the C groups on the OpenStack compute nodes now if the OpenStack compute node has somehow limited the the VM process capabilities resource usage then it would affect the entire node it would affect each you know basically the C group on on the OpenStack compute node would say limit OpenStack VMs to this much um that would limit everything on each node it wouldn't be uh I'm I'm I think they're distinct that makes sense yeah thanks I think that that does make sense I think that might be something we have to go and double check on to just shoot and that actually makes a certain amount of sense because yeah I mean the the OpenStack uh resource management um really should probably be distinct from the OpenShift resource management they need a little bit more insight into that that'll be I think another the topic for a blog post someday side commentary by someone like Dan Walter or somebody yeah fun folks here so um again thank you very much Mark and Scott and um we'll post the slides um we'll give Mark another interview to finish his links and fill in some of the blanks but um the slides will go out on the OpenShift Commons mailing list I'll send a link to that and we'll update and put this recording up on briefings so we'll be back again next week and next week we're doing identity management um so there'll be some good questions there take care and once again thank you