 So this presentation is about let's encrypt and how it's going to be applied to an infrastructure of messaging brokers based on ActiveMQ which are deployed in CoraF containers. First a little bit about me, I'm Alex Barca, I'm an Apache committer, I have contributions to Apache logging but mostly I have contributions to other open source technologies like Jenkins or Git. I'm here to talk from the operations perspective of managing ActiveMQ infrastructure and I hope you will find the information useful. A little bit about the context and the problem definition. If you have been to the previous sessions you probably heard this already, it's about a way to scale an ActiveMQ messaging infrastructure and have multiple networks integrated in a way that management of this infrastructure is done seamless. I'm going to talk more lately about the different broker but the main feature of this infrastructure was to be secure at first. We gather a lot of requirements from previous customers and we want to have like best practices of how to deploy this type of infrastructure and it started more as a playground and eventually it grew into a multi-tenant configuration of ActiveMQ that was also scalable and highly secure. We focused also on governance of the queues and topics to be able to have multiple applications being able to talk to each other. The biggest problem that we saw was not on the first deployment of an ActiveMQ infrastructure but was more on the maintenance. Different teams produce different applications and most of the time when they need to install a new version of the application they will reinstall the whole broker suite or they lock the brokers related to an application. It's very hard at least in a big company where you have multiple teams developing multiple apps to be able even to talk to each other or even operate. We saw that the biggest concern are actually related to configuration, patches. And then when we realized that we have a lot of brokers we had to think about topology and how to make it reliable. In ActiveMQ there are mainly two types of reliability. It's active passive and active active which is multi master or master slave. We chose the later one, multi master and this is because we saw that our infrastructure being large and addresses multiple availability zones you need failover between those. So getting to the operations part of it. Today the technology trends are towards microservices which try to make small applications, small deployments as a consequence but many of them. So you manage them independently and their upgrades are less riskier. All the operations aspects actually are related more to risk management and to the business side than to technological side. So when you talk about a large ActiveMQ infrastructure you talk about more about risk management. The DevOps principles were incorporating development behavior like versioning your code into operations meaning versioning your infrastructure now and being able to work together and both teams development and operations feel the same pain. Continuous delivery relied more on continuous packaging and delivering the application and having the ability to have the feedback loop from the operation side back to development. People when they talk about cloud they usually refer to having the capability to request resources on demand. So related to ActiveMQ in our problem statement we said that it's normal to just request a new broker on demand, on demand based on load or based on health. So these are the aspects that define the constraints in operations you need to have high velocity and is more driven by business because any code that is not in production is basically waste. Consistency is when you manage an infrastructure you cannot have a lot of complexity. You cannot manage each service in a different way. So you need to have a consistent way of managing all the brokers in this case. That's why most of the clients that we saw have the same broker configuration or the brokers behave in the same way. Although we saw that there is a need for brokers to have different behavior. I will talk a little bit later about this scaling. So another requirement was because everything was built on the cloud is to be able to scale out anytime you want on demand to create new brokers that automatically interconnects with the other brokers and so the network of brokers grows. And the main topic of this presentation is about securing and how, when you bring a new broker online, you can qualify that broker is the right broker to talk to. So for the solution we choose two containers, the Kurov container and the Docker container. Any containers what it offers, it offers isolation, failover and scalability. Usually those are the main reason for any containerization solution. But people when they talk to infrastructure, well, wouldn't Docker solve all the problems? Well, it kind of doesn't because when we saw the infrastructure with different types of brokers we understood that we have not only a problem of structure but we have also a problem of the behavior of the system that we had to manage both of them. With Docker infrastructure they do not manage the behavior, they manage more the structure of it of how many containers you can spin or it's very limited the behavioral aspect. So you can change how active and queue brokers behave by changing its configuration and Kurov provided these hot deployments where features and configurations can be updated on the fly and you can change the behavior of the broker without need to bring down the broker. It offers also advanced logging with the decanter it can integrate with elastic search. With the dynamic configuration you can also change it remotely either through SSH or through the Xbin URL and you have a secure framework through JAS. Now Docker containers offer a different aspect of the advantages, different aspects. They are more related to standardization and bringing consistency to your infrastructure. It also offers load balancing but in this case we use the failover of active MQ so for exactly active MQ brokers we didn't need but we need load balancing for a special type of service which would be the central configuration of the active MQ brokers. It offers auto scaling on demand and also you have versioning of the images and versioning of the containers and probably the most important aspect that we choose Docker for was to easily upgrade the patches or the OS so one of the things that we saw in the infrastructure that we configured is that people when they deploy the brokers they don't touch them. If the team that develop the application leaves or something happens that knowledge completely gets lost. If some other application comes it's with a different configuration that usually does not want to interconnect because of the risk of something fails so patch or new upgrades are very difficult in current setup. This is the Kurov architecture and few of the reasons we took it. I want to face that Docker doesn't solve all the problems. You need for different layer of containerization you have different aspects that you need to manage or take into consideration. I always like the analogy that in a container you usually do not have the merchandise as it is. For example, if you have to transport some wine the wines will be in crates and the crates will be in the container. You do not have just the wine in a container. Getting back to our problem definition we realize that we have a different life cycle for different types of brokers. As I said in current configurations brokers have a long life cycle. They last for years. We saw that the best approach is actually to try to limit the life cycle. I will explain the reasons why. When you have a new broker spinning up you need to be able to trust the communication with that broker. The best way is to use HTTPS, a TLS certificate. In order to provide that certificate you need a CA. Most companies deploy their own CA and other companies that are not so experienced they have some certificate for the CA or for their intermediate CA. Meaning that now the brokers need to have a different configuration and every time you deploy a new broker you need to accept that certificate you need to talk with the CA. There is a communication there and it's impossible to manage so signed certificates at scale. Even when you have a CA in the corporation and it is trusted there are constraints to interoperate. Using wildcard CA is not costly efficient if you use a trusted CA or trusted intermediate CA it gets to thousands of dollars at least. That's why we took a look into let's encrypt. We understood that some brokers are a little bit long lasting at the edges where clients connect to but we understood that some brokers can be optimized in terms of storage for persistence. They manage the advisories so they can manage the route and the failover. They can have processing and so on. Then we are talking about two layers of orchestration. One is for Docker containers itself and the other is for the active MQ or CAROF depends on how you want to see it. These have two responsibilities. One the Docker one is for orchestrating the structure and the availability of brokers in different regions failover for different data centers or clusters and the active MQ central or distributed and central configuration would be more for the behavior of the brokers itself. Also it allowed us to have a topology that is reactive to load or to health of the whole system. This is how we started to take a deep dive into let's encrypt. Let's Encrypt started as a project in 2012 but it eventually became available only last year about a year ago. It is now a project under Alliance Foundation umbrella. It's a standard that is a draft, is in progress, is considered to be released in September this year. Also it is a draft. Let's Encrypt is also a service that is available since April so it works pretty stable. It provides free TLS certificates, free as in beer. It has a little bit more about just generating the certificate. It has also a way to integrate with your current infrastructure and automate your process in relation to your TLS generation process. First client was Serdbot. It used to be part of the Let's Encrypt. Now it's taken out but it's still the default and recommended client for Let's Encrypt. Let's Encrypt doesn't allow wild cards. It's very important to understand that you will not be able to get intermediate seeing and from that moment on generate your own certificates. Every time you need a certificate for a domain or sub-domain you need to talk to Let's Encrypt. Let's Encrypt manages both the creation, renewal and revocation of a certificate. It follows a protocol. It's called Acme, of course, or certificate management environment. The basic idea behind it is that when you request a certificate, usually the company that gives you the certificate also validates that your company is a true company, you have the right address. In this case they cannot validate more than the fact that you own the DNS server. DNS server becomes a very, from an operations point of view as a constraint, it is a critical system so you need to take care of the security, how to manage that DNS so the access to it is not tempered with. So Let's Encrypt validates that you own the DNS and the way how it works is you create a request to Let's Encrypt, they ask you to provide the token on the DNS, they validate the token is there and say, hooray, you have the access to DNS so please here's the Let's Encrypt. The default presentations and configuration that you may see is about integration with Apache or Nginx or Webroot or standalone. What those means is that you need to have a long running process like a web server where Let's Encrypt will issue the request to make sure that you also own the server so you make a request to Let's Encrypt, Let's Encrypt says here is the nonce and please sign this nonce with your key and put it somewhere on a web server. You put it on somewhere and then Let's Encrypt based on the DNS goes to your server not based on an IP, based on DNS and then checks reverse DNS as well that you own that server and if you've been to some presentations here in ApacheCon they talk only about those scenarios. I find those scenarios actually to be less useful when you automate the infrastructure. The most useful is actually the manual one and the manual one I would have called automatic one but instead of using the DNS or C name records it uses the text entries meaning that you can issue certificates without needing even to have a server and that's what I'm going to present actually. So in manual plugin I would say the DNS and DNS one ECME protocol it goes like this. You have a script or a program that needs to manage your TLS certificates and you make a request to Let's Encrypt and Let's Encrypt says okay please prove that you're authorized to do this put this token in your DNS text records and sign it with your key and you provision that DNS text record and then Let's Encrypt will validate only against the DNS provider so in this case you do not need to have a web server at all and you can generate as many certificates as you want even if you do not have the servers online. This is very useful for a few reasons. One is that you can provision certificates before you even have the service available. As a consequence when you have the ActiveMQ Brokers live you can provision the certificates without needing to have a process even on that infrastructure that you just requested. So let's say you request some hosts on Amazon you can pre provision the certificates before your even infrastructure gets available and then it verifies and you get the certificates. The problem is that if that would be the whole problem would be nice and simple but it gets a little bit more complicated when you have different orchestration layers. As I said you have the orchestration for Docker and you have the orchestration for ActiveMQ and RUF and any operations because they need to standardize they need to make inventories of those certificates and your secrets and eventually it needs to get to Kharaf and Kharaf is any other Java process mainly they use JKS they use a CPKI but they use JKS so you need to create the certificate you need to convert the certificate from the PAM format to the JKS format you need to get the JKS to the Kharaf container and you need to reload the Kharaf with ActiveMQ or use the hot deployment to respite configuration. So this is a rather complex process and if you would need to code yourself and need to manage this aside would be hard but unfortunately let's include made it easier and when you manage the infrastructure at scale you need to have some convention over configuration. They have a very good convention for managing certificates and this is the folder structure that you will see in any on any computer that manages those certificates so in order to have a software like a certificate manager what you would need is to install the client the setbot client on your host can be in a Docker container or can be on a dedicated computer inside of your network not publicly available so if you think of security your secrets you do not want to expose them on a host that's publicly available you want to be behind the firewall so that's what I would recommend and the inventory of the certificates look something like this. The account is mainly for your let's encrypt accounts you have all the history and all the inventory of all the interactions that you had with the let's encrypt and you have both for CSR and for keys and for all the chain and full chain which is the intermediate C8 with the full with the full chain. It's awesome for all to think it's awesome to know easily that nobody tampered with your DNS especially if you have a dedicated zone where you are allowed to create your subdomains and have the brokers isolated in that way. That's why I put the CSR and keys there to see that you have all the history with let's encrypt in regards to that cert bot instance. Now if you use cert bot you will see in live and in archive the certificates which is the cert, the chain, the full chain and the private key. You will not see the P12 files or the JKS but we love the convention of let's encrypt and we said okay after you create the PEM certificate you need to have some conversions to other formats of certs and in this case what was JKS. So let's encrypt has the concept of plugins and you can plug in your DNS provider or you can plug in other type of I would say integration points for your certificate management and this is what I'm going to demo. Besides the plugins that it provides out of the box it has also the hooks which are very similar to Git hooks that help you automate the processes around pre and post creation or pre and post renewal or deletion. So this is a cert bot command where you can integrate that command with your certificates and you basically do it in one shot. So let me have the demo real quick. So for the first layer of the first layer of the configuration we use the rancher with all the stacks. These are the containers about 100 of them only on staging environment but what I wanted to show you is the certificates. So another aspect of inventories is that most of the time you don't have only one inventory you have multiple types of inventory you have inventory for your jars and words you have inventories of your docker images you have inventories of secrets and the orchestrator of your docker infrastructure or the passage or how you would like to call it will have its own inventory to manage those services at a certain moment in time. So it's the snapshot of live and used certificates. What I'm going to do now I'm going to issue a new certificate for a sub domain that I would suggest you propose if the network helps me. So the only thing that you would need I will need to drag a little bit because I do not see there very well. Is this fine? Is that fine? Okay. So the only thing that you would need is to do a git clone of this cert bot. One thing to notice about managing your infrastructure with the cert bot is that every time you run it it will try to update itself. It uses the Python virtual environment and it will try to update all the libraries to make it not only compliant with the latest changes of the ACME protocol but also to have the latest Python updates security updates you would like. That's why the cert bot that you have embedded in Ubuntu or other distributions they're always old. Most of the time it will try to execute it will not work. The best way to use it is to use it either from a docker container that you just rebuild it or run cert bot with pseudo so it updates itself and then run it again. So this is a trick that is important to know. So as I said in a docker container you just clone this one if you like but if I would do that you wouldn't see much. And let's pick a domain ACME to Miami. Okay. So first to make sure it's not there. ACME to Miami. So nothing is there. Let's try to... In most tutorials you will see instead of cert bot auto you will see only cert bot. The difference is that when you package cert bot they replace cert bot auto with the cert bot is pretty much the same. It's usually an alias or a sim link into the user bin. So it's exactly one and the same binary. As I said manual. And now you would need to provide your basic hooks. We created the... We wanted to take a look to what plugins exist. There is binding for varnish, Heroku, other DNS providers. But we wanted to have one for GoDaddy because it doesn't exist and I presume everybody or at least majority uses that. This code is publicly available on github epifocal. Let's include extra to just change to Miami. So congratulations. You have your first certificate. Now you have this structure that I said with cert pam and chain pam. What you do not see is the JKS file. And the reason is I didn't use the hook for the generation of it. Let's do it right now. It will work something like this. I tried to give you the steps. So you... So I have a code too but I generated the certificate or that's what happens with live. Of course. I have a misspelled. That sounds better. It's Miami. Used to be another domain if you paid attention. So I will renew the certificate. This is how renewal works. So it has been added to the key store by default. Any output that you have from your scripts are to error output, not the standard output. So the key to what is going to generate is on the error output. And now if we take a look at the list, we will see the JKS there. The P12 and JKS. And now the last hook would run something like this. So if we integrate both these... So now you integrate it with your renter or with whatever orchestrator engine you have. You have your certificate and your... Basically this is... By default it has all these plugins. The plugins are split in two types, authenticators and installers. That's why I recommend using the manual. The installers in Apache will install the certificate in Apache. So if you have Apache and topget behind it, and if you're doing the TLS termination it doesn't much matter. But if you use JKS in your Tomcat for whatever reason, then it won't install it there. So this is what I wanted to demo. Any questions? That one quick. So it's based on TinkerPop. It's a graph database where all the... So ActiveMQ by default takes the configurations from file. And usually the URL that you give is a file-based URL. But not many people know that you can actually use a URL. And then that configuration is actually taken out from whatever that URL will give you. So if you use this kind of URL, you can benefit from that configuration updates. So you can use any kind of URL, but that's why if you use the file you have brokers, it's came out a bit more like you said, that you can have a problem to desameulate all these configuration files and change them and send them to all... It just kind of sucks. So what if you say that benefiting from this is if you stop in a one web server basically, that can search So basically we made the naming convention based on the broker name, which has to be unique, FBHP URL for each of them. How do you make it unique? It's a bit complicated, so we use an interpog graph database to map the whole graph of brokers and clients and connections and everything. And when we know that a new broker, for instance, is required, it first defines in the cluster, generate a configuration for it with the name, then instruct the network infrastructure to be connected with the name. We know because of the name, it's going to be configured the way it was supposed to be configured. So that's where the interpog is not really serving the configuration, but it's the basis for creating the configuration and serving the IHTDP as, actually. And not only IHTPS, but for that configuration, you need high availability. It can be not only one central repository, but actually can be multiple central repositories that can matter. Four regions, yeah. And there are, I think, too few levels. So you have to react both on load, but you have to react on health issues of your system and health issues even are two types of it. One is on the availability of the service which you mainly see in the Docker and you have the healthcare service for each cluster engine. We use Rancher because it can use either Swarm or Mezos or Kubernetes as the clustering engine for your containers. So you have that health check and you also have the health check based on the traffic messages and you can always take the decision to route to a new clover. Any other questions? Anybody uses Let's Encrypt right now? You use it in the manual with hooks or without? Thank you very much.