 So welcome, and first I have some questions for you. Who's using Carrefourly? More than 10 Carrefourly instances or not? Yeah, okay. So this talk actually is more a discussion than a talk. It's about how to set up and manage a large number of Carrefourly instances. By large number I mean something like more than 20, 50 or 100 instances. And how we can bootstrap these instances and sync all together. So we're going to discuss about Carrefourly quickly, what is Carrefourly, and what the different solution we have to set up this kind of cluster. So about me, so I'm JB, probably easier than my full name, because I'm French so it's pretty hard for you I guess. I'm a software architect or talent, I'm a member of the Apache Software Foundation and I'm working on some Apache projects, so from big data to integration and container. So especially on Carrefourly beam and so on. So Carrefourly, what is Carrefourly? It's a container, it's an application container. It's a container where you can deploy any kind of application. It could be web application, it could be spring application, it could be OHGI application, etc. So you decide what you want to deploy. Is lightweight? I guess I hope. No, it's less than 20 megabytes just starting Carrefourly. And it depends what you're going to deploy on Carrefourly, of course. But by itself, Carrefourly is pretty light. And it's enterprise-ready. Enterprise-ready by the features it provides. We're going to take a look about the different services provided by Carrefourly, but thanks to this feature, you can implement your enterprise application. I think it's a great runtime for services and microservices platform, but also for IoT platform. I guess most of you use Carrefourly maybe to deploy some camera routes on it, maybe just to deploy web applications, maybe to deploy, let's say, small services applications, stuff like that. So thanks to the fact it's lightweight and enterprise-ready, you can combine these different features and exactly design the container you need. And if we take a look about the services it provides, obviously you need the OHGI framework. Basically, it's completely ideal for you. It's provided by Carrefourly itself. And thanks to that, you can leverage all the features provided by the OHGI framework if you want. You don't have to. On top of that, you have a feature resolver. Again, it's something completely ideal for you, but when you deploy a new application in Carrefourly, automatically, Carrefourly can install all the applications it needs for your application. Let's say you deploy someone in. You are not late. Let's say you want to deploy a rest services in Carrefourly. Maybe you want to use CXF to do that. In that case, without installing CXF by its own, your feature, your application can automatically install CXF as well. So it's very convenient. On top of that, you have some, let's say, ready-to-use programming model you can use in your application. Your application use Spring. Not a big deal. Spring is already supported by Carrefourly. You want to use CSR or DS, which is a OHGI way of deploying services. You can do that as well. You want to use Blueprint, which is an XML description of your services. You can do that as well. And you can combine all these different ways of setting your services inside the same container. And more than that, the blue boxes are services provided by default by Carrefourly. So of course, you can remove the services if you don't want to use it. But in the standard distribution, you already have it. So you have configuration. Most of the time when you design an application, you need some configuration where my database is located, where I don't know, my rest services is located. And most of the time, we are all developers. So maybe my friend developer has the idea to put the configuration in the com folder and another one to put in the config folder. And another one, it's a mess. So very quickly in Carrefourly, we decided to put all the configuration in the same folder. It's the ETC folder. And there are no... You can decide to change the ETC folder to another name. But basically, you know that all applications' configuration are located there. And it's dynamic. It means that if I change the configuration on the fly, Carrefourly will update the configuration in your application. So that's very interesting. Instances is the way that you can create internal instances of Carrefourly inside an existing instance. The purpose of this is to do some kind of snapshot system. So let's say you have a running instance. It runs fine and you want to deploy a new application in this instance. You don't want to pollute your existing instance because you know it's a good one. So instead of creating a new, let's say, folder, docker, and whatever, you can just create a new instance in Carrefourly, deploy your application, validate that it's okay. And if it's not, you just destroy the instance. And if it's okay, you can merge your configuration and your application to the existing instances. That's pretty convenient. Logging or deployment are going to be very fast on this, okay? I don't want to provide all the details. Maybe what I saw before is Carrefourly is enterprise ready. And you can see here JDBC, JNDI, JMS, JTA, JCA. So for example, for JTA, you need to deal with a transaction and you want to use with a persistence, let's say, I don't know, Ibernate, OpenJPA or whatever. You don't have to set up your persistence XML and all of this stuff multiple times. You just deploy the entity manager one time as a service and all the application and all part of your applications that use this persistence service will directly leverage what you deployed before. And more than that, so the blue one are provided by Carrefourly, more than that, you can extend Carrefourly with all the projects like Kamal, like CXF, like Senko, a bunch of other projects provide features that you can directly deploy in Carrefourly. So it means that you can extend, for example, this afternoon we have a talk about ActiveMQ. You can deploy ActiveMQ in Carrefourly if you want. You can mix different, let's say, typology and purposes of your container. But this is good for one Carrefourly instances, but pretty soon we want more than one Carrefourly instance. And there are different reasons why. The first reason is maybe isolation. I mean you want to dedicate your Carrefourly instance to one specific purpose. Maybe you want to deploy this kind of Kamal route only on one part of your instances. You want to be, to use one instances for an ActiveMQ broker. You want to deploy another instances just for your web part of your web application. So the first thing is isolation. The second thing, the second reason is failover. For any reason if my Carrefourly instances is failing, I would like to recover as soon as I can to limit the downtime. And pretty soon we want also scalability. So this instance, this application won't be deployed only on a unique Carrefourly instance. It will be spread to a bunch of Carrefourly instances and I can use any, let's say, part of these instances in my enterprise platform. It depends of your use cases and it depends of your applications. Okay? Maybe most of the time you're going to mix isolation and scalability. Maybe you will have a cluster of 20 instances and maybe 10 for Kamal routes and 10 for web application. Why not? But you want to manage all in the same platform. So failover. Maybe you know that in Carrefourly by itself you have a failover mechanism of a label based on the log system. You don't have to install anything else. You just start Carrefourly and you have just to set up this and you will have a failover system. So if failover system is out of the box and it's based on a log, this log could be on a file system or it could be on a database. Basically how it works, the first instance that starts take the log and become a master. The other instances try to acquire the same log that they are in slave mode. Okay? And if for any reason the log is released by the master the slave will take the log and become the new master. So you don't lose anything. All configuration, all application you deploy on your master are also deployed on the slaves. And pretty soon, as soon as the log is released or lost then a slave will become the new master and you will recover all your application. To do that, it's very easy to set up. You have one file to change in the system in the etc folder. You have one file named system.properties. And if you take a look at the bottom of this file you will see some comments about the log system. And basically it's where you store your log. So if you use a file system it means that all your Carrefourly instances have to share the same file system using NFS, CFS, whatever. Okay? So that's the only requirement. When you store the first instance you will see in the log file info log acquired. It means that the log has been acquired and this instance of Carrefourly is a master. It's the one you're going to use. And if you start another instance you will see in the log trying to lock. So it means he's waiting for the lock. He's a slave. Okay? There are no limit in the number of slaves. You can set up 10, 20 slaves if you want. The instances are all synced together. So if I deploy an application on my master this application will be deployed on the slave as well. If the master fails again the lock is released and the slave will become a master. The only drawback of this approach is an active passive system. So if you set up five slaves these five slaves are completely passive. You cannot log in on it. You cannot do anything on it. So only the master is active. So active passive is not maybe what you want but for failover it's pretty easy to set up. So another thing that you can do is instead of having active passive you want to do active active. So you want to set a bunch of carafe instances all active together and to sync the state of carafe instances on the fly. So if I deploy an application on one instance I would like to see this application deployed on all other instances. This time it's not provided by default in carafe. You have to install a new feature on top of carafe. The name is carafe cellar. So cellar is a sub-project of carafe. It provides to you the synchronization, discovery and active active setup of your cluster. It's powered by azelcast. So it means that any action you're going to do on one node will be sent to an azelcast data grid shared by all other nodes. So the state of the cluster is stored in azelcast. It's very easy to install. Feature install cellar. I'm not kidding, just this. You start your carafe instances, you do feature install cellar, it's done. Cellar is ready to be used. There are no single pawns of failure because the azelcast instances are embedded in each carafe instance. So it means that if I lose one instance the cluster state is still in sync because I have my azelcast instances running and synced together. So I need at least one node just to maintain my cluster. It's pretty convenient. You have different ways of discovering a node in your cluster. So when you start a new carafe instance to discover this node you have different mechanisms. The first one is multicast. So when a carafe instance starts it sends a multicast message on the network and the existing carafe instances running cellar so you can get this notification and update the cluster. But if multicast is not available of your network because your network administrator is a bit paranoiac for instance you can use static IP addresses. So you just list the netmask and the IP addresses where your carafe instance is running. But you can also mix instances running locally on your on-premise cluster with instances running on Amazon for instance. In azelcast you can send and broadcast messages with RWS for example and so you can discover nodes running on Amazon. Another thing that is not probably by azelcast but probably by cellar you can use Kubernetes. So it means that if you have a pod you can discover your nodes using Kubernetes. ETCD as well. There are different discovery mechanisms and you can extend and implement your own mechanism if you want. It's just one service to implement. Pretty easy. The core purpose of cellar is to synchronize the carafe instances all together but we also provide additional features like DOIGI, distributed OIGI. It means that one service is located on one carafe instance and is used remotely by all the instances. We also provide HTTP balancer. So it means that if you deploy a web application or a servlet in one carafe instance this servlet can be available on the whole cluster even if you don't deploy everywhere thanks to the load balancer provided by cellar. So how it works is pretty easy. We have a data read this one is probably by azelcast and in each cellar instances we have basically four core features these three ones are enabled by default and another one is disabled. Let's take a look. So here are my carafe instance my carafe instance I installed cellar feature and when I installed cellar I have three pieces installed the producer, the consumer and the synchronizer. What's the purpose of it? So the data read again is where I store my cluster no brainer, it's pretty easy. The producer is the pieces of cellar that send an event to the other nodes. Let's say you install an application on one node you have to notify the others but you install this application locally and they have to do the same. The producer send a cluster event from one node to another. And of course if I have a producer on one hand I have a consumer on the other hand so on the other node I have a consumer that receive a cluster event and react accordingly installing your application installing your configuration or whatever and I have one consumer but I have multiple endless and the reason is I have different cluster events depending of if it's a configuration if it's an application if it's a feature for the one who knows what is a feature in Karath it's different cluster events so I have different handle and the synchronizer I use when a new node join a cluster it has to synchronize the state of the cluster with its local state and here you can define different sync policies so for instance you can decide to do a merge first I retrieve from the cluster and so I update my configuration locally and then I push from my local node to the cluster or you can decide that the cluster is only the master so I retrieve from the state from the cluster and I override my configuration and my application locally on my node but I don't press anything on the cluster or on the other hand you can decide the node is the master and in that case you will override the cluster state with a local node state so this is possible just by configuration you can decide the sync policy you want the listener the last thing they are disabled by default but you can enable if you want is when you have a local event let's say you change a configuration in the ATC folder you can decide to broadcast this change to the cluster or not by default it's not but you can enable it it means by default when you want to install an application you have to use a dedicated cluster command cluster bundle install cluster feature install for instance if you enable the listener a simple feature install or bundle install is enough then it will be broadcasted to the cluster this is the different resources CELA can sync for you so first the features who knows the carafe features so for the others a feature in carafe is the descriptor of an application so basically a feature is a list of application components plus the configuration plus optionally some other feature that can be used in your own feature bundles is the OIGI components and configuration optionally you have OBR and even admin support it's not enabled by default it's very some specific use cases so you have different options like DOHDR, HTTP, balancer and stuff like that and another feature that you can use in CELA is the cluster group so let's say I have a big cluster of 50 instances and again I would like to mix my cluster with some isolation so I would like to deploy my application only on 5 nodes in my cluster and I would like to dedicate 10 of the nodes for let's say CXF REST services for instance in that case you can use cluster group so in cluster group you decide this is my cluster group Apachecon1 and containing these 5 nodes and I have another cluster group Apachecon2 containing 20 nodes and when you deploy an application you target to specific cluster group by default you have a unique cluster group named default containing all instances that you can dedicate some deployment you can also a node can be part of multiple cluster groups if you want so it could be for instance you can deploy 2 applications in the same node if the node is part of 2 cluster groups the purpose is just to target and isolate the deployment so CELA is good but you need a manual action first it means that first I have to install my CELA instance and then log on my CELA instance and do feature install CELA it's not really dynamic and more than that all the preparation all the setup has to be done by hand so what we want is to provide a ready to use CELA instance embedding already embedding CELA package as a docker image and be able to automatically bootstrap this so it's one approach I don't say it's the only one but it's the approach I propose here so here we're going to mix 2 things we're going to mix docker and marathon to bootstrap the instances plus CELA to synchronize all instances all together so we mix 2 technologies the first thing we're going to do it's to create a carafe distribution carafe distribution is basically carafe plus some application and CELA so I embed carafe and CELA all together by default and I create my custom distribution so here you can add your own application if you want and to do that in carafe we have a carafe method plugin that allows you to create very easily a custom distribution so to do that here I add some CELA the only thing I define in the dependencies is the location of my feature and the location of my applications and of course CELA by itself and then I just define this is the boot feature that I would like so the boot feature is as soon as I start carafe these applications will be started automatically so as a boot feature here because I would like to start CELA as soon as I start my carafe instances and so I just run mvn clean install and at the end I have a zip file and a tarboard containing my carafe distribution plus CELA plus eventually all my applications that I want first step so I have my custom carafe distribution embedding CELA in my application we are good but it's not enough now we would like to package my custom distribution as a Docker image to do that we create the Docker file so this Docker file is based on Java and it gets my my distribution of carafe plus CELA and installs in the Docker image okay then to do that the only thing that I have to do is the Docker build so the Docker build will take my Docker file and it will create a Docker image containing the Java base image plus my custom carafe distribution containing CELA and my application and here are my CELA instances we are good but now we have to bootstrap this Docker image okay the first thing to do that is to do Docker run again and again in a bunch of nodes it's not what we want it's not dynamic so it's where we are going to introduce mesos some mesos support Docker and so it can bootstrap the Docker images for you and it's mesos plus marathon so we are going to use this to do some kind of dynamic bootstrapping to setup your mesos and marathon you have to approach the first easy way so with DCOS you have all embedded so it's provided by mesosphere you have mesos plus marathon all together and it's pretty easy to bootstrap and to start with another approach you can do your own made DCOS if you want so basically you combine some mesos you start your zoom keeper you start your mesos agent you install marathon and you install marathon it's pretty easy it could be long to do bootstrap and configure and make depending on your machine but it works so it's up to you at the end it's exactly the same and after that we create a marathon JSON configuration for all applications so basically what I do here is an ID my name of my distribution is a character cellar in my case memory and the number of instances that's my initial setup so it means that here I will start with three instances of my custom character distribution and different locations so I say it's a docker this is the location of my docker image and this is the POR number and the way I expose my POR number nothing special there remember is the number of instances and after that I start my DCOS marathon dashboard so this is the initial status so I have let's say 6 CPU available on my cluster I have 7 GB of memory and for now I don't have any services so I'm in an empty let's say space now we deploy our character distribution so imagine again in my case it's character plus cellar but imagine you also add your application I do a run service and I provide my JSON configuration the one I show you before and so here you are the recap about all that we setup so especially the number of instances the number of CPU you can see the docker image CPU and all this stuff when I'm ready I just click on run and here you can see deploying free instances so it means that I'm going to take my docker image and bootstrap this images on different machines so imagine your marathon setup contains I don't know maybe 10 machines or whatever then it will spread the deployment machine again remember we use cellar so it means that these instances will be seen together from a car of perspective thanks to cellar and now we have a new task running containing multiple instances so you can see a different usage and stuff like that but now we want to scare so we started with free instances and we want to bootstrap let's say 10 instances or 20 instances so your use case is black friday you know you're going to have a lot of load during black friday and you want to bootstrap new instances very quickly so it's what we're going to do using marathon so in marathon you can go on the dashboard in the service and you can click on scale and define the new number of instances you want so you can go from 3 to 10 or from 10 to 5 whatever you want the new nodes, the key thing is the new nodes are pretty important they are synced all together again with cellar it means that if I change a configuration in one node all nodes, whatever it's new nodes or not it doesn't really matter these nodes will be synced together all together in terms of configuration and application so let's do that so I go on my service and I click on scale and then I update to 5 new instances so I go from 3 to 5 so it means that now you can see it's growing up I have more instances running and again these instances are synced all together so it means that we implemented dynamic scalability by dynamic it means that of course you still have a manual action to do is the number of instances but you don't care about if the application or deploy we don't care the only thing that you manage you is the number of instances increase or decrease it means that you may have one team dedicated for let's say your platform management and another team dedicated on let's say carafe management the configuration or whatever it means that one one thing that it's really interesting in cellar as well is you can log on one instance or another one it doesn't really matter all action we're going to do on one will be spread to the others there are no notion of cluster manager or whatever any instances is a manager of a cluster but now we can we can go forward I mean we can go further in terms of what we can provide in cellar we plan to provide a marathon scheduler by its own it means that if you set up the number of instances by hand you can define a policy in cellar that automatically deals with marathon to say I need three more instances depending on the loads for instance so it's something that we started in cellar it's not yet released it will be included in the next cellar release but it's a marathon scheduler and the purpose is again to automatically change the number of instances another thing that we plan is to provide an official docker image we have some carap docker images but they are not official and they don't embed cellar so we plan to provide two docker images one is basically different version of carap and one is carap plus cellar and of course as we can leverage DCUS in DCUS you have the universe so the universe in DCUS is the packages that you can easily bootstrap and install we plan to provide instances of this universe for DCUS for the one who knows humbun to juju you have charms in juju it's exactly the same and it's already done for juju it's not yet done for DCUS do you have any questions yes would you recommend this for like an edgey storm which in turn is distributed across my machine that's not exactly the same purpose I mean let's talk about storm or beam or whatever because I'm also working on beam they provide their own let's say providing system if you plan to deploy storm j storm or ignite or whatever in carap then it makes sense but it's not the first purpose the first purpose of this it's more to bootstrap non-applicated distributed application to a kind of platform second question obviously for non-java based network it's really java focused there's another question over there ah, that's a good question first point why docker honestly I remember I think it was two years ago we had a lot of discussion between why do you need docker when you use carap I mean carap by its own is a container so it means that deploying a container is a container maybe it doesn't make sense but why docker is just to leverage the existing let's say ecosystem like a swamp, like a mesos and marathon and all this stuff that's the only reason why there are no strong by itself carap could be a container and it was what we did at the beginning and it works, you can do the same for this the reason why is simply because docker image is very convenient just to package all together that's mostly the reason and now the second point is why mesos more than kubernetes for two reasons first reason is I take my apache cap and I prefer apache project that's not technical decision but now honestly one of the reason kubernetes in cellar in cellar we have the kubernetes pod ready for discovery and bootstrap so the reason why mesos is simply because we have the marathon scheduler available an alternative would have been to use apache overall for instance just to bootstrap the carap instances but what we did that can exactly work with kubernetes no bulb is using kubernetes in this project again this is not something it's just an example there are nothing nothing is hardcoded in cellar or carap to use only mesos of course you can switch to dnsd to etcd to kubernetes whatever using the beautiful infrastructure that you get how do you go through to decide this is a dynamic feature and this is something that I want to build for can you walk me through your process for that that's an interesting point in this approach the minimum piece in the custom distribution is just to bring cellar as part of the core let's say platform you don't have to package your application inside your custom distribution maybe and it's basically what I'm doing I just bootstrap cellar and then I use one instance in my cluster just to install my applications and then this application will be deployed on the cluster and the big advantage of doing that is just to update because when I do cluster feature update then it will update my features of my application in all nodes in my cluster that's probably the best approach to do not embedding your application in your carap distribution even if it's pretty convenient but it means that a team has to be dedicated for the installation of your applications the big advantage of using this approach and custom distribution you just provide the docker file or the tarball and the people who actually deploy the images on the cluster they don't care about what's the configuration, what's your application what's the version, they just deploy and that's all so it's some decision to take while I'm doing myself and package and while I'm leading the other team to do an update for configuration for example it could be some time problematic so yes exactly I mean the key point is your DevOps team I mean if your DevOps team is a bit focused on carap and your application and knows how to change the configuration and what's a feature and stuff like that then you as a development team you only focus on the feature you provide and that's all if your DevOps team is more my purpose and my goal just to deploy what I receive from the dev team in that case you you have to provide a package ready to run and the DevOps team doesn't care about what's the content of the image yes okay so that's a good question so basically the question is what why carap maybe where it comes from and what's the usage of carap for a developer I don't like to say that but I gotta say that carap is an OIGI container so it doesn't mean OIGI is a prime model in Java that allows you to deploy services so some components deploy services and all the components can use these services so it's a kind of SOA platform but very low level I don't like to say that because for a long time carap was only considered as an OIGI container and actually is more than that because you can deploy anything a large scope of application in carap what carap is providing to you is an environment providing services that you can directly leverage in your application you want to do logging in your application then normally let's take a concrete example you want to do some logging in your application means you have to use and define the dependency to the logging platform SLF4j, commands logging, log4j or whatever then you have to define the configuration of your logging system what's my appender, what's my category what's my different let's say basically connecting a lot of into the Java world it's a good description actually it's like an application server like we had in the past exactly so it's a mix between a container as you said it's a good style let's say application server like we had in the past weblogic, westphare and all this stuff but it's more let's say services oriented but yeah it's a container providing to you something that you can use and connect dots as you said and it's a glue between the different part of your Java component as I said it's a good platform for service and microservices application services all together so yeah it's a container so yeah there are different locking mechanism provided by configuration so you have a file based on the share file system you have JDBC you have zookeeper and you have azelcast as well so it means that the log can be on because the example for the example I use an NFS and if you take a look in the configuration file you will see different implementation yes instances yeah so this part is basically when you start carafe you have what we named the root instance okay it's the default one it's where you can interact you can type some commands because you have a complete shell commands where you can type your configuration your extra extra then what you can do is to do instance clone or instance create so instance create create a completely fresh new instances so it's a new JVM so it starts a new JVM with a dedicated file system so it dedicates directory where the files will be located if you decide to clone then you're going to take an existing carafe instances for instance the root and completely duplicate the same instance changing the port number and stuff like that and then you create a new JVM again where you can run your application so it's exactly instead of using this another alternative on your machine is just to copy the existing directory into a new directory changing the port number and it's starting yeah so basically yeah it's a new carafe instance inside your same operating system container yeah okay you're okay with the question you're done okay there's another one and Alex I will get back to you cellar is more swamp or marathon it's when you have an existing docker image cellar is more inside the docker image for synchronization so let me take a concrete example in your docker image you package cellar and that's all you bootstrap using marathon you create 3 instances or 10 instances whatever of your docker image on your cluster and then your DevOps team goes in one instance and you configure it and change foo bar to foo order in a configuration then this configuration change is not related to swamp or docker or whatever it's like internally to your application internally to docker to carafe in that case this is the purpose of cellar to sync this change to all of the instances so that's the different meaning oh yeah it happens a lot I mean in micro services platform it happens a lot if I would like to change the number of thread or if I would like to change the location of my elastic search of Cassandra database for instance because I don't know I may be set up new instances whatever in your cluster group and do the change yeah it's exactly the same question from Bob it depends what let's say what your DevOps team can do or not if you consider your docker image as a standalone and you cannot change that then it's not really useful but if you consider your DevOps team is able to change the location the number of thread or stuff like that in one cluster group then cellar is there cellar actually is very low level in carafe so if you deploy your command routes is basically packaged as a bundle or blueprint bundle or a feature whatever so in that case as soon as you install your command route so bundle or feature in one carafe instance then cellar will be able to spread the deployment no problem at all you're welcome Alex these four they are static in the docker image but they can be exposed dynamically with marathon it's not because I have in that case 8181 here he could be dynamic and managed by marathon actually that's why marathon is pretty interesting and Kubernetes and swamp does exactly the same so you're right static in the docker image but dynamic from a cluster perspective I didn't see that 0 0 means it's dynamic another question thank you so much