 This talk is with my co-worker, Natal, should be here for this talk, but he had some problems to come from France, so this alone. I want to tell you our story about DevOps and Solstack. Anyone who had already used a system configuration management here? Okay. So my name is Pablo. As you can see, I came from Argentina. So this is why I have a very bad English accent, and also I've been living in France for the last five years, and I have a very bad French accent too. So I work as a system administrator developer for the last 10 years, and as a Python developer for almost five years. So our story starts at PeopleDoc. PeopleDoc is a French startup created five years ago. It has offices in Paris and New York. The IT department has more than 40 people across different team locations, different locations in Paris or France. At PeopleDoc, we contribute a lot of open source projects. So it's a good place to start with Solstack. To describe a little more technical level of the company, here is a list of the teams. We have two Python Django teams. They work with the same stack. We use ready, set, elastic search, post-wrestling, many Django locations. Each team has different goals working on different project or product. So we have the Java Scala team. They create the backend services and some APIs used for the Django applications. We also have a new brand single-page application created with Ember.js. We use Django for the API on the backend. The last one is the DevOps team. It's the team where I work. This team is responsible for the infrastructure and the different environments like Staging, QA, and production systems. So we'll talk about tools to simplify. We use Solstack for server provisioning and configuration management. So also we have OpenStack for the cloud instances. We have a local cluster to test some integrations on the QA servers. And for the production system, we have an OpenStack provider. So we can use the same tools for all the environments, local, free-rate, or production environments. Another important tool is Neuralic. We use it for alerting and monitoring. It's a very good tool because it allows to share information between different teams. But sometimes we need to use more specific tools like Sabik, Sentry, or Munin. So this is the main tool we use every day for the production environment. The next question is why we choose Solstack? I think you know there are other solutions like Chef, Puppet, and Civil. So why Solstack? For us, there are two main reasons. The first one is because Solstack is open source. So PeopleDog, we use a lot of open source projects. So it's a very important reason. The second one is because Python. Because Solstack is quite in Python. The DevOps team uses a lot of Python, so it's a logical choice. And this leaves out of the list Chef and Puppet. So the last one is Ansible. Ansible meets these two conditions, open source Python. But Ansible is more recent than Solstack. When we take the decision to use Solstack, Ansible was at the very early stage of development. So Solstack wins. There are many other reasons to use Solstack. If you want, you can see the list of the features of Solstack in the project page. There is more technical details if you want. Before to continue, I will show you some main concepts about Solstack. So basically, there are the same concepts in all the configuration management systems. Ansible, I think, has the same concept. For example, the first one is Master Minion. Solstack works in a client-server model. So it's very simple. The master, the Solmaster, pushes the command to the Minion and the Minion sends the result back to the master. This is the first most important thing to understand about Solstack. Then, a state. Solstack states describe the state in which the systems should be in. By default, the states are the EML files. I will show you an example. Here, for example, we have a file called packages.sls. Beam and a function to install the package. The second block shows the command to install this package or this list of packages in the Minions called server1. So this is a very simple example of a state. Another important concept are pillars. Pillars are structures of data defined on the Solmaster that they are shared by all the Minions. You can store the global variables or sensitive data that will be shared by some Minions or all the Minions at the same time. Here you can see I define a list of users with the user name and user ID. So with this kind of data, I can create all these users or all the Minions everywhere. Another concept is grains. Grains are static information stored in the Minion. It's not like pillar where the information is shared between all the Minions. It's static information only specific to the Minion. For example, you can have things like the running code version, the operating system, or I don't know, many other information about the Minion. So those were some basic concepts about to understand salt. Now I will show you the workflow we use at PeopleDuck and how we use salt. This is a very simple version of the workflow. But when each team delivers a new release of their product, the DevOps teams use salt stack to deploy these products into the production system. So the important thing to see here is the DevOps team needs to create every site for each product for each team. So you have just one place to describe all the infrastructure for all your environment. And in your case, we have just one Git repository with all the states to deploy every product of the PeopleDuck. It works like this. It works pretty well. But not for longer. Why? The first one, the first problem is the bottleneck. As you can see, the DevOps team can become the bottleneck of your system. Because if you have many releases or many teams all the time, it will quickly have a problem to centralize all the deployment for only one team. Another problem may be the complexity. If you have many different stacks, remember there are Java, Scala, Python, elastic cells, Postgres, Redis. You have a lot of complex relations in this case, to manage. So it can be a problem very hard to change. Another problem is, as I said, we have only just one Git repository to store all the state. So when you start, it can be practical to have all the state in the same place. But when everything grows, it may be hard to manage all these states together. We have a lot of conflicts or different versions. Maybe one product needs a version of Postgres. So it's not easy to manage this kind of problem. So we have seen all these problems at PeopleDoc. So we have some ideas that can help. You build it, you run it. This is a quote from Bernard Bogel. The idea is to give the operational responsibilities to developers. These developers are in contact with the day-to-day operation of their product. So this can avoid the problem of the bottleneck, because each team will be responsible for the deployment of their product. So this is the first idea. Second one is a feature of a source tag called formula. The formula is our pre-written state. So it allows you to reuse a state into your projects to do many different tasks, like install packages, start a service, or many other common tasks. With the formula, you can have a modular design of your state. You can avoid the problem of monolithic states. So also this allows you or the team to write their own recipes. So it's a good idea, a good practice to use the formula. Finally, test. For each change you do in your infactual code, you need to test this change somewhere. Because if you think about infactual code, the code needs to be tested, so you need to test. It's not easy to test, because you need to simulate all your environment in your local machine or computer. But it's very complex to simulate the same infactual code, because in the production system, you can have different, I don't know, server, hardware, internet connection, or I don't know, many things can change. So it's not easy. But we start to use Jenkins and LXE, Linux containers. So this allows you to create the same services or container that you can have in an OpenStack cluster. But you can round this in the same Jenkins job. So it's not easy, because we need to think about all these kind of relations of interaction between different products, resites, states, grains, the P-LARC is different for each environment. So it's not easy. So also, if you test in your local environment, you can never be sure that that will be exactly the same in the production environment, because you don't have the same hardware infrastructure in the production system. But you need to approach or detect the problems before that touch the production system. So this is an idea, it's work. We use it for, I think, five months ago. So it's a good idea. But there is not much information about this subject on Solstack, how you can test your state. So we create our own formula to test with LXE and Jenkins. If you go to the formula repository of Solstack, there is a repository called the Jenkins formula. Right there, you can find many useful things to test your state. This is all. So thank you. Do you have any questions? Hi, so one of the reasons why I haven't used Solt recently is a couple of years ago I was doing DevOps but for building development environments. And it was fundamentally, one of the troubles was the primary thing I was debugging was Solt was being very unstable. Has that somewhat calmed down a bit, because it was essentially versions of Solt? Sorry. Has Solt gotten a lot more stable than it was two years ago, as far as my question? You mean if Solt is more stable, Solstack? Yeah, yeah, yeah. Solstack has grown very much the last one to two years, so now I think it's very, very stable, yes. And how different are you using Solt in building your development environments as well? Or is it just local, and then you deploy it, and then you use a separate setup? Sorry, I don't understand. So one of the things I was doing with Solt was using Inside Vagrant to build the near identical development environments to what we're deploying in live. Is that what you're doing, or are you still using Solt in development, are you using Solt in development as well? We use Solt in development. Yeah. Yes, yes, I, yes, and the developer team use Solstack to deploy, to deploy its products in the local environment, yes. Any more questions? Yeah, so thank you. Okay, thank you.