 So first to present myself, I'm Jung Santur, I'm a software engineer at Salando as part of the continuous delivery and deployment team. And today I'm going to present the part we took at Salando from the dark ages of self version to the enlightened age of autonomous work with it. So some quick facts about Salando. So you understand the context of my work. We are, we have our headquarters in Berlin. And we have also other offices in other places like Dublin, Helsinki, Dortmund, airport. We have more than 700, actually nowadays almost 1000 engineers. And we do lots of, almost everything we do is done in-house. So we do lots of development. Okay, so the first question when talking about gitooks is why do we need gitooks in or subversion hooks or other kinds of commitsooks in the first place? The answer to this is to enforce rules. This raises the second question. Why do we need rules? Any large group or organization needs rules? Some rules are internal to the groups, some are external. And they are very important to achieve an efficient workflow to ensure quality and transparency to all stakeholders. And in case of Salando, stakeholders go from the colleague at the next desk to the customer that is trusted as his credit card to the investor who spent lots of his money and he trusted and believed in us. Second question is how we enforce rules? Not our rules are enforced in gitooks, some rules are enforced before deployments, some rules are only monitored, they are not enforced. And so you have to look at your workflow and see what rules can be enforced a priori or with checks and what rules can be checked after you have moved on your workflow. It's vital to keep a proper balance between two types of enforcements, because if you have too many blockers on your workflow, you prevent your developers from being productive, but if you have too few checks and blockers, you risk that bad software goes live and you break the trust of all stakeholders that are in you. So now starting point, dark ages of subversion. This was the place where we were when I arrived at Salando two years ago. We had one batch script with some basic checks, a precomitook. Then we were growing as a company and we needed more rules because it was not possible for someone to keep track of everything that was happening. And it was not possible to keep a batch script anymore, it was not efficient, it was not maintainable. So we ported the subversion commitook from Pesh to Python. Around December 2013 we started moving from Git because subversion was not good enough for us anymore and we had to do something, so we started moving to Git. We still had same rules in places before and we knew we would still keep subversion for some time because we already had lots of teams and some teams wanted to move faster and others slower and not everyone wanted to move to Git, so we had to convince everyone and we decided to do migration in waves. So the first decision we took was to adapt the SVN commitook to work on both SVN and Git. So implemented Gitook as a progressive work because it's invoked at the same point on the workflow as the commitook on subversion. This is when user sense changes to the remote server and because it allowed us to reject the reference or branch without rejecting full push. The other alternative would be an update to hook, but in that one if you return an error code it will block the full push even if most of it is following the rules. Doing this at some problems, this approach at some problems. So the first problem in this plan was that Git is not subversion. They are different in fundamental ways and trying to support both would mean support for both of them. The other problem is that being distributed Git is more flexible than subversion which means that people don't use Git, same way they use subversion and they don't use Git in the same way that other people use Git. The first things that, even the first things that's aligned to migrate to Git start experimenting different workflows and this was a good thing because they were very different projects and different projects have very different requirements. So it makes sense for them to have a different set of best practices and slightly different rules and different pricing strategy. But this also forces us to rethink how we check the rules and how we implement the rules and we had to rethink the Git hooks. So we moved to plan B. Where before we had one common hook for Git and subversion, now we decided to have two different hooks. In practice this meant we forked to hook into, the subversion commit hook was aggregated, so no more changes there because we don't want to really support the subversion anymore. And we continued to work on to Git priority hook. Because different teams wanted to use different workflows, we moved away from one size fits all strategy and allowed teams to specify set rules for themselves. So this is the way we allowed teams to set rules for themselves. We have a configuration format based on YAML where Stash projects and repositories could mesh several rules sets by name. This is similar to CSS. So for example you can see there that one team could have one general rule for themselves for their project. But for a specific repository they could change part of the rules and not the whole set. And a second team could have a completely different rule set because they have different needs. The problem with this second approach, and this was a reflection we did after using the Git hooks for one year, was that we had a centralized configuration which became a bottleneck because it could only be changed by a small number of people. And sometimes we were all on vacation and it was really bad. Another more technical problem we had is that we were trying to check all the commits on a push. Because some rules apply to every single commit. For example we checked that one of the rules we had at Zalendo was that every commit message has to have a ticket. And for this we have to go through all commits and this was problematic for a number of reasons. For example Git history is not linear. So if the user merges or rebases from a branch and we try to check all the messages in the commit range for ticket ID if it matches to a branch, a commit set came from other branches by the rebase or the merge would fail the check. In the end we tried several solutions, some worked way better than others. In the end the solution for this was filtering out commits that were already in a branch because this meant they were already checked before and were not originated on this branch. The second problem we had is that when you merge to or from a long lived branch, some teams had some very long lived projects and they had branches for it and sometimes they deferred from master for two, three months and when they merged from master or to master or when they rebased they had literally ended of commits to check. Because of some of our checks took some times we had pushes that took more than half an hour to check and this was not a good experience for our developers. Related to this and one of the slowest checks we had is checking code style. Especially for Java code because we use Jallopee to validate code style and it runs on JVM and the architecture of our code validator meant we had to spawn one JVM for every changed file. So if someone pushed 200 changed files we would spawn up 200 JVMs and other people were pushing at same time and it was constantly freezing and slowing down our server. We were able to mitigate this problem by using NailGant that is a kind of server that keeps JVM alive and you can run JVM programs there and we could run Jallopee there but we had to do some work around to run Jallopee in parallel and we still had some performance issues for some reason there were more pushes than usual. The last problem of this is that the system was inflexible. Sometimes people have very good reasons to ignore rules. Sometimes they pushed something should not that they didn't want to push and they want to do a force push and we didn't allow that. Sometimes they sometimes some push was accepted by mistake and there was a bag and it has to be removed because it should not be there. Sometimes there's a bag and we rejected something that we should have accepted and because this was on our server and people didn't have access to the server they had no way to go around this. The other problem the other way we were inflexible is that because it's a remote key took and it's living on the server we only supported our internal Git server. For example you could not use our Git hook for our projects on github.com and so our open source projects were completely unsupported. In the end I could summarize all these problems as we tried to centralize Git. While Git is distributed we still saw our source control management system as centralized because we came from subversion and we were still thinking like subversion but it was clear that we would have to rethink our approach. This process of thinking about the Git hooks coincided with the usage of Zalando as part of something we call radical agility. Zalando was getting rid of all the hierarchies and giving teams more autonomy and one of the new motors was autonomy instead of control and this presented a clear path for us and in the end was the solution that we are still implementing. So we decided to move away from remote Git hooks to a set of local Git hooks One advantage of this is that Git hooks see one commit at a time so we avoided the issues where we had a set, a list of commits to check and we didn't know what came from a merge or a rebase and what was created in the branch and it's better adjusted to the distributed nature of Git because they are distributed by themselves and people run it on their local machines. We decided that they should be optional because we don't trust people. This also means that Git hooks are no longer responsible for enforcing the rules instead we are creating mechanisms to ensure that our development follows the rules before the code goes live but we don't block anyone from pushing anything right now. We did this because we believe Git hooks should be seen as a tool to help autonomous developers and not as a barrier that makes their work harder. We decided to make them extensible because as I said before one size doesn't fit all different teams have different needs and sometimes we only have a small group of developers working on this more or less full time but sometimes people have other ideas and they can implement new stuff for themselves sometimes they don't want one of our checks and they don't want to install the dependencies of our checks so by being extensible they can opt out of those dependencies and we decided to make it open source because by only being an open source it could fully support autonomous teams and this also allowed us to support our teams that want to open source their code and this also includes my team. So our new set of hooks that we call turnstile are available on GitHub and by PI so everyone is free to use them. So if you want to install it on your machine you can just use pip install turnstile core this will just install core without any of our extensions we also have some extensions on Zalendo's GitHub account so and now you want to use it on your repository. What do you have to use? Well the first step is add a turnstile eml file in the root of your repository if you use something like Travis CI or something similar the same process you just added a file to your repository and it looks something like this you have a list of checks you want to use so you just name them and you can give options to all checks that they support by themselves and every check even if you are implementing your own checks as an extension you can read this file and you have access to it so and because the way local hit hooks work you need to sim link to hooks on your .kit folder inside the repository and to make this easy turnstile supports a self-comment that's turnstile installed that will automatically sim link to hooks for you and because I was already adding support for self-comments I added several more we have to config sub-comments that right now the only thing it does is set allow you to set verbosity so if you want only to see errors you can only see the errors if you want to see everything that turnstile is doing behind scenes you can also do it we have to install command to add hooks to a specific repository we have to remove if you decide if you don't want hooks anymore for that repository the specification sub-comment is used for check if all your projects contain a valid specification of all your commits contain a valid specification and right now this means if they start with a new URL but I'm planning to support more stuff in the near future the upgrades of command will check by PI to see if turnstile and all your extensions are updated and if they are not it will offer to update everything for you and the version just brings the version and it's not that interesting so now if you want to create extensions to turnstile how can you do it the way you do it is with set of tools and entry points so if you want to add a command you just use the entry point turnstile.comments you give the name of your sub-comment and this will be used to call it from the command line and you just provide the model where your sub-comment is and you can see more on documentation later it's same thing for checks so if you want to check something with the pre-commit hook you also give it a name that will be used on the configuration and you provide the model that's going to be used so you have three entry points right now one for commands one for commit message hook the checks commit message and the other for pre-commit hook that we use for checking code style for example so what did we learn from this process first don't and this is the most important don't get stuck in the past it's very easy to make mistakes because it was always done like this and what was the best solution yesterday can be the wrong solution today and this will be mostly likely be true if you change technologies because if you change technologies you will have different limitations and different opportunities and you should rethink what you're doing and adapt to it one other thing we learned was to develop in the open because by being more transparent about what we wanted where we were going we avoided assuming too much and we avoided backing ourselves into a corner and we got early feedback that also avoided lots of issues we also learned that we should build tools not barriers software is meant to help people and to be more productive if your software is making people lose time instead of saving time you should rethink it by rethinking our GITUX we were able to move from a position where sometimes our software was entering the productivity of our engineering teams to a position where they are useful tools to help them work autonomously so any question there is one question here hi thank you for the talk my question is about the cost and the effort of any change because you were talking about your company changing I cannot hear you sorry, no? yes, thank you I was asking about the effort of changing in each step because sometimes leaving subversion may be a good thing but changing for centralized hooks to local hooks and everything out the effort yes, it's an art process and we are still working on it sometimes and actually we are doing things where we are disabling shacks on the remote hook in phases and telling people now start using this and we are trying to make it compatible so the only thing that's not working in the same way on the remote GITUX and local GITUX is specification because before we only allowed people to use gyro tickets and now we customers are autonomous we want to allow them to use whatever solution they want so now we support more stuff fortunately it's a land where we are already used to always question everything and it's one of our philosophy if you are doing the same thing all the time you should question yourself and see if there's no better way to do it and it's part of our corporate culture and that doesn't mean that people don't complain that we are changing again that we have to manage it but at least in relationship to this most feedback I have is that people are happy to be able to control when they use the hooks, how they use it and they are very happy that they don't have to rely on me and my team to change stuff for them you can provide any samples of the kind of rules you are enforcing with that? yeah so for example this is actually from one of our open source repositories I don't remember exactly what this is a very minimum set we have more than that but for example we have the specification check what it does right now is checking if your commit message starts with a new URL and you have the option allowed schemes right now and I'm saying that I only accept HTTPS URLs because on this case all specifications are done using github issues in the near future I want also to support github references so you can just put the references that are supported by github and also gyro ticket IDs because we still use it at Salando and we wanted to make it easy for people to use it we have the branch release check that in this case uses that regex expression on the bottom to check if a branch that starts with release slash something the second part has too much that one in this case it has to start with a V and follow some rules that's format we use at Salando but you can use whatever you want and to protect master branch in this case forbids you to commit directly to master because I want to enforce pull requests Any other questions? Just one more thing if you want to find more information about Salando we have a tech blog and tech.salando.com we have our github page github.com.slashsalando we have lots of open source projects there and we are going to have much more in the future we also have Twitter, Instagram account and jobs page and one of my colleagues will give a presentation in the recruiting session later today