 Hello everybody, welcome to the next session. The topic is the review and it's still very presented by Florian House. Just one comment, if you have any questions during the session, please use the Q&A section or Florian's, or Florian's, I think. And the stage is yours. All right, thank you. Hi everyone. So I wanted to share a few thoughts on something that I consider a rather important topic in our industry, which is a source code management, code review and CICD tools, and how they all relate. And that means that I'm talking about source code management, that is where we store our code and how we manage access to it, code review, how we coordinate changes to our code, testing and gating, how we make sure that those changes, when we agree on them, don't break anything, and finally deployment, how we push changes and updates out to the consumers of our code. And consumers very broadly can mean customers, it can mean users, but of course it can also mean other machines. If we're maintaining an API service and the downstream consumers of our code will typically be another service. And in case it's not obvious, what I'm talking about, source code management, code review, testing, gating and deployment, is really a large fraction of the software engineering cycle. So it's not all of it, but very nearly all of it. So the part that involves fooling around, if you will, so creative play, having ideas, that sort of thing is perhaps excluded, but substantially everything where people can be said to be building something in the software engineering organization is encompassed in these four things. And there's a few things that follow from that. So firstly, whatever tools we use in order to accomplish these four things, source code management, code review, testing and gating and CICD, those tools simultaneously influence and are influenced by a collaboration culture. It's completely ludicrous to presume that tools and culture are independent of each other and it's equally ridiculous to categorically declare that tools must be made to fit processes and not the other way around. That's not how people use tools, that's not how people work. So culture and tools always have an influence on each other and we must acknowledge that and work with it. Secondly, the scope of these four things, again, source code management, code review, testing and gating and CICD, is continually expanding as the software engineering field evolves. So to illustrate, if you looked at a few years ago at a CICD platform, a CICD platform could get away with supporting automated unit tests and then perhaps kicking off an Ansible Playbook to deploy things to the ends. And that was it and you could call that thing a CICD platform. But today what we expect out of a continuous deployment pipeline is certainly much more and it would definitely include support for say, for example, a package registry. So for your Python packages or your Node.js modules, for example, a container image registry for your Docker, Podman, OCI containers, certainly a secret store, probably the ability to deploy to a Kubernetes cluster, and those are just a few examples and I might be forgetting others. And thirdly, this is a classic example of where we must apply systems thinking. So substantially everything that a software engineering organization does is connected to this tool chain. So therefore, we cannot make changes to one part of the system without considering the consequences for the system as a whole. And that's of course not to say that we cannot make incremental changes. We absolutely can and should. It's just to say that we can't pretend that anything in the system stands alone and is disconnected from everything else. And to illustrate what I mean, let me briefly use a quick analogy. Consider, say for example, an automotive engineer who implements a design change for an engine, right? And then that design chain, say, makes the engine so much more efficient that it means a range extension of 10%, right? Excellent, great job engineer. But if in that process the designer has or the engineer has made it impossible to connect the engine to its battery or to its fuel line if we're talking obsolescent technology, then installing that new engine into a vehicle not only won't improve anything, it will actually render the whole vehicle immobile, right? So we can't afford to do that. If we want to make changes to any part of the system, we have to consider the consequences for the system as a whole. So what does that mean for responsibility? Who's ultimately in charge of the system that consists of source code management and review tools and your CI CD pipeline? Well, I hope at this stage the answer to that is a no-brainer because I'm talking about substantially or including at least substantially everything that your organizational tool chain includes. Sorry, mixed up there. But the point is the responsibility for this tests with a rest with whoever is in charge of the engineering organization and most companies that would be the CTO. And if you're a software technology company, I'd even argue that it might rest with your CEO or your MD that responsibility because your entire enterprise is substantially software engineer organization. And of course that person may delegate some of the tasks and the details of running your source code management and code review and CI CD platform, but the responsibility certainly stays with them. And that responsibility requires both an understanding of the technology itself and an understanding of how it interacts with your engineering culture and that understanding needs to be profound. And I go so far as to say that if you head up a software engineer organization and you don't have a profound understanding of this tool chain and it's mutual influence on your culture, you should find another job. And if you work in a software engineering organization and the person in charge lacks precisely that profound understanding, then you should perhaps also find another job because you deserve better. So having said all that, we can start actually talking about tools. And I'm going to talk about three of them, all of which I happen to use in some professional capacity at least weekly. So the first one is a tool chain that I think a majority of open source developers will be most familiar with, which is GitHub, whose collaboration model is based on the pull request, which originally came out of the Linux kernel community. And the GitHub pull request model replicates some of the features of the kernels model. So the collaboration model with GitHub PRs is generally fork and pull. So that means individuals maintain their own forks of an upstream code base and then they send pull requests whenever their changes are ready to review. However, in contrast to the kernel, of course, the review process then uses a web interface and not a mailing list. Each pull request generally consists of multiple commits. Those are typically expected to closely relate and serve a common purpose and merges one. And that common purpose is enumerated in a summary at the top of the pull request and GitHub calls that the PR description. A PR submitter can also mark the PR as a draft. That's a relatively recent feature in GitHub. It became available in 2019. That we had sort of an emerging convention where we would use PR descriptions with prefixes such as whip for work in progress or DNM for do not merge. But as of 2019, we have proper PR drafts. Now, GitHub PRs can be approved or ejected or merely commented on by maintainers or other contributors. And such a formal approval can also be made a mandatory requirement for merging. But by default, GitHub will let anyone merge the PR that has right permissions to the repository that the PR targets. And that includes the possibility for a maintainer to just merge a contributor's remote branch to their own local checkout and then pushing the merge branch to the target repo of the PR. And that will, of course, automatically close the PR and mark it as merged. Now, GitHub has, for quite some time, allowed maintainers to also require that PRs pass some form of automated testing. However, until relatively recently, it relied on separate testing infrastructure outside of GitHub to do that. And people would typically use CircleCI or Travis Jenkins for that. And it was also in 2019 that GitHub announced automated testing via GitHub actions workflows. But at this time, those workflows are in widespread use for CICD testing in GitHub managed projects. But it's still quite common for projects on GitHub to also allow maintainers to circumvent CICD tests and merge directly. And when that happens, it often creates a rather unpleasant situation because CICD testing is only run for contributions by outsiders or newbies and maintainers actually get a break. And that means that actual issues are often not detected until a casual contributor sends a PR. And at that point, the test breaks and leaves people confused. What does that breakage have to do with my change? And sometimes it leads to the change not even being considered because, you know, the test broke. GitHub also has something that comes bundled with it, which is the ability to maintain your own package registry and then push artifacts to it from your workflow. And interestingly, GitHub's definition of packages does include container images, so it lumps it in with that. It also contains RubyGems and NPN modules and there's a few others. Interestingly, it doesn't include Python modules. So you don't get a PyPy worker like directly from GitHub. Although, of course, you can push to PyPy from your GitHub actions workflow. Now with that, I'll move on to GitLab. And in GitLab, we have something called a merge request and that's quite similar to a GitHub pull request. Indeed, you could say they're equivalent, but there are a few noticeable differences. So firstly, the fork and pull model that we typically find in GitHub is less prevalent on GitLab. It's far more common for collaborators to work on one project and then they create topic branches within that project for each set of changes. And the project is shared. So this facilitates collaboration and single change set by multiple people because what they can do is they can simply push additional squash or fix up commits to the topic branch. What they can also do is they can force push amended commits to the topic branch and in that case, the GitLab web interface will helpfully point out differences between individual versions of a commit. That is something that GitHub presently cannot do in a PR. It's a nice GitLab feature. As with the GitHub PR, a GitLab MR is generally expected to include one or more commits and it's also expected to contain a summary that outlines its purpose. And GitLab MRs also have a draft status just like GitHub PRs do. They were actually introduced about the same time in both products. But GitLab has this one really handy feature in that it will automatically mark a merge request as a draft once you push any squash or fix up commit to the topic branch because at that point, GitLab rightfully infers that the branch still needs a squash rebase prior to merging. And as in GitHub, GitLab MRs can be reviewed in full using the web interface alone. Now, CICD in GitLab has been an intrinsic part of the GitLab review experience for some time and GitLab does include full configuration for your CI pipeline via a single configuration file in the project called .gitlab-ci.yaml. And it feels a little more intrinsic to the review process than GitHub actions do because the latter to me still feel a little bit bolted on. But in GitLab CI, that's very tightly integrated and you also have multiple options of using the CI runner. You can use shared runners, GitLab operates those for you. Those are just Docker containers that GitLab spins up on your behalf in the cloud. You can also host your own runners that you can do with your own Docker containers with Kubernetes clusters with virtual machines, even on the bare metal. And those runners also need no incoming network connectivity. You don't need to open the firewall for them. They simply connect to a service on your GitLab host and then they poll whatever jobs wait for them and then they run them. And you can also specify runners that are exclusive to a project or to a group or subgroup of projects. As in GitHub, we also have package registry to which we can push packages from the CI pipeline. This does include a few more package formats, although some of them are currently listed as being alpha or beta in status. But what you do get is actually a private PyPy worker like for your Python packages. And in addition, there's also a separate container registry for container images. So it doesn't lump that in with packages as GitHub does, but it considers the image registry a separate part of the product. And with this, I'll move on to Garrett and Zool. Now it feels a little bit awkward perhaps to call this one Garrett Zool and I've called the others just GitHub and GitLab respectively and sort of tacitly included the corresponding CI integrations with them, but there's a couple of reasons for that. So firstly, Zool is a CI CD framework that's in principle not tied to the Garrett, whereas of course GitHub actions, those only apply to GitHub and GitLab CI only applies to GitLab. Now, and also likewise, our Garrett is not tied to a specific CI CD framework. So it's perfectly feasible to run code reviews in Garrett and then use a completely different CI CD pipeline or even none at all. And the Garrett Zool combination does differ quite notably from GitHub and GitLab and I'd like to highlight some of those differences because the workflow has interesting properties. So in Garrett, we talk about a review. We don't talk about a pull request or a merge request. It's just a review. And in Garrett, you don't fork and pull and you don't use shared repositories. You don't ask anyone to pull from a branch or a fork of yours. Instead what you do in your own checkout is you run Git review and then Garrett will make a branch for you on its own and everything else then flows from there. A big difference is that unlike a GitHub PR and a GitLab MR, both of those typically contain a series of commits to land as a whole. A Garrett change is really just that. It's one change. It's a single commit and that of course also means that we don't need a separate summary for the change because the summary is of course the commit message. It is totally possible to submit a series of commits in the course of a Garrett review but what Garrett will then do is it will simply take those as individual changes, a series of changes that all depend on one another. And those are implicit dependencies and we can also express dependencies between changes explicitly by including appropriate keywords in the commit messages such as depends on and crucially these dependencies can cross project boundaries. So that is to say we can have a change in one Git repo that depends on a change in another Git repo and they must pass both and when they both pass then they both get merged and the only thing that's required is that they both use the same Garrett instance for review but it can be completely separate Git repos and we also have the equivalent of a draft PRMR in Garrett we call that a work in progress change. Now there's something really important which is that when we're using it in combination with a CI such as Zool a Garrett review project generally expects that CI tests pass on every commit without exceptions and this is in stark contrast to many GitHub or GitLab managed projects which typically only expect the head commit of the topic branch that's associated with the PRMR to pass CI and with Garrett Zool every commit must pass CI. Another interesting thing is that in Garrett Zool managed projects it's actually Zool that merges the commit so this is in contrast to projects that live in GitHub or GitLab in those pipeline run results are generally considered advisory nature and a successful pipeline run must still be confirmed by a human who clicks a merge button and in addition even a failing CI run can generally be overridden by a core committer who has the ability to merge the PRMR anyway and in Garrett Zool we typically don't have any such shortcuts so that means the only way to get changes into the repo is to pass peer review and pass the CI pipeline and in my experience and here's where we're talking about culture again this tends to create a climate of leadership by example which has a very beneficial effect on both experienced developers so seniors in a corporate setting and newcomers or juniors the Garrett Zool combination has another very interesting property which is called speculative merging and this kind of really sets it apart from other review and CI tool chains speculative merging in Zool could be a talk on its own I'm not going to get into it in much detail I'm just going to mention that what it involves is the parallel execution of CI jobs for interdependent changes and that means that with speculative merging what we get is that even complex and long-running CI CD pipelines don't hold up the development process because they're heavily paralyzed and they don't need to be serialized and this massively enhances the project's scalability there are also admittedly some downsides to the Garrett and Zool stack so for example in Garrett Zool there is no really good close integration with repository browsing so Garrett does include a plugin for that purpose it's called Githyles its user experience is rudimentary and that's perhaps putting it charitably there's a popular alternative which is to deploy Garrett with Githy but again that's not built in and your trusted Garrett Zool admin has to set it up for you and there's another snag here which is that source code browsing in Github and GitLab is very tightly integrated with your project permissions and that's also true for Githyles but that has other issues but if you want to do this with Githy there is a certain amount of administrative duplication that your Garrett Zool admin has to undergo to make the Garrett repo and project permissions also apply when browsing with Githy and Zool has no built-in package registry so Zool absolutely has ready to use jobs for pushing to a container registry or to PyPy or to NPM or whatever but you'll have to either push to upstream public registries or build your own so Zool doesn't come bundled with multi-tenant private package or image registries the way Github and GitLab do and on perhaps a more general note there is another thing that you might want to consider which in my humble opinion is an important reason why the whole Garrett Zool combination has a bit less uptake than it would otherwise deserve on its technical merits alone I'm not trying to sound overly dramatic here but people like to be in charge of their own destiny or of their own actions and software developers happen to be people and here's a bit of an issue with Zool there's quite a few things that a developer can do on their own without a support ticket without bothering an admin with Github and GitLab that they do need to ask an admin's help for in Zool so consider the following example something that I would guess is a relatively standard workflow you build a Python package you push that to a registry you roll that into a Docker apartment image you push that to a registry and then you roll it out to a Kubernetes cluster that's something that in Github or GitLab you can do as a simple project owner and with Zool you'll need an admin to at least set up and manage the registries for you and likewise another example rerunning a pipeline that's a simple click of a button or an API call in Github and GitLab and in Zool that's something you trigger via a Garrett keyword so typically a recheck but only on the pipelines where your admin has to find that trigger and if you want it to run on other pipelines as well again you'll have to bother your admin for setting that up so you came to this part of the talk where perhaps you're hoping for an answer on which of these three platforms that I mentioned you should choose for your organization or advocate for within your organization and there's no easy answer for that it very greatly depends on your priorities that is something that you will have to decide I can just give you four angles on which to base your perspective for your decision or for your advocacy so when we talk about scalability so the ability to adapt to massive organizational sizes or rapid project growth or just an absurdly large number of projects within an organization then the Garrett Zool combination will probably win hands down if you have a competent and responsive and dedicated crew to manage it when it's about getting off the ground quickly so getting started with a good usable and easily manageable review and fully integrated CI CD infrastructure you probably won't be able to be GitLab when it's about beneficial effect on your development culture again the Garrett Zool combination probably scores best because if you have a team that's great at reviews and commits and CI and doesn't cut corners or you want to build a team like that then Garrett Zool can probably really really help and finally if we're talking about the lowest barrier to entry so giving developers tools to use that they're most likely already familiar with then GitHub is very probably your platform of choice and with this I close my slides for this talk are here they're CC by SA you can use them at any time thank you for your time and I'll now take a quick look at the questions list alright and it seems that we have no questions at this time which is perfectly fine as you can see my Twitter handle is on all the slides it's on this final slide here as well so by all means please feel free to reach out to me on Twitter and I'll be happy to field your questions there thanks for your time and I hope you have a lovely rest of the conference thank you very much for your presentation Claudia it was really great thank you and hopefully see you next time cheers bye bye