 Good afternoon, everyone. Welcome to Branch by Abstraction. We are absolutely delighted to have Paul Hammond joining us today. So thanks a lot, Paul, for doing this for us. And just for people who don't know, Paul is also one of the co-creators of the Selenium project. So we're absolutely thrilled to have him. So without further ado, over to you, Paul. OK, thanks, JD. Everyone can see my first slide, and me in little form up in the top right-hand corner, hopefully. Cool. So Branch by Abstraction is a technique. This hails from 2006 or so, 2007. And materials made it into the continuous delivery book. And what we think is that we should be doing this if we're in larger Agile teams and they're trying to do something that's bigger and longer to achieve. And then as a warning or a disclaimer for this presentation, there's lots of diagrams, but there's no code shown. And we do allude to code, but don't show it. And obviously this is a source control-related practice. So let me get into this then. Oops. Click Advanced. So I have presented at Agile India before, 2006. I spoke on a topic about test-driven development, refactoring dependence injection, and then Agile's big answer to big upfront architecture, which was dependence injection. And that was a joyful experience in Bangalore. I think I flew from Salt Lake City in the States all the way to there in one go. And then two days later, I'm flying back because the project I was on was under a lot of pressure. Anyway, I also ran a Selenium workshop. So that was more unstructured. And there was people, workstations. It was slightly off-site the next day. And we had lots of fun trying to teach and Selenium and work through problems. And then maybe at that stage in 2006, acknowledge how much of Selenium had yet to be written. Anyway, I've been doing trunk-based development for a long time, advocating for it. As I said, the materials made it into the continuous delivery book. Artworks from a project was somewhat anonymized and then put into that book. I've got my own book out now, 2020, probably 10 years too late, when I should have done it. Trunk-based development and branch abstraction, this topic. So there's a short link to the book. It's on Lean Pub. And I'm still updating the odd page in it, but it's mostly complete. So here we go. If we imagine a layer cake, that is how we develop software in modern agile teams. Paul, sorry to interrupt you. Sorry for one second. There's a small toast on the conference website that says that you're sharing your screen. So there's a hide button. So if you click that, that toast message will disappear. Can you see that one? It's on the conference website. Yeah, that one. Okay, thanks a lot. Sorry for the interrupt. Things your own eyes can't see. Okay. So we have a layer cake view or representation of what the largest set of modern agile and lean practices are in the context of DevOps, in the context of safe work spaces, everything that agile signatories wrote about is still true. But what we think now is, you know, if we have solid development infrastructure, that's our own workstations and the cloudy infrastructure we might be running the likes of Jenkins in. We have source control. We do trunk-based development on top of that. And we acknowledge that the sum overlap, we do continuous integration. Of course, continuous integration is largely talked about. But quite often mal-implemented, meaning that's not continuous integration when a company is claiming to do it. They're not quite, but nonetheless, people have an understanding they should be doing it. Layered on top of continuous integration is continuous delivery. And layered on top of that is lean experiments that are really facilitated. So if you've got all of these things achieved right at the top, you can do cost-effective, quick lean experiments. And whether that experiment was successful or not, you can judge it to be a success and plan the next experiment. Now, if you've got a bad branching model, let's say not trunk-based development, you could say that your continuous integration is probably not really continuous integration and that continuous delivery could easily be a myth or a thing you can't quite attain. And lean experiments is probably impossible. So I'm trying to represent a precarious cake in a layer cake model on the right-hand side there. Now, because we need to get through this to get to branch abstraction, I'm going to enumerate some bad branching models. So this one here, we have an idea that as a large team, 20, 30, 40, 50, 60 people in a dev team, we might be doing a release one branch and then there could be a team that follows us for release 1.1 that takes daily mergers from us. So this is a real branching model. I'm not so often used, but these daily mergers in a cascading style can prove to be problematic, which I represent on the right. And I'm using green here to indicate the automatic build, you could call that the CI build, is passing. That means compile, unit test, integration test, functional test, slain, and friends or all sorts. And the red means it's not compiling or the build's not passing. And then you have an implication, if that's not working at a moment in time in the release one branch, how could you possibly merge that to the release one branch, where if you do, you're merging known broken codes. So this way of working for a large team probably has days where code isn't merged because it wasn't passing the CI build in the preceding branch in the cascade. I actually encountered this, come to that in a case study later. Another bad branching model. This one's more like the old clear case style, a mainline where people are not active. People are active on release branches. The intention is to merge that back to mainline. But let's say if we had 40 people, devs and QA automators active in the release one branch, before we've gone live with release 1.1, sorry, 1.0, we might find that somebody said we have to start the release 1.1 branch because of the lead time for all the development features we have to make for that. So what will happen is we'll start whilst release one is part complete, keep working and then after release one goes out, we'll merge code into the release 1.1 branch and then that repeats release 1.2. So in theory, the thing looks like it would on the left in a reasonable case, but in reality smaller on the right, we see there's large periods of time where say Jenkins or equivalent is not passing for the branch in question. And then you have to wonder why you're merging code from that branch if it's not passing the build. And then you're in this nightmare of broken code moving around all the place and your release branch in question is only green the day before you go live or maybe the hour before you go live and you have unplanned releases following planned releases and it's a nightmare. And I've encountered that particular model many times too in larger enterprises particularly. So this one's correlated with maybe quarterly releases, quarterly planned releases. So release one, release 1.1 and release 1.2 are the planned releases and the 0.01 releases following those are unplanned. So last one on branching models before we move to a branch of abstraction, I'm going to say in plenty of my materials that GitFlow is a bad branching model. I will understand that this year the author of GitFlow noted that there are other choices too. I'm also going to say that GitHub flow, 2008 contribution to our field from GitHub people is very close to trunk-based development. So we might say that trunk-based development has a few styles. We could say we could do this director trunk if you're in subversion, that's your only choice really. That's maybe team size 1 to 100 and 100 being like the max. But reasonably our dev teams these days are like 1 to 10 or 1 to 20. There's a second mode of operation, GitHub flow, the short-lived feature branch model and that could maybe scale up to a thousand people. That's where you have branch years and you bring those in very quickly, short amounts of time. And code review happens on that pull request branch before it gets consumed into the master, the trunk or whatever you want to call it these days. Now there's a third model, the one Google do is with a patch queue. This doesn't involve a real branch in the source control system. It's marshaled the patch. It will be replayed and merged on the head of trunk but only after code review. This one Google's managed to scale up in the many tens of thousands. Other companies are trying to copy this model if they can but that's not every company. Really it's the big companies, the one high throughput and everyone moving as a single unit on a larger plan but with relief independence for the facets, the modules, the components, the services, the applications within that larger trunk. So that would in the end be a big monorepo. Okay, branch abstraction, the problem it solves specifically. Branch diagram, time going left to right as the previous ones. We have a representation of the trunk or the main line on the bottom. We have an imagination that there's some releases 1, 1.1, 1.2, 1.3 and 1.4 and we have a thing that we're trying to achieve. So this could be non-functional, like a change in persistence technology or it could be functional like delivery of multiple parts or rectangles of a thing that have stories mapped to them that the business would understand and pay for but we know it's going to take ages to finish. In fact, we know it's going to take more than one release to finish. So what we're going to do in the old way is make a branch for that and then merge it back when it's finished and if we've got our planning rate and all of that functionality was supposed to go out for release 1.4, we merge it back when it's completed after 1.3 goes live and before 1.4 goes live. So that's our big thing to change. Our fear, and this is maybe us as devs, practitioners but also us as management, the people that see the budget that look at the larger schedule, our fear is that there could be problems in the schedule as we go and there are unplanned releases so these are the ones with dotted lines and the little 0.1 suffixes and we also imagine that there's a possibility that the team or person who's doing the big to change thing might miss their date on their completion and it would merge back after 1.4 goes out. So maybe 1.4 has to go out on that date because marketing is driving that or the business or competitors or a number of other probatives. Maybe our fear is commenting out code, merge, unmerge and a whole bunch of things that really impact 1.4 versus just slip 1.4 which is what a reasonable team would do if they had the choice. So our nightmare, the business would cancel this big to change thing. It might be invisible to them as they're changing over underpinning technology. So the business observed that there was releases going out and then all of a sudden there's not releases going out or there's a problematic thing but they do remember that releases were going out before and they were going out just well. So now they say, okay, don't do that big thing. Just carry, go back to work because they're in their mind because they saw releases as work and they've canceled this effort on you to pay masters. And the problem for you is that you probably figured you needed to do it or that you bargained that you were close and you just needed another period of time but for many reasons the business could have lost patience. So we could also, as I mentioned just now, we could be in this peril that we say we have to unmerge code or comment it out in order to get the 1.4 release out because something that was planned to be merged integrated with it and it went out with 1.4, wasn't ready and we had to do all of this work to make sure that something had to go out on the dates that 1.4 was supposed to go out. So last minute re-planning is the hallmark of waterfall. You know, agile is able to predict if we're on schedule, maybe have plans and steps and processes and practices to mitigate the fact that we're not on schedule. We can still get something out but this merge back of the big branch is like a waterfall technique and we could have had 50 people on that branch and 30 working on regular functional deliverables and the cost of unmerging a whole bunch of codes or commenting stuff out would be huge especially if that wasn't a feature it was a whole release branch. So long live branches ultimately become our way of working we showed one of those broken branch models earlier and we never get back to a high flow continuous delivery centric way of doing things. So there's that branching model that broken branching model so you had a plan to just make a branch just the once and then merge it back for the big thing but you end up in this reality where release teams are now marshalled on branches working there, nobody's working on the mainline and code only comes back to the mainline in batches when it's complete and it's coming back with a merge tool merge technique within the source control package which in itself is never defect free merge conflicts, three way merges and all sorts of things like that huge pressure on the merge team to get it done because other teams need to merge immediately after they've merged the mainline or they don't they try and leapfrog over from say the release one branch to the release 1.1 branch new problems so branched by abstraction what we do instead so this time I'm just going to represent that trunk left to right again, it's time we imagine we have release 1.1, 1.2, 1.3, 1.4 we imagine these green dots are commits on the trunk stylized these green dots are passes of builds, right so these are Jenkins woke up CircleCI, Travis whatever ran through compile, unit tests integration tests which multifaceted but in my book would include selenium or equivalent and everything passed I used to get to clients quite often and they'd ask me what percentage of red vs green do we accept within a well functioning dev team and I was always confused as to what they meant by that but they were wanting to hear 80% or something like that because that was their reality as a team that every build that runs 80% of the test pass and 20% fail so the correct answer is all the test pass and none of them fail this red dot here hopefully colourblind people can forgive me the red dot here is no handshakes no waving hands, no colourblind people this red dot here is a failing build in a high functioning team we're going to see the build automation technology itself Jenkins going to roll that back or if that was on the trunk if it's on a pull request branch we're not going to roll it back we're going to give maybe the dev in question a chance to maybe fix it so that it goes green but when it's merging back to the trunk as the pull request is consumed it's represented as green here not the interstitial red state that happened whilst that was a work in progress we could argue whether those are bad habits that we should ever be using Jenkins to find out whether we did or didn't break the build we should know in my opinion and many others we should know before we do the commit and the push so that red dot it's anomalous here but I'm just including it because we need to understand that people and timing accidents will see breakages of the build on the trunk but we're going to say it gets rolled back so in the next representation we're imagining the team mates that are active in this particular trunk there's five devs at the bottom and when I say dev I also mean QA Automator and they're busy with business as usual and they're actually doing the lime green commits the dark green ones hopefully you can see the difference in the two shades of green the dark green ones are the activities over time spanning several releases of the developer or developers who is doing the big to change thing so that could be non-functional like a change in persistence technology or an upgrade of spring or something like that that could take a while actually that's a bad example something that can be done on a piecemeal basis that takes a lot of things to achieve so we've got two representations of colour here so we can set the scene for the next slide which is what happens at the beginning of a branch by abstraction exercise from the point of view of the developer who is doing the work towards this big to this long to change thing so we're imagining that they have a rule that they constrain themselves to and that is that they can't commit anything to the trunk that would break the build for them or for anyone else so that's eternally true but their first commit for the thing we want to change is to introduce an abstraction so depending on our language let's say Java has inheritance many other languages do or they have different paradigms like functional programming but we might say whether it's inheritance or compositional or functional we will make an abstraction for the thing that should be changed from old to new and as soon as we've made that abstraction we're going to split something that was not abstracted into an implementation of that abstraction and then for short time we'll just call that the branch by abstraction change and we're going to have a mechanism in the code base that is allowing a toggle between the new states for that abstraction and the old state but this toggle is not going to do a lot because all we did was split this component into two like the abstraction and then the implementation of the abstraction and we're going to make sure before we do a commit that this passes for me and it passes for everyone else as represented by Jenkins, CircleCI and the like so from everyone else's point of view all the other developers within the team they don't see this particular change in a way that is benign to them they only focus on their functional deliverables which are the lime green pieces that are going into the trunk they're still chasing the Release 1.1 branch that's still a week away or something like that but the code that the other team mate had delivered inside the trunk is inactive from their point of view there was a famous article from 10 years back or more that talked about dark deployments I can't remember, I think it was Business Insider had an article that said something like the next version of Facebook is something you're already using you just can't see it we could call these toggles or flags or we could have any other mechanism that allows us to turn off the feature for the rest of the team because they're focused on functional deliverables and it's only active for the dev in question but it's all inside one trunk no branch is involved yet so let's roll that forwards and say partway through this exercise we're much further on the dev is working the branch for abstraction work is continuing in the same vein the same prerogative every commit they do could go live functionally anything is safe whatever way of phrasing the same constraint is it's true the developer never deviates from that but they're making more commits as they go so that could be more abstractions more implementations of abstractions that could be a refactoring of a you took a piece and you just split it into two using the refactoring tools and hopefully a JetBrains product that could be introduce interface in IntelliJ or one of five or six different refactoring menu options that amounts to the same thing they're making perhaps now second implementations of the same abstraction so let's say this was an attempt to get from a persistence technology that's now passe to a new persistence technology that the company wants to bet on we could see an abstraction for the entity in question with an old say hibernates way of implementing that components and a new say mybatis way of implementing the component so they're going to make the second implementation driven by that flag or toggle or whatever some other mechanism and all of this is within the code base that could be XML or JSON or YAML or some other technology that signals whether for this particular component we use the old or the new and from everyone else's point of view it's using the old from this developer's point of view it's using the new they could do a burn down chart based on this which allows them to do some predicting around how long it's going to take but anyway let's say release 1.1 went out and they're facing release 1.2 here they're still not complete on the larger exercise but the rest of the developers are only focused on the functional variables for 1.2 as a represent here so they do maybe see the prior work but the new stuff that's going in they're less concerned with now we could say that just keeps going keeps iterating we don't know how long it's going to take which is one of the hallmarks of an agile projects let's focus on the top right hand corner hang on I'm going to move my introduce more or refine abstractions work on implementations of those abstractions refactor hopefully in a JetBrains tool don't break the builds make sure the code remains inactive for everyone else so that's the major coding activity now each time they do a new abstraction a new implementation abstraction sorry a new separation of a component into implementation and abstraction and they're making the second implementation of the same thing in the new technology then they have an opportunity for new unit tests and they should focus on this it's probably OK to clone to copy paste the previous tests into the new one it'll become clear why that's good but for production code we would never say cutting paste is a good way of doing stuff but for test logic in a procedure like this it's OK if that's quickest way of doing it anyway so let's say they have a unit test for the old way still and they have a new unit test for the new way of doing the same thing so a new unit test not integration test you know this larger exercise should be done on a code base with tests then you know as many releases are actually happening on this timeline they're actually going live as part of those releases but with everything they're doing turned off because the toggle or the flank or whatever it is is representing the old way for the longest time until stuff is complete so completion that mythical 1.4 release when we get to that place and we're 100% happy with what we're doing we might flip the toggle for everyone so that dev5 I don't know if you can see that was dev5 dev5 has now joined the larger team again is no longer involved in the exercise which branch abstraction was doing the code is still in the code base at least initially but we've toggled it on for everyone so if we are flipping persistence technologies which is non-functional we'll now have done that for everyone the old code is still there the unit test is still passing but in production at least 1.4 that code is not active anymore it is now the dark code the code that's working is the new way of working which has been tested in every dimension unit tests, integration tests functional tests and even maybe a QA team has stood up environments and is clicking through the application making sure that it functions as it did before now if we live for a day a week or some period of time that's deemed to be safe we're going to remove the old implementation say the hibernate implementation we're going to remove the abstraction that underpin that meaning the implementation of each abstraction no longer extends or implements an interface or some other idiom and we're going to remove the toggle the govern this so we can't flip the old one back on again code is gone, toggle is gone and now our code base exists without reference to the prior technology and only source control history shows that for the same trunk now of course I think it would be best to do that after a delay because you never know when you don't change your mind and go back to the other way of working so other economic benefits the branch of abstraction process is pausable and resumable so remember we said earlier that a business group paymasters could decide to cancel our work because it's taking too long and they don't believe it can ever finish this one let's say if your competitor does something really super important and you put your best developers on the branch of abstraction effort and you want to now beat your competitor or react to your competitor's deliverables you could move those developers back to the main team get them work on functional deliverables you've pivoted and rapidly prioritized new business deliverables and dropped other in-progress work and we could go and do that and we could ship a release and then when we're breathing a sigh of relief after that effort has gone to production we could move the same developers back to the branch by abstraction effort within the same trunk and just pick up where we left off none of the code had gone stale on a branch which is one of the other hallmarks of cancelling those efforts where they were marshalled on a branch another economic benefit the business never observes no releases the releases are going out maybe on a cadence one a month, one a week one a day, ten a day it doesn't matter this longer to achieve branch by abstraction work is just working in tandem to everything else that's happening there you know if this is just a way of working the business people have only ever experienced this when they got to a company or a future employer that was doing it the other way on a branch the business people with no tech skills, no dev skills would be screaming screaming about how bad that way of working is versus their old company you know of course all of us have old company artists we remember a project which recounts our favourite and we pine for it in the years that follow and York can decide to cancel the branch abstraction effort completely without any costly unmerge or rework so let's say you know if it was a functional change rather than non-functional and we were doing it over a period of time but halfway through the effort we decided that the old way was best because we've done AB testing or competitors have all moved to the way that we think we shouldn't be doing anymore now we want to go back to it it's very easy to just shorten to the end which is removal of the abstraction removal of the toggle and removal of one of the two implementations or what was part complete as one of the two implementations so at any time we can say this effort is finished and rather than pause it because we know we're going to come back or we wish to come back we can actually remove it and that's a day's work for one of the developers to remove all of the code that's going to be turned off forever maybe it's not original intention but we're allowed this optionality born from this technique now lastly the organization can casually reprioritize this particular instance we talked about releases 1.1 to 1.3, 1.4 imagine we wanted all the feature set of 1.4 to go out before 1.3 and if we if we had a larger sophisticated design of having dark code toggles and branch by abstraction to do the longer to achieve things it's now just an issue of flipping toggles running some experiments to see how viable 1.4 releases without the 1.3 features because that's effectively what you've done fixing some bugs and then going live so that's you know hedging as a strategy is built into branch by abstraction as a larger architectural design and the toggles that come with that so I have two case studies first one's a bank it was a bank in the USA foreign exchange FX trading is where you sell dollars to buy yen for rupees whatever and vice versa and you can just trade straight or you can have futures and options and several derivative products based on foreign exchange trading and sophisticated stacks to do all of those and have large corporate and institutional account holders within that system you know including manufacturing companies they'll potentially make their product in China sell it in the US and sell it in the UK for example but they might not move the money from UK sales back to the US to then move it again to China to pay for the manufacturing they might just move the money from the UK to China and if they're doing that they're not using high street banks for foreign exchange trading they're using trading systems like this to do that anyway the banking question had three planned releases a year they had two more unplanned releases quickly following each planned release I've got a blog entry on a cynical view of bonuses based on what your release cadence is and how much release drama there is in release week and release weekend and how many unplanned releases you do following a planned release but that's more of an entertainment blog entry than anything that's super relevant so between two releases we decided to re-architect what we were doing in the code base the build system source control technologies branching model the whole shebang so we did seven things in parallel two of which were methodical branch by abstraction efforts both of them non-functional so one was to change the business messages on the wire so that would be a buy or a sell or something or other of a particular currency orders that haven't been consumed trades that have been consumed blah blah blah we wanted to change them from binary on the wire to XML and the second change we wanted to make and these weren't my decisions these are what we inherited was to change the message technology so let's say active MQ is a message technology so is Kafka and you could have gone from one to the other because somebody said it was better and maybe one is better than the other you wanted to change message bus technologies and you wanted to change the message format on the wire so both of these efforts and five other parallel things were achieved between releases these two pieces were done with branch by abstraction in parallel in the same team so the same team was like seven people and one at times or two at times people would focus on one or more aspects of this branch by abstraction plan towards a goal and as it happens they were complete inside of one release but they could have taken longer and left it toggled off all in all, all seven pieces were finished early as were all the functional deliverables for the release and the team went and pushed something out two weeks early in a three month schedule or four month schedule that's not amazingly early but it's early enough for the business to notice so they went live ahead of schedule and the business applauded the team now in terms of doing seven things in parallel probably that was foolish you know I was the guy that architected the seven things and nobody told me we shouldn't do it but if I think back somebody sure as hell should have done we were lucky next case study, airlines from years later USA again it's a major airline it's not Pan Am Pan Am doesn't exist anymore but hey if I put Pan Am's picture up then nobody can think I'm talking about a specific airline they had a C++ stack and they were moving to Java it was a rewrite they didn't have maybe a huge amount of time for their old technology even though it had done them really well within their industry and they wanted to strangle the old application in a series of releases to the new Java solution I mean if you're doing it today you might do this in Node.js but they wanted to do it without having a major rewrite period and they were doing a cascade model where the release two team was merging into the release three teams branch or the release three team was merging release two teams changes every day or every week and the release four team was trying to merge from release three and the release five team was trying to merge and it was so problematic it was slowing down all development so we presented on how to move to trunk based development its relevance explained Google were doing it and others and we said we should wrap everything up in toggles and have abstractions for all of these very rectangular pieces of the user interface there could be A or B with or without and because all of these teams were working in parallel concurrent development of consecutive releases it meant we had many toggles and many abstractions so some time later after successfully rolling out a number of releases on this basis something happened that was unplanned which was to do with an acquisition and some launch date for a bigger thing and it was pretty bad in that one of the other partners outside the organization they were going to integrate with wouldn't be ready and because that was a waterfall team that was notified late so my colleagues in question were asked are we going to be ready I've moved on by this stage are we going to be ready and my successor within the client flipped some toggles spun up a new Jenkins pipeline and said actually bar for one bug which we've already fixed we are ready and that notification was a matter of hours later after the execs within the IT department have said what is the cost of this particular re-sequencing of releases they had a matter of unmerged fear or commenting out fear but my successor had got it over the line with colleagues and went back to the same exec team and said we're good to go so at that stage I think the execs within the airline moved from fearful doubting concerns overseeing they moved to like enlightened convinced converts to trunk based developments in branch by abstraction and toggles so saving them a fortune anyway so in summary we might say branch by abstraction is instead of branch by source control and it's only for this specific instance like the long to achieve thing I'll give you a single tip if you're going to do it within your team focus on the most dependent on and least dependent component first if you had 400 components to work through which one first is going to be the most dependent on and least depending for the bank it was an object called currency everything else used that bizarrely later at a big search company I did the analysis of another hairball solution and found currency to be the same object in the middle of it that was the Jordan knot of entanglement but we didn't change that team so I run a info portal branch by abstraction com in there I re-describe everything we've just discovered in a different way to the way in which we've just discovered it so I do three tellings of the story one of them is a series of sequence diagrams if you're familiar with UML sequence diagrams that's in there there's another way of telling the same story this one's with codes and it's in a way too simple java rat but it gets it across what all the choreographed steps would be abstraction toggle removal of abstraction and then there's another one which is pictures only which is a stylized reimagination of the empire state building in New York in black and white sketched pictures and it's not my artwork I found somebody to make better art than I can possibly do and it's half possible as another way of telling the same story but at least the non coders okay so that was my last slide so we're at questions yeah paul I'm just trying to see if there are any questions guys if there are we've still got a few minutes so if you have any questions just add them to the chat so which branching strategies are ideal for the team in all cases trunk based development you know you could say github flow is close enough there's a small technicality on where you release from but for a reasonable size modern team github flow is what you should do it's a faithful reproduction of trunk based development which predates it okay this is one more by Carol says do you require a mature continuous deployment process to implement a branch by abstraction you don't know so if you were say that quarterly team that's where you're at now and had the big to achieve thing you could do whatever you needed to do with branch by abstraction and then get your deployment out on a quarterly basis now there's two CDs right there's continuous deployment which is the inter production by triggered by commit and then there's continuous delivery which is inter QA or into UAT triggered by commit and all the build steps passing so that quarterly team probably is not going to do continuous deployment but they might be doing continuous delivery meaning every successful build bounces the QA stack so that people could go in and check it and say oh look at that the new features been Vipple delivered the new feature or Paul do you want to take another one I'll take another one so ok this might be a big one but ok so is this basically a replacement for any standard source control branching how would this work for multiple component and client teams I'm going to say this is standard source control branching trunk based development is standard source control branching branch by abstraction is a technique that you will use in conjunction with trunk based developments for a specific problem that you're encountering like most functional deliverables are like a day to work on and to finish commit tests approve for code review and all that but if this is going to take you eight weeks to do and that you're doing one release a week you're probably going to borrow this branch by abstraction technique as part of your standard trunk based development branching model I hope that nailed it ok I think we're pretty much out of time so should we wind things up and guys if you want to interact with Paul directly you can find him in the lounge in a bit so yeah a lot of thumbs up coming from Paul yeah ok thanks a lot guys ok thank you everyone yeah thanks a lot Paul for doing this really appreciate it thank you everyone for watching this ok bye everyone