 Is this better, if I turn it on? Okay, so what I probably do is I start with an introduction that we usually do in workshops at my company that we run subversion workshops for people. And it might be boring for people who already know subversion to some extent, but it's also useful to review basics and sort of get caught up with terms that we use in the subversion project to describe things that happen in subversion, because often people aren't really aware of all the terminology and what it means. So usually we try to get people to forget everything they know and start from scratch. But if you use the subversion for a couple of years, then it might be a bit boring at the start, but I hope that we can quickly get to stuff that is a bit harder to run across when you're just using subversion in a basic setting and you're not really doing many complicated things with it. So now we're complete, right? We can actually just start the workshop. I don't think I know the complete list. I know the list. I know some, I saw six addresses on a mail, I was succeed on that otherwise I didn't see anything. So maybe you can already tell me a bit about what you use subversion for while we're waiting. Like is, what are you using it for? Okay, what kind of source code? Anything? Okay, and are there like many isolated projects or do you have a big project? Okay, our branch, so they all share it. I want to share a simple repository. Okay. So I'm going to repository that I'm going to use. So you have one repository and each project has a directory in it and you have, so they all share like a single revision number space as far as, so there's different ones. Yeah, there's many ways to set it up. And are there, do you have like release branches or feature branches in there? So tag releases, right? We don't tag them, but we can tag them. Okay. Right, okay. So you switch from CVS, when? Okay, so that was probably subversion one, one, four. Something like that. Okay, anyone else wants to share what they do just for me to get an impression of what kind of things you do. And what kind of products are you building there? So what kind of products? Okay, so you have copies of, hi. So you have copies of the free VST source tree. So you check out free VST from free VSTs SVN and then you have your own code like kernel modules. You can compile them externally from the source tree, right? To build it, okay. And you don't actually take free VST code and change it and keep managing patches to it. Okay, what do you use SVN for? Okay. You'd have a very different branching merging experience. And a lot of things in Git work sort of more automatically, especially if you have lots of renames in the different branches. So we're going to do that right now. We'll get to that. And you, can I ask you with you? Okay, okay, all right. And you? I run for a ministry of business, subversion. And I collect configuration data from our servers and put it into the ministry of depository so I can either be checked with us, changes from our configuration files, develop us who use it as their depository for Java and source files. And I use it for ports, subversion. Updates and source tree updates, compiling. So when you run free VST, you run as your update. And so on, yeah, yeah. Okay, do you check out your slash EDC configurations from SVN or do you copy things somewhere and then take into a working copy? I don't know, no problem. I collect them from all my servers. Okay. So you basically just stand there and you keep track of changes that people make, but you don't force the EDC system to be under version control. Basically take snapshots of. Do you use subversion for anything? We're currently doing a, who uses subversion for what round? Okay, do you also commit to the tree? No, okay. So we have no source committers, which is no problem. It's just that we're going to do exercises later that we're designed for source committers, so don't be afraid. Okay, so then what I'll do is, I also have, I also brought in some introductory training materials, which we usually use in a course that lasts for a day. So you probably won't be able to do all of it, but those who don't know subversion yet, like very well, could probably do those instead. And then it exercises, but they run you through all the basic work steps that you would do and explain some quirks along the way. And eventually after doing this, you're able to actually think about problems in terms of how subversion works. Okay, then I guess we'll start with an introduction. That is pretty much the same introduction to any workshop I would do about SVM. And I would like to know who used CVS in the past. Almost all of you are kidding. So you have a fairly good understanding of how CVS is modeled. And when you look at subversion, the first, like when you look at it from a design perspective, how it was made, what it was built, it really helps to understand what problems CVS had because subversion exists to replace CVS. It was conceived to be a project to replace CVS. And that means that the whole design is basically based on quirks of CVS that people didn't like. It's not like people went out to build a perfect virtual control system that could do anything under the sun. What they did was they looked at CVS use cases and figured out which parts didn't really work well enough for daily use or where people had problems with CVS in general. And then they fixed those with the new design and wrote an implementation of it, basically. And this implementation has been growing and people have been using it exceedingly for much more complex stuff than people would have done with CVS in the first place. So right now the subversion project has the problem that we have to integrate a lot more features that were not there in CVS into a design that is based around basically the CVS model of working but which people have been starting to use in totally unexpected ways or different ways than the system was originally intended for. This is why so much shift has been happening from subversion to other projects in the open source world because there are use cases that the distributed systems cover a lot better than subversion does because of this. So in CVS you had problems like where, right? So you know that each file is represented with a comma v file in the repository. So if I have a full file, I have a full comma v file in the repository and every version of that file is numbered 1112 and all of them refer to the comma v file in the repository, right? And I could have multiple files instead of just one. So I have a bar file which also has say three versions and all of these are also reflected in the file of bar comma v. So one of the most basic design issues of CVS is that if you want to label your software state like your product state, you have to say, okay, this is our say release one or release one zero. You have to, for each file say, in this full we use version one or two and bar we use version one or one or whatever for this state, right? So the operation to set the tag and read the tag is taken forever because you have to check every comma v file open and see if the tag is inside there which version would apply to this tag and use it. So we had people who migrated off CVS to some version who said, you can now finally label on Fridays because Fridays used to be blocked for commits because people were labeling in the repository. And one particular design point in SVN is that branching is very fast and tagging is very fast. It was supposed to be an atomic operation that completes within a couple of milliseconds. Another thing is that if you rename a file in CVS using the standard user interface not doing like repo copy hacks or whatever, you end up with a deleted like dead version of say foo is dead and then there is another foo too. And you create also a new foo two comma v file in the repository and these two have no historic link or anything. Unless you put something in a log message here that says this file is deleted and moved over there, you don't know what happened. The data model doesn't tell you. And this was another thing that people looked at in SVN when they designed it and they found a way to fix that problem. I won't go into too much detail here because you all know the stuff. You know why CVS was eventually supplanted by other systems. So in SVN, I think it was, I believe it was Jim Blandy who came up with the original design of the repository which represents a tree of snapshots and all the snapshots are numbered. And you have a couple of operations you can do on a tree. So the first revision, we say revision for each version of the tree that we create. First is empty and then you could say add a file in the directory and maybe you could create another file. You could have a directory and change one of the files, say change a line here or something. And you can of course also remove things. So you could remove the file, you just edit it, for example. And you could re-add something there with the same name but different content. So let's, is this big enough for you to see? Okay, so let's label these. So the directories get big letters. It's called A, the files are again called foo and bar. So the system does the snapshotting stuff because people wanted atomic commits. They didn't want the problem of having a network connection right within a commit when they change 30 files and they only end up changing 15 of them in the repository because the network connection broke halfway. And so any update to this tree is fully transactional. The client gives the server information upfront about what it wants to do and then the server checks once the client says, I'm done with filling in information in the transaction. The server goes and checks to see if this can still be folded into the head revision of the repository and creates a new revision if this is the case. So you either commit everything or nothing at all. And that's how they made the snapshot model. And because everything, you still use paths to address everything because you also have, like it's modeled like a file system. It's like the file system you'd have on disk. There are some issues with identifying or pointing at objects. So if I go and say, please get me the file A foo from the repository, Subversion would usually give me this. But I might remember that there was a red line at the top of the file and I don't find it. So I say, no, there must have been something else. And the reason for this is that we have to tell the system which revision we want to look for the path in. And usually on the command line, you write this as, say, give me a foo. The default would be in the header vision. But you can also specify any other revision. So this app notation, this is called the peg revision. You'd like take a peg and hold a particular revision and then look for the path in there. So if we have a foo at three, we actually get this one. Because it looks in revision three for the path A foo. And this is normally very transparent to the user. So usually you don't really see that in the interface a lot. Sometimes when people do crazy tips on mergers, they will have to use these add plus revision numbers. But generally, if people do sort of simple use cases, they don't need that. So they usually always default to either at head or some other reasonable default that Subversion chooses for them. Okay, so we had modify as an operation. We had add, delete, and there's another one which is a copy. And this is interesting in the sense that it solves the history traversal issue that CVS had. So if we copy something, like say directory and you copy bar to bar two, what you get is, it looks sort of like an ad because a new child is added to the directory here. But you also get what's called a copy from pointer. So this points at something in an older revision which it is based on. And how exactly do we create a pointer like that in this model? We know that there's a unique, one way of uniquely addressing objects, which is this. So the copy from information contains a path at a particular revision. It says this would be, the copy from would say bar from five or something like that. And you can also copy from any other older revision. You can copy from anywhere in history to the header vision. Subversion does not have a rename operation at this level in its model. So, at least currently it doesn't. And solving this has been an ongoing problem for several years. So I'm starting to get a bit pessimistic about ever fixing it. It models renames as a delete and a copy. And so we would see something like if we say rename this file to foo2, we would just, or we would see this file disappear. And bar two, and then we'd have our renamed foo, which might have the same contents. I'll do that in this case, but it's not required to have the same. Because it's a copy which can also be modified in the same revision as it's being copied. So this goes back here. And it says foo from six. And so when you see a commit that does a rename, what you will see in the log history is a delete of a foo, and an addition of a foo2. And it says something like from a foo revision six. And at the server side right now, there's nothing that connects these two pieces of information or these two operations. So when the client does an update or merge, this will happen independently in the server will just say delete this, and later it might say copy this. Or they could happen in the other way around. And this makes it very hard to give us some rearrangements people make in the tree automatically. We can find, we can spot places where this is a, where there could be a problem, but often we flag cases as problems, which are not problems. And this is annoying people who are doing a lot of reshuffling of the trees. Did you have a question? Before deleting the old one, and it does check that you can't, you can't ever change anything based on an out of date revision. So the way this works is that if you need to commit, can just look at what the working copy looks like to put all this, all this stuff in here in a small box on the top. So you have your tree in here, and now you have people actually making changes. And for that, they will need to get working copies and working copies are created with a checkout operation. And that creates possibly a subtree representation of contents of the repository. And this is also where the version is significantly different from the current, like the newer centralized systems. The working copy and the repository are totally different databases, and they work in entirely different ways. CVS had a very simple working copy where it has a CVS directory and one entries file with some lines of information about everything. Subversion has now a CQLite based database in here, which is insanely complex and does allow you to perform arbitrary remodeling of your tree inside the working copy. And it will track all your tree modifications. And for instance, correctly identify when you replace the directory, right? We had Subversion 16 with the old working copy that the old model had problems with when you deleted a directory and created new one on top of it by copying something else, say, right? So you have one tree basically replaced by another and then you delete a file inside the copy. The data model didn't know whether the deletion was part of the thing you deleted or whether it was a deletion of something in a copy. There are different layers. And so the new working copy has all this fixed and it has a layered data structure model that I won't go into details about because it's a bit too much for the session. But it is basically designed to stage arbitrary modifications to the repository based on what's called a base revision. And the base revision is what you last checked out or updated to. And that is usually, but it's written, basically Subversion keeps in its working copy, it keeps metadata, say, you checked out revision 10. So all of these nodes will have a revision 10 marker on them. So these are files or directories, it doesn't matter. And then someone else could do the same and they also check out revision 10, right? So basically for those who've already used SVN, there's nothing new here, it's just you should probably already know this. However, do you know what happens when this person over here changes a file and then runs the commit operation to transfer the change. And what happens now is that server will check if this is your question, anything has happened since revision 10 to this file, right? Because we can change any part of the tree in any revision. And so every node has something associated with it which is called the last change revision. It's the revision which last touched this thing. And if your base revision is below that, then we know that you're going to overwrite changes other people have already committed. So we don't allow it. We say you're out of date, you have to update. So you can't ever change anything unless you're fully up to date with all the nodes you want to change. And what might happen here now is that the server says, okay, everything is fine. You can make your change to the file here based on revision 10, no problem. And it says to the client, you've just committed revision 50, say, right? What can we do with the information we have here? We have three nodes all checked out from revision 10. We changed one of them and we know this node is now the same as the one in revision 50. What do we do about the others? Can commit when my check, it doesn't check. So if the actual repository revision is 49, so I would check in version 50, but I only have version 10, isn't that dangerous? That's not a problem, no. Because you see this file? Has this ever changed in the entire drawing I made? No, it hasn't. It doesn't matter which version you check out. The server will always be able to correctly apply the delta to this file because it's the same version. Oh, but it's not likely to work. If all would be changed and my change is on a node version. Oh yeah, that's a problem, but that's your problem. You can, of course, you will merge in these cases with other tree changes, but this is required to make commit a write-only operation and update the read operation. We don't want to force you to update. So this is the, we could have made a trade of the other way saying you always have to update everything, but then imagine somebody constantly updating user source, not winning the race. The race gets a lot smaller this way. And of course, people can commit stuff they haven't tested, but nothing we can do can prevent that. So this is another thing that many people don't really see in the user interface we're about to see next, but what happens to the information here, what can I do with the 10 here? Can I raise that to 50 safely knowing that my file is fine this way? Yes, right, because I just committed it. So it's the version that's in the repository now at 50. So we update this to say 50, but what do we do about the other ones? We have to leave them as they are because we don't know something between these four revisions that we're missing, something else happened over here that could maybe remove the subtree or make arbitrary changes to it. So after every commit, you have what's called a mixed revision working copy that contains something that does not exist in the repository as such. It contains nodes from separate revisions in the repository. You can basically create arbitrary mixed revision working copies and view every node from any revision you want. Basically. You can also create mixed revision working copies just by updating subtrees to other revisions. And this used to be a bigger problem before we stopped allowing people making mergers into states like this. So right now the one seven and further clients will say, please update your working copy before you run a merge because you're essentially, if you do this with a mixed revision working copy, you are essentially editing a tree that doesn't even exist in the repository in this way and this can lead to potentially crazy conflicts that people don't understand, it can make sense of. So there are situations where a subversion will tell users, please update your working copy first before you can perform this operation. But this is a fairly recent thing that we added. Apart from that, it's fairly transparent. You don't really see it, but it's happening. And it's important to know about it. So when this person over here runs an update and they have another, or the same file changed in a way, say, they will usually bump all of their revisions to 50 here, the next update, and potentially merge the nodes that they've also changed. So this is essentially how the working copy operates at a very high level. And it explains to you what the base revisions are and what the last changed revision is. It's also important to know. Essentially, in many other systems, you will always have a single base revision for the entire tree. And I'm not aware of any system that does it, likes a version does it here. But it is a consequence of the tree storage model that it uses. There are some other features which are nice to know about, which is you can put arbitrary metadata on files which are essentially key value pairs that Subversion stores for you and also versions. So you can change them, add them, modify them, and some of them influence a version's behavior, like you probably know the one that sets the end of line style of a file or expands keywords inside the, inside files. And we can add more over time. I was, before this conference, I was looking at adding log template property, which is basically an enhanced version of FreeBSD's custom patch that they use to have a log template in the commit message editor. But that is not done yet, but it's also possible to add features like that with properties. And you can also add your own and Subversion won't care as long as you don't use a name that starts with SVN column, it can be used arbitrarily, as you like. There are also revision properties which are not tied to a path like this one, but are tied to a revision. And they also start with, the names also start with SVN, SVN column. And there's currently only three of those. So there's SVN log, which stores the log message for the commit. There's SVN author, which does the, stores the author name, which is usually given to Subversion by Apache or by SSH or depending on how you authenticate. And then you have SVN date, which contains the timestamp of the commit, which you see in the log output. Basically, this is the first line you see in each log message when you type SVN log contains these two, and then the log itself is taken from this property. You can change them also, if you can use the prop edit command to modify them. By default, this is disabled because these are not versioned, so we don't want people to accidentally overwrite them and lose them. However, the admin can turn on the prop editing and allow people to fix typos and log messages, stuff like that. Any questions so far about what we've heard so far is anything new already in here, or is it all, you're all new, all that already? Okay, how are we for time? Do we have another hour or, okay, good. We have half an hour done. Coffee break is in an hour and a half. So, okay, so let's look at, so when people use these sort of update commit workflows, and they all edit the same sub part of the tree, if essentially made a system where people see each other's changes as soon as they commit and update. And there's no way for people to avoid each other because they cannot be out of date, right? So they have to react to other people's changes right away. And to avoid that, sometimes you don't want that kind of thing because maybe you want to have stable branches or stuff like that, you create copies of say, I'm gonna use the, I'm gonna try to use the previous terminology here. So we have the head branch, and if everyone was always working on head, then we would run into issues when we want to identify exactly which version we actually release to users and which version they're running and which version we can test fixes with and stuff. So the labeling and branching is a core feature of any version control system and subversion models just via copies, because if you recall, that copies always refer to things that have already been committed. We pretty much already have the tag, for example, sitting at each of those revision numbers. We just don't have a friendly name for it. We could remember this, right? We could remember head at 2,360 or something was previously nine, but that's a bit silly. So essentially subversion allows people to create aliases for these kinds of names by copying them to a speaking name. And subversion will just note down the copy from and it will exactly contain the point of the state that people have labeled. So in FreeBSD, I think what they do is they have the copy head to stable and they prepare releases and then from that, they create another branch called relaying and then they create their actual release, which is, I think, a tag and has never changed. So version models, branches as copies which are changed further and tags as copies which are not changed, but the data model doesn't really care. You can take a tag and make it into a branch. It's just a convention for the user, how the user would use it. That's all. This is the kind of branching that we do to identify versions that end users or other vendors are using as a basis for their system. And we use them so that when they have a bug that we need to fix for them, we can fix it here and give them a version that is this good state that we have branched off plus the fixes that doesn't contain crazy new features which are only half done in the head. And this is what we call release branches. And then we also have feature branches which in FreeBSD currently go to the projects directory. People can create their own branches. They're usually they copy the entire head and then they work in isolation of other people who are working in the head directory. So these people update and commit here and these people update and commit down here. And the idea is that if you're making some change which will destabilize head and prevent other people from working, maybe you break the compile for three weeks which is not very realistic, but you might have to change many subsystems which interact in some ways that you cannot just fix all at once because it takes time. And so stuff like that. And the goal of this branch though is to always be like head plus the feature that's being developed. The point is not to stop following the head branch. The point of the release branches is to actually never change. Basically the perfect release branch is created and then never change. If we could, if the code was so good that we could just ship this to users like the state over here, that'd be really nice because they could just install it and be happy forever and there would be no problem and we could just develop here. But in reality that never happens. So you anticipate that there will be problems to fix which is why all these branches are created. And of course you also need to put the bug fixes that you make on stable or these other branches over here too or vice versa, if you have an important fix on head you want to put it there. And but you want to be selective about what you get. So you don't want to merge all the changes that happen here, you just want to merge some of them. But usually you want to merge all the bug fixes down to the head, but maybe you make version number adjustment changes or other things that don't belong in the head. So you also need to be selective in the other direction. Whereas this project's thing is basically just give me all the changes from head that have happened within the last week or months or whatever. And I want to merge them and basically rebase my work on top of those so that eventually when I go back I don't have any surprises. And this is exactly how FreeBSD currently operates in the subversion repository. Now the process of transferring these changes across these different copies of things could be done manually also. And how that works is that you basically say you want to sync your project's branch to the head branch. You get a working copy of it, say from this point. Say the project was in this development at this stage. And you would say, please give me all outstanding branches from, all outstanding changes from the head. What subversion does is it follows the copy from information that has to pinpoint the common ancestor of the branches. And then it can run a three-way merge of the original file here, the modified version over here and the modified version over here. So you know how Diff3 works? Diff3, the Diff3 tool takes three files. One original one and two modified versions which are both based on the original. So you have say the original version and file A and file B and they look like this. So original was one, two, three, four, five, six and say this was seven, eight, five, six and this was five, nine. This is a simple example, two files. These could be lines or chunks of file content or whatever. And the algorithm goes and finds sections where A differs from O and sections where B differs from O. And if the sections don't overlap, it just spits out the result as a sequence and you get that result. If there are conflicts, it will say if the conflicting case is a bit better. So like here it would say A differs from O in this range and B differs from O in this range. And now you have an overlapping pattern. So the output in this case is one, two and then you have an alternative. It doesn't know what to do here, so it writes both. And this is why you see these conflict markers with the two versions when you have a text conflict in a file. And so to run a merge like this, we always need three files. And the task of the merge is to find the relevant version of the files and put them through this algorithm to produce a merged version. And so the first time you merge, subversion can really use the copy from point as a reference. So say this was revision 50 where you created the branch. The merge would say, okay, for every file I use the file here in this revision in head as my O and the file in the current head revision of the head, which would be this. As A and the file on the branch as B, for example. And then it edits the working copy accordingly. So you get local changes here, which you can then tweak further in any way you want. And you can eventually commit them to the branch. And that's when the merge happens. So a merge commit is like any other commit. It's just you could go and manually type all this, all these things in here in the working copy and you don't have to the merged command does it for you. And then you commit it back to the branch and this will say, okay, we've merged all of these changes over here. And you get a single commit, which represents many other commits that might have happened here. Basically collapses all of them in a giant change set. So the next time you do this, if you still use this version over here, you have a problem because you will reapply changes that have happened here. And you will get conflicts which don't make sense because for example, if one commit here deleted the lines, a version will try again to delete the line and the line is not there anymore, stuff like that. It's a simple example. So instead you need to move your point of reference to the time you last synced to the branch and say this was revision 100 here. You would essentially say, okay, we're merging now the O is at 100 and my B is here and my A is here. And it gets tedious if you have to keep giving subversion to revision numbers to properly identify common ancestors. But this used to be the case in subversion one four still so that if you ran merges, you had to exactly say which version you want to use as a reference for your merge. And in subversion one five, people tried to implement a system that tracks revision information for you. So you don't have to specify it manually. Maybe we'll briefly look at how a merge needs to be parameterized each time it's run. So what the merge needs to know is it needs a path one, which is usually called the merge source and it needs it at a particular revision and it needs another path at a particular revision. Usually these are called merge left and merge right. And then it needs a merge target, which is a working copy and a working copy by definition is also essentially coming from some path at some revision, but it's implicit. And so the most general kind of merge you can do in subversion is you plug, you check out your working copy from anywhere you like, from any revision you want, and you put any path you want here and in your version you want here and here and here and then it will do something. But generally you want to choose the parameters so that something sensible happens because if you choose them wrong then it will just do silly stuff. And you can look at how you would parameterize it here in this kind of sync merge. So you're syncing changes from the head and you would have, the target of course is your projects branch at the head revision of this branch, so at the newest revision. And the merge left would be the head at 50 in this case and the merge right would be the head at 100. And later on when we merge a second time, the merge left is the head at 100 and the newest revision of the head is the merge right and the head revision of your working copy which already contains all the changes you merged previously will be the target. And you can basically just type in the numbers every time but since the version one five, it records the numbers it has used in the previous merge in a property called merge for. Property is called as you're in merge for and as a general rule it will be created at the merge target. So wherever you're, whichever path you're using in plugging into the parameters as the merge target that will get merge if we recorded on it. And in this case, for instance, if the merge target is our projects branch or the working copy of the project branch and it's rooted to get a merge for property that says from head we've merged any changes that occurred between 50 and 100. It does not record anything that is part of what we call the natural history of the branch which is if you trace back the copy from and you reach the common ancestor and you go further back here. All these changes are implicitly part of that branch that won't be merged. With some tricks you could actually merge them also. You can merge from anywhere but in this similar scenario usually don't want to merge anything from there. And essentially you can think of this as a filter. So because when you run the second merge subversion will still do all the copy from tracing and stuff and find the common ancestor. However, it will then use the merger for to shift your point of reference. It filters out revisions you've already merged from there. And noteterminology merging your revision is very peculiar because revisions are global and any revision can change any branch or any number of branches. So what you should really be saying is I merged changes that happened at this path in this revision, right? I merged changes that happened in head at 51. But the revision 51 could also have changed other branches or other things, other tags or it could have not changed head at all. So I would not receive any changes but I would still get the revision marked as merged essentially saying we've checked this revision and it didn't have any changes for head. So we don't need to apply them to the work copy we're merging into. But you still get it recorded and in a sort of well-maintained feature branch like this usually you will see entire ranges merged from branches, maybe one or maybe several but you will always have continuous set of revisions usually. And if you merge, if you merge, say a bug fix from a stable branch to the head, subversion will also create merger for on the head for this revision. But if you do this a couple of times you might see merger for that says something like from stable 10 I've merged revisions 60 and 63. With a notation like that. I think in previous D people generally merged from head to stable. So it will be the other way around but a stable branch would get the merger for. And it's actually quite some history I think in the repository in terms of where merger for was created and how it was distributed in between the path and the repository and how people were supposed to get rid of it or not get rid of it. We'll get into that later. So this is the ideal model of how this is supposed to work. It gets complicated by the fact that people can essentially merge at any depth of a path and the target can also be at any depth in the repository. So sometimes you have merger for on one directory then it's child will also have merger for and the question then becomes what do we do with that kind of situation? What applies where? And the rules are that if you have merger for on a path so you have the merger for property at A and you have another directory B which also has merger for then generally the merger for on B overrides anything above it. So nothing will apply from the top of the tree to B. B has its own merger for and it's on basically it's on merger for universe. If the version finds that the merger for in A and B is essentially the same, the very strict sense, it basically what it does is it would take both property values and it will create a path adjusted version of both so that the path differences disappear. So say you have merger for here that says I've merged from head and this one said I've merged from head A. It will remove all these path differences but it's essentially pretending that this merger for was up here and then it does a text wise comparison and if that matches entirely then this merger for will be deleted. This is what we call merger for illusion. However, in practice it turns out that this rarely happens. So I think that in the original design the idea was that merger for would regularly elide upwards but because the illusion strategy right now is very simple and other strategies have been tried but they introduced problems. What happens is that you end up in some situations with lots of what we call subtree merging for that populates the tree and it's not bad as such. However, it sometimes creates problems with either bugs and subversion where it doesn't really handle the merging of the various merge of properties properly or it just makes things very slow or it complicates the answer to questions like what has been merged into this branch. Okay, so there's also non-inheritable merge for which is signified by having a revision like say R60 with a star. That essentially means that only this path has gotten the changes but the children have not even if they don't have merge info. So B could of course overwrite this but say another child C which has no merge info essentially would inherit the merge info without this revision from A. So this can happen if for example you run a merge in a working copy which contains a path that you don't have access permission for, right? So you check out the tree and some sub-directries forbidden for you in the rule side of the access rules and you don't get to see it because we want to hide it from you. We don't want people to see secret information and repositories that's what this path access feature is for. And so instead of something on the disk you just get a small marker in the database which says this path isn't here. And when a merge runs into that and has changes for that path to merge it won't apply them but it's that merge info like this on the parent to say, well we merged everything up to here but further down we can't do it. We couldn't do it in this time. Maybe next time someone else will run the merge and who has access permission and the changes will follow up. And it can also happen if you on purpose hide sub-trees of your working copy. So if you use the depth feature the sparse checkout feature, where in the working copy you can hide a path even though they exist in the repositories. So you have a very large directory that you don't really want to check out every time. You could check out the parent with what we call depth empty, parent P. And then you have a working copy of P with no children. And you can go inside there and say, update and then you set the depth of a directory P child one to infinity for example. We'll edit it here. Then C1 will appear with all its children. And other values you can put instead of infinity are immediate which gets all the immediate children or files which will only get the file children of this directory P. Oh, C1. So this can also create situations where the merge might run into paths that don't exist or that are just simply locking in the working copy. Then you get that kind of thing. Okay, so when you do this depth checkout you have what's called the ambient depth which I think is a really cool term for virtual control system to use because it sounds like some AFix twin record or something. It's recorded in the working copy as being the depth of this node. And when you run a normal update it doesn't change it. Because otherwise if you had this large directory you want to exclude, you check out the working copy and then you run update, the large directory would come down you'll probably not be very happy. So it records the depth that you used the last time and it stores it. And whenever you use the normal depth option it applies to the current operation. And we often have say you could diff your working copy to depth immediate instead of infinity to only see changes made to nodes which are within the depth of infinity, of immediate. And in this case it doesn't change the ambient depth of the nodes, right? And this is what set depth is for. So with set depth you tell the working copy look I want to have a different ambient depth for this node, right? And this is also mentioned in the FreeBSD handbook and I think people use that quite a lot because you see quite a lot of non-inheritable verger for in FreeBSD Solstice if you go look for it. I don't think it's access permissions because generally not in Solstice projects the access permissions are not the trigger for this problem because everyone can read everything but in corporations usually it's the access permissions that causes problem, but it's the same issue. Okay so we've had two types of mergers. We've had the sync and the cherry, this is what we call cherry picking here where you deliberately take particular revisions and our changes that happened in particular and merge just them. This is not as strictly the same as a git cherry pick because git treats this as a diff plus patch thing sort of more or less. It's probably equivalent but they don't track it. And so subverting actually tracks the cherry pick. You can ask a branch what has been cherry picked into you and it will be in the merger formation not in the log messages. So there's a slight difference there to git. And so this is generally used for release branches this is used for project branches. And then you have another merge which is to reintegrate merge. And why does it even exist is a big mystery to many people because a merge operation is essentially symmetric, right? I can put one branch on the left and one branch on the right and I can swap them around. And if I compare them the other way I would just get the different worded. And if I merge them the other way I would just get changes over here applied over there instead of the other way around. So why is it different? In many systems, in many virtual control systems you don't have that difference. So generally a merge operation like in git would say take the two heads that you want to merge and create a new child of them that has both of them as ancestors and merge them all in some way. And there's no concept of direction, right? And in subversion you, because of the way you parameterize the merge if you think of it in this term in terms of this it sort of makes sense. I don't think it's really the best explanation because it's really like why there are two different implementations of a merge is pretty much an implementation detail but it's possible to understand it a bit by figuring out how to parameterize and reintegrate merge these parameters and comparing that to how the sync merge is done. So we said that for the sync merge we had first one we had was the path was head and revision 50 was the merge left. The merge right was head at 100. That's the merge right. Then we have the work and copy of the project's branch. So that's always, oh yeah, the work and copy is projects. Projects at head. Okay, now if I want to merge the other way, like I've done this a few times, right? I've done it a second time putting putting the number 100 here and maybe say we have revision 200 over here. And maybe I put 200 here or something, right? The second time. So this goes on and on and on. The revisions keep changing the path down. Now if you want to merge the feature that was developed here to the head, you have to take just the changes that implement the feature but you have to omit the merge commits because those are changes from head that you already have in head. So if you do the same kind of thing, the same parameterization, just swap the paths out. So if you say merge, I'll use a new one for this. So say if you said merge projects from its common answers to 0.50 and projects at the current revision 200 into a working copy of head. What would happen? You would merge this collection of changes, including the black ones, including the ones that contain merge commits. So because this doesn't happen in Git because it implicitly already has these hidden inside the DAG, right? It doesn't see them anymore. But in Subversion they are still there. They appear as local changes relative to the head in the merge history even though they're not because these are just like, it's just a series of commits and Subversion doesn't really know or care what the changes are that are inside these commits. It will run a diff from here to here if you do this kind of, if you put in these parameters that we just put in and it will merge all that in a working copy of this and then all the changes you've merged down will come back and bounce up back at you and you get lots of conflict. And to fix that, how do you need to change this? What do I need to change around? What do I need to change in which way? Yeah, how do I do that by changing these parameters? So you could specify all the, you could cherry pick everything which is what FreeBSD does currently in some cases because the reintegrate doesn't always work for them. However, that's the intended solution because it requires you to do a lot of work. It requires you to manually pick all the changes you've already made. You basically just want everything from that branch over in the head, right? So there's a trick to do it. Basically, you think about what the O and A and B need to be. So we said the merge left is basically the original and this is one modified version, this is the other, right? If you move your original to the latest revision of head which you last synced to, what happens? So say you haven't synced up to here, right? And there are other changes over here which you don't have yet. All these changes you've got, right? You've merged them. So this can be your O. But now your other, this still needs to be the branch so that you get a diff that contains the branch changes. So essentially, instead of doing that, you do a head say at 200 and you say projects at head into a written copy of head at head which is potentially different than 200 because it's in the past. And that will give you essentially a merge that filters out the big merge changes and apply just these. Try it out, it works, it works this way. But the thing is the reintegrate option that we used to have for SV and merge does just that. It changes the parameters and then internally also which is why this isn't the full explanation, runs in a different merge algorithm, a different implementation of the merge to put just the right changes back onto the head. And we used to require users to specify the direction they merged into. So you used to have to say, okay, I'm syncing so I'm not using the reintegrate option. And now if you're going back to the head, you need to suddenly use it because you're changing merge directions. And this confused many people. It was just too cumbersome to use. And then subversion developer Julian Ford figured out how to automate the parameterization better so that subversion will can tell from the kind of merge you're doing whether you're trying to do a sync or reintegrate and then we run it automatically. So in one eight, you don't have to use this option anymore but it still works exactly the same way it will. The UI is basically just hiding it from you. So now you say, if you really want to, let's just merge everything. Going back to this, it's enough to say SVN merge and you give a merge source. And it gets all the information it needs from copy from for merging from your target. And it plugs in the parameters for you. No matter whether you do sync or reintegrate it will work or should. That's the idea. It doesn't work perfectly. It has problems with subtree merge info in some cases. And I've actually found a case where in FreeBSD you have one of the Beehive branches in projects. If you try to reintegrate it, subversion tells you, oh, you haven't really synced up the branch properly. You have some changes over here which you haven't synced up. And this means that this reintegrate trick won't work. So reintegrate requires you to have fully synced everything up to a particular vision. And it says, well, you're missing all such and such revisions from such and such path, right? It gives you a long list of things. And then you say, okay, you go back to the branch and you try to merge from the head like this and it does nothing. It says everything is already there. And this is a bug, I think. I haven't really found out why it happens but it's probably got to do with either non-inheritable ranges in immersion for or subtree merger for handling bugs. But anyway, the idea is that it should work like this. So when it doesn't work like this, usually subversion has a problem or really the user has done things like merged like into a partial working copy or merged without access permissions and created a situation where a branch isn't fully up to sync. It can also happen during merge mistakes, especially with conflict resolution that can sometimes be quite overwhelming for people who haven't practiced it a lot, especially when there are renames involved. So are there any questions about this merger for stuff and their merge tracking stuff? Was this new, did you use this before? Did any of you already use this? So another thing that becomes complicated quite quickly is dealing with what we call tree conflicts. And what happens here is that people make structural changes to a project in conflicting ways. Like someone edits a file where some other person deletes it, for example. There needs to be some sort of resolution to this because it's not possible to do both. And older subversion versions like subversion one five, I think we're doing things like this where if you have a file, say foo, and someone checks out the working copy of foo and edits it, like edits the file in here, creates a modified version, and someone else decides to delete the file and commits that. The user here does an update. What's a version one five that was, it said, oh, I have to delete this file? Well, okay, I'm gonna remove it from metadata. And then I'm gonna check it's on this state. And if it's modified, relative to the metadata, then I'm gonna keep it on disk. But, you know, because it's gone from version control, essentially before the update, you saw like a modified foo. And after the update, you saw a question mark foo, which means foo is now on version and not tracked anymore because it was deleted. So the user builds the project and everything works fine and they don't see that there's a small question mark and the SCN status output and they commit and they break the bill for everyone because some change somewhere else required this foo foo. And people who were versioning code with subversion building CT scan equipment had to get it certified from the FCC and said, if we are going to lose changes like this in our process, then we cannot use this tool. It's impossible. The FCC will never allow us to use that. It's way too dangerous. And they said, what can we done about this? And at the time, this was actually the time when I got involved in the project and my company ended up getting a small contract to try to fix this problem. And we worked with the subversion project to decide what to do about this and also with the company that builds these machines. And what we ended up doing is that we said, well, at least there should be a warning. There should be something that tells the user that something is wrong or potentially harmful. So instead of the question mark, we managed to make subversion show a conflict. And we also, so you get a C and through and you also get some information about what this conflict is. It would say something like incoming, delete versus local edit, right? So, and one other thing that's important to know here is that if you have a conflicted file or directory and you try to commit it and still mark this conflict, it's a version that says, no, you haven't dealt with that. Tell me that you've dealt with it before you can commit it. So we require people to actively touch the thing and say, I've resolved this conflict to commit their changes again and to allow them to proceed. So we're forcing the user to deal with it. The problem is that it becomes very hard if there's a lot of them. So it's a lot of manual work sometimes to resolve them and some people just get lost in the complexity and just say, yeah, yeah, resolve, resolve, resolve, I'll commit it, happens a lot. And then you get exactly the same bogus results as you had before, maybe slightly different because we also try to change the default behavior to preserve the local change in favor of the incoming change. Because that incoming change is already in the repository, the local change is not, so the local change is more important to get in there. And even then if the history needs to be fixed or needs to be reverted, whatever, it's better than having a lost change, right? So another case, for instance, is, yeah, what happened here is actually, because of that, what I just mentioned, you actually just get a conflicted through, not only, but you also get a copy of through. You'll say A plus, and Foo is now a copy of the base revision you had before the update. This is not perfect, but it allows you to just say, okay, mark, resolve, and commit, and retain your change. You're overwriting the deletion. You're basically recreating the thing from history as you had it originally, and your change makes it into the repository. If you just do the default thing that most people do is just resolve and commit. However, generally people should really start to engage their brains when they see this and try to figure out what's going on, because a deletion and subversion is possibly not just a deletion, because if you remember how the repository works, it doesn't say move anywhere, right? I said moves are modeled as copy and delete. So if you see this, I've actually changed this in 1.9 to say something else. In 1.9 it will say this, because I got annoyed at people not realizing that, and I think the interface is still very bad, and we still need to improve it, but it now says this. I think all move, it says all move. So it says incoming, delete, all move, versus local added. People still don't understand this at first sight. They will say, what are you trying to tell me? You don't know what's a move or what's a delete? Yeah, exactly, we don't know. Did you consider adding a moved to property to the file that had to be deleted? Yes, we considered so many things. It becomes really complex. Okay, you say you aren't doing that on purpose because it results in too much complexity. We are doing, yeah, what we're doing right now is the best we can do with the current model. And extending the model is something that is hard because the project has some other rules like backwards compact, and we need to keep operating against all clients. We don't want to require people to change any applications they've written against Subversion to keep working. And because Subversion is a set of libraries, it's not just a command line client. It has maybe five or six libraries with public APIs that we version and that are backwards compatible up to Subversion 1.0 at the binary level. So you can actually run code written against Subversion 1.0 with Subversion 1.8 libraries. And the client is just a small wrapper around those libraries. And Taurus SVN, for instance, on Windows, it uses the Explorer stuff to display GUI. And internally, it runs the same code as the command line client because it has all the APIs available to do that. But you said people can add their own properties that are not standard properties anyway. So why would adding a new property cause additional complexity? It won't, but it doesn't solve the problem. Because what you need to do then is the hard part is not how to record the move. We have several implementations and branches of that. The hard part is resolving the conflicts and giving people reasonable choices and operations they can use to resolve them. And no, there has to be research, research has to be done on this because there's, I mean, I've looked at Git, right? What Git does in cases like this, in their merge algorithm, in their merge algorithm implementation, they find a case where something has moved away and they run printf and scanf for an answer for what should happen inside the algorithm. I think that's stupid. What we need is something that has proper representation of the API level so that clients, no matter whether they're GUI clients or web clients or whatever, have all the options available and all offer the same functionality and are stable in the sense that they will keep offering these options for all releases since after that. And if you look at this problem space, this is one case. It gets really large very quickly because you have to consider not just updates but mergers as well. And then you have to consider add versus add. You have to consider copy versus add if you consider move versus move. If you consider move versus delete either side and just explodes, it's a lot of cases. And so what this really could be, right? This incoming delete could be incoming delete or a move. So instead of deleting this, the person also added a new foo too over here which is now in your working copy after the update, right? And Subversion doesn't know that your changes would not be here. But you should know because you know that you should go and figure out that it's a move. And so this for changes that move around lots of things, this gets really annoying and tedious. But in simple cases, it's actually pretty straightforward. So you would have to figure out, okay, what happened? Why did the delete occur? Did somebody move something or did somebody delete something? But then you find out in the log history what's going on. Maybe you talk to the person who made the change and you try to figure out, okay. Usually in the case of an incoming move, the resolution is trivial. You would just apply your edits over there once you found it, right? But in cases where you also moved it or in cases where, I don't know, people add like two Java packages with the same name and then they work on different branches and they merge them later. And they didn't know what the other person was doing. Something needs to happen that resolves the conflict in a reasonable way at the semantic level. And the virtual control system doesn't know that. How to do that. Yeah, so this for example is the case where you flag add versus add as well. You can't have the same object occupying the same path. It's impossible. And also it gets very complicated when you look at replacements. So people can also delete and add something in the same revision which will then first delete and add it in a single update or merge. And thereby it might change the copy from information that the node in the working copy has. Like maybe just say you're merging from another branch and there's a good, oh here's empty page. So say we have the head branch here and we create two branches, B1 and B2. And both of them add a new directory called D and both of them want to merge. You need to decide whether it's reasonable to have a single D or rename one of them. But I mean it's obvious if you look at it like this but the problem is that there are many, many combinations and we haven't yet found a good way of teaching the client to tell users what happened because sometimes you just lack the information. For example, if you consider delete versus delete is a very popular conflict that people get a lot. But it could be this or it could be move versus delete or it could be delete versus move or it could be move versus move. And Subversion currently has in general no way of knowing which of these it is, which of these cases happened because all the copies and deletes are disjoint. And therefore we cannot offer reasonable options currently. We just don't know. We have to tell the user, look, please figure out what happened and fix it. So whatever you're doing it. And so if you have replacements, like in this case, right, say this person commits to D first. The D will be copied by the merge from the branch to the trunk and it will have copy from pointing back to the branch. It will say this directory was copied from B2 at revision 40. And then the next person goes and wants to add their D. But now you have to copy from pointer this way or that way, which one do you use? And sometimes people would delete the D here and add the other one with different copy, with different copy from information. And so now if you do the log history, you suddenly end up here instead of there. If you go from the head revision backwards, but if you go to a revision before this was merged, you will see the other path. Because at that point, maybe I should have drawn this differently. So imagine this didn't exist, but we are here. So in this range, copy from tracing goes down here. And once we've replaced the D, copy from tracing goes this way. And if you peg your lookup here, so if you use a peg revision that is past that merge revision from B1, you will always see the B2 history. So nothing is lost. It's just that it's very awkward to figure out some of the history when you're looking from the future into it. And also, I noticed that in previous D, people complained about log history diverting from the head. So you have a file here that already existed back there. And for some reason you run a log on it and it points back to some branch instead of staying in the head. And this happens when people, I believe, run into tree conflicts. And don't really know what to do and do something. And maybe copy their files on top or run the merge again or do whatever they think is right to fix the situation. But then uncheck whether the copy from information actually still points to the head of the same branch. And then they basically create an alternate line of history for that file. And this is not a problem again. Generally, once you have this situation, you can either keep going back here to see the original history, or you can even change it back by copying stuff from the past to the front that works. You can fix it in further revisions. But it's not very convenient and it's something that also doesn't really apply to other systems because they don't have this concept. And another issue with these tree conflicts is that we don't really have a recipe that works for everything. And it's a lot of practice to fix them, a lot of understanding the data model and navigating it and figuring out what happened and putting changes in the right places and fixing up the written copy in a consistent way in several respects that you might want, like you want to make sure a copy from info is still correct or as you want it. And merge a force there or not there where you want it. And so there's a lot of freedom that the version gives to people in sort of how they use this model. But it also makes it very easy to make mistakes that people regret later because it's in history and they can't fix it. It will always be there. They can fix it in the head revision but they cannot fix it in the old ones. Do you want to walk through some more of these cases or is this not so interesting? I mean, we could probably look for another two or three to see what people would do if they run into these sorts of situations. Like have a concrete example that shows you how to resolve ad versus ad or something like that's in commands. Would that help? Or is it just enough as an introduction? I don't know. Have you ever had this kind of conflict in your use? Do you remember what happened when it did? Okay, what did you do? Ah, okay. What's interesting is that because of mixed revision working copies you can actually construct cases where you're the only person working on a repository but you still got three conflicts for yourself. So I forgot how to do that but it's a nice trick but sometimes it happens to people and they need to understand mixed revisions, copy from information and all the stuff to understand what's going on. I believe that most users get by without all of this. They just see update, add, delete and they don't see any of the, they probably don't even look at the metadata exactly or know what it means and then it becomes pretty difficult to deal with complex cases. Okay, how much time have you got left? Four. So the conflict rate is that of half an hour to the pre-decision regret. How long is the second half? Is it the same next? It just says the start time. Okay. I don't think you got this. It says start time is four hours. Okay. Good, so you guys are not asking very many questions so I'm sort of running out of interesting stories to tell. I don't know if you want to know more about anything in particular or any other thing you would like to know more about or any things you had happened to you in the past with the system that you couldn't deal with or that was surprising. Ah, the default ignores list. Yeah, well you can, but you have to add them. Yes. Okay, so there's, so there's, well that was a bit, that was changed recently. So some things we're trying to fix some of that. So a couple of the problems are like the ignore list like ignoring files, ignoring the unversion files. Right by default, when you see that in your working copy and this in status, subversion just tells you, I'm not looking at this, I don't care. And if you don't want them, if you don't want to see them in status, if you want to filter them, there are several ways of doing that. Most commonly used one is the, as we ignore property that you set on the directory and you just say ignore all text files in this directory. So the value to a multi-line list of, basically FN match patterns. You match this with APR, SFN match implementation. And then you don't see them anymore. They're still there, but you just don't see them. And you, for example, tortoise would not suggest them for commit and stuff like that. There's also the global ignore list on the client side. And that by default, this is in the subversion config file and home.subversionconfig, I think. And that by default contains some patterns like .a, .so, you know. But actually, if you want to commit them, you don't have to change this. You just add them with SVN add, even though they ignored and suddenly you will have them added. Because if you explicitly say, I want this added, then it will be. But if you add a directory recursively and it contains these files somewhere, then they will be filtered by the global ignore. So you don't end up with them by default. But you can actually still explicitly add them one by one. No, this is by default. You can comment it out and it will always be applied. But you can change it per client. Another thing that was added in 1.8 is a global ignores property, which I think it augments this one. So I'm not sure if it overrides it. Maybe it even overrides it. I have to look it up. I don't remember. But anyway, you can put the same kinds of patterns as with this one. And it's an inherited property, which is something that we added in 1.8. The problem with the original implementation of this ignore mechanism is that you had to set it on every directory explicitly because you can check out from any directory and the client had no way of knowing what is set outside of the working copy. So say you check out some deep, deep sub directory and you have an ignore property somewhere set up further up the tree. You wouldn't see it. So all the ignores would not apply to your working copy. So the original implementation required you to set as we ignore on every directory where you wanted to ignore files. And since the version 1.8, there is a concept of an inheritable property that is a property that the client will go look for in the repository in addition to local paths, even on parents and will cache the value in the local database. And now it will recache them on every update or every time it contacts the repository. But now you have a way of telling clients from the repository side what to ignore globally. You set it on the root directory of your project or your repository and it will work as long as the clients are 1.8. Another inheritable property we added was auto props. That is the same as the auto prop setting in the config file as well. And what this does is, for example, you can say any XML file should have set an SVN mime type property with the content text slash XML or whatever. You know, something like that. So this configures or you could say any text file should have set SVN EOL star to native, which would then apply recursively to all the, any time someone wants SVN add of a text file will automatically get the property and set the line style to native so that when people use different systems they will get the native new line convention in each file. So did I get this right that it's an inherited property you can set it on the server side? Yeah, once. You set it at one node. And if people check our children of that or the same node, then they will see it. But only one of the clients will do that. The other ones will ignore it. Yeah, the one feature I'd like to add is a log template because FreeBSD uses that as a custom patch currently. And there is a branch that implements a inheritable log template property where you can put a subtext and that will then fire up in the editor when you open the comment message. Editor, it will be there. But currently it's not clear whether this is enough to satisfy even FreeBSD's use case because to add some magic fields that can disappear or when people don't set them and they document their fields separately from the actual fields. So they have stuff like PR and then they have this ignore line which says anything below this will be ignored. You type a log message up here and this photos is just for your information. But here they document the PR as a number and so on. And so if people don't set the PR it will vanish from the final comment message. And I need to find a way of generically expressing this kind of these kinds of rules to actually make FreeBSD users switch because if we offer something that does not have the same behavior as their custom patch I don't think they're gonna be willing to drop the patch which I'd like them to do. But we have to find something where you can set a property and explain the rules of how the template should work and then it should work that way. I'm not yet sure how to make it so generic that everyone who uses Subversion can make use of it. If you have any suggestions I'd be very happy. Okay, no more questions? And I would suggest we start with you guys doing something. You can crack some riddles that other people working on FreeBSD have cracked in the past where they had situations they couldn't figure out and asked for help or asked each other for help and I found their readiness post so they were confused and helpless. And I devised solutions to all these problems which I will share with you at the end after you've tried to solve them. And if you don't want to do anything if you don't feel very, if you have to use Subversion maybe you will just want to sort of like walk through through all the commands that are there, what you can do with them and this will just require you to read and type on the terminal and try some stuff out and there won't be any challenge. It will just be explaining the system to you interactively as you go. So those are the things I have on offer. You can pick either of them, which ever suits you best. I think I have more copies of the FreeBSD exercises because I expected many FreeBSD developers to be. So I think I have 10 printouts of the exercises for FreeBSD and I have three or four of the others. So you can, I also have them in PDF so you can also read them on the screen if you want. And you will need Subversion command line client installed on the system, but I think I told you that in mail before it's workshop. Yeah, it's perfect. You're running current, right? So are you on OpenBSD with this? Yes. Yeah, that has the latest. Okay, so I would suggest you get up for a change and grab your exercises that you want to do. So there's four of these and 10 of these. Oh, and you also get data and you have to pass around a couple of USB sticks to get the data. But the data is only for the FreeBSD exercises. So it's not for.