 I would now like to introduce Andrew Bartlett, who is on the core of the SAMBA team to talk about packaging SAMBA 4.4 deviant, and his trials and tribulations on doing that. Hi, so I've been a SAMBA team member since 2001, and I've been working on the Active Directory DC since about 2006, so that was the SAMBA 4 project that so many of you have known and heard about for way more years than it ever actually really existed. And I've also been involved peripherally in the Debian packaging team. The Debian community has always had the most interesting SAMBA bugs, so even as a long time Fedora user, I have always hung out on the Debian packaging list because that's where you got some of the really interesting issues to chase down. And despite being a long time Fedora user, I ended up doing the Debian package merge for SAMBA 4 because I was working for Netgear at the time, and they run a Debian box inside those Netgear NASs, they're just a Debian wheezy these days, and they need packages. So they of course didn't want to be maintaining their own packages forever, so they just wanted to get the upstream ones into shape. So I also want to thank Catalyst, they keep me bed and watered while I do great things on SAMBA, and I also want to thank those fellow SAMBA team members in the Debian packaging team who took the work that I started and got polished enough that we now have it in the next release. I would very much appreciate it if folks would ask questions during the talk. This would be very boring if I'm just the one speaking, so please, if you've got some thoughts or you've been packaging other software that has some of these complexities or challenges, I'd rather go off and digression and discuss how we can do these things better. Now, most in the room you probably know what SAMBA is, but I thought I'd remind you how broad SAMBA is these days. It's more than just a file server, it's more than just a print server, it is the Active Directory demo controller, but also a number of quite fundamental libraries to a modern system. So TDB, TALAC, LDB, DCE, RPC client libraries for connecting to Microsoft Exchange servers, one of the most important things we now provide with SAMBA 4, and the Active Directory DC. I'd like to think of SAMBA as the center of the universe. Removing SAMBA from your system is rather detrimental to its operation. So our TDB is needed by 3,000 other packages indirectly, TALAC, 2,000, Lib S CME client will still remove your desktop, and the SAMBA Libs, which covers some of our other packages that use it, come to Reverse Dependency in 140 because of the evolution may be using our DCRPC libraries. So it's a fairly core and important package between the different bits that we all maintain as part of the SAMBA package, and that makes it important to get right, and also gives us some real challenges because we have a semi-monolithic system. We develop SAMBA as one whole of SAMBA. One SAMBA.tar.gz that is released, and yet you've got packages and parts of the package that you might wish to have advanced at different rates. You might really want your file server to be old and conservative, almost crusty, because it's just once a server files and you don't do that well. Or you might want to have the latest release of TALAC because we've added some useful feature there. We do have different versions for TDB and TALAC, but they do come out on the same table as SAMBA Master. We do generally require that version of TDB or TALAC when you build SAMBA so that you we basically say your system version must be the same as the version we have in Tree, because that's what we've actually tested within our order build system. We can relax those checks and some distributions do occasionally when they know that it's all OK. But in general, we don't do things in Tree. We're saying, oh, we'll do these workarounds for old versions of TDB or TALAC, we generally say, no, you'll have the current version. And so a lot of all these packages have to be upgraded all in sync, even if they don't all have to be installed at once. So TDB and TALAC do have their own build systems, though, but they use the big SAMBA build system, but they do build on their own. So it's, and one of the other important things is that SAMBA does follow the Debian rules and can build against system copies of TDB or TALAC or LEB. And indeed, that's the right way to do it, because we have quite strong rules that we don't want embedded copies of libraries because of all the challenges that brings. So SAMBA can do all these things, but it just makes the packaging worse, not better, rather than just being able to build SAMBA with the things that are in the tarball. We have to try and make sure all these other parts that we provided in the tarball are actually all correct anyway. So it's a bit complex and it makes it a little bit more off-putting to be part of the packaging effort, it seems. The other challenge that I alluded to earlier is that SAMBA is both very progressive and very conservative at the same time. One of the fundamental tensions in the SAMBA team is the tension between those who've got a stable file server that they just want to not change, not fuss with, not be innovative with, you know, that was a real, when the protocol wasn't changing for many years, that was a real feeling on the team. We've got this down, well, don't bust it, just get things, just fix things and keep people a file server they're putting in their NAS. We've also got the other end, where we were doing things like inventing Talica a while back when Trig did that work. He did a lot of work improving that and took, or LDB, or the whole Active Directory DC. And there we're trying to go the other way. We really want to be using things like we're using Python these days. We're wanting to be doing good work using modern, available things, like we want to maybe use external libraries that are unavailable everywhere. So there's a real tension, and our users are wanting their file server to remain really stable. If you've got a distribution and you're trying to pick what version of SAMBA to run, you've got the real compromise of the file server users wanting really stable, just don't mess with it. It was working. And the Active Directory users are saying, I need 2012 support. OK, so how are we going to do both of these at the same time? And how is a distribution like Debian going to cope for both of those users, giving them the packages they want? So one solution that we had to this, in a sense, was that SAMBA was packaged twice. SAMBA 4 was going to be the new SAMBA, and so a SAMBA 4 package forked to match the SAMBA 4 technology previous that we were making, particularly to support Open Change. We had SAMBA 4 in Experimental, and we had SAMBA in Main. And the two would coexist with different versions. It mirrored the reality and the sad reality that we had on the SAMBA team where we effectively did fork. And we never really wanted to admit that, and we tried not to talk a lot about it. But the team really did split in two under the one banner. And so the packages mirrored that reality. We were doing releases that were separate. We had SAMBA 3X releases continuing, and we had SAMBA 4 technology previews going at a completely different pace than unrelated to each other. So we had two lots of packages trying to mirror this new reality. SAMBA 3X was the traditional SAMBA, and SAMBA 4, including all the things you normally expect. So SMBD, NMD, Windbind, but not some of the libraries that packages like Evolution MAPI were really after. They were off in the SAMBA 4 package. And while we were at it, we packaged up the Active Directory domain controller, because the Debian attitude of if it's there, package it, and someone might want to use it that way. And that was, you know, got some people really good access into that code. We also gratuitously rewrote a number of client tools, keeping the same names, and had NSS Windbind, and we went, again, but it wasn't just here, SAMBA 3, here, SAMBA 4, and they don't need to mess with each other. Well, they used the same SMBDoc whole file. They used overlapping binary names, and so we got around to it and renamed Net from SAMBA 4 into SAMBA Tool. We had Net and Net. We had SMB Client and SMB Client. NMD Lookup and NMD Lookup. They've both gone into two different directions with different improvements. We even had different NSS Windbind and PAM Windbind, because they're looking for different copies of Windbind, because we'd forked that as well and rewritten it. This was not a very happy state of affairs, but at least when you install the SAMBA 4 packages, you got SAMBA 4, and you could sort of mostly keep it going with SAMBA 3. Alternatives were used to try and cope with two different binaries with the same name. And lots of good work was done by really smart developers like the Yomav Noi here, but it still wasn't very pretty. But neither was the situation upstream, so you can't really expect anything else. So time passes. SAMBA's gone from being just an experimental to unstable, because we want Evolution Mappy to be in unstable, because people want to be able to access their corporate email. The Active Directory DC follows into unstable, and it seems a reasonable place to go. At this point, there's still no prospect of a SAMBA 4 release. We may as well get the forked package featured so that people can use some of this software. And from testing it ends up, it goes into testing, and testing it goes into stable. In the meantime, SAMBA unhawks. We finally get our act together. We merge the Git repositories. We actually start merging the code base. We start trying to find similar bits of code, merging them together, putting them in common areas. SAMBA 10G was a previous fork of SAMBA, and a really nasty one in terms of how it impacted on the team. It was personal to some. It was painful to all. And a decade and more on those pains, the team was not willing to do that, even though we'd gone and really broken ourselves apart internally, we were not willing to break apart fully. And so the effort was really made to try and get SAMBA back together again. And that ended up being the precondition for SAMBA 4. The rule was that we would not release SAMBA 4.0. And it was very frustrating to me over many years, I feel like I couldn't release this thing. The rule the team set is you can't call it SAMBA 4.0 until there's one SAMBA 4.0. Because people who were working on SAMBA 3.6 did not want to be seen as working on the old version of SAMBA. Because SAMBA 4 was released and was replacing it all. And rightly so, they were doing really good work. And their work is now very much leading where SAMBA is going. But that was the rules. And that's where we ended up having to then make the packaging, the packaging start matching the new reality again of the SAMBA team of we're not forked with one tab or one release. We also had some changes in the way SAMBA was being updated. We couldn't keep updating unstable with new versions of SAMBA as we got them towards the 4.0 release. This situation of two different types of packages couldn't continue either. So further updates went into experimental. But we got around to our release of SAMBA 4.0 and Red Hat packages, SAMBA 4.0, and Debin is still saying that the latest is 3.6 and it's never heard of this 4.0 thing. We didn't strip out the SAMBA 4.0 package. When it got towards stable, we didn't do any work to try and put only in stable things that we thought were long-term stable parts of SAMBA. There are bits that we weren't going to be changing too much. The whole lot ended up in there. Oops. So an old, unsupported, previously SAMBA 4.0 ships, including the Active Directory DC, would beat it too because that's where we got to before the Debian freeze. And we nobody wanted to try and break the freeze to break the packages to remove the functionality because that was going against all sorts of other things that were important in Debian and also just required time and effort and things that none of us really had. So SAMBA 4.0 releases and people go, ooh, okay, Appget installs SAMBA, oh, that's SAMBA 3.6, Appget installs SAMBA 4. Oh, I've got my new SAMBA 4.0 DC. Guys, that's the one we stopped working on six months ago and there are lots of bugs we fixed since. And by the way, you're using the wrong file server because we developed a whole file server only to basically ditch it because we realized we didn't have the developers remain two file servers and the concepts, while some of them, great, did not end up. We try to patch our new file server better, but none of the code is being used. There's some interesting functionality it's got, but we've got Debian there shipping, in the stable release, shipping the file server that we don't want to maintain anymore and we don't really, we think it's well written, but not so good. And basically telling our users, there is no Debian package. If you want this, go off to a third party vendor and install their package. So Serdet, who supports our SAMBA XP conference, who's a big force behind SAMBA, has a website enterprise SAMBA and one of the things they provide is SAMBA packages that let the installers a different location in your system and generally complimentary. And so a lot of people are being suggested, we'll go over and run Serdet packages. So there's no simple upgrade path. They're so real way forward for our users. There's no one wanting to just package SAMBA, the way they've been packaged before. Because you could have, I suppose, done that. You could have said, oh, this looks just like SAMBA 3.6, just bring it over. We even had the old build system. We kept that around for one release. But no one really wanted to release a SAMBA 4.0 on Debian. They didn't include all the bits that people thought SAMBA 4.0 was, which is like the Active Directory Domain Controller and all the client libraries and things. Red Hat did though. Red Hat, they got some bits of our new advanced functionality, but none of the DC, they objected to us using Hymber Kerberos. But there's also no desire to try and keep this split up. And I don't think we could have because upstream had merged these two things together. And because we've moved to using one file server, SMBD, that we've always had and using that everywhere, we couldn't really start having the SAMBA 3.0 and SAMBA 4.0 packages be separate because they really had to be exactly the same versions and things like that. So the whole thing was basically stagnation. A large amount of work required to move forward and no one really stepping up to do any of it because it was clearly going to be months worth of work to do. And in the volunteer community, these things don't just sort of happen because one wants it to. So I stepped up. I'd done some RPMs before, so I was clearly qualified to do packaging. I figured I could learn, I was working with smart folks. And it didn't seem too hard. I had some things to start from. Netgear had had some things that they quickly hacked up and wanted improved. So we merged SAMBA 4.0. We got it into experimental in June of 2013 and we got it into unstable in October. I haven't really got many slides about exactly what the work involved there was, but it was a lot of trying to juggle which binaries to go where and trying to come up with a set of packages that wasn't insanely long yet, not just SAMBA. It's, for the first time, SAMBA has used shared libraries internally. So in the past, SAMBA essentially was statically linked. It would link dynamically against system libraries. But as far as anything that was internal, like SMBD and NBD share a lot of things in common, like loading the SMB.conf. But essentially, they were statically linked from that perspective. And more particularly, they shared the DCRPC client libraries. All the parsing of our protocols was separately in each of these binaries. And so these were now all shared. So once you've got shared libraries, you've got to put those shared libraries into packages that the right things depend on and not have dependency loops and all matter of claim. The deviant way of doing libraries is to have one library per package as the preferred way of doing it, particularly around public libraries. This turned out to be totally unmanageable. We have a lot of public libraries. And when we try to do any reasonable grouping of it, you end up just with circular dependencies. Because while the libraries themselves mostly don't have circular dependencies, there's one spot where we do, the packages, once you had reasonable groupings, the packages all did, and it just was a nightmare. So we ended up with just a SAMBA Lips package. So essentially have, which is basically which is the bulk of SAMBA. So the one that is SMBD contains is tiny because all the functionality is somewhere else in the shared libraries. And you've got this thing that has made a couple of support C files, but they end up jumping into a library. In fact, there's a big library for SMBD because we share that between SMBD and the torture process that basically does let you test some things. So it's still very difficult to split up, but we've got packages for PAMWINbind, Lips and Recline. The things that you really sort of notice on their own, and we basically don't consume directly. So we had some difficult choices to what to do about the situation in Stable. We basically ended up deciding that shipping beta 2 or SAMBA 4 to users was just dangerous. The security issues coming forward and other bits and pieces, no one was updating, no one was looking at it. So we basically emptied the packages of their binaries. So to force those packages, and I think we set some conflicts and things to try and force the packages off the system, but if you did update them, you would basically get a package without anything in there and read me saying, sorry, this has gone away. So it was pretty difficult and really shouldn't have been done in Stable. It really breaks all the rules of Debbie in Stable. And it's something we really should have had the gumption to do well before the release. We've got updated packages in, the packages were still being updated a little bit experimental, but even those were getting old. So mainly the way that we've basically told users to go forward is to then move to back ports. So the back ports do have a current SAMBA and you can install that on current Weezy. But it was a mess and telling people, your functionality's gone away, but it's over in back ports. You'll have to upgrade this a little bit. It's not how I'd like to be doing it, but at least we don't have that old code with this promise of support that we just can't make. We also have other challenges. One of the big things that Debbie is about is about using the system libraries. And so we use the system Heimdall. Because SAMBA bundles a copy of Heimdall, a very particular release of Heimdall that we then do our testing with. But we don't want to be using that in Debian. So we basically try to get our patches upstream, get those into the package that Debian's got, use then use the Debian library. Unfortunately, we have now allowed three years or so to develop between the two versions of Heimdall between SAMBA and the proper system Heimdall packages. So we've got things like segfaults in the KDC that we had to fix because a semi-private structure got changed. It's not quite clear whether they broke the rules. Well, it's clear that we're not delving into private header files. But basically, no one's really watching the ABI for KDC plugins because they're essentially we're the only one and no one's got a test. And so their patches had to be merged in with detecting which version we've got to get things right. It's still untested. Our users test it until it works. But we've updated SAMBA to current Heimdall in a test branch. I've been doing that work at Catalyst and it doesn't pass my test. So we know that there are issues that are gonna happen. And it does worry me that essentially the Debian SAMBA 4 package, we know that there are things that are breaks. And it makes things even more difficult for SAMBA 4.2 to go ahead. I had some good suggestions in the hallway track earlier about what to do for 4.2. I'm really concerned that we could have a security hole because password lockout, which is one of the features SAMBA 4.2 has, needs some hooks in the KDC to say, don't check the password if it's already locked out. Because otherwise you'll tell the user what the password, that they're still wrong with the password and let them keep on trying. We had a security hole with this. Microsoft had a security hole with this in a different area. So we probably need to actually have a runtime check that goes and tries to talk to our KDC, check the password of a user that's deliberately locked out and confirm that indeed it, we never get the password checked. We just get the check for the lockout value. Yeah? Do you have a jury bill or more like a jury run time? So the question was about whether, when we're doing this assertion about the password lockout is working, whether it should be in build or run time. I mean, I'd like to get the build time tests going, but what the suggestion was to me is that I should actually start having run time self assertions that start up to do a few self checks. So that we know that we're still running against a KDC that's actually gonna enforce the right things. I certainly don't look forward to the security release that we have because of mucking this kind of stuff up. So yes, there's some of the challenges that we've got and we're gonna have to, when we do upgrade to 4.2 it'll probably have to be combined with specially patching the Heimdall in Debian and then trying to make sure it stays that way. Password lockouts an interesting one because it's one of the few failures you can, where you can open a security hole where you don't notice it by seagull faults or those kind of things. It's just about not doing something. You're not checking a lockout flag before you do the normal thing as expected. So it's a subtle kind of bug like that unless you're really watching you can get. So I might have had a similar thing for the same kind of reasons. So I want make test in Debian. Our test suite is very comprehensive. It takes a couple of hours. I'd really like to be able to test the whole Debian package but not just as well as the KDC because of the reasons we talk about. There's some work that Red Hat's been doing to try and get make tests to run against a difference. They're code-based, they want to use MIT Kerberos and so they changed their make tests to work from something that worked on compile. We would actually override all the socket functions using a header file and a system called socket wrapper and we were able to create an entirely fake networking layer inside Samba and a number of other fake layers. One of the guys, Andreas Snyder from Red Hat has done some great work doing that with LD Preload. That gives us some great advantages because while you don't want to compile your production packages with socket wrapper, you certainly could allow them to be preloaded and so that could potentially allow us to run make test and then to be able to find these kind of issues at build time. Now, running a make test with about 10% failure rate in an automated build environment, of someone else's automated build environment is a bit challenging but it still gives us a lot better, more confidence that if it passes that that we're still actually still not screwing up. That's one of the things I'd like to go and I'd like to understand whether the Debian build and test system is actually up for doing a two hour long make test. So long-term implications of all this, we missed the boat on Weezy so we've still got an old version there. So there's no upstream support for functionality issues in the version of Samba in current Debian stable. There's even, there was for the first six months but often it's gone. Now it's in security support but it's about to leave security support as soon as we release Samba 4.2 in the next month or two. Do we, what do we promise our users? What resources do we have in the Debian packaging team for resolving bugs? I had people complaining about an issue with NTLM auth and machine accounts and it was fairly esoteric. We did some testing, it's gone on current master. We're not supporting that version anymore from a Samba team perspective. What should we be doing from a Debian perspective? What should we be telling our users about the software that's in stable and they think they can file bugs against? How do we communicate to users that the version of software that we've got in Debian stable is actually so old that it's not gonna be fixed because most bugs are too complex for the package team to deal with and upstream it's too old. Now it turns out we're a little more flexible now in that if you are, you may well find and get people to patch that. We could patch it up in Debian if you're really, really keen, if it was really bad enough. But does make me a little bit worried about just having let things get that far behind. Or do we basically end up having users basically go, oh, Debian is too old, use back ports or use a third party is the only way to get actually a supportable Samba. Is that the way we want to be dealing with core bits of our system? I'm not sure. How do we keep Samba up to date for our users? Even when Samba is in a mode of, we only make fixes to Samba that have a bug filed against it in our bug system as upstream, it seems to be very difficult to get those updated versions of Samba into Debian because Debian will only allow an upload if there's a Debian bug for every change that's in there and generally won't allow a version to be updated. Even the Samba team is doing a really good job of saying only issues with a bug attached and reviewed by two team members can be in new versions. I think that we're letting Samba get too far behind and we're not allowing our users to get the fixes that have been developed elsewhere and have considered production ready by the Samba team but Debian Stable is so conservative that we can't even do that. And I think that lets our users down. And how do we do this better next time? I mean, I hope Samba doesn't fork again. I really hope Samba doesn't fork again. That was not fun. How do we support file server and active directory user expectations? How do we try and deal with the conservative and the progressive in the same package? Should we have done the old libraries over here, this functionality there, separate projects, separate allowing things to skew against each other? Oh, you can use the old file server with a new this and that. That sounds insane to me as a developer trying to allow all manner of combinations particularly at the distributions discretion. I really liked the world we're in where we really forced know you'll use the current version. That's the one we tested with. I don't think we should let pre-release packages escape into Stable. I think we should have found them. And where we had to leave that because otherwise we were blocking other people's important work. I think we really should have cut it down to exactly what was needed by those other packages and not a thing more. As otherwise, we're just making just promises that are just really unfortunate. So everyone's been very quiet. I hope you found this interesting. These are things that we do at Catalyst. If you're interested in working on Samba and things like that, if you know see Python Git networking, Wireshark and don't think that list is getting too long already then please catch me in the hallway track. I'm trying to convince the bosses that we have a stunning candidate who wants to join me and increase the size of my Samba team and it'll be a lot easier with a name to point to. But Catalyst has been a really fun place to work for me for the last 18 months. They brought me over to Wellington to work on Samba and I would very much encourage the smart folks in this room that are looking for somewhere else to be that Catalyst might be somewhere you should think about joining. Okay, any questions? Hi, I have a whole bunch of questions. Oh, please. First of all, what was the history behind the earlier fork of Samba? Why was that happening? Why did Samba forked for the first place? So Samba 4 took off after the Samba 3.0 series. Sorry, I was referring to the original fork that was the TNG, that's right. Why did Samba TNG? Most of that history is before me and I won't really try to go into that. That was before I joined the team. But Samba 4 forked off for reasons that were relatively similar and that conservative versus progressive stuff was a bit of it for Samba 4. The file server area was getting very conservative and very resistant to change. And at that time, we thought we were building the next version of Samba. But we bit off far more than we could chew. Far more was rewritten rather than doing incremental rewrite and get it ready for release. It just kept on going as to new and fun stuff in an area that wasn't aimed back at releases. And then a strange thing happened. At that time, those who were working in Samba 4.0 were the only developers working. But one developer who had a lot of free time from the way his company had restructured was able to, he just decided to go fix some fixings in Samba 3.0 and he really got some momentum going on there. So suddenly we had two branches with momentum and that then meant there wasn't any clear pattern for where either was going. And it was just very poorly managed in hindsight by the team. We didn't really know what we were getting ourselves into. You know, we were in Dunedin and doing a first demonstration of Active Directory DC. We thought we'd be finished on that shortly after that. You know, we thought it was. And so there were lots of, but that's how Samba 3.0, 4.0, 4.0. But the 10G ones, a story for a different day. A smaller audience. This is a question. I don't want to hold the microphone if somebody else wants to ask a question. This is a question more related to the community aspect of things and you might not have an answer. But as a Samba user and systems administrator, it can be a little frustrating getting information. The wiki is good for what it covers generally, but some parts are a little out of date and it tends to be IRC or the mailing lists that have the best, most responsive place to go. It seems unusual that forums these days are normally a place to go for that kind of information, but there is no official Samba forum. Do you know if there's... Indeed, there isn't. The same team is really quite conservative, in many ways. And most of us as developers just are not interested in trying to run a forum. We work on mail. We're used to dealing with mail, you know, all these modern social networking things. We have our instant chat on IRC, we have email. It fulfills our basic needs. So really it's a developer conservative thing, so we didn't want to set up a forum where none of us would be there. So you can use... There are online interfaces that make our mailing lists look like a forum if you really need a web forum kind of look to it, but that's why we don't have a forum. My final question was related to the file server inside the early versions of SAM before. Could you elaborate on what features that would have been... Yeah, so the file server that was in early SAM before, we now call the NTVFS file server, and it was structured where the SMBD has a VFS layer that most closely resembles POSIX with a boatload of extensions on it for the things that POSIX can't do and where we've... And so it's a mix of POSIX and then some NT semantics where you can't represent them by POSIX. I should clarify, when you open a file in Windows, in Unix, you've got open, you've got three arguments, right? You know, the file name, the mode, and I think something else, you know, it's really simple. On Windows, you've got about 26, and so there are a lot more semantics that you're passing inside that open call, and they have to be atomically operated on in that open. So you get share modes, which are a way of saying no one else can touch this file, absolutely. You can set ACLs if you're getting a new file. You set what access bits you want. All manner of things can be done in that one call. So that all that gets put into and passed down to the VFS. So the Samba 3 layer tries to be like POSIX, but with these weird exceptions, a bit of a dog's breakfast, but it's relatively simple to understand because you've got open, closed, read, write, and a few odd ones. Samba 4 tried to say, well, we know that our protocol is really a thin shim on top of NTFS. So why don't we actually provide a protocol layer that with a full richness of the network protocol is passed all the way down to the module to deal with and so that we can implement, for example, a full SIFS proxy, and we have one, that takes every packet and maps it all the way down to C structures and sends it back out the other side. This should allow a file system module to implement the full semantics of NT if it's got a richer file system that can do special things because it can see the whole request, not a dumbed-down version of the request. And so it's a really good architecture for designing how Samba could be done. It was what was essentially how likewise, which was a competing S&B server, got implemented. But because so much was written from scratch and there were a lot of VFS modules that S&BD had, there wasn't, everyone said, oh, you'll never finish it because you'll need all the VFS modules that exist will have to be ported in all manner of other things. I actually wish I'd glued the two together. I reckon we could have had one on top of the other. But we, yes, that was the difference. A real different architecture. One of, we'll try to be posix plus extensions. The other try to be windows in, the windows full richness built as the layers. And so that was the difference between the two servers. And one was rewritten from scratch and one was the old one, been with a lot of really good refactoring over time. And the S&B2 file server was first built in Samba 4, but then that work was done in the Samba 3 stuff. So yeah, and sadly just didn't continue. I think it's a great architecture, but it's not anyone maintaining it. It's not got, it's mostly a nice model for where we might go in the future. So there's no plans for it to make its way eventually into? No, it remains basically a developer reference for how we could have built something. And the SIFS Proxy is really interesting because it's the only SIFS Proxy that I'm aware of. But it doesn't do S&B2 really well. It doesn't do S&B3. It's just a snapshot as it was seven years ago really when we moved from doing file server development to doing AD development. There's a great talk if you wanna understand about it all the way back in Perth, which which talks about a new VFS for Samba. And that was a long time ago, but that's when this design was first being put together and explains why we're trying to do it. So yeah, other questions? I'm more the user sort of things wanting to know how to learn more about the inner workings of Samba. Because sort of interfacing with Samba and I keep bumping across it just not working. So how do, what's the best way to learn more to try and be able to fix the problems myself with it? Turn up the debug level and just start diving around the code that's related. It's a big and difficult project to get into but that's where I'd start is just crank up the debug and just try and understand what it's doing. I don't have much more I can really suggest to a short question and answer like this. Catch me in the hallway track and tell me a bit more about what you're trying to do. Okay. One question back. My question is regarding with the Samba for trust if there's any updates on because... So trust support from is something we are working on. We have had some parts of it actually for some time but we haven't advertised it because we knew it wasn't finished. Stephen Metzmarker has just demonstrated doing an Interforest Trust between Samba and Free IPA and we thought that's really interesting because let's do some really interesting bridge work between the Unix and Windows worlds and those leading sort of servers in each of those markets. So we did a lot of work to use the Samba 3 windbind as being the only windbind going forward and the reason we did it was so that we could get further on trusts. We haven't really enabled any of the other features but we got it. We did that work because we knew that was going to be that was the blocker for doing trusts. So nothing's official. Nothing will be in Samba 4-2 yet but some slow progress is being made. There's a surprising amount of it working under the hood but we haven't been willing to start telling people it's there until we finish some things but it's not like there's no code and no hope. It's more that we know there's still more to do. So maybe you'll see it in 4-3. Best way to do it is to step up and help us work on the development. That's, you know, there's lots of stuff. Sammy's hard to start on but the only way we're going to get forward is if we get more developers interested in things and I'm very happy to help new developers get up to speed. Any other questions? He said, oh, in an hour and a half. One last question before we go. Are you aware of any NAS vendors or distributions that are working on getting Samba 4 into their products? Well, Samba 4 is in many products but as the Samba 4.0 file server and the 4.1 file server. In terms of, if you ask them about the Active Directory DC, I'm not aware of... FreeNAS, I think, was looked at at one point but, you know, there's... I'm not aware of any, you know, physical buyer box vendors doing it that way. You're better to install it on a generic distribution. Anyway, thank you. Thank you, Andrew. So next up, we have got Angus Lee's talking about getting the Rust programming language or bootstrapping a new language into Debian. So there'll be a slight pause as you get things set up.