 Alright, so I'm going to be talking about packaging Tomcat for Linux distributions. Most of this is going to focus on RPMs, probably because I'm a co-maintenor for Fedora, but in light of the last talk, talking about the Windows distribution, I'm going to also mention one of the Red Hat products, the J-Balls web server. We ship RPMs for Tomcat for that and Windows and Solaris, so you can talk about that because it's packaged a little differently than the Fedor ones. I effectively just said all this stuff. I'm going to briefly go over the Apache Tomcat distribution, making the assumption that everybody knows how this laid out and how that works. I'm just going to mention what happens there just so I can draw a correlation between the two. I'm also going to briefly mention Snaps, because someone on a free note talked to me about it and it was sort of interesting, so I'm going to try and see if anybody's using that. Alright, click about me. I work at Red Hat, obviously. I supported Tomcat, HDPD, and J-Balls EAP, which is an enterprise application platform, mostly J-Balls Web, which is a fork from Tomcat from forever ago. I supported that for about three years. I've moved around inside of Red Hat, so I worked in the business intelligence group as a business intelligence engineer doing ETL and data warehousing. That was radically different than what I was doing before. I didn't like that, so I moved back. Now I'm in engineering, and I'm a committer for Tomcat, and also a co-maintainer for the Fedora Tomcat package. I maintain the real Tomcat package, and also work on the J-Balls Web server Tomcat packaging. They're all slightly different. I've been using open source since right after college, which sort of dates me a little bit. Let's see. Yeah, I'm a recent committer of Tomcat. Just got Commit Rights last year. Oh, I'll put an interesting fact about me. Something that most people find interesting is that I'm Native American. A lot of people, I guess not here so much because it's more international audience, but at home around Raleigh and stuff, I always get weird questions about my ethnicity. I don't really understand why, but... Oh, and I also apparently have a strong southern accent. Lucas isn't here to heckle me, so that's nice. If I start talking faster, it tends to come out more, so I'll try and slow down. Yes! How's the goal? Okay, so Hoaxing yesterday was talking about building Tomcat with Maven. We had a need to do that. One of our build systems uses Maven, and we needed to build Tomcat with Maven recently. I wrote this wrapper, which basically has a bunch of different sub-modules that are all named really funny. It's a whole bunch of directories, and they all just have palm files on them. It uses the ant run plug-in. This is just a wrapper of the ant build XML. That sort of decoupled me from having to maintain my own stuff with Maven. Now, if any of the targets change, I can just change it right there. I don't know if you guys can see that. I build the package sources, the embedded jars, compile web apps, all that stuff. And then some of the other of these guys, the assemblies, package stuff differently. We don't ship the examples in the JWS distribution, so we strip those out and package them separately in a zip, and we use that zip for QA testing. QA uses the zip for testing and stuff. So if anybody wants to talk about how we did that, we can do that after the fact. I thought it was sort of a novel thing. Okay, why use distributions? I break this down to a list of pros and cons, and I try to anticipate the questions that were going to be asked. Using Tomcat in a distribution isn't a popular method, I guess, for most ASF members. Everybody tries to drive customers to using the ASF distribution, or at least reproducing problems on the ASF distribution, which basically removes all the potential for problems inside the Linux distribution or whatever. So here are some pros. Packaging Tomcat or using Tomcat from a distribution makes it easy to install and maintain. You can install the package just like you would like anything else instead of having to go to the internet and download it. The distribution tests Tomcat with all the libraries, all the other libraries that the distribution provides. So like Log4j, for example, is one you can easily just point to the Log4j jar that Fedora provides and use that. Systematability. This one's sort of interesting. This mostly applies to RHEL. For Fedora and for JWS, we upgrade to whatever the latest revision is. So recently I upgraded from 7075 to 77 on the Extras library for RHEL6. I generally do that just because it's easier for me. But RHEL, for example, Rhett Enterprise Linux, only backports only backports specific bug fixes that customers request or security fixes that meet the criteria for being fixed. And I say that because RHEL6 is older. So only critical and important CVs get addressed there. But RHEL7, we fix everything. And then one of the things I thought of this morning was support for EOL versions. So Tomcat 6 is EOL upstream. But it's included in RHEL6. So we have to continue to provide support for that. However painful it may be until RHEL6 goes into life. Sure. Yeah, I'll talk about that in the Fedora section. Alright, cons. I'm sure you guys have a million more. So one thing that I forgot to mention before is that I'm not a Tomcat user. I don't have any systems that are running Tomcat presently. I only work on Tomcat and packaging Tomcat. So a lot of the use cases that I know of and that I care about are ones that our customers use and they care about and also the community. So I may be a little more detached than I want to be from use cases and such. I just wanted to throw that out there. Alright, so obviously whenever you add another layer between Tomcat and the end user, there are potential to have bugs there. There are actually quite a few of those in Fedora until recently, but I think most of them are addressed now. It's difficult for developers to use. If your developers don't know how to use the operating system and don't have any administrative skills, then they won't know how to install, start, stop, tech logging, all that sort of stuff. So that requires specific skills for distribution. And one that I thought of yesterday was that there are configuration changes between your visions, then those don't get included in the update because we mark configuration files so they don't get overwritten whenever you do a YUM update. So if there are any changes between the two versions, additions or removals of attributes or whatever, those don't get included so you'd have to manually do that. I try and protect against that as much as I can, but if something was removed, I can't remove it for you. Now without clobbering your whole config. Okay, so distribution overview. Hmm. Yeah, it's definitely noteworthy. And the Tomcat community does a really good job of linking commits together by revision saying this is a follow-up for that, this is a fix for that. So it makes it sort of easy to kind of backtrack. Always do that whenever I grab a fix. Always check and make sure that revision is not listed anywhere else or if that VZ number is not anywhere else or whatever. But yeah, that can happen to you. Okay. So all the distributions that repackage Tomcat go through these four basic steps to download the sources and build Tomcat and any extras that you need. We don't distribute the Catalina WSJR so we don't provide the Web Services implementation stuff. So we don't build that. And then they add any OS-specific stuff like service scripts for RHEL, their SystemD for Ubuntu, their SystemD scripts. And for RHEL also there's a couple of configuration files that give you the equivalent feature set that you would get by using setM and I'll talk about that some more in a minute too. And then they distribute the package with their package manager, for example. All right, so I said this is going to be brief. VZ Tomcat distributes a self-contained archive and then there are individual archives that you can also add later. So the API documentation, the embedded Tomcat jars and the logging adapters for log4j1.x they're built and provided in separate zips that you can get. Right, so now I'm going to talk about my favorite. So Fedora packaging. I'm just going to sort of breeze through this stuff. I'm doing pretty good on time so this is what I'm going to talk about. So I don't know. I didn't really, I guess I maybe made some assumptions about you guys' level of knowledge on packaging or using a Linux distributions package but the RPM format is what Red Hat flavors of Linux use. Fedora, Red Hat Enterprise Linux, CentOS and there may be some other spins I don't know about or work with regularly. Everything you do know about an RPM is defined in the spec file and the spec file is used to build the RPM package or packages. In the case of Tomcat, it has a parent package and then there are a whole bunch of stuff packages underneath that I'll talk about in a second too. I have a link to the spec for Tomcat but it's like super complex. I'm also a code maintainer for Tomcat Native and its spec file is much simpler. So this is what a spec file will look like. So you have these things that are required to build the package so you have Java to build, APR, OpenSSL, etc. You get a description that you can see if you run like YAMEMFO or RPMEMFO and then you go through the steps of building. So there's a preparation step, build, install, clean and then you can define things to happen, post install or post uninstall or pre uninstall and so on and so forth. And then basically it just here, it puts the files where they need to go. So Lidder which is a macro for user share and then it drops the shared object there. One of the things that is done in the Tomcat package is that we sort of amend the library pass so that it knows where to find Tomcat Native. So you can do YAMEMSTALL Tomcat, start it up, everything's fine. If you want to use Tomcat Native, you just do YAMEMSTALL Tomcat Native, restart Tomcat and then it picks up the APR libraries. Like I said, the Tomcat spec, we can look at it real quick, but it's huge in comparison and does lots of scary stuff and has lots of bugs probably somewhere. But yeah, so you have lots of sources. These things are specific to Fedora, got a JSVC service. These three are wrapper scripts that do the initialization and start the daemon. Then you have a name service for multiple instances, a couple patches. We build with Java 8 so remove the compiler options for v9. That is OSGI. Are you generating OSGI component services over Java? Yes, I think so. Excellent. Okay, bring it on. Oh, okay. Well, I don't personally know how, but these existed before. But yeah, cool. So then you have all of your build requirements and these are required for build time. And then these things are required whenever you install and it recommends installing Tomcat Native. It doesn't actually do anything, I don't think. It may tell you that you can use Tomcat Native somewhere, but I've never seen it. Okay, so yeah, this thing is super long. It does all sorts of stuff. It builds maven things for whatever reason. Okay, so back to this. Alright, so packaging breakdown. This list here is the Tomcat parent package and then all these are sub packages which basically means that we don't include all the libraries in the Tomcat package. They're all in Lib. We break up the web apps so that you can install a doc web app separately, the Java doc on the system. You can install JSDC support separately. This is the root and examples web apps. This really only applies to Fedora and maybe Ubuntu, but we don't install the examples web apps on the JVolts web server distribution and then the admin web app. So you notice that there is a Tomcat Lib which has all the libraries, almost all the libraries, and then there's JSP, EL, and server API. They're all in their own individual packages and I'll talk about why that is in a second. Okay, so directory layout. This is sort of where all the things go. It's different than the ASF distribution because we sort of put things in weird places. Well, weird. So the weird places are defined by the file system hierarchy standard. So examples of things like that would be executables go in user band, global configuration files, and Etsy. Log files would be on Vara log, like these. So user share Tomcat is the cataloging a home for the Tomcat installation, and then these are a bunch of sim links that point to the various places. So if you really wanted to, you could just refer everything to user share Tomcat and the sim links will take care of everything else. Okay, so back to the API sub packages. I'm not even sure this is really still used, but update alternatives on Red Hat Linux updates links to things on the system. So for this example, I picked servlet. So if you refer to Etsy alternative servlet, it points you to the Tomcat drawer, which is right there. So you can install the servlet API without having to install the Tomcat RPMs in case you wanted to build a servlet or something like that. So that sort of frees you from having to install the entire Tomcat distribution or any of the other Tomcat libraries. You can just install the servlet API, the EO, or JSP API RPMs. It's a lot of letters. Right, so to install and configure Tomcat, use the package manager. For Red Hat Flavors of Linux, it's yum. You can do yum install Tomcat. This will give you a working minimal Tomcat installation. It includes Tomcat, configuration files that it needs to run, but it doesn't include any of the web apps. And I think that's it. I think it just excludes the web apps, which you can install separately. For example, I picked the Tomcat admin web apps. So if you wanted to install the manager and host manager apps, you would just do a yum install Tomcat admin web apps, restart your Tomcat instance. And there you go. Configuring Tomcat is like I mentioned before, the Red Hat Flavors introduce to configuration files. So you have ... These two things give you the same capabilities that you have with Setem, the shell script that you would like to find some stuff in. For example, you can enable the security manager by putting it in either one of those. So there are two because if you wanted to have multiple installations of Tomcat you can copy some things around and you can copy the service script and you can still use the operating systems facilities to start, stop, and configure that. I always get these confused that I write it down. Yeah. So, as the Systemfig Tomcat overrides any of the properties you define in Tomcat Conf. So that is because when you make a copy of the service script, you get your own copy of the Systemfig configuration file, but the Tomcat.conf is global. So all the services use Tomcat.conf and they all have their own individual Systemfig files that will override any of the properties you put in the Tomcat.conf. Okay. Starting and stopping the service is pretty easy. I've got examples on the next slide. Yeah. So there's a start, stop, and status commands for SystemCTL. This looks kind of wonky, but this is from Fedora25. So, in previous versions, whenever you start or stop Tomcat using the system v scripts, it dumps all the standard out, standard error, and the Catalina died out. And Fedora, like 20 and later, we moved from sysv to system d, which sort of changed the way things are logged. Standard out, standard error, and a system d unit goes to the journal, to the system's journal. So that's why you see the stopping logging in here whenever I check the status of it. So system d prints starting and stopping message and stopped message, and then any messages in between that it gets from standard out, standard error, it dumps to the journal. That allows system administrators to add. Yeah. So this is the, it just uses Julie by default, the Tomcat Julie logging. So we don't change any of the way that it logs. But instead of, in the sysv or nit scripts, the standard out and standard error is routed to Catalina.out. We just don't route it in the system d wrappers. So it just goes to the journal. That lets your administrators, like, pretty easily check what's going on with their service status that they wanted to, instead of having to check Catalina out. Right. For the normal standard error, standard out, they're regular logging. Yeah. Oh, well, it still logs to the Catalina.date.log file inside of our log. Just anything that would be standard out, standard error goes to the journal versus Catalina out. So if you run a vanilla Tomcat distribution now, you start it up, you get Catalina out and Catalina.date.log, instead of getting Catalina out and the other log in this, all the output logging goes to the journal, but then you still have your standard logging in the log file. Does that answer your question? I guess I was asking why the standard logging wasn't also put to the journal. Yeah, I mean, I wouldn't expect, you know, in this case, whenever you're troubleshooting a problem, I think it's easier for you to see whatever snippets of the thing that you've initiated and the journal versus looking at a log file for it. But if somebody thinks that's a change, I'm moving to fixing or correcting that. Okay, so there's been a lot of time on that. For updating, you can just use the package manager again, yum update, which basically what that does is it removes all the files that Tomcat installed previously, except for your configuration files, or files that are marked as configuration files in the spec. And it installs all the new jars. So you remove all the old stuff, and then you get the new stuff in an update. Like I mentioned before, it's one of the cons. If there are any configuration changes, it's up to the user to make those, because I don't want to clobber whatever configuration you have in your server.xml whenever you do an update. So I just don't touch that. And then removals, pretty self-explanatory. Yes? Oh, yeah, yeah. Yeah, I suppose I could do that. Okay, so for contributing, there's not really a whole lot going on as far as bugs go. But, I mean, if you guys have any, feel free to jump on their log one. If you want to fix it, jump on there and fix it. The fedora infrastructure in my opinion is a lot easier to understand than the Ubuntu one. If you were to look at the package overview here. So basically everything that you need to know about the package and what's going on with the package you can find on this page. These are all the different distributions that we are currently supported. Apparently I have a version in test. And then, for updates, stop spinning. Okay, anyway, apparently that's not going to work. But for updates, I have to submit a new update for each one, for each of the operating system. So I have to submit an update for master, well not master, Fedora 25, 24, 23, any of the versions that are supported now have to submit individual updates for those. So if you're ever curious what the status of an upcoming Tomcat update is, you can go to the update system and check it and see where it is. If you have a Fedora account, you can also give me karma. If you get plus three karma, then I can push the stable sooner than the mandatory week or two week waiting period, depending on which distribution it is. For the extra packages and rel, it's 14 days. So whenever an update has to sit in the testing repository for 14 days, unless somebody gives it plus three karma, and then I can promote it to stable. You also still have access to the testing repositories if you guys wanted to. You can just enable that on your system and install the test packages. Those aren't actually past QA yet, so if you wanted to test them, you can do that. Okay, on to something I know slightly less about. Ubuntu distribution, I'm going to draw some comparisons to Fedora. The majority of this stuff is going to be the same except they don't include the alternative stuff I don't think. Oh, and one thing I forgot. I didn't mention on the pros of using a distribution that I intentionally meant to do. SCLinux, I don't know if you guys ever used SCLinux before, but if you installed the distribution package of Tomcat, you get the protection that SCLinux offers through the SCLinux policy for Tomcat. So your Tomcat instance can only bind to specific ports, can only write to specific files, so on and so forth. That is, as far as I know, also available in Debian. I think by default Debian, Ubuntu installs AppArmor instead of SCLinux, but you can switch it if you want. And I don't know if there's an AppArmor policy for Tomcat. I'll check on that. Okay, so as you can see here, AppGet or App is similar to Yum's package manager. In this case I grabbed the package so I can inspect it. It is pretty much the same thing. Just in a different format. So these are all the packages that are available for Tomcat 8. Ubuntu offers on Fedora, there's only one Tomcat package and then it gets updated as we go along. So at some point it changed from Tomcat 7 to Tomcat 8. On Ubuntu, there's a Tomcat 7 and then there's Tomcat 8. And I think they included 8.5. Well, I know they included 8.5 somewhere because it broke the free IPA server but I don't know exactly where that was. So this is similar to what we had in the Fedora distribution. So you got the admin package here if you want to install the administrative web apps, docs, examples. And then this thing is, if you wanted to run multiple installations it gives you some sort of facility to do that if you install the user package. So Ubuntu also follows the file system hierarchy standard. The Catalina home is barlib. Tomcat 8. And for some reason they use relative pass further some links instead of linking to like var log and var cache. I'm not really sure why that was done. But okay. So Ubuntu and Fedora, like I said, they're virtually the same. They both follow FHS and they both offer package managers that you can use to install, update, or remove Tomcat. This is how you install. If you guys are getting tired of seeing the same stuff I can breeze through this. So one note about the difference between Ubuntu, at least Ubuntu 16 and Fedora 25 is that Ubuntu moved from sysv to systemd. They offer a systemd service but it just wraps the sysv stuff for backwards compatibility. So if you notice the status command here is a lot less verbose than the status command on Fedora distribution was. Because everything is going to Catalina out instead of going to the journal. Update and remove. So a quick note about contributing. I was hoping that Emmanuel was going to come because he is a maintainer and recently knew Tomcat Committer also. I'm not a maintainer and I'm not really involved with the community very much other than talking to him sometimes. So if you wanted to contribute there, this is sort of what their infrastructure page looks like. And maybe you'll see why it's confusing. Oh yeah. Okay. So compared to the Fedora package overview, it's very plain, I guess. I had a really hard time trying to find where the sources for the package was. So that's sort of, I can only really find the summary of the stuff. Okay. Not bashing them or anything, it's just different. Alright. So snaps. I said a brief overview and then I wrote all the stuff. I just wanted to stick some links up here for you guys to check out. I don't know if it's anybody using snaps or ever heard of snaps before. Okay. So snaps is basically a platform agnostic distribution of packages. So in this case, which is kind of weird, I don't really understand how this works, but it's given to Michael and honorable mention here. His blog is at the bottom. You can read all about it if you want. But basically it's just another distribution of Tomcat that you can do on any system instead of having to worry about the platform specific thing. It's sort of like just a big tar ball. Everything is self-contained, but instead of having to go and download it from the ASF, you can say snap install. And it uses the system D facilities for logging and such too. So I don't know if you guys are interested in that or not. I didn't see anybody raising their hand. So I'm going to skip over that. Alright, discussion. And I have 15 minutes. So like I said at the beginning of this talk, I wanted this to be more of a discussion than just me talking about stuff. So ask away. Chris? Yeah. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Hmm. Yeah. Yeah, I think maybe if we did like XML includes, and I laughed when you said ConfD because there is now a ConfD directory in the Fedora distribution of Tomcat. But it's because system D doesn't allow for shell expansion. So instead of sourcing Tomcat Conf, there's a thing where it doesn't really have a shell and it cries if you try and use a variable in there. So the ConfD directory just sources or the wrapper script sources, whatever is on ConfD and you can do shell expansion that way. I haven't thought about trying to break up the server XML because I don't really think it's that big. And I mean, I mark it as a configuration file. So install time, install it once. And every time you update, I just leave it alone because I don't want to clobber wherever you have. The WebXMLs of like the Manager app and the Who's the Manager app, Docs, all that stuff is marked as config. Yeah. Hmm. I don't know. I mean, I don't, I don't mean I don't really care because if it's done, I may probably be helping to do it anyway. So it made a lot of sense. I'll say package that too. So it made a lot of sense to do it that way. But for Topcat, I don't know if we want to spend cycles doing that when there's other things to do. Unless people think so. We can always pull the user's list and ask if people will think that that would work for them. But as far as who does it, I don't care. So there's cases to run multiple instances of Topcat. How much do you look at supporting parameterized system B scripts? You know, you can put a parameter in the file name and then inject that into maybe the name of the big file inside the system. Hey. I'll see. So we have these Topcat name services. So whenever you get ready to have another instance, you can just create a Catalina-based directory on the varlib Tomcat's thing. Oh, that hurt your eyes. Sorry. Oh, really? Oh, come on. Dang it. I always forget where the server scripts live. Can you see that now? All right. Yeah, we use the name variable here for named Tomcat instances. So we sort of do that already, I guess. I'll see a bug. Okay. Do you have a question? Any more questions? Yeah. So the Red Hat stance is a little different than the Fedora stance. So with Fedora, I just update, like every month whenever we release an update, two-ish weeks later, I push another update up. So I mean, nobody's cried too much about me doing that unless we introduce your regression, because the only group that depends on Tomcat that I know of at this point is the free IPA folks. They use it to host their IPA server stuff. So sometimes whenever I bump a revision and a regression creeps in, they get really upset about it. Yeah, so all the bills are usually billed one or billed two on the RPMs, but they have the Tomcat version number. So that's what it will look like. Just eight or 40. The only time you would really have to figure it out is if you see the build release number here increasing. If the release number is increased, then there may be a patch that is for Tomcat, or it may just be a patch that was for the wrapper or whatever. I feel like I was going to say something else. Oh, I was going to say real quick too. So the Red Hat policy for backporting is different than the Fedora policy is. Especially for the production phase two and three products like REL6. So for REL6 we only push out critical and important CVE fixes. If there's a customer that has a problem, like a bug, we can push that out too. So we don't just say we only fix stuff whenever there's a CVE. Like if you had a problem with your REL distribution, or if you wanted a feature that was included in the later version of Tomcat, we would like pick that in, so I mean it's not super difficult, and then push out an update. Not generally. For the case that Mark was talking about earlier, I guess there wasn't a good commit note. I think that was somebody's phone. I don't think there was a commit message that linked the two together because they were separate. I don't remember if I got it before or after they did, or maybe it was because we're on a later version. I didn't have a problem because I'm not really sure what. So I was going to say it. I think that the way that we're releasing versions now is probably the optimal solution. As long as we continue to clearly delineate whenever there's a bug, like this fixes this bug, if there are any follow-up commits, point to the original one. Because in cases that the same code wasn't changed twice, the patch may be tweaked to apply, but you're missing whatever it was. Generally I try to include as many patches to make it apply clearly as I can. Just do some review, see if this is a functional change, or if this is a syntax change, or whatever. And then if it makes sense to grab it, or if Remi tells me to grab it, then I'll give those. A lot of times she tells me not to, because it's a very factoring, which is kind of funny. Any more questions? No, I mean I couldn't... I couldn't tell you who uses the RPMs, or if anybody even uses them. And I think at the Red Hat level, that's not something that we can easily track because of the way that the system updates are pushed out, and they're sort of locked down so that not everybody can just jump on our repository and download them. So I don't really know that we could even publish that sort of stuff. We used the large distribution because we had to support multiple platforms. So we downloaded that and we continued running that. We simply moved to 8.40, but immediately we got some... So then... Yeah, so that's another oddity of using a distribution like Rail, for example. Fedora, because we track the same version numbers. It's not really a big deal, but we have a lot of customers that come in and have a list of CVs and say, my security scanner says that I'm vulnerable to these 100,000 CVs. What's wrong? And generally, we have a portal for CV issues, so if you wanted to go look it up, you could go to the customer portal, look up a TV, there's a nice little table that says this is affected, it's fixed in this version or whatever. But yeah, security scanners don't know that. They just check the version and say, your version's like five years old, so you should probably update. When in reality we've been backporting bug fixes and security fixes in there, incrementing the release number that I mentioned earlier and pushing those fixes out to customers. Should I put something in? I do not. I haven't heard of doing one. I've got 30-ish seconds. Anybody else have another question? So I noticed when you showed those versions that you were supporting in the store, was that listed? Is that because we already have seven now and it includes Tomcat? So Tomcat's included in Rail 6. The version that's included in Apple is later than the Rail 6 version is. So Rail 6 includes Tomcat 6 based on 6.024, which is ungodly old. But the Apple 6 distribution is wherever the latest Tomcat 7 releases. I guess the reason why there isn't an Apple 7 is because Rail 7 uses Tomcat 8. There isn't a release after Tomcat 8 except for 8.5 for right now. So I might push Tomcat 8.5 version into Apple 7 at some point, but I don't know. Do you have new features offering them for the retail policy of trying to keep the stable version for a long time and only patching for security and stuff? Is that a problem with the right new features coming out of Tomcat? Generally no, I don't think. Customers just open a bug and say we need this fixed. I don't see too much traffic for Tomcat bugs on Rail. Actually, 80% of the bugs that I fixed for Rail are ones that I filed because I found problems. Hmm. So yeah, that won't make it into Rail until like Rail 8. Well, we also offer the J-Balls web server product. It's a bad name, honestly. At this point, it's really just Tomcat. So if users wanted a later version of Tomcat than what Rail offers, they could purchase separate entitlements that provide the JWS distribution. Right now that includes Tomcat 7070 with some CVE fixes on top of it and 8036. So you can sort of pick which version that you want to run, but it's a separate thing. I don't see how we could backport HTV2 support Tomcat 80. So it won't be included. But as far as Apple goes, so the extra packages repo, that's not a Red Hat supported package. So I can do whatever I want to there just like I do for Fedora. So I can just continuously push updates to Apple 6 and give customers the latest if they want it. I don't think so. I mean, it's generally pretty easy for me. Like I said, I'm pretty vigilant about checking commits and making sure that stuff makes sense and that things are included. I also have access to the QE test suite. So every time I backport something around the QE test suite, I'll run the unit test. I have a Jenkins box that has three jobs for each version or each branch of Tomcat at this point and runs it nightly in my house. And that also runs OpenSSL, Tomcat Native. I think there might be some HDPD tests in there. I mean, if something is broken, I don't really know about it. And as long as we continue to adequately comment on commits and revisions, I'm happy. Is there anything else that we could be doing to help out the Tomcat community? I mean, I was hoping that developing this presentation and it's super worthy like this because I intentionally wanted to be able to point somebody to it and have it decouple from a talk and say, this is sort of how this works. And I generally try and answer those. And most of the people that come into the Free Note channel aren't asking about distribution specific stuff versus sending emails to the knowing list. I think I'm kind of whipping people in to come here and ask me questions about stuff. Because the answers on the usual lists are usually, yeah. Okay, so I'm over time. This is my information. Also, pound Tomcat on Free Note. If anybody uses IRC, use it for work. So I'm always online. That's it.