 So let's get started, nine o'clock. So welcome to How I Stopped Warring and Learn to Love Open Source, much like when a new team kicks off a new project, the first thing you do is design the T-shirt. One of the things I do when I decide what I wanna talk about at a conference is come up with an interesting title. For some reason, Dr. Strange Love kept on sticking in my mind, and for those of you who do not know How I Stopped Warring and Learn to Love the Bomb is the subtitle of the movie, Dr. Strange Love. It was actually toying of uploading that picture as my head shot to Apache, because I think that would've been pretty funny, but didn't know if that's strained the fair doctrine use. So what I'm gonna talk about, I'm gonna talk about, first I'm gonna let you know a little bit about where I come from. Gonna talk about why we made the move to an Apache Tomcat-based application server for our product. Gonna talk about some of the extensions we've made to Apache in order to run in our environment, and I'm gonna wrap up with some lessons that we learned in the process. I've got more slides here than I'm gonna have time to present, and I think the demos will be a little more interesting, so some of this stuff I'm gonna skip over pretty quickly. Also, I do have some very hairy technical details about the stuff we did to spring security that are hidden from the presentation, but are included in the slides if you download them. So first I wanted to do a disclaimer. Some of the things we decided, you may say, why are you doing that? The choices we made were made for our requirements, and certainly are not gonna be, may not be correct for you. So take what I say is not what you should do, but as what we did. So I wanna, it's been a while since I actually talked to a group that wasn't part of our user community, so I'm gonna be probably using some terms that you aren't familiar with, so I wanted to get them out in advance. When I say Paz, I'm talking about the Progress Application Server. This is an application server based on Tomcat that contains a core set of capabilities that we use for different products within our company. PazOE is the specific flavor of the app server that we use for OpenEdge, which is the division I work for. ABL stands for Advanced Business Language. I wouldn't put this on the slide, but it really is the 21st century marketing term for 4GL, which is where our company comes from. An ABL application, and this is important. Inside Tomcat, we support multiple ABL applications, and an ABL application can contain one or more web applications running inside an instance. And this, you know, does generate some challenges for us, especially in things like logging and things like that, that I'll be getting into. So who are we? Well, Progress was first introduced at Comdex 1983. So it'll give you some idea of some of the other things that were announced at that Comdex, which is actually quite famous. PCDOS 2.1, that came out. And the reason that came out is because the IBM PC Junior was also announced. Finally, the big one, Microsoft Windows, was announced even though it would be a few years before it would actually ship. And finally, machine 30 years ahead of its time, the Cardiff Electric Giant. Which is a joke for those of you who watch how I catch fire. So August 1984, we shipped our first version of Progress 2.2. It wasn't completely due to the fact that nobody would buy version one of a product back in those days, but that was our first version we shipped, it did change from the original 1.0 that we did announce. So some of the timeline that happened since that original announcement, I joined Progress in 1994. In 1998, we introduced our Progress App Server, which I worked on, and that was our, what I'm gonna refer to as our classic App Server. That was in version nine of our product. In 2003, we rebranded our product from Progress to OpenEdge, and released OpenEdge App Server 10.0. In 2014, we released our first version of our Progress application server based on Tomcat. And that was in our OpenEdge 11.5 release, and that was based on Tomcat 7. And finally, last month, we released version 11.7 of our product, which is now based on Tomcat 8.5. So first thing I wanted to do is show you what our classic App Server architecture looked like so you can see the bar that we had to meet in order to create this product. So our product consists of a, our core App Server consists of a broker process that's written in Java, and a pool of agents, which are native code. Each agent can support one ABL session. Normally, you hook up to a database. With these, with this core, we supported an ABL client, as well as some Open Clients Java, and at that time, ActiveX, which has since been replaced with .NET. Then came the servlets. So our first server that we did was our App Server Internet Adapter, or AIA. And it just basically tunnels our App Server protocol over the HTTP protocol. This was well before the Tomcat days when I did this and did give me the first opportunity to write an HTTP client in C. And also, this was the days of a servlet exec in J run. So what do you know? One of those servlet containers actually didn't support chunking at the time, HTTP chunking at the time, so I had to write some hacky code around it so that no one, when our messages ended coming back. And just last month, a customer reported a bug in that old hacky code that showed up due to changes in our message structure over the years. Next, we created a Soap Adapter for web services. And it's based on Apache Soap, and it still is based on Apache Soap today. And by Apache Soap, I don't mean Apache Axis, I mean Apache Soap. In fact, I bet you we probably are the biggest, our community is probably the biggest user of Apache Soap out in the wild today, I would think. I'm sure everyone else moved to Axis over those years. And finally, more recently, we created our REST Adapter when we were doing our mobile work. And this is a more modern based architecture based on Camel and CXF. We also had some ancillary processes. We had an admin server. It's required for our app server. It manages starting, stopping, managing, and all those other things. It uses RMI to communicate with everything, which is one of the reasons why we moved away from this. We also created a name server. It uses UDP and allows clients to find where the app server is located based on an app service name. And last, we also had a web application server. This actually predated or delivering our regular application server called WebSpeed. And it's based on the same architecture, but it uses a completely different agent process that uses a completely different model. Whereas our normal app server is an RPC based model. This one is a streaming based model designed for the web. So why do we develop a new app server? Well, our current app server wasn't really designed for the cloud. I mean, we do have customers running it up on Amazon and what have you, but I do it all the things like RMI, admin server, things like that. It wasn't ideal for the cloud, especially in a high transaction, you wanna bring servers up and down stuff. Also, since it was all proprietary, we didn't work well with third party tools. We lacked integrated security. For most of our life cycle, we've given our customer some tools to implement security, but it's really up to them to implement it for their applications. We started with Spring Security Framework when we really started RestAdapter and we were able to extend that over to all our transports. It was not standard space, I talked about that. And we wanted a single application server platform for multiple products. Outside of OpenEdge, we have some other products. Corticon is a business rules management system and role basis in a path development environment. All these are Java based and we didn't want everyone using their own version of Tomcat and customizing it and having to support multiple versions. So we wanted a single platform that we can use throughout the company. So deciding on a platform. Well, our first choice wasn't Tomcat. We decided to go with Eclipse Virgo with Tomcat and for many reasons. We did a scorecard on different application servers and Tomcat was even considered, because at that time we knew Tomcat. Tomcat was the boring, safe choice. It wasn't the fun, new, exciting thing. So we were gonna go with Eclipse Virgo and the reasons why? Well, performance. First test we did with getting some of our legacy code into Eclipse Virgo showed it was actually better performing than just straight Tomcat, which was surprising. I don't know why. The OSG architecture, I mean, what architect doesn't love that? The control you get, your users aren't gonna screw up your system. You've got full control, everything's locked down. The administrative console, these are things that we wouldn't have to write ourselves but are required for a commercial application server. That's spring integration. It was already done. And built in diagnostics for when what things go wrong. So a few months into the project, we switched to Tomcat and the reasons why? Well, difficulty is getting legacy code to run. We weren't coming into this with writing a completely new code base. We were coming into this with a huge number of legacy lines of code that we needed to get run in the platform and be compatible with our other products. Pushback from other groups. The other products I mentioned, Corder-Conn and Rollbase, they weren't too happy with OSGI because they had some experience with it. We didn't. So one of the goals of this was to have a common platform and if we chose Virgo, these other groups may say, no, no thanks. We're gonna just stick with Tomcat. We couldn't no longer fight the server to meet the deadline. This project actually took quite a long time. We actually landed up slipping. We were supposed to go out in our 11.4 release, but we actually slipped. So we had an 11.5 release about four months after 11.4 and 11.4 is like the orphan release. It has no service packs, no nothing. It just was out there because we're agile and we have to release when we can, even though we've got dates we gotta meet. So finally, let's talk about the past architecture. So first and foremost, it's Tomcat. Initially it was 7.0.42. Our current shipping version is 8.5.11. We are going to 8.5.15 in our 11.7.1 release which will be coming out in the next month or two. We'll extend, but not customize, of Core Apache Tomcat server. We will support the deployment of any compliant web application in Tomcat and we don't create dependencies on our core password from any of our products. So for instance, in Open Edge when we extend it, we had to create a lot of extension points in order to drop stuff in, in order to extend it so that the other groups don't have to worry about any of our requirements. We're adding value to standard Tomcat and I hope you'll agree with me when I show you a little bit about what we've done. We simplified management of server data to know from automation scripts. A lot of our work went into our scripting framework for managing our servers and I think it's one of the coolest parts of our additions to Tomcat. We have an administrative friendly command line tool for common server tasks and I'll be showing you that. Full support for Tomcat instances. How many people use the instance architecture in Tomcat? Just curious. That's one of the things that I never knew about until we started this project was this whole Tomcat instance architecture. I thought you had a Tomcat server and that was it. But the instance architecture was a great feature that allows us to be more like our classic app server and how our customers expect things to work. This is one of our more controversial decisions we made. Common location for third-party ISV products and web applications. We enabled a shared loader and we used that to intercatalyne a home common lib to include a lot of our third-party jars and stuff like that. This causes problems especially if you're trying to deploy standard non-progress web applications I'll get into that in the end. Drop in exceptions to customize Tomcat runtime environment. Again this goes back to the point we can't really create dependencies between our products on this. We wanted a core standard platform and drop in exceptions to customize creation of Tomcat instances. When you create a Paz-OE instance that's different from creating an instance for some of our other products and we have a specific tailoring that gets called for those environments that you can just drop in. And then we removed the unsecure management in root web application and we distribute those as extras. We go out in two flavors. We have a development version which includes a root application that's our application that supports our transports and stuff as well as everything's open. And we also have a production version which locks everything down and everything's disabled by default and you need to go in and turn things on in order to get them to work. And that's kind of what I just said, the predefined security production great Tomcat features. So some of the features that we've configured in our platform is authentication realms, ACP session management support, Java security manager integration, multiple server instance support, we enable the filters for whitelist, blacklist checking in both the IP and other valves logging. We give you a way of turning on and off logging easily. The JMX console is turned off by default even though you can turn it on and use it as normal ways. We normally recommend you use JMX locally. We did add full set of management beams for our extensions into Tomcat. We predefined HTTP and EJP 13 connectors. Some are enabled by default, others aren't. Tomcat single sign-on is configured. We use a session ID size of 22. Not sure why that's important but. We include an SSL Java key store with a test certificate that's self-signed so that in order to test SSL while getting a normal certificate, you're good to go out of the box for development. The web crawler valve is enabled as well as memory leak monitoring. So our extensions to it, we created a single scriptable command line utility tool called TCManager. The script itself is TCMan for short. That allows you to do some of the most common server administration stuff. We integrated spring security and spring MVC support into our product. On the open-edged side of things, we really haven't touched what the spring framework can do in our stuff. We use it for our rest adapter but there's a lot of stuff in there that we'd like to take a look at and take advantage of in the future. We include the common's HTTP client. I know somebody was saying yesterday at the meetup that it would be good if Tomcat just had its own support for this stuff. We included it. We have a spring security authentication. We support digest file, LDAP, active directory and we're extending that as more requirements come along. One of the things that we've done is we've externalized server.xml values into property files for easy maintenance and that's one of the things I'll be showing you. Also, we created a method for externalizing and enabling and disabling of individual server xml features so that you can do that from a command line. Our customers, we don't like them going in and editing xml files by hand. We can help it. We included a secure root web application. That's just a blank web application by default in our core pass server if you're doing a specific one for open edge for instance and it's ours. And then our extras directory contains all the things that come with Tomcat plus some other things that we needed. And finally, Windows service support which is actually a lot more difficult than I thought other than just running the Tomcat 7W or 8W, a big environment that we had to set up in order to get that to work. So, yes. Yes. This is a replication of Tomcat experience with Windows service, SCM, do you think it would be fine to do this for it? Would it be notifying? Yeah, I mean. So, yeah, so one of the things at least normally our web applications load and our issues have to do with our agent processes and whether you're hooked up to the database. That's all done sort of through logging. So, with a Windows service you can start up your service but it may not actually be healthy. And again, yeah, we only recommend using Windows services for production server when things are working right. But yeah, we normally recommend against it because we find issues where we lose critical error information sometimes. Right. I mean currently we're facing the same problem where some of the apps they may have been to start up as expected. Right. Exactly, yeah, yes. We encountered the same issue and haven't got a solution for that. So, our PaaS command line tool TCman for Unix, we support, we have to support multiple platforms, we support Windows and Linux as well as AIX, HP, UX, and Solaris. So, for our Unix variants it's just a shell script and for Windows we use PowerShell. We manage each instance independently and you can manage all instances by working on the home PaaS instance. We record our instances in a file in our configuration directory so that you do, we have this registration and registration process I'm gonna show you. And each instance is assigned an alias name that we also use for the JVM route for when you're doing clusters. So, some of the actions I'll be showing these, also I've got some stylish placemats that we hand out to our customers that go over some of the hand views and with that I'm going to switch over to demo mode and show you a little bit about our TCmanager script. So, this is an environment window that sets up our environment for OpenEdge and we have a script called PaaSman that's on the path that just wraps Catalina Home, Ben, TCman. So, if I do PaaSman you get your standard help screen. You get your standard help screen that shows the different options that you can do that we support and a lot of it. Some of it requires the Tomcat Manager to be installed which we do for development but it's not required. So, for instance if I look at what instances I got currently installed, just give it PaaSman instances and you see I got two instances installed right now. OE AG1 is our authentication gateway and the interesting thing about that instance is that we deliver it just basically as zipped up pre-configured instance since it's a specific implementation of a secure token server. We just, Tomcat allows us to generate an instance to zip it up and then we can deploy it on both Unix or Windows and just register it and have it go. OEPaS1 is our standard development instance. So, let's take a look at how we create an instance. You got full command line help feature, so create. So, when you create an instance, so when you create an instance you can specify a number of different attributes such as your ports, your web apps directory. If you are the Tomcat manager application and another web application we called OEManager which provides a JMX or a REST interface on top of our JMX beans, they run using container security so here we allow you to set the credentials at creation time. One of the things we found was that a number of our internal people who put instances up on Amazon for development and demo purposes or whatnot, sometimes those machines get stolen if you leave the Tomcat defaults in there. We also allow you to specify whether it's going to be a development or production version of the instance. So, let's go through and create an instance here. So, one of the things that we do is, dash f means let's deploy every, we allow you to configure your Catalina home instance with a number of web apps and then if you specify the dash f it'll deploy those web apps along with the development and just the root application. And I'm going to take the do verbose output so it's interesting. I'm just going to take the default ports at this point. You just give it an A least name, apcon1. So, it'll go through and it'll just create a Tomcat instance. And I'll be going into the details of what a Tomcat instance is directory wise for those of you who aren't familiar with it. Because this is our development server, all our transports are enabled and everything is opened up. Also, we turn off our stuck session valve for development server because when you're debugging it's easy to create stuck sessions. So, now if I do a Passman instances, you see we now have a new instance, apcon1 that we just created. So, let's take a look at configuring that. First thing I want to do is I'm going to look at the server.xml file that was created. So, this is my Tomcat instance directory that I'm going to comp. So, one of the things that we do in order to enable this scripting of turning features on and off is we delineate them with some special sauce. So, for instance, let's take a look at the access log feature. So, right now our access log is on. You see this XML tag there and we have our access log configured. Some of these things have been externalized to property files, such as condition and less. So, you can then set that in a property file with having go in and modify your XML file. So, let's see how we configure features of the instance. So, Passman features gives you a list of features that we support and it gives you the current settings of what they're on. So, if I want to turn off my access log because it's chatty, I can do a Passman feature. Because we're using instances, I got to use my instance syntax which is specified by capital Y. We give it our instance alias name. And then we just do access log. So, now if we go back to our server.xml file, you'll see that it's been modified. If we go down to where our access log was, you'll see that it is now uncommented. We don't, I mean, it's not rocket science here. We just take off the ending XML tag and make it a whole comment. Now you're able to turn on and off features easily. So, let's take a look at it running. So, after a few seconds, it will be up and running. So, this is our development server welcome. Let's see, wrong port. 1810. This is our welcome page for our development server. It gives you the version of Open Edge, the version of Tomcat it's running on and some other information. The PaaS version is the core PaaS version that we're running on, which I talked about. Also, we install the Tomcat manager by default for a development server. And as you see, for our development server, we have a number of web applications installed. Our root application is our main one that supports our clients, Tomcat manager. OEDBG is a debuggler, is used for debugging our stuff and was our first foray into the world of web sockets. It was actually pretty cool. We actually had to do web sockets in native code too, so we were able to find this library called Libnopole that we were able to set up communications using web sockets between Tomcat and our agent process. And finally, our OED manager web application, which is, like I said, a REST interface on top of our JMX means. So, back to the presentation. And let's just stop the server. So, Tomcat instance architecture, what made this possible for us to do and is an awesome feature that I highly recommend to everybody. Basically, we have a runtime configuration that shares common libraries and scripts with our home server installation. Each instance is a full Tomcat server process with its own set of ports. We offer, you know, it gives you the lightweight expansion of the number of servers for load bouncing and scaling. It can have its own configuration or actually set its own deployed web applications. So, that's like what I showed you when we used the dash app, we copy over some pre-configured stuff. You can also deploy your own web applications into any instance. It can have its own shared web application libraries, which we use heavily. It could be pre-configured in packages, deployable unit, and ISV and on-premise installations. A lot of our customers, and that's exactly what we took advantage of for our secure token service. Lifetime can spam multiple home paths on installs and installs. And this is a great feature for us that I'll be getting into some more detail on. When you update your home instance, you end up updating all instances because we are using the shared loader. So, when you have any library updates, by copying them into the home instance, you'll have to get picked up by any instances out there. And basically, any shared libraries that are in the instance itself don't have any effect on any other instances or your home instance. So, let's look at the instance runtime. So, DLC is where our product gets installed to, and our home instance is found under Service Path OE. So, when you create an instance, we create basically a set of directories where you tell it to. We then copy over a number of scripts that get copied and tailored and tailored basically setting up Catalina Home and Catalina-based environment variables. We do a full copy of all our properties. We copy over our root.war and any other web apps that you may want to do. And then we do a full copy of any stuff that's specific to our product. So, at runtime, what happens is, you start up your instance, Tomcat Live and the bin get used from the home instance, as well as our shared loader, which is all the stuff in Common Live. The stuff that gets picked up from the instance are all the properties, the logging, the temp work, the web apps that are deployed in there, as well as the open edge stuff. So, when you do this, you create a number of instances off your home directory. You can then easily put up an HTPD in front of it, support clustering. Also, you can deploy web applications into your home instance that get copied over, or you can deploy them directly into a specific instance. Also, there's our preconfigured instances that you can just unzip and register. So, when we upgrade instances, and this wasn't as simple as this slide leads you to believe that we learned when we moved from Tomcat 7 to Tomcat 8, or from Spring Security 2 to Spring Security 3. But basically, you would install a new version. For instance, if you're going from open edge 11.6 to 11.7, you'd install 11.7. You would unregister all your instances on your 11.6. You would register them with your 11.7 instance, and it would pick up all the new stuff, as long as none of the configuration files have changed. Unfortunately, that's not always the case, so we have to do a little extra work in that regard we found out. You can then undeply your older version, and you're good to go. So, the next thing that we did was Spring Security. It's a big thing for our customers moving forward, especially given running on the cloud and all those other things that are happening today. So, originally, because we don't create the applications our application partners do, our use of Spring Security have to be based on external XML files, and our customers don't write Java code, so other annotations are not gonna work for them. So, in our original configuration, you required a customer, and man, we edit XML files with hard-coded values. And because of that, we can't update those files ourselves directly. Also, there was a ton of files, and a lot of them contained similar information. So, for instance, we had form local, form LDAP, form AD, you know, all those different types of configurations that really complicated things. We had no GUI tools to simplify the administration of these files, and the number of files is very large, and would only get larger as we had support for other things like OAuth and things like that. And there's a high maintenance, because the properties are specific to web applications, and they weren't shared among multiple web applications. And, for instance, one of our requirements is that multiple web applications can be basically created as a thought of as a single application. So, in 11.6, what our customers would have to do, they'd edit web.xml, and select one of the 12 files we provided, depending on their security requirements, such as form local and what have you. They'd edit the XML file for each user account source, where are they getting their information from, and they also edit the XML file for URL access for a web and resident web transports, or APSV transport, which is our app server protocol, and so just use basic authentication. So, for release testing, you go in and you'd edit the web. Add XML file, and you select your test file and logins, and the real issue was upgrade in patches. When we went from Spring Security 2 to Spring Security 3, that required a change in the XML namespace and all those XML files. So, you cannot take a web application that we created in 11.6 and deployed to 11.7 without modifying that. Nothing else changed, just that stupid namespace. So, for 11.7, what we did was we separated the configuration from the XML. So, we created a property file, much like we did with a lot of the Tomcat properties and externalized it, and for your URL access controls, we use a CSV file to get those out of the XML. So, now you go in and you edit the user accounts, you edit the property file for the user account source and test accounts, and for upgrades in patches, we have free control over overwriting those XML files because they're pretty much static, and it just picks up the stuff on the properties. So, in order to turn on security, there's basically two properties that you have to set, and then there's more properties depending on your specific situation. But we have a login model. By default, it's anonymous, but then you select whether you're using basic form. Container, single sign-on. And then we have an off-manager, which is local x local l dot oe realm, which is a bridge into our user's own homegrown security account information, and then active directory. So, if we look at that, this is our development environment. It's based on Eclipse, and when you create a web application, we have our directories here. It looks very familiar with our web inf. And our OAB, a security properties file, is where you go in. It's very well documented, and you go in and you set your property. So, for instance, our off-manager is local, and our model is anonymous. If I wanted to enable this for form, all I have to do is just change that to form. And then we have our CSV file, which is where you put in your URLs and your access controls, as well as roles and stuff like that. So, if we were to take a look at this real quick, I'm going to start up my server here. And if I hit my sec test, is the web application I gave it when I created it. Web is our web transport, and then hello is my web handler in this simple example. This is just our default hello world when you create a web application. So, to enable security, all I need to do is go in and change my property file to form from anonymous. I'll save it, and server will get republished. Now, one issue that we did find in 11.7, which was just released is that, unfortunately, the development environment doesn't reload the context. So, I have to do that manually. And I give it my web application. I have to give the Tomcat Manager password. And sec test is my web application. So, and all this does is it basically calls the, we have a little utility that calls the Tomcat Manager via HTTP and does a reload on it. So, now if I go to my web application, boom, I get my form login screen, and now I have authenticated access. So, some of the challenges that we've come across. So, our classic app through the world is thought of like this. You know, it's very nice, relaxing, you know, kind of tranquil, while our progress applications are for open edge can consider this, you know. We have no control over things like, you know, the pool water temperature or anything like that, but it has cool water slides and it's a lot more fun. So, some of the challenges was, you know, we no longer control the environment we run in. Our company has a compatibility mandate of we support one major version behind and one major version forward. So, a lot of our customers are still on version 10 of our platforms, so that means, you know, we're supporting 20 years of legacy code back when we delivered nine in 1998 to current 11.7 for those customers. We have to be a little more, it's a little more challenging in order for us to support this here, and we're running into things like, you know, updating spring security requires, you know, updated namespaces and things like that. But it's well worth the challenges to allow us to deliver this product. Logging. Logging is an issue for us. As I said, we had a requirement of multiple web applications. We wanted to log to the same log file because they were the same ABL application. Our core PAS server ships with Java Commons logging included, but we chose to go with log back in SLF or J, and that does cause some issues, especially in the bridges, because a lot of third-party products, you know, use Commons logging, we needed to include the bridge. So, for instance, in our Corticon product, which uses Commons logging, if we include the bridge and their stuff, then it causes issues with their logging. That doesn't work. So that's at GRLibrary Hell. Even though we do tell our customers that, yeah, sure, they can deploy any Tomcat, you know, any Tomcat-compatible web application into our server, chances are, when they actually start doing that, we're going to run into these compatibility issues that we're going to have to work through with them. You know, it's not the end of the world, and it'll be easily solved. It will require some manual integration. You know, multiple products installing their own private version of the same file is problematic, depending on how your class loaders pick it up. You could be, you know, using an older version of a third-party library that don't work with your application. And finally, you know, applications that store their temp work data inside the web application is a problem for us. So if we look at where we were, this is our classic app server, and what we delivered with Tomcat is a single application server. We moved all our serverless inside our OEABL.war, our transports of web, AIA, SOAP, and REST. Our OE Manager Warp provides our REST API. The broker process we had in our classic application server is now a lightweight session manager because we no longer have to manage connections. Tomcat does that for us. And we replaced our single agent process with a multi-session agent that can support multiple ABL sessions in a single OS process. So for more information, if you're interested in actually playing around with the TC Manager framework, and really it is just Tomcat under the covers, you can download our classroom edition. It includes a fully functional PAS OE development server, and that's the URL for that. We also have other products that you may be interested in, quarter pound and roll base. Those are the URLs. One final thing I want to show before I wrap up is if you do grab the, grab the classroom edition, we do include a PAS running dot text with Tomcat just like running dot text. It shows all the modifications that we made to Tomcat, such as, you know, turning on things, moving things into property files. Pretty much anything you want to know how we go from downloading a standard Tomcat to turning it into PAS is included here, and it's mostly up to date. There may be some things, you know, a little out of date like, it still says Tomcat 7 there, even though we're on Tomcat 8 for the Windows stuff. And then we talk about the extras where you can find things that both come with Tomcat that aren't installed by default as well as some other things. So with that, I am done. Any questions? Yes? You said there was a potentially non-governmental influence in the way of sharing the language across the world. Yes. So, like, apart from virtual conflicts, so do you see any more challenges to that? I mean, you said it is all of the problems. Right. It comes with its own problems. Yeah. What do you mean? So, you know, our biggest problem has to do with, that we ran into was the SLF for Jay Bridge for Java Commons Logging, like I said, that has adverse effects on other things, other products that use Commons Logging. Another thing is we need to make sure that all our libraries have version numbers in their file names, because what we do when we go and update, install a service pack or whatnot, we copy these new versions over that we go through and we have to, we look at all the file names and we take all the ones that are older and we move them into an archive directory so that we don't have multiple versions of the same third-party library there. So there's a lot of management tasks involved in having to manage that properly. What does, I don't know, did this make you gain the ability to be able to install a service pack and have all our customers deploy instances update to new versions without touching them? That's the big thing for us. Yes. So, a customer is running their phone now, so we're saying you don't support their application, but you're essentially patching them for them? No. So, yeah. Right. You just, you say, congratulations, you've got the new versions, but you're not probably QA in their products, so you might be replacing a library because they're online. Well, yeah, our customers are not Java programmers. They're ABL programmers, so they're running in our environment, so we do... So their applications are actually your applications? Exactly. I mean, I kind of like to think of it as that old BASF commercial. You know, we don't make the applications, we make the applications possible. Our customers are basically application partners who have been on our platform for a number of years and sell our applications to their customers, and that's one of the reasons why, as a 4GL company, we're still in business because of that lock-in and inertia. So, you're actually essentially simplifying the upgrades of not only the application server, but also the application provider. Correct, yeah, they're... They have an upgrade into the server, which includes the library. Exactly, and our customers aren't calling those libraries directly, our code internally is what's using that functionality for the most part, as far as open edge is concerned. So, yeah, if a customer did install a third-party web application that we didn't know about, they very well could run into difficulties with versioning, but I think we can figure out ways to help them around that. Any other questions? Yes? Yeah, so the... An older version of the slides are already uploaded to the Apache cons site, and I will be updating them with this version, which I've been working on since the last... So, that's a good question. I'm glad somebody asked that, because I was prepared to answer that. None of it's open source, but we would not be adverse to donating it to Apache should the Tomcat community think that it would be useful. Now, the one caveat with that is that I have no actual resources to vote to that process, but we would be happy to work with people who do have the time and resources that are possible. Now, our TC manager stuff is really Tomcat... I mean, it's open-edged agnostic. Our core server, although it's tied to a particular version of Tomcat, is basically only Tomcat. We just basically work around the edges, replacing hard-coded values with property values and all that stuff, but it's definitely something we'd be willing to work with somebody on if the community thought it was valuable. The spring security stuff, that's really specific to our use cases and probably would not translate well general purpose-wise. Yes? Can we cover this at the... Basically, the changes in server.xml were big. The whole connector architecture and stuff like that. Originally, we had... But nothing that was insoluble? Oh, no, not at all. It's just basically learning what the changes were. I don't know if there's a good document on that process. I did not find it. Yeah. That would also be helpful if you knew exactly the types of things that changed. For instance, like the scanning of libraries. That's a little problematic, because we use the shared loader and we have a ton of libraries in there. I didn't realize that that property name was 25. I was running around. It's hard enough to get the logger to dump out what jars it's having problems with that it's doing. Once I figure that out, then I go in and change it, nothing happens. It's stuff like that. There's nothing earth-shaking and certainly not something I'm complaining about giving the value that Tom Capra buys us. Anything else? This session was so much technical, so much tied to Tom Capra and everything that at least I think would go with it. But this guy is looking at the title. So we are glad we got into this. In the title of the session suggested it is going to be a generic talk, but we can be so relevant to it. Anything else? Okay, great. Thank you for coming.