 Rydyn ni'n gwybod. Rydyn ni'n gwybod, ydw i'n gwneud o ddweud o bwyllfa ac yn fanbyrwyr i Ffaven 1.0. Roedd yma'r ddweud, mae gwirionedd mewn gweithio yn y UK. Rydyn ni'n gwybod Ffaven, mae nhw'n gwybod yma. Rydyn ni'n gwybod, mae nid yw'n gwybod, rydyn ni'n gwybod i'n gwybod i'n gwybod i'n gwybod i'n gwybod. Rydyn ni'n gwybod i'n gwybod i'n gwybod i'r ysgweithio'r 1.01 ar y dyfodol. Fabean is a facility for developing and running multi-tier server benchmarks across multiple systems. You can also use it for doing microbenchmarking as well. Maybe a specific component such as an FTP server or a HTTP server. This open source is developed by Sun and it's available at that particular URL, which I'll give you a reminder of later at the end of the slides. Fabean has two major components, the harness, which automates the whole process of a benchmark run and also provides a container for your benchmark driver code. Again, we'll look at some examples of benchmark driver code in a minute. We have a driver framework which provides an API for you to develop benchmark drivers, also services and tours and also provides a component model. I've also mentioned there's two main aspects to it but there's also analysis as well. Fabean provides comprehensive analysis of results and data it collects from the systems involved in the test. The best way to introduce Fabean is to show you some of what it does and what it gives you. This is a typical Fabean benchmark run. You'll see at the top here we have information such as the operations per second for this run, the number of operations that were run and where it passed or failed. Beneath that, you have several tables and two of the tables are displayed here. An important concept in Fabean is the operation. I'll mention operations quite frequently and an operation is basically a specific use case. An actor's interaction with a system, for example, a user accessing a web server or a web page. The two tables we have displayed here are the operation mix, which shows you the number of operations that were performed successfully, the number that failed and the actual mix that we had of that particular type of operation. What we're seeing here, this particular benchmark run was for a social networking application. For a social networking application we have operations such as homepage, login, ad event or event detail. They're all of these operations, specific user interactions with a system. Down here we have response times. We have response times for all of the operations involved, average, maximum, 90th percentile response time and standard deviation. Fabean also gives you a detailed results page with graffin of key performance indicators, things like throughput and response times. There are a few others as well about what the driver's done and what work is done to perform these particular metrics. Depending on what you've done in terms of deploying tools across the systems in your benchmark run, you'll get specific stats for each system. You can drill down for each system, look at statistics such as CPU utilisation or look at the NIC and see what kind of throughput is going through the NIC. That's the kind of stuff you get from Fabean in terms of the graphical display and the reporting. Fabean architecture, we have two major run-time components within Fabean. We have a master. The master coordinates all of the systems involved in the test. The other aspect of it, the other part of it, is the agent. We have several agents running on the other systems that are actually involved within the test. In this case, at the top here, we see a bunch of driver systems. They're actually JVMs. They don't need to be specific systems, so you may find these free running on the same machine. They're instances of a JVM. Each one has an agent, and these machines are driving load into the system on the test down here. This gray box contains the entire system that's on the test. We have web servers, proxy servers maybe at the front, and we also have databases, memcash de-instances, those kinds of things. All of those systems will be running agents, and Fabean can control them all through the master. You have to give Fabean certain information to get everything working. You have to provide information about all of the systems involved in the test. For example, the web servers, the application servers, database servers, memcash de-instances, reverse proctors, host port pairs for the most part. You also have to provide a benchmark driver, but Fabean gives you some samples you can use to develop that, and we'll look at some of the code that you would use to develop a benchmark driver in later slides. You also have to provide a configuration. The configuration data is details about what the run will do, how long it will run for, the number of simulated users, etc. Fabean gives you back some of the stuff we've already mentioned. It gives you a harness that runs all your driver code for you, that manages all of the systems within the run, that gathers the data at the end of the run, makes sure the statistic tools are running on all of the systems, brings it all together at the end and presents you the data in graphical and tabular form. It also provides you a driver API, along with all of the plumbing to run your Fabean benchmark driver. We'll talk a little bit more about that in another slide. You also get configuration screens where you provide all of the input data about the systems involved in the test, a couple of examples in a second. These are all also generated from a configuration file, a very basic configuration file that you provide, and when you deploy the benchmark, the configuration screens are all generated for you. An example of that is one of four tabs in a configuration. You see at the top here we have four tabs. The Java tab here allows you to configure the Java runtime. The driver tab is where you provide information about the run, how many simulated users, what the steady state time will be, how many driver agents will run, how many systems will be generating load, that kind of thing. This screen is the data service tab. This is where you configure all of the backend systems. In this case, we have a database server, a file store, and some Memcached instances, although we haven't provided any data about them. That just means we probably don't not use any on this particular test. We also have the ability to say whether we want to reload these particular resources at the beginning of each run. Also, we have opportunity to provide faban with information about where the configuration files are, where the log files are, and how to stop and start the services. Also, what tools we want to run for any particular given run. On the web server configuration tab, in this case, we are testing the Apache OLEO social networking application. We are using the Rail service and the NGINX service. In this case, we get two boxes, we get one for the app server, and we get one for the proxy server. Here, we are using FIN, this is the Rail service. We are saying we want to use the FIN back-ends. We are going to run 32 of them. The starting port will be 8,000. Here, we are saying we want to use NGINX, what the port is, and then we provide all this other information again about how to start and stop those particular processes, and where to get the log files, where to get the configuration data and such like. Also, again, we can provide information about what tools we want to run. Services and tools. A service is something that you want to run throughout the duration of the benchmark run. It needs to be in place before the benchmark can actually start. That could be something like a MySQL instance, initialised with a set of data that you've provided. You generally don't start testing with a raw database. You'll load a database up initially. It could also be the number of a service that provides like FIN instances, like we've seen before on the previous configuration page, or a service that initialises a file store with clean data for each run. It also provides a plugable infrastructure for you to develop your own, and it also provides a basic set of services and tools that you can use. I forgot to mention what a tool is. A tool is something like a wrapper around VM stat that will run during the steady state period of a run to measure specific statistics about the system, a monitoring tool, something like a memcash these stats tool, that kind of thing. A lifecycle of a favon benchmark run, where as far as services and tools are concerned, we start off with a configuration phase, and we end up with a post-processing phase. In between, we start off our run with a ramp-up period, where we're starting all of the load of threads, all of the driver threads to simulate the load. We're warming up caches and such like until we get to a point where we're in steady state, and this is a point where we want to actually do the measurements. We actually want to test or measure the performance of response times and the throughputs during steady state. What happens with a tool? A tool starts only in steady state, because you don't really want to measure the system while it's not in steady state. You wait to this point here, and then you start all of the tools. All of the tools will run until the actual steady state period ends with a ramp down here. Services have to be in place right from the beginning, so they start here, and they end in the post-process phase. Looking at the favon driver, oh, I do apologise. We seem to have run out of screen. So a favon driver is... Sorry? A favon driver is a Java class that's used to drive load to your system on the test. It's something you write. It uses Java annotations. It's basically a plain old Java object, but annotated with Java annotations provided by the favon API. It's basically a plain old Java object, so a favon API. It uses these annotations to define metadata and also define operations, which we talked about earlier. So an operation is a specific interaction with a system. It may well be that in this case you get a HTML page initially. You do the get, and then you load all the static information that you need to render that page, the JavaScript files, the CSS files and such like. You pull them back from the server. Then you scrape the HTML to find out what images are on the page and then you can load them from the server or you can load them from local cache if you're simulating a cache. Another example would be for a form data, HTTP post request, the response code may be a redirect in which case the operation may act on a redirect and redirect the user to another page. Not effectively viewing the page, but using the user itself. And it also validates the success of the operation. If the operation fails, it throws an exception and faban registers that as a failure. Down here we have an operation mix so you can't see this and I think it's quite a bit underneath it but you'll see it on the sample code. But an operation mix is basically the transition from one operation to another. So you perform an operation and then you'll do something else on your system. An operation mix defines how that happens. It could be a simple sequence like do this, then do that and then do that. A sequence is constantly repeated by every single user or it could be something more complicated and that's what we start talking about using finite Markov chains to simulate randomness within a system. But you can do all of that and you get that absolutely for nothing. It's a bit more complicated to programme that so this is a typical not necessarily typical but this is a faban driver class annotated at the bottom there I don't know if you guys can see it is the class itself, ui driver and then we have a whole bunch of metadata annotations that we provide. The first two annotations are purely metadata we won't need to go into those but they're the ones you provide to the system and then we provide a matrix mix and we mentioned operation mix is before on a previous page this could be something very simple it could be just a very simple transition from one operation to another or you can use some real intelligence in it and you can use a matrix mix this takes two arguments it takes an array of operations and it takes a mix which is a matrix and we have one row for every operation and one column for every operation and this allows you to define a state transition from one operation to another and also a waiting of what will be the preferred operation to run at the next stage the next operation run maybe we've got a heavy weight here that after we've run the home page we'll do an event detail and also using this you can actually prevent certain state transitions from happening at the bottom we have this thing a negative exponential and we're using this thing called a cycle time a cycle time is a space between one operation and the start of one operation and the start of the next you can also define a think time we're saying we want to run with a cycle mean of 5,000 milliseconds with a 2% deviation and we want it to be a negative exponential so Fabian will actually vary the cycle time between operations to fit a negative exponential response graph so some of them may be 25 seconds some of them may be 0 some of them might be about 5 and a couple of methods annotated so we have do home page which is a home page operation and we have do add event which is an add event operation we can say the maximum 90% our time automatic timing so Fabian will start the timing when we enter the method this one here is using manual timing we enter this method when we call this event and if that's given you any kind of interest please go here and find Fabian because it's really cool