 Okay, let's start it. Thank you so much and welcome to our talks, especially the last session of the last day of the conference. We really appreciate it. Okay, I am Catherine Dip. I'm a solution architect and performing engineer at IBM. My name is Tat Chen and I am a software engineer and performance engineer also at IBM. Yes, hi, and I am Paul Van Ack, software and performance engineer as well, also at IBM. Okay, to start with, let's go through the agenda that we'll go through today. So we'll first give an introduction and some detail about per kid performance benchmark itself. We will then go on and introduce a tool that we have implemented named per kid elastic shirt publisher. This is the tool that will be used to send the results and analyze results, et cetera. Then we'll have a demo of how you would use the publisher and Kibana to analyze the data. A quick summary and then wrap up today's session. When we talk about performance analysis in OpenStack, what kind of tools come into your mind? Anyone? Exactly. So when we talk about performance, we would pretty much think about rally and rally is such a good tool, a well-known tool, suit the purpose of what we do. But like anything else in performance testing, I personally believe that there's no one tool, one size fits all tools. And I believe we would use different tools for different purpose. To illustrate the idea, let's look at a number of sample tools. It doesn't mean to be a complete list, but let's look at a number of tools. Look at rally. As we all know, this is an OpenStack project. The strength of rally is in data control plan performance analysis. What you do there, and especially at scale, you would use there to measure your provisioning time, whatever configuration, how many networks. But the net result is you want to know how fast can you put up a VM. You also want to know, you use rally to know if I put the VM sequentially one at a time versus the performance if I do concurrent user, what would be the difference? Or if I just continually provision the number of VM and pack my compute node, what would be the characteristic of this provisioning time? Does it increase as the number of density of VM in the compute node increase? Is there a saturation point? So rally is a tool is really good to looking at controller plan performance. Neutron performance, Neutron server performance, et cetera. Looking at the next tool I want to talk about is Perkich Benchmarkers. So this is an open source performance measurement tools that originated by Google. Like in Perkich, the strength of it is it includes a number of popular benchmark for data plan performance measurements. So what is data plan performance measurements? So in controller plan performance, let's take one example. You will be interesting to know if I put a VM, how long does it take to put that VM, et cetera. For data plan performance in this context, you would like to know now that the VM already put up, now that I install my application or my middleware, for example, like a database on there, what would be the performance of that database? That is what we are talking about. We're talking about whatever things, whatever application workload that you put on top of that. The data plan analysis will typically use a benchmark, and we have a number of popular benchmarks out there, and a number of them are included in Perkich Benchmarkers. Then if you want to simultaneously look at the performance of the data plan and the control plan together, you would use something like a spec cloud IIS 2016. This is a spec standard that comes out from the spec corporation. As you know, if you would like to publish your benchmarking data, this is a place with credibility that you will have your data put there. One of the tools that included in the spec cloud IIS 2016 is CB2. CB2 is an open source that originated by IBM. It comes with just like Perkich, 25 plus workloads in there. The thing about spec cloud 2016 is there's two parameters, one is the metric that was defined by the spec, especially specifically for cloud. One is elasticity. So if you are in a cloud, your capacity should be unlimited, or somehow limited, but it's more than your traditional data center. Elasticity is if at any time my workload increased, and I need more capacity, these things should be automatically spin up and put your workload in there and run. The second parameter is scalability. So for this kind of tool, you would use it if you would like to publish your data as an official benchmark metric, and at the same time want to look at both data and controller plans. And Browbeet, there are sessions talking about Browbeet at this summit. This is another OpenStack project. The strength of this one is you have an Ansible Playbook that you can install, including the tools, and also maybe some of your infrastructure like your elastic search environment, your cabana, et cetera. So if you would like a tool that do all of this to you with some standard dashboard that they included in there, this is the tool that you would be using. But in short, what we're saying is there's number of tools in there. It's the user that you will choose the correct tools for your purpose. In today's talk, we talk about Perkitt Benchmarker. It is one of the tools that we would like to introduce to you. So, Paul. Thank you, Catherine. So of course, as Catherine mentioned, Perkitt Benchmarker is an open source cloud performance measuring framework originated by Google. Roughly around early last year, Perkitt is hosted and managed on github.com. So anyone can submit a pull request if they want to make a change. The link is provided right here. So the strength of Perkitt is that it provides wrappers and workload definitions around several existing popular benchmarks. So for determining what benchmarks... Yeah, so for determining what benchmarks are included, typically the community is engaged to determine representative benchmarks that might represent cloud performance. Areas that are considered are things like reproducibility, ease of use, and relevance to customers. So right now, we'll last that I checked, Perkitt Benchmarker has around 28 benchmark tools configured. Run straight out of the box. So the benchmarks are run in their default states and configurations that is not tuned at all to provide... Not tuned at all to in favor of any specific provider. Now, there are currently 11 cloud providers enabled. We'll get into what those are in a bit, but the goal is to have benchmark diversity with good coverage and minimal overfitting. So when you run Perkitt Benchmarker, when it initiates a test, VMs are provisioned in the cloud. Perkitt Benchmarker installs a selected benchmark in VMs, runs the benchmark, then reports and collects the results. Now, you don't have to have Perkitt provisioned to VMs. You can also specify existing VMs or external points in the config file and optionally pass that in. The idea is the same. You pass in your benchmark tool in your config file slash cloud provider and let Perkitt do its work. So next slide. So here are the variety of benchmarks that Perkitt currently has support for. In the top table, you have what are system-level benchmarks. In the lower table, you have what are application-level workloads. So in the microbenchmarks table, you might see the three key area of interest are storage, CPU and network. So you might have one core mark, which is for CPU performance. Or you might run iPerf or netPerf, for instance, if you're interested in network performance, like if you wanted to measure throughput. In workloads, you have big data slash IoT, high-performance computing slash scientific computing. Simulation and web benchmarks. You have all the YCSB under big data, which is Yahoo Cloud Serving Benchmark. Good for data store workload testing. Also, for instance, you might want to run Tomcat as a web benchmark. Now, this is a list of the supported providers that Perkitt currently has. Of course, Google Cloud is up there. It's a Google run project. And, of course, OpenStack. Now, installation of Perkitt Benchmarker is very trivial. Get clone the repository. Then it's a Python project, so you pip install the requirements file inside the repository. And then each provider has its own requirements file that you have to also install. So we care about OpenStack, so we just install the OpenStack requirements file. And once you have that, running Perkitt is actually very simple as well. So for OpenStack, here's an example run. We're measuring the core mark score for an OpenStack VMs. So first, of course, source the RC file. Get your OpenStack credentials in your environment. Then run the pkb.py script inside the root folder. Then specify your Cloud provider. Here we specify OpenStack, then the benchmark. Benchmark or benchmarks. Here we're just running core mark. Of course, here's the cloud-specific options. And then you can optionally put metadata in the arbitrary key value pair to help you identify a test run and what was being tested at a later time. Here's a list of some OpenStack provider-related flags. Of course, you might see many of these flags should be familiar to many of you as users of OpenStack. For example, OpenStack image username. That's just the SSH username for the cloud image you're booting into. Now, here are some sample results. So Perkitt, by default, returns results in JSON. Of course, you can optionally return them as CSV or whatever, but here we have... You might see that the metric field in the JSON result is highlighted as the core mark score. Then you have the test that was run. Here we ran core mark, the value in unit. Since it's a score, there's no unit. And, of course, we see the metadata that we had passed in in the metadata dictionary. So typically, you might just import these into... You might import these into a spreadsheet or whatever. We prefer a systematic way of being able to analyze and visualize our data. So that's where we... That's where we came up with what Ted's going to talk about. Here, I'll pass it off to Ted. Okay, so for some people who would like to have a systematic way to analyze a test result, Google and Perkitt have provided a tool for you to look at your own data. So the current approach for the Perkitt is to use the Perkitt Explorer. To use the Perkitt Explorer, you will need to run the same command. You pick your own benchmarks, and then you choose your own cloud provider. But additionally, in your Perkitt command, you're going to specify the BigQuery table of yours. So at the end of the Perkitt test, all of your data result will be automatically uploaded to your own Google BigQuery table. And then you can deploy your Perkitt Explorer to the Google App Engine. And then over there, you can analyze your test results. Now, of course, for some people who owns their own data center or would like to save their test result elsewhere, we have implemented the Perkitt Elastic Search Publisher. So the publisher allows users to leverage popular open-source tools such as Elastic Search and Kibana. Elastic Search, as you may know, is a search engine that stores and indexes JSON documents. And Kibana is an interface plugin that provides visualization capabilities on top of the data indexed by the Elastic Search. So we have implemented the code directly inside the Perkitt benchmarker. And we have also created and submitted a pull request to the Perkitt community. And currently, we're working with the community to have it merged. So how do we use the Elastic Search Publisher and Kibana? So first, similarly, we're going to pick our own benchmarks. Second, we will pick our cloud provider. And then additionally, what we're going to provide is our own Elastic Search URI. So at the end of the result, all of your data in result would be automatically sent to your own Elastic Search server. And then can be visualized and analyzed using the Kibana dashboard. Now, Kibana dashboard, you can analyze your data by creating charts, bar charts, or line charts. So here is the full command to run your Perkitt with the Elastic Search Publisher. So all you need to do is, in the bottom, specify your own Elastic Search Publisher, Elastic Search Server URI. Okay, in the demo next, I'm going to show you how to build a chart to compare a call mark score using Kibana. Now, before that, let's look at one more time the JSON result returned by the Perkitt. One of the important metrics of the Perk... of the call mark benchmark is the call mark score. So here you will also see the value of the call mark score. And the test is call mark. And here, inside the metadata, you will see machine type, which is your flavor, and one dot large. So I have been running a number of Perkitt tests to collect the call mark result using different flavors of VM. So what I'm going to do next is that I will compare the call mark score for the VMs of different flavors. So next, I'm going to show you a video recording that I made earlier because I'm not quite sure the network speed. So we prerecorded this recording earlier. Okay, so this is my own Kibana dashboard. Oh, thank you. So this is my own Kibana dashboard. So on the top, you'll see three tabs that you'll be used very often. Discover, Visualize, and Dashboard. So under the Discover tab is where you're going to see your test result. Or you can search for a particular test. Under the Visualize tab is where you're going to build your charts. Here are all the charts you can build. Under the dashboard is where you can put all of your charts and search result into one place. So first, let's start with the Discover. It is the first step to build a dashboard and chart. So let's search call mark. You will see only the test result containing call mark will return and they are highlighted. So for some people who like to build more precise search, they can also use the Lucene search syntax. So in this case, there are only three results returned. Let's click on one of the results. So as you can see, this is the exact JSON file that I showed you earlier in the slide. It has the value and it has the test that says call mark and it has the metrics says call mark score. And under the metadata, it also has the VM flavor. So this gives you a very detailed view of one of your result samples. Now on the left, you will see a list of fields. These fields are only related to the document on the right. So now we can customize the look of your result to make it much easier to read. Now let's add some fields and columns into your table. So you can rearrange the table by clicking the left and right or you can delete a column. This will make your result much, much easier to read. So right now you're looking at call mark. We have three call mark test, value and metric and so on. Only three test result. So now let's change the time filter from 15 minutes to one hour. So right now we are seeing 12 results. So time filter, in this case, the time filter shows me the test result within the past one hour. So after you are happy with your search, what you're going to do next is to save it. So let's click on the save search so we can save the search and then we'll type in some names and then we'll save it. So this would be our first step of building a chart. You always start with the search and then once you're happy with the search, you're going to save it. So the second step is visualize. Now we're going to click on the visualize tab. Under the visualize tab, you will see a list of charts you can build. So in this case, we're going to build a vertical bar chart from a search that we just saved earlier, which is the call mark score. So what you see right now is a bar chart that only has a single bucket. This bucket contains all the 12 test results that was collected within the last one hour. So what we're going to do next is that we're going to split the single bucket into smaller buckets by the VM flavors. So what I'm going to do is that I will click on the x-axle and then I will split the bucket and then using the turn aggregation. Now the turn aggregation will give you the unique turns inside a field. So what field are we going to use? We're going to use the machine type which contains the VM flavor. So after we do that, we can click on the play button. The play button will apply the change. Okay, so now what you're looking at is four buckets. Each bucket is a flavor, large, medium, small, and tiny, and they all have three test results in there. But the test result count isn't what we are interested in. So for the y-axle, let's get the average of the core mark score for each bucket. After you select that, you can apply change by clicking the play button. So what you are looking at right now is large flavor will have a higher average core mark score. Smaller flavor will have a lower core mark score. If you want to change the y-axle, you can do so in the custom label and hit the play button again. Okay, so once you are happy with your charts, again, you can also save it and by clicking at the save button, type in a name to save it, I'm going to type core mark flavor compare. So after you finish your visualization, the next step is to build your dashboard. So the dashboard is a place that you can put all of your charts, your searches together. Now let's click on the add button here. So let's select the chart that we just built earlier. And then we will also put the search, you can resize it. And then next, we're going to also add the search that we created earlier. Now this is the search result that's associated with the chart on the left. So I've also built another line chart of iPerf earlier. So I'm going to put my line chart here. And also as well as the search result next to it, side by side. So in my line chart, I only have two dots there, two data points there. So I'm going to load more data, so I'm going to resize the chart. And then I'm going to change the time filter again from one hour to 24 hours. So now you see more data displayed inside my iPerf chart. And also on the right hand side, you'll see the test result also updated itself. You can also apply the filter or change the filter by dragging on the charts. And the same on the right hand side, the search result will also update itself. You can click on it to display more details. Now the dashboard, once you're happy with it, you can also click on the save button here and save it. So here would be the end of my demo. With this, I will return to Catherine. So as you just see here, the focus of today's talk is mostly on the tool. Notice that we haven't gone into any of the performance data itself. And that would be another talk. But the idea here is how can you use a tool? How can you simplify your work by using existing tool in terms of test Measurement, also saving data, and also visualizations? And I truly believe that you would use different tools for different purpose. And Perkitt is one of the tools that we introduced to you today. If it fits your purpose, of course, the next step will be looking at the specific benchmarking for whatever problem, whatever performance analysis that you would like to do and draw into there. And that is where the expertise of performance engineering come in. With that, I would like to see whether we have any questions. The pool request, yes, but it has not. You mean the pool request? No, Perkitt itself. Yeah, Perkitt itself is, yes. It's on GitHub. Perkitt is in the GitHub. The electric search publisher is submitted but has not merged. It should be merged soon, hopefully. They left some comments. They were very receptive about it, so we addressed those comments and they're just waiting to hear back from the Google. Shouldn't you? Yeah. It's like an open-set review. We go through a few cycles and they're generally very positive, but the style and whatever have to match whatever Perkitt is. Yes, please? Absolutely. Absolutely. Absolutely. So that is the theme that we're trying to introduce today is use whatever tool that you think is you probably most familiar with and you have the expertise in house. That is the tool. But more important, you don't build some things. But you see how Cabana, you can use Cabana as an interface to looking at elastic search engine or even with your log data you can look at it. That's the idea. The idea is open source and the community to contribute to that open source together. We can make our job much easier. That's the theme that we try to convey today. So to answer your question, one of your questions, you said we should use the log stash. So by default, it does output the result to a file, to a specified location. So if you configure your own log stash server to get the data from the file, it's up to you if you want to do that, too. Right. Anything else? Yes, please? Are you using this right now as far as, like, on any files you're testing? We are exploring right now. We were able to test how many, how many benchmarks so far. We tested about, we test all the standards set. Yes. So that's about 19. One note, one note on here is, as any tool, if it's out of the box, is what gets you into the door. That configurates, you choose any of the benchmarker here. The standard, the default configuration tool is for everyone to use a minus to your purpose. That is where the performance expertise is coming in. You know what area you want to looking at, and that's where you tune your configuration file to whatever that purpose. And we hope with that kind of work, it will be a lot of white paper coming out. But for a tool out of the box, the idea is get the user, it's a very short time to get into the door. That is the idea. Yes. The workload as Paul showed here is a standard workload. Let's say you build your own workload. Then you have to bring that workload into Perkit. Then you have to go through the, kind of like, a plug-in to bring your own workload inside. Yeah. Then you have to follow. The idea is if you use one of the 28, whatever, then you don't have to do anything more than your configuration. You still have to do configuration. Out of the box is not going to be suit whatever your purpose. The idea here is without Perkit, you will have to build a server, you have to install like the YCSB, the Yahoo Cloud Service Benchmarker, and then you configure it. But with Perkit, it is installed for you, and you just run with standard configuration. Yes. Isn't that additional for storage benchmarks, representing the streaming, logging? Yes. Exactly. I don't know what the other one is, but there are some sort of representations. Right. Typical. Yes. Yeah. Yeah. That is where the engineers come in and the performance engineers come in. Yes. Okay. Well, thank you so much for coming in. We really appreciate it. It's just late, and we all need to want to get out of here. Thank you.